Manifest lists and manifest files

Description

This lesson opens the two middle links in the chain: the manifest list — this hearing's evidence index, in the module's analogy — and the manifest files — the evidence folders. Unlike the metadata file (lesson 3), neither one is JSON. Both are Avro, a compressed binary format designed to store lists of records compactly — the same kind of design decision that led Parquet to be binary and columnar instead of plain text. This lesson doesn't try to read them as text — that is, on purpose, the mistake lesson 6 uses to teach the correct one — it inspects them with the right tool: PyIceberg's table.inspect.manifests().

Connection to the module. Lesson 3 ended by pointing at the manifest-list field inside the metadata file's snapshot — a path to an .avro file. This lesson follows that path, and the path one level further in: from the manifest list to the manifest files that list enumerates.

An analogy: the evidence index, and the folders it enumerates

A hearing's evidence index doesn't contain the evidence — it's a short list: "folder A, with 3 new exhibits from this hearing; folder B, inherited from the previous hearing, unchanged." Each evidence folder (manifest file), in turn, has its own more detailed sub-index: what specific exhibit each one contains, how many are new at this hearing, how many were inherited, how many were withdrawn. This lesson shows that a manifest list enumerates manifest files — in Kiosko's case, just one, because there was only one write — and that a manifest file in turn enumerates actual data files — also just one, in this same case.

Worked example: following the pointer, with the right tool

Step 1 — The current snapshot already gave you the path to the manifest list

# inspect_manifests.py
import os

from pyiceberg.catalog import load_catalog

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.fact_orders")

print("=== metadata.json points to the current snapshot's manifest LIST ===")
snap = table.current_snapshot()
print("manifest_list:", os.path.basename(snap.manifest_list))

What to expect (the name includes your own run's snapshot_id, different each time):

=== metadata.json points to the current snapshot's manifest LIST ===
manifest_list: snap-<snapshot-id>-0-<uuid>.avro

table.current_snapshot() returns the same object you already saw in module 1's lesson 6, and its .manifest_list attribute is, literally, the manifest-list field you read directly from the JSON in this module's lesson 3. The manifest list itself — the .avro file — you're not going to open as text: it contains a list of manifest files, encoded in compressed Avro format, and the right way to read it is through PyIceberg's inspection API, which does exactly that work for you.

Step 2 — table.inspect.manifests(): one row per manifest file

print("\n=== table.inspect.manifests() -- one row per manifest FILE ===")
manifests = table.inspect.manifests().select(
    ["path", "partition_spec_id", "added_snapshot_id",
     "added_data_files_count", "existing_data_files_count", "deleted_data_files_count"]
)
for row in manifests.to_pylist():
    print(f"  path: {os.path.basename(row['path'])}")
    print(f"  partition_spec_id: {row['partition_spec_id']}")
    print(f"  added_snapshot_id: {row['added_snapshot_id']}")
    print(f"  added_data_files_count: {row['added_data_files_count']}")
    print(f"  existing_data_files_count: {row['existing_data_files_count']}")
    print(f"  deleted_data_files_count: {row['deleted_data_files_count']}")

print("\nnumber of manifest files listed:", manifests.num_rows)

What to expect (added_snapshot_id is your own run's snapshot_id; the rest of the structure and the business values are deterministic):

=== table.inspect.manifests() -- one row per manifest FILE ===
  path: <uuid>-m0.avro
  partition_spec_id: 0
  added_snapshot_id: <snapshot-id assigned in your run, different each time>
  added_data_files_count: 1
  existing_data_files_count: 0
  deleted_data_files_count: 0

number of manifest files listed: 1

table.inspect.manifests() returns a normal pyarrow.Table — the same kind of object you already used in module 1's lesson 7 — with one row per manifest file. In this case, exactly one row: the single manifest file that exists, with added_data_files_count: 1 — confirming that manifest file registers one new data file, added in the snapshot you captured — existing_data_files_count: 0 — it inherits no file from an earlier write, because this was the first one — and deleted_data_files_count: 0 — it deletes nothing. path is the path to the manifest file itself, the same <uuid>-m0.avro pattern you already saw in module 1's lesson 6 diagram.

Diagram: the complete chain, two levels, one single file at each

flowchart TB
    SNAP["current snapshot\n(inside metadata.json)"]
    MLIST["manifest list\nsnap-<snapshot_id>-0-<uuid>.avro\n(1 entry: points to the one manifest file)"]
    MFILE["manifest file\n<uuid>-m0.avro\nadded_data_files_count=1\nexisting=0, deleted=0"]
    DATA["data file\n00000-0-<uuid>.parquet\n(lesson 5)"]

    SNAP -->|"snap.manifest_list"| MLIST
    MLIST -->|"lists (1 row)"| MFILE
    MFILE -->|"lists (1 row)"| DATA

Notice that, with a single write, every link has exactly one element: one snapshot, one manifest list, one manifest file, one data file. This "one to one" chain is as simple as it gets, and it's deliberately this guide's starting point — in a real table, with many writes accumulated, a single manifest list typically enumerates several manifest files (one per relevant write operation, or grouped by compaction), and each manifest file can enumerate several data files. Module 7 of this guide, on maintenance, comes back to this same chain once those numbers are no longer all "1."

Going deeper: why Avro, not JSON, for these two links

It's worth understanding why Iceberg picks a different format for the cover sheet (JSON) than for the index and the folders (Avro). The metadata file gets read whole, once, every time someone opens the table — it's relatively small (a few KB in this guide), and its readability as JSON helps debug problems by hand, as you did in lesson 3. The manifest files, on the other hand, are designed to scale to millions of data files in a real production table — think of kiosko_orders_at_scale.parquet, the 10-million-row table module 5 of this guide inherits from spark-and-distributed-processing-guide — a text format like JSON would be enormously heavier to store and read at that scale, while Avro, binary and compressed (deflate, visible in the file itself if you inspect it byte by byte in lesson 6), stores the same information in a fraction of the space, with its own schema embedded in the file's header — so a reader doesn't need any external schema to decode it. This is the same design decision, applied to the same problem, that already justified why the data files themselves are columnar Parquet and not CSV — lesson 5 of this module revisits it from that angle.

Common mistakes

Trying to count manifest files by summing manifest list entries by hand, opening the .avro with a hex editor. What happens: someone, driven by genuine technical curiosity, tries to decode the manifest list byte by byte to count how many manifest files it enumerates. Why it happens: it's a valid way to learn the Avro format in the abstract, but it's completely unnecessary work to answer the real question. How to spot it: if you find yourself writing an Avro decoder by hand to answer "how many manifest files does this snapshot have?", stop — that question already has a one-line answer. How to fix it: table.inspect.manifests().num_rows answers exactly that question, already decoded, already in a format you can filter and aggregate with pyarrow — this lesson's worked example does it in step 2.

Confusing added_data_files_count with the total row count of data. What happens: someone sees added_data_files_count: 1 and assumes it means "one row of data," instead of "one data file (which can contain many rows)." Why it happens: in Kiosko's case, with only 40 rows in a single file, the confusion doesn't produce a wrong number by coincidence — but it does produce bad intuition for when volumes grow. How to spot it: compare added_data_files_count (from table.inspect.manifests()) against total_rows (from table.scan().to_arrow().num_rows, already used in module 1's lesson 7) — if you expected them to be the same number by definition, revisit this lesson. How to fix it: added_data_files_count counts files, not rows — a single Parquet file can (and typically does) contain thousands or millions of rows; module 5 of this guide, with 10 million rows in fact_orders_at_scale, makes this distinction far more visible, because there a manifest file is going to register several data files, each with many rows inside.

Thinking a manifest list and a manifest file are the same type of file, just with different names. What happens: someone, seeing both are .avro, assumes they play the same role and the distinction is just naming. Why it happens: they share extension and binary format, so at a glance (with file, for example) they look identical. How to spot it: if your code or your explanation treats "manifest list" and "manifest file" as synonyms, revisit this lesson's diagram — each one enumerates a different kind of thing (one enumerates manifest files, the other enumerates data files), and they live at different levels of the chain. How to fix it: the naming pattern helps tell them apart on disk: a manifest list always starts with snap-<snapshot_id>-; a manifest file ends in -m<number>.avro (-m0.avro, -m1.avro, ...). Lesson 6 of this module uses exactly that pattern to identify each one from the terminal.

Exercises

Exercise 1 — Reproduce both steps yourself. On your own machine, with module 1's state available, run this lesson's full script. Confirm you see added_data_files_count: 1, existing_data_files_count: 0, deleted_data_files_count: 0.

See solution

If your table has the single snapshot module 1 left, your output should exactly match this lesson's in structure and business values — only the snapshot_id/added_snapshot_id and the UUID-based file names are going to be different. If manifests.num_rows isn't 1, check how many times you ran a write against this table.

Exercise 2 — Compare the manifest file's name against module 1's lesson 6 pattern. Go back to module 1's lesson 6 diagram (06-loading-kioskos-fact-orders-into-iceberg.md) and compare the name pattern it predicted for the manifest file (metadata/<uuid>-m0.avro) against the real path you got in step 2 of this lesson. Do they match?

See solution

Yes — the <uuid>-m0.avro pattern module 1 predicted, without having verified it with code yet, is exactly the same pattern table.inspect.manifests() confirms in this lesson. The -m0 indicates it's the first (and, in this case, only) manifest file associated with that write; if a single write operation ever generated more than one manifest file (something that can happen with large volumes, outside this small table's scope), you'd see -m1, -m2, etc.

Exercise 3 — Prediction: what would change in table.inspect.manifests() if module 3 did a second append() instead of an overwrite()? Without jumping ahead to module 3, predict: if instead of an overwrite() (which replaces content) a second table.append() ran against this same table, would you expect table.inspect.manifests() to return 1 row or 2 rows? Justify your answer by thinking about what the existing manifest file has done so far.

See solution

It depends on whether Iceberg decides to reuse the existing manifest or create a new one, but the typical case — and the one you'll see in practice — is that a second append() creates a new manifest file (with added_data_files_count: 1 for the new file), and the resulting snapshot references, through a new manifest list, both that new manifest file and the one that already existed — now with "existing" status instead of "added," from the new snapshot's point of view. The central point, more important than the exact number, is that the first write's manifest file doesn't get modified or deleted: a new manifest list simply references it again, alongside what's new. This is the same "nothing gets overwritten" guarantee you already saw in the metadata file (lesson 3) and in the catalog (lesson 2), now one level further into the chain.

Summary and next step

In this lesson you followed the path of the manifest list lesson 3 found in the snapshot, confirmed it's not JSON — it's binary, compressed Avro — and inspected it with the right tool: PyIceberg's table.inspect.manifests(), which returned one row per manifest file, with added_data_files_count: 1, existing_data_files_count: 0, deleted_data_files_count: 0 for the single manifest file that exists so far.

Before moving on you should be able to: explain the difference between a manifest list and a manifest file; explain why both use Avro instead of JSON; and use table.inspect.manifests() to count how many manifest files a table's current snapshot has.

Lesson 5 follows the arrow one last time: it opens the data file this single manifest file enumerates — and, unlike this lesson, you're going to be able to open that file directly, because it turns out to be the same Parquet you already know from the six previous guides in the ecosystem.

Resources

• Apache Iceberg — official documentation, "Table Spec," the "Manifests" and "Manifest Lists" sections, the formal definition of both Avro formats. iceberg.apache.org/spec. In English.
• PyIceberg — API reference, table.inspect.manifests() and the exact columns it returns. py.iceberg.apache.org/api. In English.
• Apache Avro — official file format documentation, the spec that explains the header with embedded schema and the compression this module observes in lesson 6. avro.apache.org/docs/. In English.
• This guide's DESIGN doc — the explicit warning that manifests are Avro, not JSON, and must be inspected with the PyIceberg API, never opened as text. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.