Module overview: the anatomy of an Iceberg table

Why this module exists

Module 1 ended with a promise deliberately kept only halfway. You installed PyIceberg, created the kiosko catalog, and loaded Kiosko's forty rows for the week into kiosko.fact_orders with table.append(). You confirmed, with table.current_snapshot() no longer being None, that a real snapshot got created. And you saw, in that module's lesson 6 diagram, a list of files that appeared on disk after that load: a data file, a manifest file, a manifest list, and two metadata files. But that module never opened a single one of those files. It named the chain — catalog → metadata → manifest list → manifest files → data files — without yet showing what's inside each link, or why the order matters.

This module opens that box. It doesn't add a single new row to kiosko.fact_orders — the table you're going to inspect is exactly the one module 1 left behind, with its single snapshot; instead, it walks you through, one by one, the five links in that chain: first the catalog, the SqlCatalog backed by SQLite you installed in M1's lesson 4; then the metadata file, readable JSON, which declares the schema and the list of snapshots; then the manifest list and the manifest files, which aren't JSON — they're Avro, a binary format a text editor can't show you without help; and finally the data files, which turn out to be exactly the same Parquet you already know from the six previous guides in the ecosystem. By the end of this module you'll be able to answer, with evidence from your own disk, the question this module exists to answer: what points to what, and why is none of it ever overwritten?

The case that runs through the module: the same table, without a single new row

This module doesn't add data to Kiosko. kiosko.fact_orders still has exactly forty rows, the same 106.15 total revenue (S01=38.3, S02=38.8, S03=29.05) module 1's lesson 7 verified, and — this matters — a single snapshot, the one table.append() created in that module's lesson 6. Every script in this module starts from that same state: the kiosko catalog loaded, the kiosko.fact_orders table with its forty rows, not a single write since then. You're going to see that same snapshot show up, again and again, from different angles — as a row in a SQLite table, as a block inside a JSON file, as a row in a pyarrow.Table returned by table.inspect.snapshots() — and in every angle you're going to recognize the same numbers.

An analogy: a court case file

Think about how a real court case file is organized. It isn't a single document — it's a chain of references, each pointing to the next, and none ever replacing the previous one:

• The cover sheet of the file is the first thing anyone consults: it states the case number, the date of the most recent hearing, and — most importantly — which evidence index to go to for that hearing's current evidence. The cover sheet doesn't contain any evidence itself.
• The evidence index for this hearing is a list: "exhibit 1, in folder A; exhibit 2, in folder B; exhibit 3, in folder C." The index doesn't contain the evidence — it points to where it is.
• Each evidence folder contains a batch of related exhibits, with its own sub-index: "this folder has 3 photos, added at this hearing; 0 photos inherited from earlier hearings; 0 photos removed."
• And finally, inside each folder, there are the actual photos — the real evidence, exactly as it looks, with no more layers of indirection.

If a new exhibit gets added at the next hearing, nobody rewrites the old cover sheet, or the old index, or the old folders — a new cover sheet gets filed, pointing to a new index, which can reuse the old folders (if that evidence is still valid) and add new folders for what changed. The whole case file, with all its earlier versions, keeps existing, available to anyone who needs to reconstruct "what did this case look like at hearing 2?"

This guide uses exactly that chain for Apache Iceberg: the cover sheet is the metadata file (lesson 3); the evidence index for this hearing is the manifest list (lesson 4); the evidence folders are the manifest files (also lesson 4); and the actual photos are the data files — Parquet, the same as always (lesson 5). The catalog (lesson 2) is, in this analogy, the court clerk's desk that tells you, without you having to search through a hundred old cover sheets, "the current file for this case is at this exact cover sheet, right now."

Diagram: the whole chain, at a glance

flowchart TB
    CAT["Catalog: kiosko\n(kiosko_catalog.db, SQLite)\n<< the court clerk's desk >>"]
    META["Metadata file\n(metadata/*.metadata.json)\n<< the case file's cover sheet >>"]
    MLIST["Manifest list\n(metadata/snap-*.avro)\n<< this hearing's evidence index >>"]
    MFILE["Manifest file(s)\n(metadata/*-m0.avro)\n<< the evidence folders >>"]
    DATA["Data file(s)\n(data/*.parquet)\n<< the actual photos >>"]

    CAT -->|"metadata_location points to"| META
    META -->|"current snapshot.manifest-list points to"| MLIST
    MLIST -->|"lists"| MFILE
    MFILE -->|"lists"| DATA

Notice the direction of the arrows: each link points forward, never directly containing what the next link holds. The catalog doesn't know the table's schema — that lives in the metadata; the metadata doesn't know which Parquet files exist — that lives, indirectly, behind the manifest list and the manifest files. Each layer solves a different problem, and that separation is exactly what makes it possible for adding a new data file (a future write) to not require rewriting any old data file.

The map of this module

Lesson    Which link in the chain it covers
────────  ──────────────────────────────────────────────────────────────
L1        (this one) The full map of the chain, before opening anything
L2        The catalog: kiosko_catalog.db as a pointer to the current metadata
L3        The metadata file: readable JSON, schema, partitioning, snapshots
L4        The manifest list and the manifest files: Avro, not JSON
L5        The data files: the same Parquet you already know
L6        Full walkthrough on disk, with real shell commands
L7        The same chain, now with PyIceberg's inspection API
L8        Project: the complete anatomy of kiosko.fact_orders, mapped

Lessons 2 through 5 walk the chain from top to bottom, one link per lesson, each with real code run against kiosko.fact_orders. Lesson 6 repeats the full walkthrough, but this time from the terminal, with ls, file, and cat, so you see with your own eyes the difference between a readable file and one that isn't. Lesson 7 shows that PyIceberg already packages that same inspection into four one-line methods — table.inspect.snapshots(), table.inspect.manifests(), table.inspect.files(), table.history() — which you're going to use constantly for the rest of this guide. And lesson 8 brings the previous seven together into a single script that maps the table's complete anatomy, end to end.

The boundary: what does NOT belong in this module

This module inspects a 100% local catalog: kiosko_catalog.db, backed by SQLite, exactly the same one you installed in module 1's lesson 4. Production catalogs — REST, AWS Glue Catalog, Unity Catalog, Polaris — solve the same underlying problem (a single pointer to the current metadata, with concurrency control) over cloud-managed infrastructure, with real authentication and permissions; this guide names them only in module 7, without implementing them. Here, "catalog" means, unambiguously, a row in a SQLite database on your own filesystem.

No new write belongs in this module either. kiosko.fact_orders ends this module with exactly the same forty rows and the same single snapshot it started with — creating a second snapshot, and traveling between the two, is precisely module 3's job.

Common mistakes

Thinking "inspecting the anatomy" requires modifying the table. What happens: someone, arriving at this module, expects the lessons to ask them to load more data or create a new table. Why it happens: the ecosystem's earlier modules — and module 1 of this guide itself — almost always ended with a new write. How to spot it: if you finish this module and table.inspect.snapshots() shows more than one row, check which script you ran — no worked example in this module calls table.append(), table.overwrite(), or any other write operation. How to fix it: this module is read-only over the state module 1 left behind; if you accidentally wrote something new, you can still keep using the table — you'll just see more than one snapshot in later lessons than they themselves predict.

Trying to open an .avro file with a text editor and concluding it's "broken" or "corrupted." What happens: someone, with the healthy curiosity to look at everything, opens a manifest file or a manifest list with cat, vim, or their operating system's text editor, sees a mix of readable text and unreadable symbols, and assumes something got damaged. Why it happens: the metadata file is readable JSON, so it's reasonable to expect the other files in the metadata/ folder to be too. How to spot it: if you see fragments of recognizable text (like the schema's JSON) mixed with nonsensical characters, that's exactly what's expected from a compressed Avro file — it's not corruption, it's PyIceberg's correct format. How to fix it: lesson 4 and lesson 6 of this module show, with real evidence, why this happens and how to inspect those files correctly — with PyIceberg's table.inspect.manifests()/table.inspect.files(), never by opening them as plain text.

Exercises

Exercise 1 — Reconstruct the whole analogy, without looking back. Without rereading this lesson's analogy section, write from memory the five links in the court-case chain (cover sheet, evidence index, evidence folders, actual photos, court clerk's desk) and which real Iceberg piece each one corresponds to.

See solution

Court clerk's desk → catalog (kiosko_catalog.db); case file's cover sheet → metadata file (*.metadata.json); this hearing's evidence index → manifest list (snap-*.avro); evidence folders → manifest file(s) (*-m0.avro); actual photos → data file(s) (*.parquet). Order matters: the clerk's desk takes you to the current cover sheet, the cover sheet takes you to this hearing's index, the index takes you to the folders, and the folders take you to the photos — never the reverse, and there's never a jump that skips a link.

Exercise 2 — Prediction: how many files do you expect to find in each category? Based on what you learned in module 1 — a table created empty (lesson 5), and then a single load with table.append() (lesson 6) — predict how many JSON metadata files, how many manifest lists, how many manifest files, and how many data files you should find in kiosko_warehouse/kiosko/fact_orders/ before running any lesson in this module.

See solution

Two JSON metadata files (one from the empty table in M1's lesson 5, another from the first snapshot in lesson 6), a single manifest list (the one existing snapshot), a single manifest file (that snapshot added a single batch of data), and a single data file (the forty rows fit into one Parquet). Lesson 6 of this module confirms this prediction by counting the actual files on disk.

Exercise 3 — Explain in your own words why "nothing gets overwritten" isn't just a slogan. In 2-3 sentences, and without looking at module 1 yet, explain what concrete evidence — not a marketing promise — you'll be able to show by the end of this module to prove Iceberg never overwrites an existing file.

See solution

There's no single correct answer — it's a hypothesis to verify — but the concrete evidence this module is going to show is counting files: you're going to see, on disk, two distinct .metadata.json files (one from the empty table, another from the first load) coexisting in the same folder, neither one deleted. That coexistence — not a claim in some text, but a real ls showing both files with different timestamps — is the proof that "nothing gets overwritten" is a verifiable behavior, not just a promise.

Summary and next step

In this lesson you walked through, without running any inspection code yet, the full map of the chain this module is going to open: catalog → metadata → manifest list → manifest files → data files, with the court-case-file analogy as your guide. You confirmed this module doesn't add a single new row to kiosko.fact_orders — it only opens what module 1 already left written.

Before moving on you should be able to: name the five links in the chain, in order; and explain why each link points forward instead of directly containing what the next one holds.

Lesson 2 opens the first link: the catalog, the simplest of the five pieces, and the one that makes it possible for any reader to find the current metadata without having to guess.

Resources

• Apache Iceberg — official documentation, "Table Spec," the formal definition of the metadata → manifest list → manifest file → data file chain this module walks through. iceberg.apache.org/spec. In English.
• PyIceberg — API reference, the table.inspect.* methods this module's lesson 7 uses in depth. py.iceberg.apache.org/api. In English.
• This guide's DESIGN doc — the "Table anatomy" section (M2), source of the exact list of commands this module runs. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.