Project: Kiosko's table anatomy, mapped

Description

This project closes the module. You walked the catalog (lesson 2), the metadata file (lesson 3), the manifest list and the manifest files (lesson 4), the data files (lesson 5), the same walkthrough from the terminal (lesson 6), and PyIceberg's four inspection methods together (lesson 7). One step is left: a single script that walks all five links in the chain, end to end, and prints a complete map of kiosko.fact_orders's anatomy — with final assert statements that automatically confirm the whole chain is consistent.

Connection to the module. This project doesn't introduce any new concept — it's the final integration of the seven previous lessons, exactly the way module 1's lesson 8 integrated its own five central lessons. It literally revisits the question that opened this module in lesson 1: what points to what, and why is nothing ever overwritten? This project answers that question with a single script, run end to end, against Kiosko's real table.

An analogy: the case file's complete map, at a glance

Lessons 2 through 7 of this module had you walk the whole case file, piece by piece: the desk, the cover sheet, the evidence index, the evidence folders, the actual photos, and finally the court's quick-lookup system. This project is the moment to draw, on a single sheet, the complete map of that walkthrough — like the diagram an experienced archivist prepares for a complex case, showing at a glance how each piece connects to the next, with no one having to walk the whole hallway again to understand it.

The material: everything this module inspected, in one place

You need, in the same working directory where you completed module 1 (with kiosko_catalog.db and kiosko_warehouse/ already created):

(module 1's directory, reused unchanged)
├── kiosko_catalog.db
├── kiosko_warehouse/
│   └── kiosko/fact_orders/...
└── kiosko_table_anatomy_mapped.py    (this project)

This project is read-only — it doesn't create any new table, it doesn't add a single row. If you didn't keep module 1's directory, you can recreate it by running that module's closing project again (kiosko_first_iceberg_table.py, M1 lesson 8) in a new directory before continuing.

The reference solution, verified

# kiosko_table_anatomy_mapped.py -- module 2 closing project
# maps the full chain: catalog -> metadata -> manifest list -> manifest files -> data files
# over kiosko.fact_orders, exactly as module 1 left it (a single snapshot).
import json
import os
import sqlite3
from urllib.parse import urlparse

from pyiceberg.catalog import load_catalog


def main() -> None:
    print("=== Kiosko: complete anatomy of kiosko.fact_orders ===\n")

    warehouse_path = os.path.abspath("kiosko_warehouse")
    catalog_db_path = os.path.abspath("kiosko_catalog.db")

    # --- Link 1: the catalog ---------------------------------------------
    conn = sqlite3.connect(catalog_db_path)
    cur = conn.cursor()
    cur.execute(
        "SELECT metadata_location, previous_metadata_location FROM iceberg_tables "
        "WHERE table_name = 'fact_orders';"
    )
    metadata_location, previous_metadata_location = cur.fetchone()
    conn.close()
    print("Link 1/5 -- catalog (kiosko_catalog.db)")
    print(f"  metadata_location:          {os.path.basename(urlparse(metadata_location).path)}")
    print(f"  previous_metadata_location: {os.path.basename(urlparse(previous_metadata_location).path)}")

    # --- Link 2: the metadata file ----------------------------------------
    metadata_path = urlparse(metadata_location).path
    with open(metadata_path) as f:
        metadata = json.load(f)
    snap = metadata["snapshots"][0]
    print("\nLink 2/5 -- metadata file (readable JSON)")
    print(f"  format-version: {metadata['format-version']}")
    print(f"  columns in the schema: {len(metadata['schemas'][0]['fields'])}")
    print(f"  number of snapshots: {len(metadata['snapshots'])}")
    print(f"  current snapshot -- operation: {snap['summary']['operation']}, "
          f"total-records: {snap['summary']['total-records']}")

    # --- Links 3-5: via PyIceberg (manifest list/files, data) -------------
    catalog = load_catalog(
        "kiosko", type="sql",
        uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
    )
    table = catalog.load_table("kiosko.fact_orders")

    manifests = table.inspect.manifests()
    files = table.inspect.files()
    history = table.history()
    snapshots = table.inspect.snapshots()

    print("\nLink 3/5 -- manifest list (referenced from the snapshot)")
    print(f"  manifest_list: {os.path.basename(table.current_snapshot().manifest_list)}")

    print("\nLink 4/5 -- manifest file(s)")
    print(f"  number of manifest files: {manifests.num_rows}")
    for row in manifests.select(["added_data_files_count", "existing_data_files_count"]).to_pylist():
        print(f"    added_data_files_count={row['added_data_files_count']}, "
              f"existing_data_files_count={row['existing_data_files_count']}")

    print("\nLink 5/5 -- data file(s)")
    print(f"  number of data files: {files.num_rows}")
    for row in files.select(["file_format", "record_count"]).to_pylist():
        print(f"    file_format={row['file_format']}, record_count={row['record_count']}")

    print("\n=== Final verification: a single snapshot, consistent chain ===\n")
    print(f"table.history() -- entries: {len(history)}")
    print(f"table.inspect.snapshots() -- rows: {snapshots.num_rows}")
    print(f"table.inspect.manifests() -- rows: {manifests.num_rows}")
    print(f"table.inspect.files() -- rows: {files.num_rows}")

    assert len(history) == 1, f"expected 1 history entry, got {len(history)}"
    assert snapshots.num_rows == 1, f"expected 1 snapshot, got {snapshots.num_rows}"
    assert manifests.num_rows == 1, f"expected 1 manifest file, got {manifests.num_rows}"
    assert files.num_rows == 1, f"expected 1 data file, got {files.num_rows}"
    assert table.current_snapshot().snapshot_id == history[-1].snapshot_id
    assert previous_metadata_location != metadata_location

    total_rows = table.scan().to_arrow().num_rows
    assert total_rows == 40, f"expected 40 rows, got {total_rows}"

    print(f"\nAll checks passed: consistent chain, {total_rows} rows, 1 snapshot throughout the module.")


if __name__ == "__main__":
    main()

What to expect (verified by running the actual python3 kiosko_table_anatomy_mapped.py, against the exact state module 1 left; the UUID-based file names and the snapshot_id inside manifest_list are your own run's, different each time — the rest of the structure and every business value are deterministic):

=== Kiosko: complete anatomy of kiosko.fact_orders ===

Link 1/5 -- catalog (kiosko_catalog.db)
  metadata_location:          00001-<uuid>.metadata.json
  previous_metadata_location: 00000-<uuid>.metadata.json

Link 2/5 -- metadata file (readable JSON)
  format-version: 2
  columns in the schema: 7
  number of snapshots: 1
  current snapshot -- operation: append, total-records: 40

Link 3/5 -- manifest list (referenced from the snapshot)
  manifest_list: snap-<snapshot-id assigned in your run>-0-<uuid>.avro

Link 4/5 -- manifest file(s)
  number of manifest files: 1
    added_data_files_count=1, existing_data_files_count=0

Link 5/5 -- data file(s)
  number of data files: 1
    file_format=PARQUET, record_count=40

=== Final verification: a single snapshot, consistent chain ===

table.history() -- entries: 1
table.inspect.snapshots() -- rows: 1
table.inspect.manifests() -- rows: 1
table.inspect.files() -- rows: 1

All checks passed: consistent chain, 40 rows, 1 snapshot throughout the module.

Notice the four assert statements before the final message: they're not decorative. assert previous_metadata_location != metadata_location confirms, with code — not just a visual ls like in lesson 6 — that the catalog explicitly tells apart "the current one" from "the previous one," and that both are different values, neither empty. The other three assert statements confirm, precisely, that lesson 7's four methods — history(), inspect.snapshots(), inspect.manifests(), inspect.files() — all agree on the same number: 1. If any of the chain's five links were inconsistent with the others — for example, if the catalog pointed to a metadata file saying "2 snapshots" while table.history() found only 1 — one of these assert statements would fail immediately.

Diagram: the five links, mapped in a single walkthrough

flowchart TB
    A["Lesson 2:\ncatalog\n(kiosko_catalog.db)"] --> B["Lesson 3:\nmetadata file\n(JSON, 7 columns, 1 snapshot)"]
    B --> C["Lesson 4:\nmanifest list + manifest files\n(Avro, 1 manifest file)"]
    C --> D["Lesson 5:\ndata files\n(Parquet, 40 rows)"]
    D --> E["Lesson 6:\nsame walkthrough,\nfrom the terminal"]
    E --> F["Lesson 7:\nthe 4 table.inspect methods\ntogether, consistent"]
    F --> G["This project:\na single script,\nautomatic asserts"]
    G --> H["Module 3:\nsnapshots and time travel\n(a SECOND snapshot, finally)"]

Closing lesson 1's promise, point by point

What lesson 1 promised	Evidence this module delivered it
Name the five links in the chain	Lesson 1: the court-case-file analogy, mapped link by link
Open the catalog, and confirm it only stores an address	Lesson 2: iceberg_tables with metadata_location/previous_metadata_location, nothing more
Open the metadata file, readable JSON	Lesson 3: 7-column schema, empty partition-specs, 1 snapshot, all read with json.load()
Tell manifest list apart from manifest files, and why they're Avro	Lesson 4: table.inspect.manifests(), 1 manifest file, added_data_files_count=1
Confirm the data files are the same Parquet as always	Lesson 5: direct pq.read_table(), same field_id as in the declared schema
Walk the chain on disk, with no Iceberg code at all	Lesson 6: ls, file, head — exactly 5 files, readability confirmed by type
Use PyIceberg's inspection API as a toolkit	Lesson 7: history(), inspect.snapshots(), inspect.manifests(), inspect.files(), consistent
Confirm nothing got overwritten throughout the module	This project: previous_metadata_location != metadata_location, whole chain consistent

No Kiosko data changed during this module — kiosko.fact_orders still has exactly forty rows, the same 106.15 revenue module 1's lesson 7 verified, and the same single snapshot this module started with. What changed is your ability to answer, with first-hand evidence, exactly what's behind that snapshot.

Common mistakes

Running this project on a directory where you already moved on to module 3 (or further). What happens: someone runs this script after already doing a second write against kiosko.fact_orders (getting ahead to module 3), and this project's assert statements fail, because there's now more than one snapshot. Why it happens: this project deliberately assumes the exact state module 1 leaves — a single snapshot — because that's the simplest possible anatomy to learn without noise. How to spot it: if AssertionError: expected 1 snapshot, got 2 (or a higher number) shows up, you already advanced your table's state beyond what this project expects. How to fix it: this isn't an error in your table — it's exactly expected if you already did a second write; run this project on a separate copy of module 1's state, or simply move on: module 3 is deliberately going to use a new table (kiosko.dim_product) so as not to interfere with the single-write anatomy this project documented.

Being surprised the project doesn't create any table or load any new data. What happens: someone, used to module 1's closing-project pattern — which did build a table from scratch — expects this project to do the same. Why it happens: the "closing project = build something end to end" pattern was already established in the previous module. How to spot it: if you look for a call to catalog.create_table() or table.append() in this lesson's script and don't find it, that's correct — it's not there on purpose. How to fix it: nothing to fix — this whole module is read-only over what module 1 already built; the goal is understanding the existing anatomy, not creating a new one. Module 3 picks construction back up, with a new write against a different table.

Running the script without having completed module 1 first. What happens: someone tries to run this project in a directory where kiosko_catalog.db and kiosko_warehouse/ never existed, and the script fails immediately trying to connect to an empty SQLite database or load a table that doesn't exist. Why it happens: unlike module 1's closing project, this project isn't self-contained — it explicitly depends on the state that earlier module left behind. How to spot it: if you see an error related to an empty iceberg_tables or a NoSuchTableError for kiosko.fact_orders, check whether your working directory has the right kiosko_catalog.db. How to fix it: run module 1's closing project (kiosko_first_iceberg_table.py) first, in the same directory you're going to run this script in, or copy kiosko_catalog.db and kiosko_warehouse/ from where you already completed it.

Exercises

Exercise 1 — Run the whole project yourself, against your own module 1 state. In the directory where you completed module 1, add kiosko_table_anatomy_mapped.py and run python3 kiosko_table_anatomy_mapped.py. Confirm you see the five links and the final "All checks passed" message.

See solution

If your kiosko.fact_orders table has exactly the state module 1 left (one append(), forty rows, one snapshot), the output should exactly reproduce this lesson's structure: five numbered links, followed by the final verification with all four counts at 1 and a total of 40 rows. The UUID-based file names and the snapshot_id inside the manifest list's name are going to be different from what's shown here — that's exactly expected, not an error.

Exercise 2 — Break an assert on purpose, and watch it fail. Temporarily change the line assert files.num_rows == 1, ... to assert files.num_rows == 99, ..., run the script again, and observe what happens. Then revert the change.

See solution

The script should fail with AssertionError: expected 99, got 1 (the message includes the actual value found, 1, alongside the expected value you broke on purpose, 99) — execution stops right at that assert, never reaching the final success message. This exercise confirms, the same way the equivalent exercise in module 1's closing project already did, that this script's assert statements are real checks, not decoration — if any link in the chain were genuinely inconsistent, the script would tell you loudly and immediately.

Exercise 3 — Explain, in your own words, why this project verifies consistency between the four inspection methods, and not just each one's result on its own. In 3-4 sentences, justify why comparing len(history), snapshots.num_rows, manifests.num_rows, and files.num_rows against each other — all must equal 1 — is a stronger check than simply confirming each one, on its own, didn't throw any error.

See solution

A method not throwing an error only confirms the call was syntactically valid and the corresponding file could be read — it doesn't confirm that file's content is coherent with the rest of the chain. If, through some unlikely corruption, the metadata file said "there are 2 snapshots" but one of their manifest lists had been mistakenly deleted, table.inspect.snapshots() could keep working with no error while table.inspect.manifests() failed or returned a number that doesn't add up. Comparing all four results against each other — they must all be consistent with the same number of real writes — is the same "verify, don't trust" discipline module 1's lesson 7 already demanded about Kiosko's revenue, now applied to the table's internal structure instead of its business values.

Summary and next step: closing this module

With this project you close module 2. You walked the complete chain — catalog → metadata → manifest list → manifest files → data files — five times, each time from a different angle: direct SQL, raw JSON, PyIceberg's inspection API, the terminal, and finally this single script that ties everything together with automatic assert statements. You confirmed, with evidence from your own disk, that kiosko.fact_orders has exactly one snapshot, one manifest file, and one data file — the simplest possible anatomy, and the exact starting point the rest of this guide builds on.

None of the four problems named in module 1's lesson 2 (atomicity, history, schema evolution, hidden partitioning) is solved yet — that's exactly right at this point in the guide. What you have now is something different and necessary: you know, with file-level and field-level precision, where each piece of the truth about an Iceberg table lives, and why that separation into layers is what makes it possible for module 3 to add a second snapshot without touching a single byte of the first.

Where you go next. Module 3 — Snapshots and time travel — is where this anatomy stops being static for the first time: you're going to create kiosko.dim_product, with no history column at all, do an overwrite() with the P002 change (snacks/0.60 → health-snacks/0.68), and use table.scan(snapshot_id=...) to recover the earlier state — the first time in this guide you're going to see, with your own eyes, two snapshots coexisting in the same chain this module just mapped.

Resources

• PyIceberg — official documentation (quickstart) and API reference, the complete inspection flow this project integrates. py.iceberg.apache.org · py.iceberg.apache.org/api. In English.
• Apache Iceberg — official documentation, reference version 1.11.0, the complete "Table Spec," the formal source for the five links this module walked. iceberg.apache.org/spec. In English.
• This guide's DESIGN doc — the full map of the eight modules, including module 3 which follows. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.