Reading snapshots and history with PyIceberg

Description

Lessons 2 through 6 opened each link in the chain separately — sometimes with direct SQL, sometimes with json.load(), sometimes from the terminal. This lesson brings together the toolkit you're going to use constantly for the rest of this guide: PyIceberg's four inspection methods — table.history(), table.inspect.snapshots(), table.inspect.manifests(), table.inspect.files() — run side by side, over the same table, so you see how they complement each other.

Connection to the module. This lesson doesn't discover any new file — everything you're going to see here you already saw, scattered, in lessons 2 through 6. What this lesson contributes is the consolidated view: the same information, but accessible with four one-line calls, without having to open SQLite or parse JSON by hand.

An analogy: the court file's quick-lookup system

After having walked the archive hallway in person (lesson 6), this lesson is the equivalent of the court finally giving you access to its digital lookup system: instead of walking to the physical shelf every time, you type a query and the system gives you back, in seconds, exactly the same information you would have found by walking — the complete hearing history, the summary of each evidence folder, the inventory of each photo. The system doesn't know anything you couldn't have found by hand; it just makes it faster and less prone to counting mistakes.

Worked example: the four methods, side by side

Step 1 — table.history(): the simple list of snapshots, in order

# snapshots_and_history.py
import os

from pyiceberg.catalog import load_catalog

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.fact_orders")

print("=== table.history() ===")
history = table.history()
print(f"number of entries: {len(history)}")
for entry in history:
    print(f"  snapshot_id={entry.snapshot_id}  timestamp_ms={entry.timestamp_ms}")

What to expect (snapshot_id and timestamp_ms are values assigned in your own run; the structure and the number of entries are deterministic for the state module 1 left):

=== table.history() ===
number of entries: 1
  snapshot_id=<snapshot-id assigned in your run, different each time>  timestamp_ms=<timestamp assigned in your run>

table.history() returns the simplest list of the four: one entry for every snapshot that ever existed, with its snapshot_id and the exact moment (timestamp_ms, milliseconds since epoch) it was created. With a single append() in module 1, this list has exactly one entry — the same snapshot_id you captured in the snap_id variable in that module's lesson 6.

Step 2 — table.inspect.snapshots(): the same list, with more business detail

print("\n=== table.inspect.snapshots() ===")
snaps = table.inspect.snapshots().select(
    ["committed_at", "snapshot_id", "parent_id", "operation", "summary"]
)
print(f"number of rows: {snaps.num_rows}")
for row in snaps.to_pylist():
    summary = dict(row["summary"])
    print(f"  committed_at: {row['committed_at']}")
    print(f"  snapshot_id:  {row['snapshot_id']}")
    print(f"  parent_id:    {row['parent_id']}")
    print(f"  operation:    {row['operation']}")
    print(f"  summary.added-records: {summary['added-records']}")
    print(f"  summary.total-records: {summary['total-records']}")

What to expect (snapshot_id and committed_at are your own run's; parent_id: None, operation: append, and the counts are deterministic):

=== table.inspect.snapshots() ===
number of rows: 1
  committed_at: <date and time assigned in your run>
  snapshot_id:  <snapshot-id assigned in your run, different each time>
  parent_id:    None
  operation:    append
  summary.added-records: 40
  summary.total-records: 40

table.inspect.snapshots() returns a pyarrow.Table — the same kind of result as table.inspect.manifests() and table.inspect.files(), already used in lessons 4 and 5 — with more columns than table.history(): parent_id (the snapshot this one "descends" from — None because this is the table's first, with no ancestor at all), operation (the exact type of write: append in this case), and summary, a map with the same keys you already saw inside metadata["snapshots"][0]["summary"] in lesson 3 — added-records, total-records, and the others you already recognize. Notice parent_id: None: this column is the one that's going to make sense in module 3, when a second snapshot has this first one's snapshot_id as its parent_id — the ancestry chain that makes it possible to reconstruct, in order, how the table got to its current state.

Step 3 — table.inspect.manifests() and table.inspect.files(): the rest of the chain, one line each

print("\n=== table.inspect.manifests() -- number of manifest files ===")
print(table.inspect.manifests().num_rows)

print("\n=== table.inspect.files() -- number of data files ===")
print(table.inspect.files().num_rows)

What to expect:

=== table.inspect.manifests() -- number of manifest files ===
1

=== table.inspect.files() -- number of data files ===
1

The same numbers you already got in full detail in lessons 4 and 5 — here just the count, to make the symmetry clear: one snapshot, one manifest file, one data file. This is exactly lesson 4's "one to one" chain from its diagram, now confirmed with all four methods together.

Step 4 — Confirm current_snapshot() and history()[-1] are the same truth, seen from two angles

print("\n=== table.current_snapshot().snapshot_id == history[-1].snapshot_id ===")
print(table.current_snapshot().snapshot_id == history[-1].snapshot_id)

What to expect:

=== table.current_snapshot().snapshot_id == history[-1].snapshot_id ===
True

table.current_snapshot() — the method you already used in module 1's lessons 5 and 6 to capture snap_id — and table.history()'s last entry always point to the same snapshot: the current one. This isn't a coincidence specific to this table — it's a structural guarantee: current_snapshot() reads current-snapshot-id directly from the metadata file (lesson 3), and history() builds its list by following that same file's snapshot-log field, whose last entry, by definition, always matches the current active snapshot.

Diagram: the four methods, and which link in the chain each one answers

flowchart TB
    T["table (kiosko.fact_orders)"]
    T --> H["table.history()\nsimple list: snapshot_id + timestamp"]
    T --> S["table.inspect.snapshots()\nsnapshot_id + parent_id + operation + summary"]
    T --> M["table.inspect.manifests()\none row per manifest file"]
    T --> F["table.inspect.files()\none row per data file"]

    H -.->|"answers: 'which snapshots existed, in order'"| Q1["metadata.json: snapshots[]\n(lesson 3)"]
    S -.->|"answers: 'what happened in each snapshot'"| Q1
    M -.->|"answers: 'which manifest files does the current one have'"| Q2["manifest list + manifest files\n(lesson 4)"]
    F -.->|"answers: 'which data files does the current one have'"| Q3["data files\n(lesson 5)"]

Going deeper: why four methods exist, and not just one "that tells everything"

It might seem simpler for PyIceberg to offer a single method, something like table.inspect.everything(), that returned the whole chain at once. The reason it doesn't exist is the same reason the chain has several links in the first place: every question has a different cost to answer, and every method is designed for the minimum cost of the question it answers. table.history() only needs to read the current metadata file — cheap, always. table.inspect.snapshots() too, with a bit more detail per row. But table.inspect.manifests() needs to open the manifest list (and, depending on the implementation, potentially each manifest file) to build its result — more expensive, proportional to how many manifests exist. And table.inspect.files() is the most expensive of the four: it needs to open every manifest file to enumerate every individual data file — in a table with millions of files, like kiosko.fact_orders_at_scale could become in module 5, this method does far more work than table.history(). Separating these four methods, instead of one that always does the most expensive work, is a design decision that respects the same indirection chain this whole module taught: you don't pay the cost of opening manifest files if you only need to know how many snapshots exist.

Common mistakes

Calling table.inspect.files() repeatedly inside a loop, without saving the result. What happens: someone, needing the file count several times in the same script, calls table.inspect.files() every time they need it, instead of saving the result once in a variable. Why it happens: in a short script, with a table as small as kiosko.fact_orders, the extra cost is imperceptible, so the habit doesn't get corrected in time. How to spot it: if your script calls table.inspect.files() (or .manifests()) more than once with no change to the table between calls, revisit this lesson's Going deeper section on each method's relative cost. How to fix it: save the result in a variable the first time — exactly like files = table.inspect.files() did in lesson 5 — and reuse it; on a large table, with many manifest files, this habit avoids repeated, unnecessary work.

Confusing table.history() (a Python list) with table.inspect.snapshots() (a pyarrow.Table), and using the wrong syntax on each. What happens: someone tries to use .select([...]) on table.history()'s result, or iterates with a for row in ...to_pylist() over table.history() as if it were a pyarrow.Table, and gets an attribute error. Why it happens: both methods answer related questions (which snapshots exist), so it's easy to assume they have the same result shape. How to spot it: if your code fails with AttributeError: 'list' object has no attribute 'select' (or the reverse error, 'Table' object is not iterable directly), check which of the two methods you're using. How to fix it: table.history() returns a normal Python list of SnapshotLogEntry objects — iterate it with a plain for, as in step 1 of this lesson; all four methods under table.inspect.* return pyarrow.Tables — filter them with .select([...]), count them with .num_rows, list them as dictionaries with .to_pylist(), as in steps 2 and 3.

Expecting table.inspect.manifests() and table.inspect.files() to always return the same number of rows. What happens: someone, after seeing that both methods return 1 on kiosko.fact_orders, assumes this is a general rule — that there are always as many manifest files as data files. Why it happens: in this table's current state, with a single small write, the "one to one" chain makes them coincide by chance. How to spot it: if on a table with several accumulated writes (something you're going to see starting in module 3) you expect both numbers to keep matching, revisit this module's lesson 4 diagram. How to fix it: a single manifest file can enumerate several data files — for example, if a large write gets split into several Parquet files by size — so table.inspect.files().num_rows is, in general, going to be greater than or equal to table.inspect.manifests().num_rows, never necessarily equal.

Exercises

Exercise 1 — Reproduce the four methods yourself, and verify the final equality. On your own machine, with module 1's state available, run this lesson's full script. Confirm step 4 prints True.

Exercise 2 — Calculate, yourself, how many milliseconds passed between committed_at and now. Using the timestamp_ms you got in step 1 of this lesson, write a line of code that calculates how many seconds have passed from when that snapshot was created to the moment you run the calculation. (Hint: use time.time() only for this one-off comparison — never to generate Kiosko business data, which is this guide's hard rule.)

import time

elapsed_seconds = time.time() - (history[0].timestamp_ms / 1000)
print(f"Seconds since commit: {elapsed_seconds:.1f}")

Exercise 3 — Prediction: if you called these four methods on a freshly created table, with no append() yet, what would you expect each to return? Remembering module 1's lesson 5 (table.current_snapshot() returned None before the first load), predict what each of this lesson's four methods would return on that same empty table.

Summary and next step

In this lesson you ran PyIceberg's four inspection methods — table.history(), table.inspect.snapshots(), table.inspect.manifests(), table.inspect.files() — side by side, over kiosko.fact_orders, and confirmed all of them consistently describe the same state: one snapshot (operation: append, 40 records), one manifest file, one data file. You also confirmed table.current_snapshot() and table.history()'s last entry are always the same truth seen from two different angles.

Before moving on you should be able to: choose which of the four methods to use depending on which question you need to answer; explain why four separate methods exist instead of just one; and reproduce all four on your own table.

With the whole chain walked — three different ways: pure code (lessons 2-5), the terminal (lesson 6), and the dedicated inspection API (this lesson) — lesson 8 closes the module with a single project that brings the previous seven together into one map of kiosko.fact_orders's complete anatomy.

Resources

• PyIceberg — API reference, the complete table.inspect section, with the list of every available method (snapshots, manifests, files, entries, partitions, and others later modules of this guide are going to use). py.iceberg.apache.org/api. In English.
• PyIceberg — API reference, Table.history() and the SnapshotLogEntry class. py.iceberg.apache.org/api. In English.
• Apache Iceberg — official documentation, "Table Spec," "Snapshots" section, the formal definition of parent-snapshot-id that makes the ancestry chain between snapshots possible. iceberg.apache.org/spec. In English.
• This guide's DESIGN doc — the exact list of the four inspection methods this module had to run against kiosko.fact_orders. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.