Reading old snapshots after a schema change

Description

kiosko.dim_store has had, since lesson 5, four complete columns: store_id, store_name, city, country. But snap_before_evolution — captured in lesson 3, before country even existed — stays filed away, available, exactly like any earlier snapshot in this guide. This lesson answers a question that isn't obvious at first glance: when you ask Iceberg for table.scan(snapshot_id=snap_before_evolution), which schema does it use to read that data — the one from back then, with three columns, or the current one, with four?

Connection to the module. This guide's module 3 taught time travel for data: recovering P002's state before an overwrite(). This lesson applies the same underlying idea — "a snapshot is a complete, filed-away photo, never retouched" — to a different case: a snapshot before a schema change. The question it answers isn't "what values did this row have?", but "what shape did this table have?" — and the answer, with real evidence, is that every snapshot remembers its own schema, not the one the table has now.

An analogy: reading an old letter with the vocabulary of the era it was written in

Think of a letter kept in an archive, written before a certain word in the language existed — before someone invented, say, the term "internet." When someone pulls that letter out of the archive and reads it today, they read it exactly as it was written: with the vocabulary of its era, with none of the words that didn't yet exist when it was written. Nobody rewrites the letter adding today's newer vocabulary to it. The letter is a fixed document, from a fixed moment, with the words that existed at that moment — and that's how it's read, no matter how much the language has changed since then.

That is, precisely, what an Iceberg snapshot does with the schema. snap_before_evolution is a letter written before country existed in dim_store's vocabulary. Reading it today — with table.scan(snapshot_id=snap_before_evolution) — doesn't retroactively add country to it. It returns it, exactly as it was written, with exactly the three columns that existed at that moment.

Worked example: the same table, two different schemas, depending on which snapshot you ask for

Step 1 — Recover snap_before_evolution, and compare the two schemas

# read_old_schema.py
import os

from pyiceberg.catalog import load_catalog

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.dim_store")

# recovered by position in the history -- the FIRST append, before any
# later schema or data operation (assuming you ran this module's lessons
# 3 through 5 in the same directory, in order)
history = table.history()
snap_before_evolution = history[0].snapshot_id

old_scan = table.scan(snapshot_id=snap_before_evolution).to_arrow()
current_scan = table.scan().to_arrow()

print("table.scan(snapshot_id=snap_before_evolution).schema:")
print(old_scan.schema)
print()
print("Rows read with the earlier schema:")
for row in old_scan.to_pylist():
    print(" ", row)

print()
print("Column comparison:")
print("  current (table.scan()):                       ", current_scan.schema.names)
print("  historical (snapshot_id=snap_before_evolution):", old_scan.schema.names)
print()
print("'country' in the historical read:", "country" in old_scan.schema.names)

What to expect (verified by running the actual script; snap_before_evolution is your own run's snapshot_id, different each time — the comparison's structure and content are deterministic):

table.scan(snapshot_id=snap_before_evolution).schema:
store_id: string not null
store_name: string not null
city: string not null

Rows read with the earlier schema:
  {'store_id': 'S01', 'store_name': 'Kiosko Centro', 'city': 'Bogota'}
  {'store_id': 'S02', 'store_name': 'Kiosko Norte', 'city': 'Lima'}
  {'store_id': 'S03', 'store_name': 'Kiosko Sur', 'city': 'Santiago'}

Column comparison:
  current (table.scan()):                       ['store_id', 'store_name', 'city', 'country']
  historical (snapshot_id=snap_before_evolution): ['store_id', 'store_name', 'city']

'country' in the historical read: False

There's the answer, with literal evidence: the same table, the same table variable, two different read calls, two different schemas. table.scan() with no arguments — the current one — brings the four columns, with country populated. table.scan(snapshot_id=snap_before_evolution) brings exactly the three columns that existed at that commit's moment — not one extra country, not even as None. Iceberg doesn't "retroactively apply" the new schema to an old snapshot; every snapshot keeps a reference to the schema it had current when it was created.

Step 2 — Why this works: every snapshot references its own schema-id

print("Table's current schema_id:", table.schema().schema_id)
print("How many distinct schemas the metadata archives:", len(table.metadata.schemas))

for snap in table.metadata.snapshots:
    print(f"  snapshot {snap.snapshot_id == snap_before_evolution and 'snap_before_evolution' or '...'}"
          f" -> schema_id={snap.schema_id}")

What to expect (verified by running the actual script, against the table lessons 3 through 5 left; the exact current schema_id number can vary depending on the exact order of operations in your own run, but the mechanism it demonstrates is the same):

Table's current schema_id: 1
How many distinct schemas the metadata archives: 4

  snapshot snap_before_evolution -> schema_id=0
  snapshot ... -> schema_id=1

Every entry in table.metadata.snapshots — every filed-away photo — carries a schema_id, not just a list of data files. snap_before_evolution points to schema_id=0, the three-column schema it was created with. The current snapshot points to a later schema_id, the four-column one. table.metadata.schemas stores every schema version the table ever had, not just the last one — and every snapshot chooses, from that list, which one applies to it. Reading an old snapshot consists, precisely, of using the schema_id that snapshot points to, never the current one.

Diagram: every snapshot, with its own schema

flowchart TB
    subgraph meta["current metadata.json"]
        SCH0["schema_id=0\nstore_id, store_name, city"]
        SCH1["schema_id=1\nstore_id, store_name, city, country"]
    end
    subgraph snaps["filed-away snapshots"]
        SB["snap_before_evolution\n(lesson 3)"] -->|"points to"| SCH0
        SV["current snapshot\n(lesson 5)"] -->|"points to"| SCH1
    end
    SB -.->|"table.scan(snapshot_id=snap_before_evolution)"| R0["3 columns,\nno country"]
    SV -.->|"table.scan()"| R1["4 columns,\nwith country"]

Going deeper: what would happen if the old snapshot had a column that no longer exists

This lesson showed the case of a column added after a snapshot — country doesn't exist reading snap_before_evolution. The symmetric case is also true, though this guide doesn't run it on dim_store so as not to lose Kiosko's business thread: if you'd read snap_before_evolution after lesson 4 had, say, dropped the city column — hypothetically, that's not what happened — that old snapshot would still show city, with its original values, because city's field_id still exists in that snapshot's Parquet file, and the schema_id=0 it points to still declares it. A column dropped from the current schema doesn't disappear from the snapshots that had it — it stays part of their own photo, readable precisely, exactly like any other archived data. This is consistent with the guarantee this module's lesson 3 already quoted from the official documentation: "Dropping a column or field does not change the values in any other column" — and it doesn't change the values of snapshots that existed before the DROP either.

Common mistakes

Assuming table.scan(snapshot_id=...) always returns the current schema, with new columns shown as None. What happens: someone, familiar with how country behaved within the current schema after lesson 3 (None for all three rows), expects to see that same None when reading snap_before_evolution, instead of the column simply not showing up. Why it happens: it's a reasonable, but incorrect, generalization of "a column with no value shows up as None" applied to the wrong case — that rule applies to existing rows read with the current schema after an add_column, not to snapshots that predate the column existing at all. How to spot it: if your code expects a country key in every row of old_scan.to_pylist(), and instead gets a KeyError, check whether you're reading the correct snapshot. How to fix it: always check old_scan.schema.names before accessing a column by name in a historical read — if that column isn't in the list, it's not going to be in any row, not even as None.

Trying to identify snap_before_evolution by a fixed position in table.history(), without verifying. What happens: someone assumes history()[0] always corresponds to snap_before_evolution, no matter how many additional operations ran before. Why it happens: in this module's exact flow, as described, it does turn out to be position zero — and it's easy to generalize that particular result. How to spot it: if you ran any additional experiment before lessons 3 through 5 — for example, if you repeated some lesson more than once against the same catalog — position zero might no longer correspond to what you expect. How to fix it: module 3's lesson 4 already taught the robust technique for this exact case — walk table.history() and check each candidate's content (here, each snapshot's schema with table.metadata.snapshots[i].schema_id) instead of trusting a fixed position. The safest approach for this particular module is capturing snap_before_evolution in a variable the moment it's created, as lesson 3 did, and not relying on recovering it later.

Exercises

Exercise 1 — Reproduce the schema comparison yourself. With kiosko.dim_store in the state lessons 3 through 5 left, run step 1's script from this lesson. Confirm old_scan.schema.names has exactly three elements, and current_scan.schema.names has exactly four.

See solution

If you followed lessons 3 through 5 in order, on the same directory, your output should exactly match this lesson's: three columns in the historical read (store_id, store_name, city), four in the current one (with country added at the end). The recovered snap_before_evolution is going to be a different integer than this lesson's — that's exactly expected.

Exercise 2 — Explain why table.metadata.schemas can have more than two entries, even though you only see two schemas "in use." Based on this module's lessons 3 and 4, explain why dim_store's metadata archives four distinct schemas (schema_id 0 through 3), even though only two of them — the original and the current — really matter for the business.

See solution

Every update_schema() operation that produces a column structure different from any already archived one creates a new schema_id — and this module's lesson 4 did several: adding country (a new schema_id), renaming store_name to outlet_name (another one), renaming it back (which, if the resulting structure exactly matches one already archived, can reuse that schema_id instead of creating a new one), adding temp_notes (another one), and dropping it. Iceberg archives every distinct structure the table ever had, not only the ones that "matter" for today's business — it's the same never-lose-history philosophy you already saw with data snapshots, applied here to schemas.

Exercise 3 — Prediction: what would happen if you read a snapshot captured AFTER populating country, but BEFORE temp_notes existed? Without running it, predict: if you captured a snapshot_id right after lesson 5's overwrite(), and temp_notes had never existed in that version of the table (imagining a different lesson order), what columns would you expect to see reading that snapshot?

See solution

Exactly the four business columns: store_id, store_name, city, country — with no temp_notes, because that column never came to exist in the current schema at the moment that hypothetical snapshot would have been captured. This exercise reinforces the lesson's central idea: a snapshot remembers the exact schema the table had at the instant of its own commit, neither before nor after — never "every column the table ever had at any point across its complete history."

Summary and next step

In this lesson you confirmed, with real evidence, that table.scan(snapshot_id=snap_before_evolution) still reads kiosko.dim_store with its original three-column schema — no country — even though the table's current schema already has four. You saw the exact mechanism that makes it possible: every snapshot archives a reference to its own schema_id, and table.metadata.schemas stores every schema version the table ever had, not just the last one.

Before moving on you should be able to: read a snapshot before a schema evolution and confirm it brings the correct schema for that moment; and explain, in your own words, why every snapshot references its own schema_id instead of always using the current one.

With schema evolution fully verified — add, populate, rename, drop, and reading the past without any of it mixing together — lesson 7 takes a step back to answer, with full precision, the question that opened this module: what exactly does the word "ACID" guarantee in the context of a single Iceberg table?

Resources

• PyIceberg — API reference, table.scan(snapshot_id=...), table.metadata.schemas, table.metadata.snapshots, the three entry points this lesson uses to compare schemas across snapshots. py.iceberg.apache.org/api. In English.
• Apache Iceberg — official documentation, "Evolution," "Correctness" section, the formal guarantee that dropping or adding a column doesn't change any other column's values — including already-archived snapshots. iceberg.apache.org/docs/latest/evolution. In English.
• This guide's DESIGN doc — the "Schema evolution" section (M4), the exact source of the table.scan(snapshot_id=snap_before_evolution).to_arrow() check this lesson runs. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.