Module overview: snapshots and time travel

Why this module exists

Module 2 opened kiosko.fact_orders's whole box without writing a single new row. You walked the chain catalog → metadata → manifest list → manifest files → data files, and at every link you found a single snapshot: the one table.append() created in module 1's lesson 6. That module ended with a question deliberately left open, raised in its own lesson 1: if "nothing ever gets overwritten" is Iceberg's central promise, what exactly happens when the table does change? Where does the earlier state go?

This module answers that question with the example you already know by heart, because you solved it twice before, with two different techniques. In data-modeling-for-analytics-guide you built dim_product_scd by hand: valid_from, valid_to, is_current columns, and a MERGE INTO you wrote with your own hands so P002's old row wouldn't disappear when unit_cost changed from 0.60 to 0.68. In dbt-analytics-engineering-guide you automated that exact same technique with dbt snapshot: the same three ideas — keep the old version, mark when it stopped being current, know which one is current — now with less typing, generated by dbt_valid_from/dbt_valid_to/dbt_scd_id. Both solutions are correct. Both, also, ask the modeler to design and maintain history columns.

This module does the same thing — recovering category='snacks', unit_cost=0.60 after the table already changed to category='health-snacks', unit_cost=0.68 — without declaring a single new column. kiosko.dim_product is going to have, from start to finish, exactly four columns: product_id, product_name, category, unit_cost. No valid_from. No is_current. The history doesn't live in a column — it lives in the snapshot mechanism module 2 already showed you exists, and that this module, for the first time in this guide, actually uses.

The case that runs through the module: the P002 change, again

The thread is the same as always: P002 Energy Bar changes from category='snacks', unit_cost=0.60 to category='health-snacks', unit_cost=0.68, effective 2026-08-15 — after Kiosko's forty real orders, all between August 3 and 9. The correct result for any query over those forty orders is still category='snacks', with a margin of 10.8 for that category; the broken result — the one that applies the new cost to sales that already happened — still gives health-snacks, margin 9.36. They're the same two numbers you already saw in data-modeling (module 5) and in dbt (module 5).

What changes here is the table. This module creates kiosko.dim_product from scratch — it didn't exist before this module — with one row per product and no history column at all. It first loads it with the V1 values (P002 still snacks/0.60), captures that moment's snapshot_id in a variable, and then completely overwrites it with the V2 values (P002 now health-snacks/0.68). The table's current state, from that moment on, is V2 — just like in real life, where a store's product catalog always reflects today's price and category, never a mixed-together history. Recovering V1 doesn't come from a column the table deliberately keeps around — it comes from asking Iceberg, precisely, "show me what this table looked like at the snapshot before the change."

An analogy: the snapshot of the supermarket shelf

Think about the person who restocks a supermarket shelf. Every time they restock — every time what's there changes — they take a photo of the whole shelf, exactly as it ended up, and file it with the date. They never retouch yesterday's photo. They never delete an old photo to "fix it" with today's merchandise. If you ask them "what was on the shelf last Tuesday?", they don't give you their opinion or reconstruct it from memory — they go to the archive, look up Tuesday's photo, and show it to you exactly as it looked that day.

That is, precisely, what an Iceberg snapshot is, and what time travel is. Every write — every append(), every overwrite() — is a new photo of the whole shelf, filed alongside every previous one, none retouched. Asking "show me the shelf as it was at snapshot X" — what this module calls, with PyIceberg's exact syntax, table.scan(snapshot_id=X) — is exactly asking the archive keeper "pull up that day's photo for me." The keeper doesn't need anyone to have asked them, beforehand, to write down in a separate notebook "this is what changed, and since when" — that's what valid_from/valid_to do in data-modeling and dbt. The photo archive, on its own, already contains the whole history, with no one having designed a column for it.

Diagram: two writes, two photos, one trip through time

flowchart LR
    T0["kiosko.dim_product\njust created, empty"] -->|"table.append(V1)\nP002 = snacks / 0.60"| S1["snapshot snap_v1\nthe shelf's photo BEFORE"]
    S1 -->|"table.overwrite(V2)\nP002 = health-snacks / 0.68"| S2["snapshot snap_v2\nthe shelf's photo NOW\n(= current_snapshot)"]

    S2 -.->|"table.scan()\nno arguments"| R2["reads snap_v2\nP002 = health-snacks / 0.68"]
    S1 -.->|"table.scan(snapshot_id=snap_v1)\nTIME TRAVEL"| R1["reads snap_v1\nP002 = snacks / 0.60"]

snap_v1 doesn't disappear when snap_v2 gets created — it stays filed, available, exactly the way the shelf's old photo stays in the supermarket's archive. The only thing that changes is which of the two photos the front desk shows you by default when you don't ask for a specific date.

The map of this module

Lesson    What it solves
────────  ──────────────────────────────────────────────────────────────
L1        (this one) The full map: two writes, two snapshots, one trip
L2        This guide's first real write against a second table:
          kiosko.dim_product, loaded with V1 -- a new snapshot
L3        The second write: table.overwrite() with V2 -- the P002 change
L4        Why the snapshot-id is never hardcoded, and how to capture it right
L5        table.scan(snapshot_id=snap_v1) -- the trip through time, executed
L6        The payoff: P002's correct margin (10.8), with no column at all
L7        The honest limit: what time travel does NOT solve
L8        Project: dim_product, historized via time travel, end to end

Lessons 2 and 3 create the two writes that make everything else possible. Lesson 4 deliberately pauses on a hard rule for this entire guide: a snapshot_id isn't a number you can jot down and reuse from one run to the next — it has to be captured in code, always. Lessons 5 and 6 are the trip through time itself, and the verification that it recovers the same correct result you already know from two previous guides. Lesson 7 is, on purpose, the most important of the eight: it clearly states what this technique does not solve, so you don't leave this module thinking time travel generally replaces a row-level SCD-2 design. Lesson 8 brings the previous seven together into a single project.

The boundary: what does NOT belong in this module

This module doesn't touch any table's schema — adding, renaming, or dropping a column is, precisely, module 4's job. It doesn't touch partitioning or volume either — kiosko.fact_orders_at_scale, with its 10 million rows, only arrives in module 5. And, the most important of the three boundaries: this module does not solve the general case of a dimension that changes many times, with facts spread across several of those versions. That general case still needs row-level SCD-2 — by hand, as in data-modeling, or automated, as in dbt. Lesson 7 of this module states that explicitly, with a real, executed example showing, with real evidence, exactly where that limit sits.

Common mistakes

Expecting this module to add history columns "just in case." What happens: someone, familiar with data-modeling's valid_from/valid_to or dbt's dbt_valid_from/dbt_valid_to, expects this module's kiosko.dim_product to have them too, "to be safe." Why it happens: two previous guides in the ecosystem solved this exact same problem exactly that way, and it's natural to assume the third does the same. How to spot it: if your own kiosko.dim_product schema has more than four columns, revisit step 2 of this module's lesson 2. How to fix it: this module's whole point is that those columns are not necessary — the table has, and must keep having, exactly four columns: product_id, product_name, category, unit_cost.

Confusing "time travel solves history" with "time travel solves EVERY possible history." What happens: someone finishes this module's lesson 6, sees P002's correct margin (10.8) recovered with no column at all, and concludes Iceberg makes everything data-modeling and dbt taught about SCD-2 obsolete. Why it happens: Kiosko's example, with a single P002 change and every fact happening before that change, is the most favorable possible case for time travel — and it's easy to generalize from a favorable case to "it always works this way." How to spot it: if you can't explain, with a concrete example, a scenario where time travel does not give the correct answer, you haven't reached lesson 7 yet. How to fix it: read all of lesson 7 before considering this module closed — it's, on purpose, the lesson with the most important pedagogical point of the eight.

Exercises

Exercise 1 — Before starting, predict: how many snapshots is kiosko.dim_product going to have by the end of lesson 3? Without having read lessons 2 and 3 yet, and based only on this lesson's analogy (one photo per write), predict how many snapshots you're going to see on kiosko.dim_product after loading V1 and overwriting with V2 — two write operations, as this lesson's map describes them.

See solution

The most reasonable prediction, with this lesson's information, is two: one per write operation (append(V1) and overwrite(V2)). It's worth flagging ahead of time that this module's lesson 3 is going to show the real answer is more nuanced — table.overwrite() can, internally, generate more than one snapshot in a single call — but the "one write, at least one snapshot" intuition this lesson teaches is correct, and it's the foundation lesson 3 builds the full nuance on top of.

Exercise 2 — Explain, without looking back, the difference between the "current" state and the "recoverable" state. In 2-3 sentences, explain what it means for snap_v1 to still be recoverable after snap_v2 becomes the current snapshot — and why this is different from having "lost" the earlier state.

See solution

"Current" is what table.scan() shows you by default, without you asking for anything more specific — it's the archive's most recent photo, the one the catalog points to right now. "Recoverable" means the earlier photo is still physically filed, available to anyone who explicitly asks for it with table.scan(snapshot_id=snap_v1) — it wasn't lost, wasn't moved somewhere else, and requires no special restoration process. The key difference is that Iceberg never had to "make a backup copy" of snap_v1 before writing snap_v2 — snap_v1 was never touched at all; writing snap_v2 simply filed a new photo next to the old one, overwriting nothing.

Exercise 3 — Name, from memory, the two earlier techniques this module replaces for Kiosko's favorable case. Without looking back, name the two guides in the ecosystem that already solved the P002 change, and the exact technique each one used.

See solution

data-modeling-for-analytics-guide (modules 4-5) solved the P002 change by hand, with valid_from/valid_to/is_current columns in dim_product_scd, maintained with a MERGE INTO explicitly written in SQL. dbt-analytics-engineering-guide (module 5) automated the same idea with dbt snapshot, generating dbt_valid_from/dbt_valid_to/dbt_scd_id with nobody writing the MERGE INTO by hand. Both techniques depend on history columns designed by a person; this module reaches the same correct result without declaring any.

Summary and next step

In this lesson you learned module 3's full map: two real writes against a new table, kiosko.dim_product, with no history column at all, and a trip through time that recovers the state before the P002 change. You saw the supermarket-shelf-photo analogy, the two-write diagram, and the explicit boundary of what this module does not solve — the general case of a dimension with many changes, which still needs row-level SCD-2.

Before moving on you should be able to: explain, in your own words, what a snapshot is and what time travel is, using this lesson's analogy; and name the two earlier techniques in the ecosystem this module reproduces with a simpler table.

Lesson 2 creates kiosko.dim_product and gives it its first real write: the V1 values for Kiosko's four products, with P002 still snacks/0.60.

Resources

• Apache Iceberg — official documentation, "Table Spec," the formal definition of a snapshot as a table's complete state at a given instant. iceberg.apache.org/spec. In English.
• PyIceberg — API reference, table.scan(snapshot_id=...) and table.overwrite(), this module's central pair of operations. py.iceberg.apache.org/api. In English.
• data-modeling-for-analytics-guide DESIGN doc — source of dim_product_scd with valid_from/valid_to/is_current and the MERGE INTO this module reproduces with no columns. src/guides/data-modeling-for-analytics-guide/DISENO.md. In Spanish.
• dbt-analytics-engineering-guide DESIGN doc — source of dbt snapshot and dbt_valid_from/dbt_valid_to/dbt_scd_id, the automated version of the same technique. src/guides/dbt-analytics-engineering-guide/DISENO.md. In Spanish.
• This guide's DESIGN doc — the full map of the eight modules, including this module's exact boundary. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.