Capturing the snapshot-id, never hardcoding it

Description

Lessons 2 and 3 already followed this rule without fully explaining it: every time this module needed a snapshot_id, it captured it in a variable — snap_v1 — immediately after the write that created it. This lesson pauses on that discipline, turns it into an explicit rule, and shows what breaks if you don't follow it. It also solves a practical problem: what do you do if you didn't capture the snapshot_id in time, and need to find it later?

Connection to the module. Without this lesson, lesson 5 — the trip through time itself — would depend on a number nobody explained why it's so fragile. This lesson is the "why" before the "how": understanding that a snapshot_id isn't a piece of Kiosko business data, but an identifier the engine itself generates at the instant of the commit, is the foundation for why the rest of this guide — and the whole ecosystem — never writes it as a literal anywhere.

An analogy: a camera's roll number, not the photo's date

Think again about the supermarket's photo archive. Every photo has a date — that's business data, predictable, the same for anyone asking "which photo is from August 15?" But the archive also assigns each photo an internal roll number, generated by the camera at the exact moment the photo is taken, with no relation to the date or the content. That roll number is exactly what a snapshot_id is: an identifier that exists because the system needs to tell one photo apart from another, generated at the instant of capture, and that would be different if the same photo had been taken a second earlier or later. Nobody memorizes a photo's roll number to find it later — you search by date, by event, by what the photo shows. snapshot_id works the same way: it isn't what you search for, it's what you use once you already know which photo is the right one.

Worked example: proof it isn't reproducible, and how to recover it if you lost it

Step 1 — Real evidence: two identical runs, two completely different snapshot_ids

This isn't a theoretical warning. I ran this module's lessons 2 and 3 full script — create kiosko.dim_product, load V1, overwrite with V2 — twice, in two different working directories, from scratch each time. The code was exactly the same. This is what each run captured:

Run A -- snap_v1: 2791049306460028584   snap_v2: 8316902092849711638
Run B -- snap_v1: 5083159773583532361   snap_v2: 6033084327179618444

Not a single digit in common between Run A's snap_v1 and Run B's snap_v1 — beyond both being integers of the same magnitude, there's no predictable relationship between them at all. If lesson 5's code had written table.scan(snapshot_id=2791049306460028584) as a literal — copied from an earlier run, or from this same lesson — it would have worked in Run A, and would have failed (or, worse, would have pointed to a snapshot that doesn't exist, or the wrong one) in any other run, including yours, right now, on your own machine.

Step 2 — The correct way: capture in the same block that writes

# the correct pattern, already used in this module's lessons 2 and 3
table.append(pa_table_v1)
snap_v1 = table.current_snapshot().snapshot_id  # <- captured IMMEDIATELY, the very next line

# ... more code, later, in another lesson or another point in the script ...

table.overwrite(pa_table_v2)
snap_v2 = table.current_snapshot().snapshot_id  # <- captured IMMEDIATELY, again

table.current_snapshot() always returns the currently active snapshot — so if you call it immediately after the write you care about, before any other operation, you're guaranteed to be capturing exactly that write, and not a later one someone else (or another process) might have made in the meantime. Capturing late — "I'm going to need V1's load snapshot_id eventually, I'll look it up when the time comes" — is exactly the habit that breaks this guarantee.

Step 3 — If you genuinely lost it: recover it by content, not by position

Suppose that, for whatever reason, you didn't capture snap_v1 at the time, and now kiosko.dim_product already has lesson 3's three snapshots — append, delete, append. How do you find which of the three is "the photo of P002 before the change"?

# recover_snap_v1.py -- if you didn't capture it at the time
import os

from pyiceberg.catalog import load_catalog

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.dim_product")

history = table.history()
print(f"table.history() has {len(history)} entries\n")

recovered_snap_v1 = None
for i, entry in enumerate(history):
    rows = table.scan(snapshot_id=entry.snapshot_id).to_arrow().to_pylist()
    p002 = next((r for r in rows if r["product_id"] == "P002"), None)
    category = p002["category"] if p002 else "(0 rows -- intermediate delete snapshot)"
    print(f"  history()[{i}]: P002.category = {category}")
    if p002 is not None and p002["category"] == "snacks":
        recovered_snap_v1 = entry.snapshot_id

print(f"\nsnap_v1 recovered (last snapshot with P002 in 'snacks'): found = {recovered_snap_v1 is not None}")

What to expect (verified by running the actual script; the internal snapshot_ids aren't shown, only position and business content, which are deterministic):

table.history() has 3 entries

  history()[0]: P002.category = snacks
  history()[1]: P002.category = (0 rows -- intermediate delete snapshot)
  history()[2]: P002.category = health-snacks

snap_v1 recovered (last snapshot with P002 in 'snacks'): found = True

Notice the method: you did not assume "the first snapshot" (history()[0]) is always snap_v1 just because it's at position zero — in this table, with only two writes, it happened to be true, but that position isn't guaranteed in general (a table with more accumulated history could have snap_v1 at any position). What you did was verify each candidate's actual content with table.scan(snapshot_id=...), and keep the last snapshot where P002 still had category='snacks' — the precise definition of "the state before the change." This technique — walk table.history(), scan each candidate, and decide by business content, not by position — is the robust way to recover a snapshot_id you didn't capture in time.

Diagram: capturing at the moment, versus reconstructing later

flowchart TB
    subgraph correcto["Correct way -- capture immediately"]
        W1["table.append(V1)"] --> C1["snap_v1 = table.current_snapshot().snapshot_id\n(on the same line of code)"]
    end
    subgraph tardio["If you missed it -- reconstruct by content"]
        H["table.history()"] --> L["walk each entry"]
        L --> SC["table.scan(snapshot_id=candidate)\nchecks the business content"]
        SC --> D["keep the last one\nmatching the condition you're looking for"]
    end

Going deeper: why timestamp_ms isn't hardcoded either

The same rule applies to the other piece of data every table.history() entry carries: timestamp_ms, the commit's exact moment in milliseconds since epoch. It's tempting to think a timestamp, unlike a snapshot_id, really is "predictable" — after all, you roughly know when you ran your script — but in practice it's just as fragile as a literal: it depends on the clock of the machine where the commit ran, on how long the write took, and on any retry that happened internally. This whole guide — just like dbt-analytics-engineering-guide with product_updated_at and spark-and-distributed-processing-guide when measuring execution times — follows the same discipline: any commit timestamp gets captured in a variable immediately after it's generated, never written as a literal in code and never predicted ahead of time. table.snapshot_as_of_timestamp(timestamp_ms, inclusive=True) — a real PyIceberg method, verified against API 0.11.1 — lets you travel through time by date instead of by snapshot_id, but the timestamp_ms you pass it always has to come from a captured value (for example, history[0].timestamp_ms), never from a number you wrote by hand thinking "this was roughly around that time."

Common mistakes

Copying a snapshot_id from this guide's documentation, or from an earlier run, and pasting it as a literal in new code. What happens: someone, reading this guide's examples, sees a sample snapshot_id somewhere — from a lesson, from an error message, from their own run yesterday — and copies it directly into a new script, instead of capturing it with table.current_snapshot() or recovering it with step 3's technique. Why it happens: a large integer looks like normal data, the same kind as a product_id or a store_id, which really are stable across runs. How to spot it: if your code has a snapshot_id=<a number of 18-20 digits written directly> instead of a variable, check where that number came from. How to fix it: any snapshot_id in your code must come from a call to table.current_snapshot(), table.history(), table.snapshots(), or table.snapshot_by_id() — never from a copied literal, not even "just to test."

Confusing "the oldest snapshot" with "the snapshot I need." What happens: someone, with a table that accumulated many writes across several modules — something you're going to see later in this guide — assumes table.history()[0] always corresponds to the state they're looking for, without checking the content. Why it happens: in this lesson's example, with only two writes, history()[0] did turn out to be snap_v1 — and it's easy to generalize that particular result into a general rule that isn't one. How to spot it: if your table has more than two or three accumulated snapshots and you trust a fixed position in the history without having checked its content, you're at risk of this mistake. How to fix it: always use this lesson's step 3 technique — walk the candidates and check with table.scan(snapshot_id=...) what they actually contain — instead of assuming a fixed position in table.history().

Exercises

Exercise 1 — Reproduce the two-run comparison yourself. Run this module's lessons 2 and 3 twice, in two completely different working directories (two separate kiosko_warehouse//kiosko_catalog.db pairs). Write down each run's snap_v1. Confirm they're different.

Exercise 2 — Implement content-based recovery to find snap_v2 (not snap_v1). Adapt this lesson's step 3 script to instead find the snapshot_id of the last snapshot where P002 has category='health-snacks' — that is, snap_v2, without having captured it in lesson 3.

recovered_snap_v2 = None
for entry in table.history():
    rows = table.scan(snapshot_id=entry.snapshot_id).to_arrow().to_pylist()
    p002 = next((r for r in rows if r["product_id"] == "P002"), None)
    if p002 is not None and p002["category"] == "health-snacks":
        recovered_snap_v2 = entry.snapshot_id

print("snap_v2 recovered:", recovered_snap_v2 == table.current_snapshot().snapshot_id)

Exercise 3 — Explain, in your own words, the difference between a product_id and a snapshot_id. In 2-3 sentences, contrast why "P002" is safe to write as a literal in this guide's code — in fact, it does so all the time — while a snapshot_id never is.

Summary and next step

In this lesson you confirmed, with evidence from two real runs, that a snapshot_id is never reproducible across executions — not even with the same code and the same business data. You saw the correct way to capture it (immediately after the write that generates it) and the way to recover it if you didn't do so in time (walking table.history() and checking each candidate's content, never trusting its position). And you saw the same rule applies to every commit's timestamp_ms.

Before moving on you should be able to: explain why a snapshot_id is never written as a literal in code; capture a snapshot_id immediately after a write; and recover a lost snapshot_id by checking each candidate snapshot's content, instead of assuming a fixed position in the history.

With snap_v1 correctly captured — lesson 2's, or the one you just recovered in this lesson — lesson 5 uses it for the first time to travel through time for real: table.scan(snapshot_id=snap_v1).

Resources

• PyIceberg — API reference, table.current_snapshot(), table.history(), table.snapshots(), and table.snapshot_by_id(), the four entry points for getting a snapshot_id without hardcoding it. py.iceberg.apache.org/api. In English.
• PyIceberg — API reference, table.snapshot_as_of_timestamp(timestamp_ms, inclusive=True), the equivalent method for looking up a snapshot by date instead of by identifier. py.iceberg.apache.org/api. In English.
• dbt-analytics-engineering-guide DESIGN doc — source of the same discipline applied to product_updated_at, never CURRENT_TIMESTAMP, this rule's direct precedent in the ecosystem. src/guides/dbt-analytics-engineering-guide/DISENO.md. In Spanish.
• This guide's DESIGN doc — the complete hard rule about never hardcoding a snapshot-id or a commit timestamp, with the justification for why they can't be determined ahead of time. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.