Expiring old snapshots, safely

Description

This is the lesson where kiosko.dim_product finally gets pruned. Starting from the thirteen snapshots lesson 3 left behind — three from module 3, ten from five redundant nights — you're going to really run table.maintenance.expire_snapshots(), with no caveats at all: pure-Python PyIceberg 0.11.1, no Spark, no JVM. You're going to explicitly protect snap_v1 — the only snapshot that makes recovering P002's original state possible — and you're going to confirm, with assert, that time travel still works exactly the same after pruning. And you're going to discover something Iceberg's general documentation doesn't clarify for this specific version: expire_snapshots() in PyIceberg 0.11.1 doesn't delete any physical file — only metadata.

Connection to the module. This is the only maintenance operation in this module that really runs, verified against PyIceberg 0.11.1's installed source code. Lessons 4 and 6 document representative operations; this one executes.

An analogy: pruning the album, with grandma's photo marked in advance

This module's lesson 1 promised a two-part pruning rule: throw out the duplicates, never tear out the page grandma wants to see. This lesson applies that rule literally, not metaphorically. Before touching a single page of kiosko.dim_product's album, you're going to make an explicit list of what gets pruned and what gets protected — you're never going to trust the system to "guess" which photo is the important one. That list, in code, is exactly what you already know from module 3: snap_v1, captured in a variable, never hardcoded.

Worked example: expire_snapshots(), with snap_v1 protected

Step 1 — Starting from lesson 3's state: thirteen snapshots, snap_v1 captured

This script assumes you already ran lesson 3's script in the same working directory — kiosko.dim_product with thirteen archived snapshots. If you need that run's snap_v1, capture it again from table.history(): it's always the first one in the list.

# expire_dim_product.py -- safe pruning of kiosko.dim_product
import os

from pyiceberg.catalog import load_catalog

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.dim_product")

history = table.history()
snap_v1 = history[0].snapshot_id                    # the first one -- V1, before P002's change
current_snapshot_id = table.current_snapshot().snapshot_id  # the last one -- today's current

print(f"BEFORE: {len(history)} snapshots")
print("snap_v1 (protect, needed for time travel):", snap_v1)
print("current (protect, it's the current state):", current_snapshot_id)

What to expect (verified by running the real script, on the state lesson 3 left behind; the snapshot_ids are from your own run, different every time):

BEFORE: 13 snapshots
snap_v1 (protect, needed for time travel): <snapshot-id assigned in your run>
current (protect, it's the current state): <snapshot-id assigned in your run>

Step 2 — Build the pruning list: everything except what's protected

all_ids_in_order = [entry.snapshot_id for entry in history]
to_expire = [sid for sid in all_ids_in_order if sid != snap_v1 and sid != current_snapshot_id]

print(f"\nsnapshots selected to expire: {len(to_expire)} of {len(all_ids_in_order)}")

What to expect:

snapshots selected to expire: 11 of 13

Eleven of thirteen: the ten produced by the five redundant nights, plus the intermediate delete snapshot P002's real change's overwrite() had already left behind since module 3. None of the eleven adds anything snap_v1 (the "before") or the current one (the "now") doesn't already cover — exactly this lesson's pruning criterion: you need the two ends of the business history, not every intermediate stop.

Step 3 — Really run expire_snapshots()

table.maintenance.expire_snapshots().by_ids(to_expire).commit()
table.refresh()

history_after = table.history()
print(f"\nAFTER: {len(history_after)} snapshots")
for row in table.inspect.snapshots().select(["snapshot_id", "operation"]).to_pylist():
    print(" ", row["operation"], row["snapshot_id"])

What to expect (verified by running the real script):

AFTER: 2 snapshots
  append <snap_v1>
  append <current>

From thirteen down to two. table.maintenance.expire_snapshots() — the builder table.maintenance returns — accepts the complete list with .by_ids(...), and .commit() applies the change in a single transaction, with the same conditional atomic catalog update you already know from lesson 2. Notice a detail that confirms the pruning was surgical: the two surviving snapshots are, both, of type append — the intermediate delete from P002's real change disappeared alongside the ten from the redundant nights, because neither of the two protected snapshots needs it to reconstruct itself.

Step 4 — Verify time travel is still intact

v1_rows = table.scan(snapshot_id=snap_v1).to_arrow().to_pylist()
p002_v1 = next(r for r in v1_rows if r["product_id"] == "P002")
print("\nAS OF snap_v1 (post-pruning) P002:", p002_v1["category"], p002_v1["unit_cost"])

current_rows = table.scan().to_arrow().to_pylist()
p002_current = next(r for r in current_rows if r["product_id"] == "P002")
print("current (post-pruning) P002:", p002_current["category"], p002_current["unit_cost"])

assert len(history_after) == 2
assert p002_v1["category"] == "snacks" and p002_v1["unit_cost"] == 0.60
assert p002_current["category"] == "health-snacks" and p002_current["unit_cost"] == 0.68
print("\nasserts OK -- time travel intact, current state intact")

What to expect (verified by running the real script):

AS OF snap_v1 (post-pruning) P002: snacks 0.6
current (post-pruning) P002: health-snacks 0.68

asserts OK -- time travel intact, current state intact

This is this lesson's central promise, delivered with evidence: you pruned eleven of thirteen snapshots — 85% of the archived history — and time travel to snap_v1 still returns, byte for byte, the same result it returned before pruning. Nothing the business still needed was lost.

Discovery: PyIceberg's expire_snapshots() doesn't delete files — only metadata

Apache Iceberg's general documentation, on its "Maintenance" page, describes snapshot expiration like this: "Data files are not deleted until they are no longer referenced by a snapshot that may be used for time travel or rollback. Regularly expiring snapshots deletes unused data files." That sentence is true for the Java implementation and for Spark's action. It is not true, verified by reading its own source code, for PyIceberg 0.11.1's table.maintenance.expire_snapshots().

Verify it yourself, by counting physical files on disk before and after pruning:

import os

data_dir = os.path.join(warehouse_path, "kiosko", "dim_product", "data")
on_disk = [f for f in os.listdir(data_dir) if f.endswith(".parquet")]
print("\nphysical .parquet files on disk, AFTER expire_snapshots():", len(on_disk))

tracked = table.inspect.all_data_files().num_rows
print("files tracked by some current snapshot:", tracked)

What to expect (verified by running the real script, in lesson 3's same directory, which had seven physical files before this lesson):

physical .parquet files on disk, AFTER expire_snapshots(): 7
files tracked by some current snapshot: 2

Seven files are still on disk. Two are tracked. The metadata expire_snapshots() rewrote no longer mentions the other five — but nobody deleted them from the filesystem. This isn't a bug in this guide or in PyIceberg: it's exactly what its implementation does, verified by reading ExpireSnapshots._commit()'s source code (in pyiceberg/table/update/snapshot.py): it builds a single RemoveSnapshotsUpdate — an instruction telling the metadata "stop listing these snapshot_ids" — and applies it to the in-memory metadata model. At no point in that class, or in the function that applies the update (pyiceberg/table/update/__init__.py), is there a single call to delete a file. expire_snapshots() in this version is, precisely, a pure metadata operation: fast, safe, reversible in the sense that it never destroys bytes — but it doesn't free a single byte of storage on its own.

A real alternative, verified separately: older_than(dt)

.by_ids([...]) was this lesson's main technique, aligned with this guide's discipline of never depending on datetime.now(). But table.maintenance.expire_snapshots() also offers .older_than(dt) — it expires any unprotected snapshot with a timestamp earlier than a given value — verified in an isolated demo, with a timestamp captured from a real snapshot, never from the system clock:

# isolated demo -- does NOT touch kiosko.dim_product, only confirms older_than() works
snaps = table_demo.inspect.snapshots().select(["snapshot_id", "committed_at"]).to_pylist()
threshold = snaps[1]["committed_at"]      # committed_at captured from the 2nd snapshot, not datetime.now()
table_demo.maintenance.expire_snapshots().older_than(threshold).commit()

What to expect (verified by running this real demo, on an isolated 3-snapshot table): the snapshot with committed_at strictly earlier than the captured threshold gets expired; the other two — including the one that defined the threshold — survive, because older_than compares with strict <. table.history() goes from 3 to 2 entries. This form most resembles a recurring production maintenance plan — "expire everything older than N days" — but it needs, in real production, a datetime calculated from the system clock at the moment maintenance runs — perfectly reasonable for a scheduled maintenance job (not for generating Kiosko's business data), which is why this guide demonstrates it separately, without mixing it into kiosko.dim_product's main flow.

And if you try to expire the current snapshot, on purpose

try:
    table.maintenance.expire_snapshots().by_id(current_snapshot_id).commit()
except Exception as e:
    print(f"{type(e).__name__}: {e}")

What to expect (verified by running the real script):

ValueError: Snapshot with ID <id> is protected and cannot be expired.

PyIceberg always protects the current snapshot (the table's main reference's HEAD) and any snapshot marked as a tag or branch — the _get_protected_snapshot_ids() method, visible in the same source code quoted above, calculates that list before applying any expiration, and by_id() explicitly checks it before accepting your request. You don't need to exclude the current snapshot from your list yourself for extra safety — but this lesson did it anyway, in Step 2, because explicit, verified pruning is always preferable to depending on an implicit system protection.

Diagram: from thirteen to two, with both ends intact

flowchart TB
    subgraph antes["BEFORE -- 13 snapshots"]
        A1["snap_v1\nprotected"] --> A2["...11 pruning candidates..."] --> A3["current\nprotected"]
    end
    antes -->|"expire_snapshots().by_ids(11 ids)\n.commit()"| despues
    subgraph despues["AFTER -- 2 snapshots"]
        B1["snap_v1\nsnacks/0.60"] -.->|"metadata rewritten,\nno direct connection"| B2["current\nhealth-snacks/0.68"]
    end

Common mistakes

Assuming expire_snapshots() freed disk space immediately. What happens: someone runs expire_snapshots(), sees table.history() went from thirteen to two, and reports "the table now weighs a lot less" without having measured the real size on disk. Why it happens: the operation's name — "expire," "prune" — suggests a complete cleanup, and the Iceberg project's general documentation (written with Java/Spark in mind) reinforces that expectation. How to spot it: measure the data/ directory's real size before and after, like this lesson did — if it didn't change, your client isn't physically deleting files. How to fix it: in PyIceberg 0.11.1, treat expire_snapshots() for what it is — a metadata operation that makes files stop being tracked, not deleted — and follow up with lesson 6 (remove_orphan_files, representative in this environment) to complete the physical cleanup.

Choosing which snapshots to protect "by eye," instead of explicitly building the list from table.history(). What happens: someone, with a real table with hundreds of snapshots, tries to remember from memory which ones are "the important ones" and builds the protection list by hand, risking forgetting one. Why it happens: with few snapshots — like this lesson's thirteen — doing it from memory seems manageable; with hundreds, it no longer is, and the careless habit persists. How to spot it: if your protection criterion can't be expressed as code anyone can audit and rerun, you run the risk of accidentally expiring a snapshot that was actually needed. How to fix it: always express the protection criterion as explicit code over table.history() or table.inspect.snapshots() — like this lesson's Step 2 did with a simple list comprehension — never as a list of IDs copied by hand from an earlier query.

Exercises

Exercise 1 — Reproduce the full experiment yourself, and confirm the four numbers. With lesson 3's state available, run this lesson's four steps. Confirm 13 before, 11 selected, 2 after, and that Step 4's two final asserts pass.

Exercise 2 — Try to expire a snapshot_id that doesn't exist, and watch the error. Call table.maintenance.expire_snapshots().by_id(999999999999999999).commit() on the table this lesson left behind.

try:
    table.maintenance.expire_snapshots().by_id(999999999999999999).commit()
except Exception as e:
    print(f"{type(e).__name__}: {e}")

Exercise 3 — Explain, in your own words, why this lesson builds to_expire as "everything except what's protected" instead of "a fixed list of IDs I know I want to delete." Think about what would happen if, between lesson 3 and this lesson, someone had added a sixth redundant night without you knowing.

Summary and next step

In this lesson you really ran table.maintenance.expire_snapshots(): from thirteen snapshots down to two, with snap_v1 explicitly protected and time travel verified, with assert, intact after pruning. You discovered, with evidence from the source code itself and a direct filesystem measurement, that this operation in PyIceberg 0.11.1 rewrites the metadata but doesn't delete physical files — seven files are still on disk, even though only two are tracked. You also verified the automatic protection of the current snapshot, and a real alternative (older_than) for when the pruning criterion is "everything before a date," not an explicit list.

Before moving on you should be able to: build a list of snapshots to expire from table.history(), explicitly protecting what the business still needs; and explain why PyIceberg's expire_snapshots() doesn't free disk space on its own.

Lesson 6 completes the cleanup this lesson left pending: the five orphan files, still on disk, that no current snapshot tracks anymore.

Resources

• PyIceberg — API reference, table.maintenance and ExpireSnapshots (by_id, by_ids, older_than, commit), this lesson's complete foundation. py.iceberg.apache.org/api. In English.
• Apache Iceberg — official documentation, "Maintenance," "Expire Snapshots" section, source for the quote about file deletion this lesson contrasts with PyIceberg 0.11.1's verified real behavior. iceberg.apache.org/docs/latest/maintenance. In English.
• PyIceberg — source code, pyiceberg/table/update/snapshot.py (ExpireSnapshots class) and pyiceberg/table/update/__init__.py (RemoveSnapshotsUpdate application), the direct evidence that this version doesn't delete physical files. Installed locally with pip install "pyiceberg[sql-sqlite,pyarrow]". In English.
• This same guide, module 3, lesson 4 — source of the discipline of never hardcoding a snapshot_id, applied in this lesson to the pruning list. 04-capturing-the-snapshot-id-never-hardcoding-it.md. In Spanish.
• This guide's DESIGN doc — module 7's section, "safely expiring old snapshots." src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.