Module 7: Catalogs Maintenance And Delta Lake By Contrast

Why snapshots accumulate cost

Next →

Description

This lesson rebuilds kiosko.dim_product with the exact state module 3 left it in — V1 loaded with append(), P002 changed to health-snacks/0.68 with overwrite(), three snapshots total — and adds what no previous module needed: the passage of time. You're going to simulate five more nights of a real operational pipeline that reloads the complete product catalog every night without checking whether anything changed. You're going to measure, with table.history() really run, exactly what that lack of checking costs.

Connection to the module. Lesson 1 promised evidence, not an abstract claim that "snapshots accumulate cost." This lesson is that evidence: executed code, real numbers, before and after.

An analogy: the photographer who doesn't check the roll before repeating the shot

Go back to module 3's supermarket shelf: every complete restock produces a new photo, archived forever. Now imagine an employee who, every single night, without fail, photographs the whole shelf again — even on nights nobody touched a single product. After a week, the archive has eight nearly identical photos of the same unchanged shelf, mixed in with the two photos that do matter: the one from before the real restock, and the one from after. Nobody planned that waste — nobody simply told the employee "check first whether anything changed, and only then take the photo."

Worked example: rebuilding the state, and adding five redundant nights

Step 1 — Module 3's exact state: V1 loaded, P002 changed

# accumulated_history.py -- rebuilds module 3's state and adds 5 redundant nights
import os

import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import DoubleType, NestedField, StringType

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")
os.makedirs(warehouse_path, exist_ok=True)

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
catalog.create_namespace("kiosko")

dim_product_schema = Schema(
    NestedField(field_id=1, name="product_id", field_type=StringType(), required=True),
    NestedField(field_id=2, name="product_name", field_type=StringType(), required=True),
    NestedField(field_id=3, name="category", field_type=StringType(), required=True),
    NestedField(field_id=4, name="unit_cost", field_type=DoubleType(), required=True),
)
table = catalog.create_table("kiosko.dim_product", schema=dim_product_schema)

pa_schema = pa.schema([
    pa.field("product_id", pa.string(), nullable=False),
    pa.field("product_name", pa.string(), nullable=False),
    pa.field("category", pa.string(), nullable=False),
    pa.field("unit_cost", pa.float64(), nullable=False),
])
DIM_PRODUCT_V1 = [
    {"product_id": "P001", "product_name": "Bottled Water 600ml", "category": "beverages", "unit_cost": 0.40},
    {"product_id": "P002", "product_name": "Energy Bar", "category": "snacks", "unit_cost": 0.60},
    {"product_id": "P003", "product_name": "Instant Coffee Sachet", "category": "beverages", "unit_cost": 0.35},
    {"product_id": "P004", "product_name": "Phone Charger Cable", "category": "electronics", "unit_cost": 2.10},
]
DIM_PRODUCT_V2 = [
    {"product_id": "P001", "product_name": "Bottled Water 600ml", "category": "beverages", "unit_cost": 0.40},
    {"product_id": "P002", "product_name": "Energy Bar", "category": "health-snacks", "unit_cost": 0.68},
    {"product_id": "P003", "product_name": "Instant Coffee Sachet", "category": "beverages", "unit_cost": 0.35},
    {"product_id": "P004", "product_name": "Phone Charger Cable", "category": "electronics", "unit_cost": 2.10},
]

# A -- module 3, lesson 2: V1 loaded with append()
table.append(pa.Table.from_pylist(DIM_PRODUCT_V1, schema=pa_schema))
snap_v1 = table.current_snapshot().snapshot_id
print("A) append(V1). snap_v1 captured.")

# B -- module 3, lesson 3: P002's REAL change, with overwrite()
table.overwrite(pa.Table.from_pylist(DIM_PRODUCT_V2, schema=pa_schema))
print("B) overwrite(V2) -- the real 2026-08-15 change.")

print("\nAfter A+B (module 3's state):", len(table.history()), "snapshots")

What to expect (verified by running the real script; snap_v1's snapshot_id is from your own run, different every time):

A) append(V1). snap_v1 captured.
B) overwrite(V2) -- the real 2026-08-15 change.

After A+B (module 3's state): 3 snapshots

Exactly the same number module 3, lesson 3 confirmed: append, delete, append — three snapshots for two writes, because overwrite() with no filter resolves internally as a complete removal followed by a complete load.

Step 2 — Five nights, same pipeline, no real change

# Kiosko's nightly pipeline reloads the complete catalog every night,
# without checking whether the source system really changed anything -- overwrite() never compares.
nights = ["2026-08-16", "2026-08-17", "2026-08-18", "2026-08-19", "2026-08-20"]
for night in nights:
    table.overwrite(pa.Table.from_pylist(DIM_PRODUCT_V2, schema=pa_schema))

print("After 5 redundant nights:", len(table.history()), "snapshots")

What to expect (verified by running the real script):

After 5 redundant nights: 13 snapshots

Five nights, each with table.overwrite(DIM_PRODUCT_V2) — the same content, with no different business data at all — and the history went from 3 to 13 snapshots: ten new ones, none with any information that wasn't already in the previous snapshot. Notice the number: it isn't five new snapshots, one per night — it's ten, because every filterless overwrite() still resolves as delete + append, the same module 3 lesson applied ten more times.

Step 3 — The cost, measured in files, not just snapshots

all_files = table.inspect.all_data_files()
live_files = table.inspect.files()
print("all_data_files().num_rows (every data file some snapshot still tracks):", all_files.num_rows)
print("files().num_rows (files the CURRENT snapshot needs):", live_files.num_rows)

current = table.scan().to_arrow().to_pylist()
p002_current = next(r for r in current if r["product_id"] == "P002")
v1_rows = table.scan(snapshot_id=snap_v1).to_arrow().to_pylist()
p002_v1 = next(r for r in v1_rows if r["product_id"] == "P002")
print("current P002:", p002_current["category"], p002_current["unit_cost"])
print("AS OF snap_v1 P002:", p002_v1["category"], p002_v1["unit_cost"])

What to expect (verified by running the real script):

all_data_files().num_rows (every data file some snapshot still tracks): 7
files().num_rows (files the CURRENT snapshot needs): 1
current P002: health-snacks 0.68
AS OF snap_v1 P002: snacks 0.6

This is the number that sums up the whole problem. kiosko.dim_product today still has exactly four rows — not one extra business row — and to answer any query about its current state, Iceberg only needs to open one data file (files().num_rows == 1). But seven data files still exist and are still tracked by some not-yet-expired snapshot (all_data_files().num_rows == 7) — six of them each contain a complete, redundant copy of the same four products. And time travel still works exactly like in module 3: snap_v1 still recovers snacks/0.60, with none of the ten writes in between having affected it at all.

Why this happens: three mechanisms, none new

Nothing you just saw depends on hidden or unexpected behavior — it's three rules this guide already taught, one per module, now combined:

  1. Data files are immutable (module 2). No overwrite() modifies an existing Parquet file — it always writes a new one. Ten redundant writes produce, at minimum, ten opportunities to write a new file (in this case, five of the ten operations were delete, which don't write a file — hence seven files, not ten).
  2. A snapshot doesn't disappear on its own (module 3). Every one of this lesson's thirteen snapshots stays archived, with its own snapshot_id, available to whoever explicitly asks for it — exactly the mechanism that makes recovering snap_v1 possible ten writes later.
  3. table.overwrite() never compares, it always replaces (module 6, lesson 6, by contrast with table.upsert()). That lesson's Exercise 2 already demonstrated that upsert() with data identical to what's already on file produces UpsertResult(rows_updated=0, rows_inserted=0) — zero new snapshots. This lesson's nightly pipeline used overwrite(), the wrong tool for "reconfirm without knowing if anything changed": by design, it has no way to notice that nothing needed to be written.

The combination of all three explains the whole cost: files that never get modified in place, snapshots that never expire on their own, and a write tool that never asks "is this needed?" None of the three is, on its own, an Iceberg design flaw — they are, on the contrary, exactly the guarantees that made module 3's time travel possible. The cost is that guarantee's price when nobody actively manages it.

Diagram: thirteen photos of the same shelf, two of them matter

flowchart LR
    S1["snap_v1\nappend, V1\nsnacks/0.60"] --> SD["delete\n(module 3)"]
    SD --> S3["append, V2\nhealth-snacks/0.68"]
    S3 --> N1["night 1: delete+append\n(identical to S3)"]
    N1 --> N2["night 2: delete+append"]
    N2 --> N3["night 3: delete+append"]
    N3 --> N4["night 4: delete+append"]
    N4 --> N5["night 5: delete+append\n= CURRENT snapshot"]

    S1 -.->|"protected -- time travel"| KEEP1["needed"]
    N5 -.->|"current -- needed"| KEEP2["needed"]
    SD -.->|"noise"| WASTE["candidate to expire"]
    N1 -.->|"noise"| WASTE
    N2 -.->|"noise"| WASTE
    N3 -.->|"noise"| WASTE
    N4 -.->|"noise"| WASTE

Common mistakes

Assuming table.overwrite() with identical data is a "no-op" because the business result doesn't change. What happens: someone, familiar with systems where "saving the same thing that was already there" does nothing, assumes this lesson's five nights shouldn't have cost anything. Why it happens: in many traditional relational databases, an UPDATE that changes no value can, depending on the engine, generate no new transaction record. How to spot it: if your snapshot count after a "no change" overwrite() isn't exactly zero additional, check which operation you used — this lesson's Going Deeper section already explains it: overwrite() never compares. How to fix it: if your pipeline needs to "reconfirm without knowing if anything changed," use table.upsert() (module 6) — it's the tool designed exactly for that case, with the zero cost module 6, lesson 6's Exercise 2 confirms.

Confusing "the file is still on disk" with "the file is still part of the current table." What happens: someone sees all_data_files().num_rows == 7 and concludes table.scan() is going to read all seven files on every query, making the table seven times slower than expected. Why it happens: it's easy not to distinguish between "every file some archived snapshot still tracks" (all_data_files()) and "the files the current snapshot needs to answer a normal query" (files(), with no arguments). How to spot it: compare the two numbers, like this lesson's Step 3 did — if files() (current) is much smaller than all_data_files() (full history), your table is healthy in terms of normal read performance, even though it has pending cleanup work. How to fix it: the six redundant files' cost isn't a read cost — it's a storage cost (bytes on disk nobody needs) and a metadata cost (manifests that grow without adding anything). This module's lesson 5 solves exactly that cost, without touching a normal query's performance, which was already correct from before.

Exercises

Exercise 1 — Reproduce the full experiment yourself, and confirm the three numbers. In a new directory, run this lesson's three steps. Confirm 3 snapshots after A+B, 13 after the five nights, 7 total tracked files, and 1 current file.

See solution

If your environment has PyIceberg 0.11.1 installed, your output should match this lesson's four numbers exactly. The snapshot_ids are going to be different from the ones shown here — that's exactly what's expected, and it's precisely why snap_v1 gets captured in a variable in Step 1 instead of quoted as a literal.

Exercise 2 — Calculate how many of the thirteen snapshots correspond to delete operations and how many to append. Use table.inspect.snapshots().select(["operation"]) on the table this lesson left behind, and count each type.

See solution 
ops = [row["operation"] for row in table.inspect.snapshots().select(["operation"]).to_pylist()]
from collections import Counter
print(Counter(ops))

Counter({'append': 7, 'delete': 6}). One initial append (V1, Step 1) and six later overwrite()s (P002's real change plus the five redundant nights), each resolved as delete + append — six additional deletes and six additional appends, plus the first one: 7 appends total, 6 deletes total, sum 13. Every delete writes no new file — it only stops referencing the previous snapshot's file — which is why the physical file count (7) is lower than the count of append operations that did write something.

Exercise 3 — Prediction: if the nightly pipeline had run for a full year (365 nights) instead of five, what would happen to how long table.inspect.files() takes to respond? Think about module 2, lesson 7's Going Deeper section, on inspection methods' relative cost.

See solution

table.inspect.files() (with no arguments, only the current snapshot) would keep responding almost instantly — it stays, always, at one file, no matter how many redundant nights went by, because the current snapshot's content never changed. What would grow, proportionally to the number of nights, is how long table.inspect.all_data_files() and table.inspect.snapshots() take — each has to walk a list of manifests that grows with every commit, even though none of those commits added a single row of business value. Module 2, lesson 7 already explained this: every inspection method pays a cost proportional to what it has to walk through, not to what the table "really contains" in business terms — three hundred sixty-five redundant nights would inflate that cost with no normal query ever noticing.

Summary and next step

In this lesson you measured, with executed evidence, exactly how cost accumulates in an Iceberg table: five nights of a pipeline that reconfirms without checking turned 3 snapshots into 13, and left six redundant data files alongside the single one the current table needs. You identified the three mechanisms that explain it — immutable files, snapshots that don't expire on their own, overwrite() that never compares — and confirmed that, despite all that noise, snap_v1's time travel kept working with no degradation at all.

Before moving on you should be able to: explain why all_data_files() and files() (current) can differ so much; and calculate, for any sequence of writes, how many snapshots a filterless overwrite() is going to produce.

Lesson 4 shifts axis: instead of old snapshots nobody needs anymore, it looks at the small files problem — even within a single current snapshot, many small writes can fragment the table into more files than necessary.

Resources

  • Apache Iceberg — official documentation, "Maintenance," "Expire Snapshots" section: "Snapshots accumulate until they are expired," the phrase that sums up this whole lesson. iceberg.apache.org/docs/latest/maintenance. In English.
  • PyIceberg — API reference, table.inspect.all_data_files() vs. table.inspect.files(), this lesson's Step 3's central distinction. py.iceberg.apache.org/api. In English.
  • This same guide, module 6, lesson 6 — source of the Exercise 2 that demonstrates table.upsert() with identical data produces zero new snapshots, this lesson's Going Deeper section's central contrast. 06-pyicebergs-upsert-the-python-native-alternative.md. In Spanish.
  • This guide's DESIGN doc — module 7's section, "why snapshots accumulate cost over time." src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.
← PreviousCatalogs beyond local: REST, Glue, Unity Catalog, PolarisNext →Compacting small files