Module overview: catalogs, maintenance, and Delta Lake by contrast

Why this module exists

Modules 1 through 6 built, piece by piece, a real Iceberg table: you loaded it (module 1), mapped its complete anatomy — catalog, metadata, manifest list, manifest files, data files (module 2), time-traveled over it (module 3), evolved its schema and partitioning with no rewriting at all (modules 4 and 5), and applied partial changes to it with MERGE INTO and upsert() (module 6). At no point in those six modules did you ask something any team operating this table in production is going to ask sooner or later: what happens to kiosko.dim_product if nobody touches it for six months, except to write to it every day?

The answer, with evidence you're going to generate yourself in lesson 3, is uncomfortable: it grows. Not the rows — they stay at four, one per product — but the number of archived snapshots, the number of physical Parquet files nobody needs to read anymore, and the metadata file's own size. Every write you made in modules 3 through 6 — every append(), every overwrite(), every upsert() — deliberately left a permanent trace: no data file ever gets deleted by a normal write, no snapshot disappears on its own. That's exactly what makes module 3's time travel possible. And it's exactly what, with no maintenance at all, turns a healthy table into one that's expensive to scan and expensive to store.

This module solves three questions no previous module touched:

1. Who's the catalog when it isn't a SQLite file on your laptop? Modules 1 through 6 used a kiosko catalog backed by SQLite, perfect for learning and perfect for $0 cost, but no real company puts its production lakehouse on a single .db file on one disk. This lesson names, without implementing any of them, the four catalogs that really get used in production: REST, AWS Glue Data Catalog, Unity Catalog, and Apache Polaris — and exactly what guarantee each one solves.
2. How do you prune a table without losing the time travel you do need? expire_snapshots, small-file compaction, remove_orphan_files — the three maintenance operations Iceberg's official documentation groups under "Maintenance." You're going to really run one, with pure-Python PyIceberg 0.11.1, and document, with the same honesty as always, why the other two don't run in this environment.
3. Why does the market name Iceberg and Delta Lake in almost the same breath? This is the only one of this guide's eight lessons that mentions Delta Lake — once, by contrast, without building a single Delta table.

The case that runs through the module: kiosko.dim_product, six months later

This module doesn't invent a new table to talk about maintenance. It goes back to kiosko.dim_product — the same table module 3 left with three snapshots (append of V1, delete + append from P002's change) and that module 6 explicitly confirmed it didn't touch. This module rebuilds that exact state in a new working directory, and adds what it was missing for the maintenance problem to be real, not hypothetical: five more nights of an operational pipeline that — with no bad intent, just without checking whether it was needed — reloads the complete product catalog every night, even when nothing changed. You're going to see, with table.history() really run, how those five redundant nights multiply the number of archived snapshots by more than four, without adding a single bit of new business information.

And, separately, you're going to see the other classic maintenance problem — small files — on a new, dedicated table, kiosko.fact_orders_daily_batches: the same forty-order week as always, this time loaded day by day instead of in a single append(), exactly how it would arrive in a real production pipeline that receives one file per day.

An analogy: pruning the wedding album without tearing out the pages grandma still wants to see

A wedding photo album, over the years, accumulates something nobody planned: the photographer delivered twenty near-identical copies of the same toast shot — the photographer fired in burst mode, and nobody bothered to discard the duplicates before archiving the whole roll. Over time, the album weighs three times what it should, and finding the exact toast photo gets slower, not because organization is missing, but because there are too many nearly identical pages to search through.

Pruning that album properly has two rules, not one. The first: throw out the duplicates — nineteen of the twenty toast copies can disappear without anyone noticing the difference, exactly what expire_snapshots does with snapshots nobody's ever going to query again. The second, the one a careless pruner breaks: never tear out the page grandma still wants to see — the maybe-blurry photo of the exact moment the grandfather who's no longer with us arrived, the only copy of that instant that exists. That is, precisely, this guide's snap_v1: the snapshot that keeps how P002 was before 2026-08-15, the one that makes recovering the correct margin (10.8) possible via time travel. Pruning this table properly means deleting the nineteen redundant toast copies, and explicitly protecting the photo that still matters.

Diagram: where you came from, where you're going

flowchart TB
    M3["Module 3:\nkiosko.dim_product\n3 snapshots -- snap_v1 protected"]
    M6["Module 6:\ndidn't touch dim_product,\nconfirmed it explicitly"]
    M3 --> M7A["This module:\n+5 redundant nights\n= 13 snapshots, 7 files, 1 live"]
    M6 -.-> M7A
    M7A --> L2["L2: production catalogs\nREST / Glue / Unity / Polaris\n(named, not implemented)"]
    M7A --> L3["L3: why it accumulates cost\n(real evidence)"]
    L3 --> L4["L4: compacting small files\n(representative -- Spark)"]
    L3 --> L5["L5: expire_snapshots\n(REAL, pure PyIceberg)"]
    L5 --> L6["L6: remove_orphan_files\n(representative -- Spark)"]
    L4 --> L7["L7: Delta Lake by contrast\n(quoted, not implemented)"]
    L6 --> L7
    L7 --> L8["L8: closing project,\nautomated assert"]
    L8 --> M8["Module 8:\nlakehouse capstone"]

The map of this module

Lesson    What it solves
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Why this module exists, the 5 redundant nights
          case, and the full map
L2        Production catalogs -- REST, Glue, Unity Catalog,
          Polaris -- named by their guarantee, none implemented
L3        Why snapshots accumulate cost -- real evidence:
          13 snapshots, 7 files, 1 live
L4        Compacting small files -- kiosko.fact_orders_daily_batches,
          7 live, tiny files (representative: Spark)
L5        expire_snapshots() -- REAL, pure Python PyIceberg, with
          snap_v1 protected and verified
L6        remove_orphan_files -- the 5 orphan files L5 left behind,
          verified with real code, deleted with Spark (representative)
L7        Delta Lake by contrast -- once, same problem,
          flat log vs metadata tree, 2026 convergence
L8        Project: Kiosko's maintained table, end to end

A technical warning, stated from here

This module runs entirely on pure-Python PyIceberg 0.11.1 — the same vehicle from modules 1 through 5 and 7 through 8, no JVM, no Spark. But, unlike those modules, here you're going to find this whole guide's first case where PyIceberg can't really run everything the market calls "Iceberg maintenance." It was verified, against PyIceberg 0.11.1's installed source code and against its official API reference, exactly what exists and what doesn't:

Operation	Does it exist in pure-Python PyIceberg 0.11.1?	Treatment in this module
expire_snapshots	Yes — table.maintenance.expire_snapshots().by_id(...)/.by_ids(...)/.older_than(...).commit()	L5, really executed
Compaction (rewrite_data_files)	No — no equivalent method exists on Table or on table.maintenance	L4, representative (Spark)
remove_orphan_files	No — no equivalent method exists	L6, representative (Spark)

And an even finer distinction, which lesson 5 develops with evidence: even the expire_snapshots() that does run in PyIceberg doesn't delete any physical file — it only rewrites the metadata to stop referencing the expired snapshots. Iceberg's official documentation, written with the Java/Spark implementation in mind, says expiring snapshots "removes data files that are no longer needed" — that's true for Spark's action, but not for PyIceberg 0.11.1's method, verified by reading its own source code in lesson 5. No previous guide in this ecosystem had to draw this fine a distinction between "what the project's general documentation says" and "what the installed Python client literally does" — this lesson does, with evidence, because it's exactly the kind of difference that breaks a real maintenance plan if nobody verifies it first.

The boundary: what does NOT belong in this module

This module doesn't connect to any real managed catalog: naming REST, Glue, Unity Catalog, and Polaris doesn't mean having an AWS account, a Databricks workspace, or a Polaris server running — that, with real credentials and permissions, is aws-core-services-guide's territory. It isn't a FinOps module either: calculating how much it costs, in dollars, to keep old snapshots in S3 — or deciding how often to run expire_snapshots on a production cluster based on budget — belongs to cost-optimization-caching-guide; here "maintenance" is local table hygiene, verified with before/after counts, not a cost system. And Delta Lake, in lesson 7, gets named exactly once, with its own syntax quoted against its official documentation — this guide never builds a second, parallel implementation.

Common mistakes

Assuming "PyIceberg 0.11.1" and "Apache Iceberg" always have the same capabilities. What happens: someone reads Apache Iceberg's general documentation — mostly written with the Java implementation and Spark's actions in mind — finds an operation described there with complete naturalness, and assumes pip install pyiceberg gives them access to exactly the same thing. Why it happens: the Apache Iceberg project is one, with one specification, but three independent client implementations — Java, Python (PyIceberg), Rust — and each implements a different subset of the full specification, at its own pace. How to spot it: if you're going to depend on a maintenance operation in a 100% Python pipeline, first verify, like this module does, that the method really exists in the installed version — dir(table.maintenance), or PyIceberg's official API reference, never the project's general documentation alone. How to fix it: this module's lessons 4 and 6 are the permanent example of how to proceed when the answer is "no, not yet": document the real syntax of the alternative that does exist (Spark, in this case), explicitly mark it as representative, and don't fake an execution that didn't happen.

Thinking "accumulating snapshots" is always a design mistake that has to be avoided from the start. What happens: someone, after seeing lesson 3's five-redundant-nights problem, concludes module 3's table "was built wrong" from the start, or that it should have been cleaned up after every write. Why it happens: it's easy to confuse "this generates a cost that has to be managed" with "this is a flaw that has to be prevented." How to spot it: if you finish this module thinking every append() or overwrite() should be followed by an immediate expire_snapshots(), revisit lesson 5 — the snap_v1 protecting module 3's time travel is, literally, an "old" snapshot that deliberately doesn't get expired. How to fix it: accumulating snapshots isn't the mistake — not having an explicit policy for which ones to protect and which ones to prune is the mistake. This module teaches exactly that policy: never expire what the business still needs for time travel, always prune what was pure operational noise.

Exercises

Exercise 1 — Before reading lesson 2, predict: what do the four production catalogs you're going to learn about have in common, beyond their names? Think about what this guide's module 2 already taught about the kiosko/SQLite catalog: what minimum guarantee does any Iceberg catalog have to meet, local or production?

See solution

All four — REST, Glue, Unity Catalog, Polaris — meet the same minimum guarantee you already saw in module 2 about the kiosko catalog: keeping a single pointer to each table's current metadata file, and updating that pointer atomically (never halfway) when a new write is confirmed. The difference between kiosko/SQLite and the four production catalogs isn't that guarantee — all of them have it — it's how many simultaneous writers, from how many different engines, and with what concurrency-control mechanism they sustain it. Lesson 2 develops that difference with evidence.

Exercise 2 — Calculate yourself: if the five-redundant-nights pipeline had used table.upsert() instead of table.overwrite(), how many new snapshots would it have produced? Use what you already confirmed in module 6, lesson 6's Exercise 2 (running upsert() a second time with identical data).

See solution

Zero. Module 6, lesson 6's Exercise 2 already demonstrated, with executed evidence, that table.upsert() compares every row against what already exists, and produces no new snapshot when there's no real change (UpsertResult(rows_updated=0, rows_inserted=0), table.history() not growing). If this module's nightly pipeline had used upsert() instead of blind overwrite(), the five nights with no real changes wouldn't have left any additional trace — the whole problem this module's lesson 3 is going to directly quantify wouldn't have existed. This is the same pattern module 6's lesson 7 already anticipated: the choice of write tool has consequences beyond the immediate result.

Exercise 3 — Explain, in your own words, why this guide chooses by_ids() with snapshots captured in a variable instead of older_than() with a "now minus a few days" timestamp for lesson 5. Think about this guide's hard rule about datetime.now().

See solution

older_than(dt) needs a threshold datetime, and building that threshold from "the current moment minus N days" would imply, somewhere in the code, a call to datetime.now() or equivalent — exactly the source of non-determinism this guide's hard rule bans from any code feeding a "What to expect" block. With by_ids(), instead, the list of snapshots to expire gets built by reading table.history() at run time and filtering by the snapshot_ids you already know in advance (snap_v1, the current one) — no wall clock enters the decision, so anyone who runs this module, at any moment, gets exactly the same structural result. Lesson 5 does run older_than() once, in an isolated demo, to show it exists and works — but Kiosko's main flow uses by_ids(), for this same determinism reason.

Summary and next step

In this lesson you saw why this module exists: six modules in this guide built a real Iceberg table, but none asked what happens to that table as time passes with no maintenance at all. You learned the full map of the eight lessons, and the central technical warning: PyIceberg 0.11.1 really runs expire_snapshots, but not compaction or remove_orphan_files — both get documented as representative, with Spark's exact syntax, verified against the official documentation.

Before moving on you should be able to: name this module's three maintenance operations and which of the three really runs in this environment; and explain why "pruning" and "protecting snap_v1" aren't contradictory goals.

Lesson 2 doesn't touch kiosko.dim_product yet — it names, one by one, the four production catalogs this module never implements, and the exact guarantee each one solves.

Resources

• Apache Iceberg — official documentation, "Maintenance," the source for the three operations that organize this whole module. iceberg.apache.org/docs/latest/maintenance. In English.
• PyIceberg — API reference, the table.maintenance section, the only reliable source for what really runs in pure Python. py.iceberg.apache.org/api. In English.
• PyIceberg — PyPI, current version 0.11.1, the same one you installed since module 1. pypi.org/project/pyiceberg. In English.
• src/paths/data-engineering-ecosystem/VALIDACION.md — the market audit that explicitly asks for this guide to cover "compaction and catalogs, which is where the open discussion is." Internal repo document. In Spanish.
• This guide's DESIGN doc — the full map of the eight modules, including module 7's complete section. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.