Module overview: schema evolution without rewriting

Why this module exists

This guide's module 1, in its lesson 2, visited four times the Kiosko ecosystem ran into the same ceiling. The first of those four was data-engineering-foundations-guide (module 6): the overwrite-partition pattern — deleting a date's partition, then inserting that date's new data — which really does solve the duplicated-rerun problem. That guide was honest about its own solution's cost, with a quote this lesson is going to revisit word for word in lesson 2: between the DELETE and the INSERT there's a real moment where the partition sits empty, and if the process dies right there, the date is left without data. Two separate operations, run in sequence, with a risk window in between — that is, precisely, the opposite of an atomic operation.

This module closes that pending promise. You're going to see, with executed evidence — not a brochure claim — why an Iceberg write never leaves the table "halfway" between one state and the next, not even when that write, internally, does more than one thing at once (you already saw this in module 3: table.overwrite() can produce a delete snapshot and an append snapshot in the same call). And you're going to use that same atomicity guarantee as the foundation for building something new: evolving a table's schema — adding a column, renaming it, dropping it — without touching a single Parquet file that already exists.

The case that runs through the module: kiosko.dim_store learns where each store is from

Up to this module, kiosko.dim_store didn't exist in any Iceberg catalog in this guide — it's a new table, Kiosko's three stores you already worked with in the six previous guides in the ecosystem: S01 Kiosko Centro (Bogotá), S02 Kiosko Norte (Lima), S03 Kiosko Sur (Santiago). This module creates it with three columns — store_id, store_name, city — and then adds it a fourth: country, deterministically derived from city (Bogotá→Colombia, Lima→Peru, Santiago→Chile). It also adds, and drops, a scratch column called temp_notes, just to demonstrate with evidence that a DROP COLUMN doesn't rewrite any data file either.

What makes this case interesting isn't the final result — a three-store table with one more column is, on its own, pretty undramatic. What's interesting is how it gets there: with no already-written Parquet file ever touched, no migration that locks the table, no reader querying kiosko.dim_store at that instant ever seeing, not even for a microsecond, a schema halfway through changing.

An analogy: the census form, not anyone's door

Imagine a national census that already covered half a city, house by house, with a three-question form. Halfway through, someone decides the form needs a fourth question — country of birth, say. There are two ways to handle this. The bad one: go back and knock on every already-surveyed door, so they fill in the new question from scratch — expensive, slow, and probably impossible if some of those houses no longer have anyone answering. The good one: add the question to the form from that point forward, leave blank (or fill in with a known value, if it can be deduced) the answer for those already surveyed, and keep surveying the rest of the city with the four-question form. Nobody goes back to knock on a door already knocked on.

That is, precisely, what this module does with kiosko.dim_store. Adding country doesn't rewrite the three rows that already exist — nobody "knocks on the door again" for S01, S02, or S03 — the new schema simply starts applying from that moment forward, and the old rows are left available to be filled in, in this case with a value that really can be deduced from data they already had (city). And lesson 2's atomicity is the other half of the same idea, with a different analogy: a census office's front desk that opens or updates a record in a single transaction, never halfway — you're never going to find a record with the name already changed but the address still old, because the desk never hands out any partial result to whoever asks while the transaction is in progress.

Diagram: where this module starts, where it lands

flowchart LR
    A["foundations M6:\nDELETE + INSERT\nreal risk window,\nquoted literally"] --> B["Lesson 2:\nwhy an Iceberg write\nIS atomic\n(executed evidence)"]
    B --> C["Lesson 3:\nadd_column('country')\nmetadata only, 0 files touched"]
    C --> D["Lesson 4:\nrename + add/drop 'temp_notes'\nsafe, same mechanism"]
    D --> E["Lesson 5:\npopulate country from city\nBogota->Colombia, Lima->Peru,\nSantiago->Chile"]
    E --> F["Lesson 6:\nsnap_before_evolution\nstill reads the old schema"]
    F --> G["Lesson 7:\nwhat 'ACID' guarantees here,\nprecisely"]
    G --> H["Lesson 8:\nProject: dim_store\nevolved, end to end"]

The map of this module

Lesson    What it solves
────────  ──────────────────────────────────────────────────────────────
L1        (this one) The full map: from foundations' quote to the final project
L2        Why overwrite-partition was never atomic -- and why Iceberg's
          table.overwrite() is, with executed evidence
L3        add_column() on kiosko.dim_store: a metadata operation,
          zero Parquet files touched
L4        rename_column() and delete_column(): the same guarantee, with
          temp_notes added and dropped in the same lesson
L5        country deterministically populated from city -- the new
          column's real payoff
L6        table.scan(snapshot_id=snap_before_evolution) still reads
          the 3-column schema, no country
L7        What "ACID" guarantees in this exact context -- no more, no less
L8        Project: dim_store evolved, end to end, with assert

Lessons 3 and 4 build the mechanism — adding, renaming, dropping a column — with no business data yet, on temp_notes as a practice column. Lesson 5 is the real payoff: country, populated with Kiosko's three countries. Lesson 6 verifies something that isn't obvious at first glance: a snapshot before the schema evolution still reads with its own schema, not with the one the table has now. Lesson 7 takes a step back and precisely, without overstating it, answers what the word "ACID" guarantees in the context of a single Iceberg table — which isn't the same as a full transactional engine with multiple tables. Lesson 8 brings the seven pieces together into a single project.

The boundary: what does NOT belong in this module

This module evolves the schema of one local table, with a catalog 100% on your own machine. What doesn't belong here is publishing versioned schema contracts as a governance artifact — which columns can change, who approves the change, how downstream consumers get notified — which is exactly data-reliability-and-governance-guide's job. Partitioning kiosko.dim_store or any other table doesn't belong here either — that arrives in module 5, with kiosko.fact_orders_at_scale. And this module doesn't touch kiosko.fact_orders or kiosko.dim_product, the two tables from modules 1 through 3: kiosko.dim_store is a new, isolated table, built specifically for this module.

Common mistakes

Expecting "schema evolution" to mean the same thing as "changing the data." What happens: someone, on hearing "add a column," assumes Iceberg automatically rewrites every row with a calculated value for the new column. Why it happens: in many everyday tools — a spreadsheet, a web form — "add a column" and "fill it with data" happen in the same gesture, so it's natural to expect the same here. How to spot it: if after add_column("country", ...) you expect to see the three countries already populated, without having run any additional step, revisit lesson 3 — the real result is country=None for all three rows. How to fix it: this module deliberately splits into two separate steps what looks like a single action: lesson 3 adds the column (metadata, instant, zero rows touched); lesson 5 populates it with real data (a normal write, with its own snapshot). They're two operations of a different nature, and this guide treats them as such.

Confusing "atomic" with "instant" or "with no cost at all." What happens: someone interprets an atomic Iceberg operation as taking no time, or thinks "atomic" means Iceberg is, in general, faster than any alternative. Why it happens: the word "atomic" in everyday language sometimes gets used as a vague synonym for "fast" or "simple." How to spot it: if your definition of "atomic" doesn't mention the word "indivisible" or explain what happens if the process fails halfway through, you still don't have the precise definition this module uses. How to fix it: "atomic" precisely means an operation happens as a single indivisible unit from the outside — either it's seen complete, or it isn't seen at all; never halfway. It says nothing about speed. This module's lesson 2 builds this definition with real evidence, not a loose analogy.

Exercises

Exercise 1 — Before starting, recover data-engineering-foundations-guide's exact quote. Without looking back yet, try to remember (or look up in this guide's module 1, lesson 2) the exact phrase that guide used to describe the overwrite-partition pattern's risk. Write it down, because this module's lesson 2 revisits it word for word.

See solution

The central quote, from data-engineering-foundations-guide's module 6 lesson 3's Going deeper section, says: "There's a real cost to this decision, and it's worth naming honestly: between the DELETE and the INSERT, there's a moment where the partition sits empty. If something fails exactly at that moment — the process dies halfway through — the date is left temporarily without data [...]". That same guide acknowledges, in the following sentence, that wrapping both operations in an explicit transaction "is a real improvement over this design" — exactly the improvement an Iceberg write delivers out of the box, with nobody having to wrap anything by hand.

Exercise 2 — Predict: how many kiosko.dim_store rows are going to have country=None at some point in this module? Based only on this lesson's map (without having read lessons 3 and 5 yet), predict how many of dim_store's three rows are going to pass through, even momentarily, a state with country=None.

See solution

All three. add_column("country", ...) in lesson 3 adds the column at the schema level without touching any existing data file — so, until lesson 5 populates it with a real overwrite(), all three rows (S01, S02, S03) have country=None simultaneously. This isn't a transient error you need to rush to fix: it is, precisely, lesson 3's central point — demonstrating that adding a column is a metadata operation, independent of whether that column already has real values or not.

Exercise 3 — Name, from memory, the difference between what module 3 already demonstrated about overwrite() and what this module is going to demonstrate about update_schema(). In 2-3 sentences, explain what type of operation each one is, and why both end up guaranteeing the same thing (atomicity) even though they change different things.

See solution

table.overwrite(), which module 3 already used for the P002 change, is a data operation: it replaces a table's rows, and can internally produce more than one snapshot (delete + append), as that module's lesson 3 revealed. table.update_schema(), this module's protagonist, is a metadata operation: it changes the table's column structure — adding, renaming, dropping — without touching any data file or creating a new snapshot. Both end up guaranteeing atomicity for the same underlying reason, which lesson 7 of this module develops in depth: any change to an Iceberg table's state — whether data or schema — gets confirmed with a single atomic move of the catalog's pointer, never in two separate steps an external reader could catch halfway through.

Summary and next step

In this lesson you learned module 4's full map: module 1's pending promise — why an Iceberg write really is atomic, unlike data-engineering-foundations-guide's overwrite-partition — and the new case that comes with it — kiosko.dim_store, with a country column added without rewriting any file, and a temp_notes column added and dropped to prove it. You saw the census-form analogy, and the explicit boundary of what this module doesn't solve.

Before moving on you should be able to: explain, in your own words, the difference between a data operation and a schema operation in Iceberg; and recall from memory data-engineering-foundations-guide's exact quote about the overwrite-partition risk, because lesson 2 revisits it directly.

Lesson 2 is where the real comparison happens: the same quote, the same risk, and the executed evidence for why an Iceberg write doesn't have it.

Resources

• data-engineering-foundations-guide DESIGN doc — source of the overwrite-partition pattern and the exact quote about the risk window between DELETE and INSERT, which this module's lesson 2 revisits. src/guides/data-engineering-foundations-guide/DISENO.md. In Spanish.
• Apache Iceberg — official documentation, "Reliability" (Serializable Isolation, Optimistic Concurrency), the formal foundation for the atomicity this module demonstrates with code. iceberg.apache.org/docs/latest/reliability. In English.
• Apache Iceberg — official documentation, "Evolution" (Schema evolution, Correctness), the formal foundation for the safe schema evolution this module runs. iceberg.apache.org/docs/latest/evolution. In English.
• PyIceberg — API reference, table.update_schema() with add_column/rename_column/delete_column/update_column. py.iceberg.apache.org/api. In English.
• This guide's DESIGN doc — the full map of the eight modules, including this module's exact boundary. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.