Evolving the `PartitionSpec` without rewriting

Description

kiosko.fact_orders_at_scale ended up partitioned by store_id, with ten million rows spread across three files. But Kiosko also frequently needs date-range queries — "this week's revenue," "yesterday's orders" — and today order_ts isn't part of the PartitionSpec at all: any date filter has to open all three files in full, exactly like you saw in the previous lesson's exercise 3 with product_id. This lesson adds that second dimension — DayTransform over order_ts — with a single line of code, and confirms, with evidence, something that might sound impossible: the ten million rows that already exist don't get touched.

Connection to the module. This lesson does, with partitioning, exactly what all of module 4 already did with schema: update_schema().add_column() never rewrote an existing Parquet file; update_spec().add_field(), this lesson's protagonist, doesn't either. It's the same underlying mechanism — Iceberg separates the table's definition (metadata) from its data (Parquet) — applied this time to how the files are organized, not to which columns they have.

An analogy: the carrier adds a second sorting criterion

Going back to the carrier: so far he organizes his bags only by recipient. This lesson is the moment he decides to add a second criterion — within each recipient's bag, also separate by day. And notice what the carrier does not do: he doesn't empty already-full bags to reorganize them under the new criterion. That mail already filed away stays exactly where it is, organized by the old rule. The new criterion applies only to mail that arrives from now on. That's exactly what you're going to confirm in this lesson with real code.

Worked example: update_spec(), with evidence nothing gets rewritten

Step 1 — The state before evolving

# l6_evolve_spec.py
import os

from pyiceberg.catalog import load_catalog
from pyiceberg.transforms import DayTransform

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")
catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.fact_orders_at_scale")

files_before = sorted(f["file_path"] for f in table.inspect.files().to_pylist())
snapshots_before = len(table.history())
print(f"before evolving: {len(files_before)} data file(s), {snapshots_before} snapshot(s)")
print("current spec before evolving:")
print(table.spec())

What to expect (verified by running the real script, continuing on lesson 5's same table):

before evolving: 3 data file(s), 1 snapshot(s)
current spec before evolving:
[
  1000: store_id: identity(3)
]

Three files, a single snapshot — the one table.append() created in lesson 5 — and the spec you already know. This is the exact starting point you're going to compare against the final result.

Step 2 — Evolve the spec: add order_day

with table.update_spec() as update:
    update.add_field("order_ts", DayTransform(), "order_day")

print("\ncurrent spec after evolving:")
print(table.spec())

What to expect:

current spec after evolving:
[
  1000: store_id: identity(3)
  1001: order_day: day(7)
]

One line of code, and the table's PartitionSpec now has two fields: the original store_id (field_id=1000) stays exactly the same, and a new field, order_day (field_id=1001, as lesson 4 anticipated: the next available number after 1000), which truncates order_ts (field_id=7 in the schema, hence day(7)) to its calendar date. Notice you did not pass source_id as a number this time — you passed "order_ts", the column name — unlike lesson 5, where the initial PartitionSpec was built by hand for a table that didn't yet exist, update_spec() operates on a table that's already alive, with a real schema it can ask "what's order_ts's field_id?" for you.

Step 3 — Confirm nothing got rewritten

files_after = sorted(f["file_path"] for f in table.inspect.files().to_pylist())
snapshots_after = len(table.history())
print(f"\ndata files after evolving: {len(files_after)}")
print(f"snapshots after evolving: {snapshots_after}")
print(f"identical files before/after (not one got rewritten): {files_before == files_after}")

assert files_before == files_after, "evolving the spec must NEVER touch an existing data file"
assert snapshots_before == snapshots_after, "evolving the spec never creates a new snapshot"

What to expect:

data files after evolving: 3
snapshots after evolving: 1
identical files before/after (not one got rewritten): True

The same three files, with exactly the same names, and still a single snapshot. update_spec() didn't write a single byte of new data — it only updated the table's definition in a new metadata file, the same kind of "instant" operation you already saw with update_schema() in module 4. Ten million rows, evolved in how they're organized, without moving a single one of them.

Diagram: what changed, and what didn't

flowchart LR
    subgraph antes["Before update_spec()"]
        A1["3 data files\nstore_id=S01/S02/S03"]
        A2["spec: store_id only"]
        A3["1 snapshot"]
    end

    subgraph despues["After update_spec()"]
        B1["THE SAME 3 files\n(not one touched)"]
        B2["spec: store_id + order_day\n(new spec_id)"]
        B3["1 snapshot\n(unchanged -- not a data write)"]
    end

    A1 -.->|"identical"| B1
    A2 -->|"update_spec().add_field()"| B2
    A3 -.->|"identical"| B3

Going deeper: why this is safe, and what "new spec" means

It's worth being precise about what update_spec() actually produced. It didn't modify the original PartitionSpec — that spec, with spec-id=0 and a single field (store_id), still exists, archived in the table's metadata tree, exactly as lesson 5 left it. What it produced was a new spec — spec-id=1, with two fields — and it updated the table's "current spec" pointer toward that new spec, for any future writes. It's exactly the same pattern you already know from snapshots: nothing ever gets overwritten, a new version always gets archived and a pointer gets updated.

This explains, at once, why partition evolution is safe even while other processes are reading or writing the table at the same time: the three existing data files stay associated, in the manifest that describes them, with the original spec-id=0 — nothing about how they're organized changed, so no reader that already knew how to interpret them breaks. Lesson 7 is going to make this distinction visible with table.inspect.partitions(), showing each group of files' spec_id, side by side.

Common mistakes

Expecting update_spec() to repartition the existing data under the new scheme. What happens: someone, after running this lesson's step 2, expects table.inspect.partitions() to show the ten million rows already grouped by order_day too, with real date values. Why it happens: if you're coming from a system where changing the partitioning implies a full reorganization of the data (a blocking ALTER TABLE, for example), it's natural to expect the same behavior here. How to spot it: if you query table.inspect.partitions() right after evolving the spec and expect to see order_day values for the old rows, revisit this lesson's step 3 — the file count didn't change, so how those existing files get described didn't change either. How to fix it: lesson 7 shows the exact result — the old rows keep showing up under the original spec_id, with order_day=None, precisely because they were never rewritten. Only rows written after the evolution carry a populated order_day.

Calling update.add_field() outside the with table.update_spec() as update: block. What happens: someone tries to use the update object outside the with, or saves it in a variable to reuse later somewhere else in the script, and gets an error or unexpected behavior. Why it happens: PyIceberg's with ... as update: pattern — the same one you already used with update_schema() in module 4 — groups several operations into a single transaction that gets committed automatically when the block exits; the update object isn't meant to live beyond that with. How to spot it: if your code saves a reference to update and uses it after the with block has already ended, revisit this lesson's worked example again. How to fix it: every update.add_field() (or remove_field(), rename_field()) call you want grouped into a single spec evolution goes inside the same with block — exactly like module 4's project already did with several chained schema operations.

Exercises

Exercise 1 — Reproduce the evolution yourself, and confirm the three numbers. With lesson 5's kiosko.fact_orders_at_scale available, run this lesson's three steps. Confirm 3 files before and after, 1 snapshot before and after, and the new spec with two fields.

Exercise 2 — Evolve the spec a second time, adding BucketTransform over franchise_id. Using what you learned in lesson 4 about high cardinality, add a third partition field: update.add_field("franchise_id", BucketTransform(8), "franchise_bucket"). Confirm the spec now has three fields, and that the data files still haven't changed.

from pyiceberg.transforms import BucketTransform

files_before_second = sorted(f["file_path"] for f in table.inspect.files().to_pylist())
with table.update_spec() as update:
    update.add_field("franchise_id", BucketTransform(8), "franchise_bucket")
print(table.spec())
files_after_second = sorted(f["file_path"] for f in table.inspect.files().to_pylist())
print("files unchanged:", files_before_second == files_after_second)

Exercise 3 — Explain, in your own words, why this lesson never creates a new snapshot. In 2-3 sentences, connect this observation to what you already know from module 3 about which operations create snapshots and which don't.

Summary and next step

In this lesson you evolved kiosko.fact_orders_at_scale's PartitionSpec, adding order_day (DayTransform over order_ts) alongside the original store_id, with a single line of code: update.add_field("order_ts", DayTransform(), "order_day"). You confirmed, with asserts on the exact count of files and snapshots, that the ten million rows already loaded weren't touched at all — the same guarantee module 4 already demonstrated for schema evolution, applied here to partition evolution.

Before moving on you should be able to: explain the difference between the original spec-id and the new one; reproduce the evolution on your own table; and anticipate that the old rows are going to show order_day=None in any inspection, because they were never rewritten.

You have an evolved spec, but you still haven't seen, with your own eyes, how the old data (under the original spec) coexists with the new data (under the evolved spec) inside the same table. Lesson 7 adds a new franchise — the first one to arrive after this evolution — and uses table.inspect.partitions() to show both schemes, side by side, in the same query.

Resources

• PyIceberg — API reference, table.update_spec(), update.add_field()/remove_field()/rename_field(). py.iceberg.apache.org/api. In English.
• Apache Iceberg — official documentation, "Partitioning," "Partition Evolution" section (the formal guarantee that evolving a spec doesn't rewrite existing data). iceberg.apache.org/docs/latest/partitioning. In English.
• This guide's DESIGN doc — module 5's section, with the spec evolution's exact specification (update_spec().add_field("order_ts", DayTransform(), "order_day")). src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.