Module 5: Hidden Partitioning And Partition Evolution

Reading old and new partition schemes, together

Next →

Description

Lesson 6 evolved kiosko.fact_orders_at_scale's PartitionSpec without touching any of the three existing files. This lesson runs the definitive test: it adds a new franchise — number 250,000, the first one to arrive after the evolution — and uses table.inspect.partitions() to show, in a single query, how the ten million old rows and the forty new rows coexist under two different partition schemes, within the same table, with no conflict at all.

Connection to the module. This is the lesson that closes the module's central argument: lesson 1 promised both schemes were going to coexist; lesson 6 proved nothing gets rewritten; this lesson is the visual evidence, with real data, of that coexistence. Lesson 8 integrates the seven previous lessons into a single project.

An analogy: the carrier receives mail under the new rule

Going back to this module's carrier: lesson 6 left him with a new rule — also organize by day, in addition to by recipient — but without having touched the old mail. This lesson is the moment genuinely new mail arrives, and the carrier files it using the rule that's in effect now: by recipient and by day. If you ask him right now "show me everything you have filed," he shows you two kinds of record, with no contradiction between them: the old mail, filed under the old rule (recipient only), and the new mail, filed under the new rule (recipient and day). Neither one is "wrong" — each one precisely reflects the rule that was in effect at the moment it got filed.

Worked example: a new franchise, under the evolved spec

Step 1 — Add franchise 250,000

# l7_read_old_and_new_partitions.py
import os
from datetime import datetime

import pyarrow as pa
import pyarrow.compute as pc
from pyiceberg.catalog import load_catalog

from kiosko_scale import generate_orders_at_scale

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")
catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.fact_orders_at_scale")


def revenue_of(arrow_table: pa.Table) -> float:
    line_revenue = pc.multiply(pc.cast(arrow_table.column("quantity"), pa.float64()), arrow_table.column("unit_price"))
    return float(pc.sum(line_revenue).as_py())


PA_SCHEMA = pa.schema([
    pa.field("order_id", pa.string(), nullable=False),
    pa.field("franchise_id", pa.int32(), nullable=False),
    pa.field("store_id", pa.string(), nullable=False),
    pa.field("product_id", pa.string(), nullable=False),
    pa.field("quantity", pa.int32(), nullable=False),
    pa.field("unit_price", pa.float64(), nullable=False),
    pa.field("order_ts", pa.timestamp("us"), nullable=False),
])

# franchise #250,000 -- the first one to arrive AFTER evolving the spec.
# Same deterministic generator as always, no random.
NEW_FRANCHISE_ID = 250_000
new_rows = list(generate_orders_at_scale(1))
for r in new_rows:
    r["franchise_id"] = NEW_FRANCHISE_ID
    r["order_id"] = r["order_id"].replace("F000000", f"F{NEW_FRANCHISE_ID:06d}")
    r["order_ts"] = datetime.fromisoformat(r["order_ts"])
new_pa_table = pa.Table.from_pylist(new_rows, schema=PA_SCHEMA)

table.append(new_pa_table)
print(f"franchise {NEW_FRANCHISE_ID} added: {new_pa_table.num_rows} new rows, "
      f"under the current spec (with order_day)")

What to expect (verified by running the real script, continuing on the table evolved in lesson 6):

franchise 250000 added: 40 new rows, under the current spec (with order_day)

Forty new rows, loaded with the same table.append() as always — no extra argument at all, with no need to tell Iceberg "use the new spec." The engine already knows what the current spec is, and applies it automatically to any new write.

Step 2 — Confirm the total, and S01's revenue including the new franchise

row_count_now = table.scan().to_arrow().num_rows
revenue_now = revenue_of(table.scan().to_arrow())
print(f"total now: {row_count_now} rows, revenue {round(revenue_now, 2)}")
assert row_count_now == 10_000_000 + 40
assert round(revenue_now, 2) == round(26_537_500.00 + 106.15, 2) == 26_537_606.15

s01_now = table.scan(row_filter="store_id == 'S01'").to_arrow()
s01_revenue_now = revenue_of(s01_now)
print(f"S01 now: {s01_now.num_rows} rows, revenue {round(s01_revenue_now, 2)}")
assert round(s01_revenue_now, 2) == round(9_575_000.00 + 38.30, 2) == 9_575_038.30

What to expect:

total now: 10000040 rows, revenue 26537606.15
S01 now: 4000016 rows, revenue 9575038.3

The total went up by exactly 40 rows and 106.15 in revenue — the full new franchise, not one row more or less — and the hidden filter by S01 still works exactly like in lesson 5, now including the 16 S01 lines the new franchise brought (4,000,000 + 16 = 4,000,016 rows, 9,575,000.00 + 38.30 = 9,575,038.30 in revenue). The filtered query never had to find out the spec changed halfway through.

Step 3 — table.inspect.partitions(): both schemes, in the same table

print("=== table.inspect.partitions() -- both partition schemes ===")
partitions = table.inspect.partitions().to_pylist()
partitions_sorted = sorted(partitions, key=lambda p: (p["spec_id"], str(p["partition"])))
for p in partitions_sorted:
    print(f"{p['spec_id']:>7}  {str(p['partition']):<45}  {p['record_count']}")

spec_ids_present = sorted({p["spec_id"] for p in partitions})
print(f"\nspec_id present: {spec_ids_present}")
assert spec_ids_present == [0, 1]

spec0_rows = sum(p["record_count"] for p in partitions if p["spec_id"] == 0)
spec1_rows = sum(p["record_count"] for p in partitions if p["spec_id"] == 1)
assert spec0_rows == 10_000_000
assert spec1_rows == 40

What to expect (full output; spec_id=0 is lesson 5's original spec, spec_id=1 is the one evolved in lesson 6):

=== table.inspect.partitions() -- both partition schemes ===
      0  {'store_id': 'S01', 'order_day': None}         4000000
      0  {'store_id': 'S02', 'order_day': None}         3250000
      0  {'store_id': 'S03', 'order_day': None}         2750000
      1  {'store_id': 'S01', 'order_day': datetime.date(2026, 8, 3)}  4
      1  {'store_id': 'S01', 'order_day': datetime.date(2026, 8, 4)}  2
      1  {'store_id': 'S01', 'order_day': datetime.date(2026, 8, 5)}  1
      1  {'store_id': 'S01', 'order_day': datetime.date(2026, 8, 6)}  2
      1  {'store_id': 'S01', 'order_day': datetime.date(2026, 8, 7)}  3
      1  {'store_id': 'S01', 'order_day': datetime.date(2026, 8, 8)}  3
      1  {'store_id': 'S01', 'order_day': datetime.date(2026, 8, 9)}  1
      1  {'store_id': 'S02', 'order_day': datetime.date(2026, 8, 3)}  2
      1  {'store_id': 'S02', 'order_day': datetime.date(2026, 8, 4)}  2
      1  {'store_id': 'S02', 'order_day': datetime.date(2026, 8, 5)}  1
      1  {'store_id': 'S02', 'order_day': datetime.date(2026, 8, 6)}  2
      1  {'store_id': 'S02', 'order_day': datetime.date(2026, 8, 7)}  2
      1  {'store_id': 'S02', 'order_day': datetime.date(2026, 8, 8)}  3
      1  {'store_id': 'S02', 'order_day': datetime.date(2026, 8, 9)}  1
      1  {'store_id': 'S03', 'order_day': datetime.date(2026, 8, 3)}  2
      1  {'store_id': 'S03', 'order_day': datetime.date(2026, 8, 4)}  2
      1  {'store_id': 'S03', 'order_day': datetime.date(2026, 8, 6)}  1
      1  {'store_id': 'S03', 'order_day': datetime.date(2026, 8, 7)}  2
      1  {'store_id': 'S03', 'order_day': datetime.date(2026, 8, 8)}  3
      1  {'store_id': 'S03', 'order_day': datetime.date(2026, 8, 9)}  1

spec_id present: [0, 1]

There they are, side by side, the two schemes this module promised since lesson 1. The first three rows (spec_id=0) are the ten million original rows — grouped only by store_id, with order_day=None because those files never stored that dimension (it didn't exist when they were written). The following twenty-three rows (spec_id=1) are the new franchise — only 40 rows total, spread across real store_id and order_day combinations, with readable dates (datetime.date(2026, 8, 3), not the raw integer you saw in lesson 4) — because those files were written after the evolution, with the full dimension available.

Notice something else that confirms, with data, this module's discipline: S03 has no row at all for 2026-08-05 — neither in the old spec (aggregated, you wouldn't be able to tell) nor in the new one — because Kiosko's real week never had an S03 order that day (check RAW_ORDERS from module 1: Wednesday only had two orders, from S01 and S02). The per-day breakdown you see here is exactly the same real week you've known since the ecosystem's first guide, now visible through a new partitioning.

Diagram: two specs, one table

flowchart TB
    T["kiosko.fact_orders_at_scale"]
    T --> S0["spec_id=0: store_id only\n10,000,000 rows\norder_day=None"]
    T --> S1["spec_id=1: store_id + order_day\n40 rows (franchise 250,000)\norder_day populated"]
    S0 -.->|"lesson 5's files,\nnever rewritten"| F0["3 data files"]
    S1 -.->|"lesson 7's files,\nwritten under the new spec"| F1["several small files\n(one per store_id x order_day\ncombination)"]
    Q["table.scan(row_filter=\"store_id == 'S01'\")"] -->|"reads BOTH specs\nwith no distinction for the caller"| S0
    Q --> S1

Common mistakes

Thinking spec_id is a real column of the table, queryable with row_filter. What happens: someone, on seeing spec_id in table.inspect.partitions()'s result, tries to write table.scan(row_filter="spec_id == 0") to read only the old rows. Why it happens: spec_id shows up alongside real business columns (store_id, order_day) in the same result table, so it's easy to assume it has the same status. How to spot it: if your row_filter mentions spec_id and fails with an unknown-column error, revisit this lesson — spec_id is metadata about the partitioning, generated by table.inspect.partitions(), never a column of the table's business schema. How to fix it: spec_id is introspection information, useful for understanding how the table is organized internally (exactly this lesson's purpose) — it isn't part of a normal query's row_filter, which only knows real columns like store_id, product_id, or order_ts.

Expecting order_day=None in the old rows to mean those rows have a missing or corrupted date. What happens: someone, on seeing order_day: None for the ten million original rows, worries order_ts got lost or loaded incorrectly for that data. Why it happens: None is usually associated with "missing data" in the data-quality sense, not with "this row never computed this derived value." How to spot it: if your concern is about order_ts's quality in the old rows, verify directly with table.scan(row_filter="franchise_id < 250000").to_arrow().column("order_ts") — you're going to see order_ts is complete and correct in every row. How to fix it: order_day is a partition value, not a business schema column — it doesn't exist as a queryable column at all, it's a property of how table.inspect.partitions() groups files. The old files simply never computed that grouping, because the partition field didn't exist when they were written; the real business data, order_ts, was never at risk.

Exercises

Exercise 1 — Reproduce both specs' coexistence yourself. With lesson 6's evolved table available, run this lesson's three steps. Confirm 10,000,040 total rows, 9,575,038.30 for S01, and both spec_ids (0 and 1) present in table.inspect.partitions().

See solution

If your table arrived at this lesson in exactly lessons 5 and 6's state, your output should match number for number. Pay special attention to spec_id present: [0, 1] — if you only see [1], your table probably had no data prior to the evolution (check that lesson 5 loaded the ten million rows before evolving the spec in lesson 6); if you only see [0], this lesson's new franchise append() probably didn't run.

Exercise 2 — Add a second new franchise, and confirm it also lands under spec_id=1. Repeat this lesson's step 1 with NEW_FRANCHISE_ID = 250_001, and confirm with table.inspect.partitions() that both new franchises' rows (250,000 and 250,001) show up under the same spec_id=1 you already saw — the spec doesn't evolve again just because more data arrived.

See solution 
NEW_FRANCHISE_ID_2 = 250_001
more_rows = list(generate_orders_at_scale(1))
for r in more_rows:
    r["franchise_id"] = NEW_FRANCHISE_ID_2
    r["order_id"] = r["order_id"].replace("F000000", f"F{NEW_FRANCHISE_ID_2:06d}")
    r["order_ts"] = datetime.fromisoformat(r["order_ts"])
more_pa_table = pa.Table.from_pylist(more_rows, schema=PA_SCHEMA)
table.append(more_pa_table)

partitions_v2 = table.inspect.partitions().to_pylist()
spec_ids_v2 = sorted({p["spec_id"] for p in partitions_v2})
print("spec_id present after the second franchise:", spec_ids_v2)
assert spec_ids_v2 == [0, 1]

The result is still [0, 1] — no new spec got created just from writing more data. A new spec_id only shows up when someone explicitly evolves the PartitionSpec, with update_spec(), exactly like lesson 6 did. Writing data, no matter how many new franchises, always uses whatever spec is current at that moment — it never creates a new one on its own.

Exercise 3 — Explain, in 2-3 sentences, why this table never needed a "migration" or "backfill" operation for order_day on the old rows. Contrast this with what you'd probably expect in a traditional relational data warehouse, where adding a computed column to an existing table usually does require an explicit backfill step.

See solution

In a traditional relational warehouse, adding a column with derived data (like order_day from order_ts) typically means a massive UPDATE or a backfill job that walks every existing row and computes the new value — an expensive operation, that can take hours on a large table, and that blocks or competes for resources with normal traffic. No backfill was needed here because order_day is never a column of the table in the traditional sense: it's a partition value, computed only to decide which file to write a row into at the moment it's written. The old rows never needed that retroactive calculation because, for business reading purposes (order_ts is still there, complete), they never lacked anything — the only thing "missing" is a file grouping that, for those rows, simply never happened, and there's no need for it to happen.

Summary and next step

In this lesson you added a new franchise to kiosko.fact_orders_at_scale, after having evolved its PartitionSpec in lesson 6, and confirmed with table.inspect.partitions() that the ten million original rows (spec_id=0, store_id only) and the forty new rows (spec_id=1, store_id + order_day) coexist, with no conflict, within the same table. You also verified lesson 3's hidden query still works exactly the same — row_filter="store_id == 'S01'" — with no need for the code to find out the spec changed halfway through.

Before moving on you should be able to: read a table.inspect.partitions() output with multiple spec_ids, and explain what each one means; and explain, unambiguously, why the old rows never needed a backfill.

You have all seven pieces complete: Spark's visible cabinet, the hidden query, the three transforms, the partitioned table at scale, evolution without rewriting, and both schemes coexisting. Lesson 8 integrates them into a single end-to-end project, with automated asserts on each one.

Resources

  • PyIceberg — API reference, table.inspect.partitions(), its result's exact structure (spec_id, partition, record_count). py.iceberg.apache.org/api. In English.
  • Apache Iceberg — official documentation, "Partitioning," "Partition Evolution" section (the formal guarantee that data written under different specs coexists with no conflict in the same table). iceberg.apache.org/docs/latest/partitioning. In English.
  • This guide's DESIGN doc — module 5's section, with the exact expected verification (table.inspect.partitions() showing both partition schemes coexisting). src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.
← PreviousEvolving the `PartitionSpec` without rewritingNext →Project: Kiosko's `fact_orders_at_scale`, partitioned