Module overview: hidden partitioning and partition evolution

Why this module exists

This guide's module 1, lesson 2, named, one by one, four times Kiosko already ran into the ceiling of Parquet as a plain file. The fourth of those four was spark-and-distributed-processing-guide (module 7 of that guide): fact_orders_at_scale.write.partitionBy("store_id").parquet(...), ten million rows written into folders — store_id=S01/, store_id=S02/, store_id=S03/ — the Hive pattern any big-data engine recognizes. That pattern works: filtering by store_id avoids reading the other two stores' folders. But the price of that performance is explicit, and this guide deliberately left it pending since module 2, in its lesson 3: "kiosko.fact_orders isn't partitioned [...] This guide's module 5 introduces the first real partition scheme." That moment has arrived.

The price of Spark's partitioning is this: to take advantage of it, you have to know it exists. Someone who writes WHERE store_id = 'S01' without knowing the table is partitioned by store_id gets the correct result, but not necessarily the cheapest execution plan — and if the folder convention changes (a second partition column gets added, a name changes), any external tool that depended on that physical structure stops working with no warning at all in the business code. The folder structure is the contract, and that contract lives outside any catalog, in the memory of whoever wrote the pipeline.

This module solves exactly that problem, with the same mechanism you already saw solve similar problems in modules 1 through 4: moving the responsibility from the engineer to the engine. kiosko.fact_orders_at_scale, the table you're going to build here, is going to have a real partition — as real as Spark's, with the same file-pruning benefits — but the query that takes advantage of it is never going to mention a folder. And you're going to go one step further than Spark allows without friction: you're going to evolve the partition scheme after the data already exists, without rewriting a single row of the ten million you already loaded.

The case that runs through the module: kiosko.fact_orders_at_scale, really partitioned

This module inherits, without regenerating the argument for why it exists, the synthetic at-scale dataset that spark-and-distributed-processing-guide already justified in its module 4: real Kiosko — three stores, forty orders in a week — never produces a volume that justifies thinking about partitioning. generate_orders_at_scale(250_000) multiplies Kiosko's real week once for each of 250,000 simulated "franchises," with no random, without running the calendar forward: exactly ten million rows, total revenue 26,537,500.00, with the same proportional breakdown by store as always: S01=9,575,000.00, S02=9,700,000.00, S03=7,262,500.00.

What this module adds, for the first time in this guide, is an Iceberg table with a real PartitionSpec at creation time: IdentityTransform over store_id, the same criterion Spark already used for its folders. You're going to load the ten million rows, confirm that a query filtered by store_id == 'S01' — no folder, no path, nothing physical in the code — still gives exactly 9,575,000.00, and evolve that PartitionSpec to add a second partition dimension (order_day, truncating order_ts to its date) without touching a single file of the ones that already exist.

An analogy: the mail carrier who doesn't need you to tell him the mailbox

Imagine an apartment building with a new mail carrier. The old way of working — Spark's way — is like a building where every resident has to memorize which floor and which wing they live on, and tell the carrier every time: "I'm on floor 3, north wing." If the carrier is new, or if the building reorganizes its floors, anyone who doesn't know the current structure is left without mail, or the carrier has to check the whole building, floor by floor.

Iceberg's hidden partitioning is a different carrier: one who already knows, on his own, which bag each resident's mail ended up in, with nobody having to tell him the floor number. You ask him for "Ana's mail from August" and he handles the rest — he checks his own internal index, goes straight to the right bag, and hands you exactly what you asked for, without you ever having had to know which bag it was in. And if one day the carrier decides to reorganize his bags — adding, say, a sub-bag per day in addition to the sub-bag per resident — the old mail stays exactly where it was: only the new mail starts showing up in the new sub-bag too. Nobody has to relabel the old mail. That reorganization, without touching what already exists, is exactly what this module calls partition evolution.

Diagram: from the visible folder to the hidden bag

flowchart TB
    subgraph SPARK["Spark (inherited, spark-and-distributed-processing-guide M7)"]
        A["fact_orders_at_scale.write\n.partitionBy('store_id')\n.parquet(...)"]
        A --> B["kiosko_orders_at_scale.parquet/\nstore_id=S01/ store_id=S02/ store_id=S03/"]
        B --> C["Whoever reads HAS TO know\nthat 'store_id' organizes the folders"]
    end

    subgraph ICEBERG["This module: kiosko.fact_orders_at_scale"]
        D["Initial PartitionSpec\nIdentityTransform(store_id)"]
        D --> E["table.append(10M rows)\nthe engine decides the layout"]
        E --> F["table.scan(row_filter=\n\"store_id == 'S01'\")\nnever mentions a folder"]
        F --> G["Lesson 6: update_spec()\nadds DayTransform(order_ts)"]
        G --> H["Lesson 7: inspect.partitions()\nold and new files coexist"]
    end

    C -.->|"same store_id,\nsame result,\ndifferent responsibility"| D

The map of this module

Lesson    What it solves
────────  ──────────────────────────────────────────────────────────────
L1        (this one) From Spark's visible folder to Iceberg's hidden bag
L2        How Spark partitioned Kiosko by folders -- the starting point,
          with a real demonstration of Hive layout on disk
L3        The same query, without knowing the layout -- the central contrast
L4        The three partition transforms: IdentityTransform, BucketTransform,
          DayTransform, and when to use each one
L5        kiosko.fact_orders_at_scale created, loaded with the real 10M rows,
          hidden query for S01 = 9,575,000.00, with pruning evidence
L6        update_spec().add_field(DayTransform) -- evolution without rewriting
L7        inspect.partitions() -- the two partition schemes, coexisting
L8        Project: fact_orders_at_scale partitioned, hidden, evolved,
          end to end

Lessons 2 and 3 build the contrast before touching any at-scale table: lesson 2 shows, on disk, what Spark already built; lesson 3 shows that the same business question gets answered without that physical knowledge. Lesson 4 gives precise vocabulary to the three transforms you're going to need. Lessons 5 through 7 are the real execution, at full scale: create, load, query, evolve, and confirm both partition schemes coexist without conflict. Lesson 8 integrates the seven pieces into a single script.

The boundary: what does NOT belong in this module

Why ten million synthetic rows exist, and why it's correct to keep using them to measure something real, was already justified in depth in spark-and-distributed-processing-guide (module 4 of that guide) — this module inherits that justification, it doesn't repeat it or regenerate it conceptually; it does rebuild the data, with the same deterministic generator, because it needs to really load it into an Iceberg table. Distributed computing in depth — shuffle, in-memory partitions, Catalyst — remains spark-and-distributed-processing-guide's territory; this module never runs Spark, only pure PyIceberg. MERGE INTO and native upserts, this guide's fourth operational guarantee, arrive only in module 6, with Spark as a SQL client — not before. And managed partition catalogs in the cloud (partitions over AWS Glue Catalog or S3 Tables with thousands of real files) get named, without being implemented, in module 7.

Common mistakes

Thinking "hidden partitioning" means Iceberg doesn't partition anything, or that "everything is slower without folders." What happens: someone, on hearing "hidden," assumes Iceberg gives up on the physical organization of the data, and that all file pruning is done "by hand," checking each Parquet one by one. Why it happens: the word "hidden" sounds, at first hearing, like a synonym for "absent." How to spot it: if you expect table.inspect.files() on kiosko.fact_orders_at_scale (lesson 5) to show a single giant file with all the rows mixed together, revisit this lesson again. How to fix it: Iceberg does organize the data physically — you're going to see, in lesson 2, that it even writes folders with the same store_id=S01/ pattern as Spark — what changes is who needs to know that structure to take advantage of it. With Spark, the reader. With Iceberg, only the catalog itself.

Confusing "partitioning by store_id" with "ordering by store_id." What happens: someone expects that, within a single Parquet file, S01's rows appear grouped and ordered before S02's, as if partitioning were the same thing as an ORDER BY. Why it happens: both concepts organize data by a column's value, so it's easy to mix them up if they were never explicitly distinguished. How to spot it: if your mental model of partitioning includes the word "order," revisit this module's lesson 4 — a PartitionSpec decides which file each row falls into, never what position within that file. How to fix it: partitioning is a file layout decision (how many files, and which rows go in each one); the internal order of rows within a file is a separate topic (sort order), which this guide doesn't build — Iceberg supports it, but it's outside this module's declared scope.

Exercises

Exercise 1 — Name, from memory, the fourth time plain Parquet wasn't enough. Without looking back, remember: which previous guide in the ecosystem ran into the problem this module solves, and with what exact code?

See solution

spark-and-distributed-processing-guide, in its module 7: fact_orders_at_scale.write.partitionBy("store_id").parquet(...) — ten million rows written into Hive folders (store_id=S01/, store_id=S02/, store_id=S03/). The problem isn't that this partitioning doesn't work — it works, and the Spark guide itself even shows PushedFilters/partition pruning in its .explain() — the problem is that taking advantage of it requires whoever queries it to know, in advance, that this folder structure exists and which column organizes it.

Exercise 2 — Trace the analogy yourself. In your own words, using this lesson's mail carrier analogy, explain what an Iceberg table's PartitionSpec represents.

See solution

The PartitionSpec is the internal index the carrier keeps on his own: the rule that says, for each new row, which bag it must go into, with neither the sender (whoever writes) nor the recipient (whoever queries) having to know that rule by heart. When someone writes table.append(), the engine consults that index and decides the layout; when someone queries with row_filter="store_id == 'S01'", the engine consults that same index again to know which bags it can ignore entirely. The physical "bag" is still a Parquet file on disk — lesson 2 shows it to you firsthand, with Spark's layout — what changes is that, on Iceberg's side, that index lives in the table itself, not in the head of whoever uses it.

Exercise 3 — Prediction. Before reading lesson 6: if kiosko.fact_orders_at_scale already has ten million rows loaded under a PartitionSpec that only uses store_id, and you then add a second partition field (order_day, truncating order_ts), what do you expect happens to the ten million rows that already exist? Are they going to "receive" the new partition field, or are they going to stay as they were?

See solution

There's no single way to phrase it, but the correct answer — which lesson 6 confirms with executed evidence — is that the ten million existing rows stay exactly as they were: their data files aren't touched, and they keep "living" under the original PartitionSpec (only store_id). Only rows written after the evolution are going to be organized according to the new scheme, with both fields. It's the same logic you already saw in module 4 with schema evolution: adding something new never forces "updating" what already existed.

Summary and next step

In this lesson you saw the exact point where module 4 left you — a pending promise since module 1, about Spark's folder-based partitioning — and the full map of how this module solves it: hidden partitioning first, partition evolution after, both on kiosko.fact_orders_at_scale, the ten-million-row table this guide inherits, without regenerating, from spark-and-distributed-processing-guide.

Before moving on you should be able to: explain, in your own words, what cost Spark's folder-based partitioning has that Iceberg's hidden partitioning eliminates; and name this module's two central experiments (hidden query, evolution without rewriting) that you're going to really run in lessons 5 through 7.

Lesson 2 begins where module 4 ends: showing you, on disk, exactly what Spark already built — the real starting point for this module's whole contrast.

Resources

• Apache Iceberg — official documentation, "Partitioning" (hidden partitioning, transforms, partition evolution without rewriting data) — the formal foundation for this whole module. iceberg.apache.org/docs/latest/partitioning. In English.
• PyIceberg — API reference, PartitionSpec/PartitionField, the transforms (IdentityTransform/BucketTransform/DayTransform), table.update_spec(), table.inspect.partitions(). py.iceberg.apache.org/api. In English.
• spark-and-distributed-processing-guide DESIGN doc — source of fact_orders_at_scale.write.partitionBy("store_id").parquet(...), generate_orders_at_scale(), and the exact numbers for the at-scale dataset. src/guides/spark-and-distributed-processing-guide/DISENO.md. In Spanish.
• This guide's DESIGN doc — the full map of the eight modules, including module 5's section. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.