How Spark partitioned Kiosko by folder

Description

Before contrasting anything, this lesson builds the real starting point: the folder layout that partitionBy("store_id") produces, the same move spark-and-distributed-processing-guide (module 7 of that guide) already ran over kiosko_orders_at_scale.parquet, Kiosko's ten million rows at scale. This guide doesn't reinstall Spark to reproduce it — this guide doesn't touch Spark until module 6, stated that explicitly — so you're going to rebuild the same kind of layout with pyarrow, the library you already know from module 1, and see it on disk with your own eyes.

Connection to the module. Lesson 1 promised a contrast; this lesson builds the "before" half of that contrast, with real evidence on disk, not an abstract description. Lesson 3 is going to take exactly this same business question — "give me S01's data" — and answer it against Iceberg without any of the physical knowledge this lesson is going to require.

An analogy: the filing cabinet with hand-labeled drawers

Think of an office filing cabinet, one of those with physical drawers. Someone decided, at some point, to organize the documents by client: one drawer per client, with a hand-written label taped to the front. The system works well — as long as the person looking for a document knows the organization is "by client" and knows how to read the exact label. If someone new shows up at the office, with nobody explaining the system to them, they can open drawer after drawer until they find what they're looking for — it works, but it's slow and depends on guessing — or they can ask someone who already knows the system. What the cabinet doesn't do is tell you, on its own, "the document you're looking for is in the third drawer" — that intelligence lives in the head of whoever organized the drawers, not in the furniture.

That is, precisely, what this lesson builds: a real filing cabinet, on disk, with one drawer per store, and you're going to find out firsthand what happens when someone tries to use it without knowing the label convention.

Worked example: rebuilding Spark's layout with pyarrow

Step 1 — Generate a slice of Kiosko at scale, and write it partitioned by store_id

generate_orders_at_scale(), spark-and-distributed-processing-guide's deterministic generator (module 4 of that guide), rebuilt here identically so this lesson is self-contained — no random, no datetime.now(), Kiosko's same real week multiplied per franchise:

# kiosko_scale.py -- identical to spark-and-distributed-processing-guide M4L4
from typing import Iterator, Dict, Any

KIOSKO_WEEK = [
    {"order_id": "ORD-1001", "store_id": "S01", "product_id": "P001", "quantity": 3, "unit_price": 0.55, "order_ts": "2026-08-03T08:14:00"},
    # ... the full 40 rows of Kiosko's real week (August 3-9, 2026)
]


def generate_orders_at_scale(num_franchises: int) -> Iterator[Dict[str, Any]]:
    """Repeats KIOSKO_WEEK once per synthetic franchise. No random, no
    datetime.now(): franchise_id walks range(num_franchises) in fixed order."""
    for franchise_id in range(num_franchises):
        for row in KIOSKO_WEEK:
            yield {
                "order_id": f"F{franchise_id:06d}-{row['order_id']}",
                "franchise_id": franchise_id,
                "store_id": row["store_id"],
                "product_id": row["product_id"],
                "quantity": row["quantity"],
                "unit_price": row["unit_price"],
                "order_ts": row["order_ts"],
            }

# l2_hive_style_write.py
import os
from datetime import datetime

import pyarrow as pa
import pyarrow.dataset as ds

from kiosko_scale import generate_orders_at_scale

OUT_DIR = "kiosko_orders_at_scale_hive"

NUM_FRANCHISES = 25  # small demo -- 1,000 rows, enough to see the folder
rows = list(generate_orders_at_scale(NUM_FRANCHISES))
for r in rows:
    r["order_ts"] = datetime.fromisoformat(r["order_ts"])

pa_schema = pa.schema([
    pa.field("order_id", pa.string(), nullable=False),
    pa.field("franchise_id", pa.int32(), nullable=False),
    pa.field("store_id", pa.string(), nullable=False),
    pa.field("product_id", pa.string(), nullable=False),
    pa.field("quantity", pa.int32(), nullable=False),
    pa.field("unit_price", pa.float64(), nullable=False),
    pa.field("order_ts", pa.timestamp("us"), nullable=False),
])
pa_table = pa.Table.from_pylist(rows, schema=pa_schema)

# the same move as fact_orders_at_scale.write.partitionBy("store_id").parquet(...)
# from spark-and-distributed-processing-guide: whoever WRITES decides,
# explicitly, that "store_id" becomes folder structure
ds.write_dataset(
    pa_table, OUT_DIR, format="parquet",
    partitioning=ds.partitioning(pa.schema([("store_id", pa.string())]), flavor="hive"),
)
print(f"rows written: {pa_table.num_rows}")

What to expect (verified by running the real script, with a representative slice of 25 franchises — 1,000 rows — instead of the full 250,000, so this cabinet stays small enough to explore by hand; lesson 5's real table does load the full 10,000,000):

rows written: 1000

Step 2 — Look at the cabinet on disk

for root, dirs, files in os.walk(OUT_DIR):
    depth = root.replace(OUT_DIR, "").count(os.sep)
    indent = "  " * depth
    print(f"{indent}{os.path.basename(root) or OUT_DIR}/")
    for f in sorted(files):
        print(f"{indent}  {f}")

What to expect:

kiosko_orders_at_scale_hive/
  store_id=S01/
    part-0.parquet
  store_id=S03/
    part-0.parquet
  store_id=S02/
    part-0.parquet

There's the cabinet: three drawers, each with its label hand-written right into the folder name — store_id=S01, store_id=S02, store_id=S03. This is exactly what spark-and-distributed-processing-guide's kiosko_orders_at_scale.parquet already has on disk, at full scale: the same column=value/ pattern, the convention the Hadoop ecosystem has been using for over a decade.

Step 3 — Find out, firsthand, what happens if someone doesn't know the convention

import pyarrow.dataset as ds

print("--- 'Naive' read: just pyarrow, without telling it anything about partitioning ---")
plain = ds.dataset(OUT_DIR, format="parquet")  # no partitioning=... -> doesn't recognize store_id
plain_table = plain.to_table()
print("columns seen without declaring the partition scheme:", plain_table.schema.names)

print("\n--- 'Layout-aware' read: you have to DECLARE the hive scheme ---")
aware = ds.dataset(OUT_DIR, format="parquet", partitioning="hive")
s01_only = aware.to_table(filter=(ds.field("store_id") == "S01"))
print("S01 rows (declaring partitioning='hive'):", s01_only.num_rows)

What to expect:

--- 'Naive' read: just pyarrow, without telling it anything about partitioning ---
columns seen without declaring the partition scheme: ['order_id', 'franchise_id', 'product_id', 'quantity', 'unit_price', 'order_ts']

--- 'Layout-aware' read: you have to DECLARE the hive scheme ---
S01 rows (declaring partitioning='hive'): 400

Notice something that isn't a minor detail: in the "naive" read, store_id simply isn't there among the columns. It's not that the column is empty or incomplete — it disappeared, because its real value lives encoded in the folder name, not inside any Parquet file. A reader who doesn't know, in advance, that it has to declare partitioning="hive" can't even filter by store_id — the column, for that reader, simply doesn't exist. To recover it, someone has to explicitly tell the reader what the convention is (partitioning="hive"), and that "explicitly telling it" is, precisely, the physical knowledge lesson 1 of this module promised to eliminate.

Diagram: where the knowledge lives

flowchart LR
    W["Writer:\n.write.partitionBy('store_id')\ndecides the convention"] --> D["Disk:\nstore_id=S01/\nstore_id=S02/\nstore_id=S03/"]
    D --> R1["Reader WITHOUT knowing\nthe convention:\nstore_id doesn't even show up"]
    D --> R2["Reader THAT declares\npartitioning='hive':\nrecovers store_id, can filter"]
    R2 -.->|"the knowledge lives\nIN EACH READER,\nnot in the table"| W

Going deeper: this isn't a pyarrow flaw

It's worth being precise about something this lesson's example could make look like a library problem: pyarrow.dataset does know how to read Hive layouts — step 3 proves it, with partitioning="hive". The point isn't that there's no way to read it correctly; the point is that doing it correctly requires an explicit declaration, made by every reader, every time. If someone reorganizes the layout tomorrow — adds a second partition column, swaps store_id for region — each one of those readers has to update its own declaration, in its own code, on its own. There's no single place where that convention lives, versioned, queryable — it lives duplicated, as many times as there are readers. This is exactly the responsibility lesson 3 is going to show transferred to a single place: the table itself.

Common mistakes

Assuming the folder name (store_id=S01) is purely cosmetic, and that the real value is still in some column of the Parquet. What happens: someone opens one of the part-0.parquet files directly with pq.read_table(), bypassing pyarrow.dataset, and is surprised not to find store_id among the columns. Why it happens: it's intuitive to think a column used for partitioning still exists, "just in case," inside every file — but pyarrow.dataset's hive flavor removes it from the physical file, precisely because it's already encoded in the path. How to spot it: if pq.read_table("kiosko_orders_at_scale_hive/store_id=S01/part-0.parquet").schema.names doesn't include store_id, that isn't a bug — it's the expected behavior of a Hive-partitioned dataset. How to fix it: to recover store_id as a column, always read through pyarrow.dataset with partitioning="hive" (or the equivalent in whatever tool you use) — never by opening the individual loose files.

Thinking that partitioning by folders is "free" in terms of maintenance. What happens: someone assumes that, once the Hive layout is written, there's nothing more to think about — any future query "just works." Why it happens: as long as nobody changes the partition scheme, the system effectively looks invisible and costless. How to spot it: ask yourself what would happen if, six months from now, Kiosko decides it also needs to partition by month in addition to by store — how many existing readers would have to update their code? How to fix it: any change to a Hive layout's partition scheme requires, in the general case, rewriting all the data with the new folder structure, and updating every reader that depended on the old one. This module's lesson 6 shows the exact contrast: Iceberg evolves its PartitionSpec with neither of those two consequences.

Exercises

Exercise 1 — Reproduce the cabinet yourself, and open it with and without partitioning="hive". Run this lesson's three steps on your own machine. Confirm store_id disappears from the naive read, and that it reappears — with 400 rows for S01 — when you declare partitioning="hive".

See solution

If you followed the three steps, your output should match this lesson's exactly: rows written: 1000, a tree of three folders store_id=S0N/, columns with no store_id in the naive read, and 400 rows of S01 in the layout-aware read (25 franchises × 16 S01 lines per franchise = 400, the same proportional pattern you already know from module 1).

Exercise 2 — Calculate how many files would exist with 250,000 franchises, if the layout didn't change. With this lesson's 25-franchise slice, each drawer (store_id=SNN/) has exactly one file (part-0.parquet). If you wrote the full 250,000 franchises with this lesson's same ds.write_dataset(), in a single batch, how many files per drawer would you expect, at minimum?

See solution

At minimum, one per drawer — three total — exactly as in this lesson: ds.write_dataset(), when it receives a single pyarrow table in memory (with no explicit write partitioning across multiple batches), writes one file for each distinct value of the partition column within that call. The real number can be higher in a distributed system like Spark, where each in-memory partition (module 4's topic in spark-and-distributed-processing-guide) that holds rows for a given store_id can generate its own file — that's why a table partitioned by store_id at real scale almost never has exactly three files, but several per store. This module's lesson 5, on the full Iceberg table, is going to show the real number a single table.append() call produces.

Exercise 3 — Explain, in 2-3 sentences, why step 3's "naive" read doesn't throw an error. Instead of failing with a clear message like "partitioning declaration missing," pyarrow.dataset simply omits store_id silently. Why do you think that's a more dangerous behavior than an explicit error?

See solution

An explicit error stops you immediately and forces you to fix the problem before moving on. store_id's silent omission is more dangerous precisely because it doesn't stop you: the script keeps running, produces a result — except that result no longer has the column you might have needed for filtering or grouping later — and the error only shows up later, as a confusing KeyError or an incomplete business result, far from where the problem actually originated. This is the same class of silent risk you already saw in module 1 about why a loose Parquet file can't warn you if something went wrong in its own write.

Summary and next step

In this lesson you built, with pyarrow, the same kind of layout spark-and-distributed-processing-guide already left on disk at full scale: a cabinet with one physical drawer per store, store_id=S01/, store_id=S02/, store_id=S03/. You confirmed, with direct evidence — not a claim — that a reader who doesn't know that convention loses the store_id column entirely, and that recovering it requires explicitly declaring the layout at every read point.

Before moving on you should be able to: describe, in your own words, what information lives in a Hive folder's name and what information lives inside the Parquet file; and explain why that knowledge scattered across readers is, precisely, the cost this module is going to eliminate.

Lesson 3 makes the real contrast: the same question — "give me S01's data" — answered against an Iceberg table, with the code never mentioning a single folder.

Resources

• Apache Arrow — official documentation, pyarrow.dataset.partitioning() and the hive flavor (the exact API this lesson uses to write and read the folder layout). arrow.apache.org/docs/python/dataset.html. In English.
• Apache Iceberg — official documentation, "Partitioning," "Partitioning in Hive" section (the formal description of the pattern this lesson rebuilds). iceberg.apache.org/docs/latest/partitioning. In English.
• spark-and-distributed-processing-guide DESIGN doc — source of fact_orders_at_scale.write.partitionBy("store_id").parquet(...) and of kiosko_orders_at_scale.parquet, the real full-scale layout this lesson rebuilds at smaller scale. src/guides/spark-and-distributed-processing-guide/DISENO.md. In Spanish.
• This guide's DESIGN doc — module 5's section, with the explicit contrast between visible folder and hidden partitioning. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.