Module 4: Partitions And The Cost Of Shuffle

Generating `kiosko_orders_at_scale`, deterministically

Description

Everything you saw in lessons 2 and 3 was real, but invisible in cost: forty rows demand nothing of Spark, no matter how many times they get reorganized. This lesson builds this entire guide's first genuinely new piece of data — not new business data, but data that's synthetic, deterministic, and declared as such — so that partitioning and shuffle stop being an abstract idea and become something measurable: kiosko_orders_at_scale, ten million rows, built by replicating Kiosko's same forty-order week once for each of 250,000 simulated "franchises."

⚠️ Explicit declaration, so it doesn't get lost for the rest of this module. Kiosko, at its real scale — three stores, forty orders in a week — never produces a volume that justifies Spark. Nobody needs a distributed engine for forty rows. This dataset represents no real business projection for Kiosko: it's a deliberate construction, with the sole purpose of letting you genuinely feel shuffle and partitioning on a laptop. Every lesson in this module that uses it is going to declare this, unambiguously.

Connection to the module. Lessons 2 and 3 built the mental model — what a partition is, what triggers a shuffle — over data where the cost is invisible. This lesson is the module's hinge: from here on, lessons 5 through 8 measure that same mental model with numbers that actually matter.

An analogy: the same recipe, multiplied for a banquet

Imagine you have the exact recipe for a dish that serves four people — the same ingredients, the same quantities, the same order of steps, every time. Now imagine you need that same dish for a banquet of two hundred fifty thousand people: you don't invent a new recipe or improvise random quantities — you multiply the original recipe, exactly, two hundred fifty thousand times, and put a label on each portion so you know which table it belongs to. Each portion's taste is identical to the original recipe's; the only thing that changes is how many times you repeated it, and a label that wasn't needed before. That's exactly what generate_orders_at_scale() does: it takes Kiosko's real week — forty orders, not a single value changed — and repeats it, mechanically and verifiably, once per synthetic franchise, adding a single new column (franchise_id) that didn't exist before.

Worked example, part 1: the generator function

# kiosko_scale.py
from typing import Iterator, Dict, Any

# The same real Kiosko week from modules 1-3: 40 orders, 3 stores, 4 products,
# seven real calendar dates (2026-08-03 to 2026-08-09). Not a single value changes here.
KIOSKO_WEEK = [
    {"order_id": "ORD-1001", "store_id": "S01", "product_id": "P001", "quantity": 3, "unit_price": 0.55, "order_ts": "2026-08-03T08:14:00"},
    {"order_id": "ORD-1002", "store_id": "S01", "product_id": "P002", "quantity": 1, "unit_price": 1.20, "order_ts": "2026-08-03T08:20:00"},
    {"order_id": "ORD-1003", "store_id": "S02", "product_id": "P003", "quantity": 2, "unit_price": 0.75, "order_ts": "2026-08-03T08:31:00"},
    {"order_id": "ORD-1004", "store_id": "S01", "product_id": "P004", "quantity": 1, "unit_price": 4.50, "order_ts": "2026-08-03T09:02:00"},
    {"order_id": "ORD-1005", "store_id": "S03", "product_id": "P001", "quantity": 5, "unit_price": 0.55, "order_ts": "2026-08-03T09:15:00"},
    {"order_id": "ORD-1006", "store_id": "S02", "product_id": "P002", "quantity": 2, "unit_price": 1.20, "order_ts": "2026-08-03T09:47:00"},
    {"order_id": "ORD-1007", "store_id": "S03", "product_id": "P003", "quantity": 1, "unit_price": 0.75, "order_ts": "2026-08-03T10:05:00"},
    {"order_id": "ORD-1008", "store_id": "S01", "product_id": "P001", "quantity": 2, "unit_price": 0.55, "order_ts": "2026-08-03T10:22:00"},
    {"order_id": "ORD-2001", "store_id": "S01", "product_id": "P002", "quantity": 1, "unit_price": 1.20, "order_ts": "2026-08-04T08:05:00"},
    {"order_id": "ORD-2002", "store_id": "S02", "product_id": "P001", "quantity": 4, "unit_price": 0.55, "order_ts": "2026-08-04T08:40:00"},
    {"order_id": "ORD-2003", "store_id": "S03", "product_id": "P004", "quantity": 1, "unit_price": 4.50, "order_ts": "2026-08-04T09:12:00"},
    {"order_id": "ORD-2004", "store_id": "S01", "product_id": "P003", "quantity": 3, "unit_price": 0.75, "order_ts": "2026-08-04T09:50:00"},
    {"order_id": "ORD-2005", "store_id": "S02", "product_id": "P002", "quantity": 2, "unit_price": 1.20, "order_ts": "2026-08-04T10:15:00"},
    {"order_id": "ORD-2006", "store_id": "S03", "product_id": "P001", "quantity": 6, "unit_price": 0.55, "order_ts": "2026-08-04T10:33:00"},
    {"order_id": "ORD-3001", "store_id": "S02", "product_id": "P004", "quantity": 2, "unit_price": 4.50, "order_ts": "2026-08-05T08:10:00"},
    {"order_id": "ORD-3002", "store_id": "S01", "product_id": "P001", "quantity": 1, "unit_price": 0.55, "order_ts": "2026-08-05T08:22:00"},
    {"order_id": "ORD-4001", "store_id": "S01", "product_id": "P001", "quantity": 4, "unit_price": 0.55, "order_ts": "2026-08-06T08:10:00"},
    {"order_id": "ORD-4002", "store_id": "S02", "product_id": "P003", "quantity": 2, "unit_price": 0.75, "order_ts": "2026-08-06T08:45:00"},
    {"order_id": "ORD-4003", "store_id": "S03", "product_id": "P002", "quantity": 1, "unit_price": 1.20, "order_ts": "2026-08-06T09:20:00"},
    {"order_id": "ORD-4004", "store_id": "S01", "product_id": "P004", "quantity": 1, "unit_price": 4.50, "order_ts": "2026-08-06T09:55:00"},
    {"order_id": "ORD-4005", "store_id": "S02", "product_id": "P001", "quantity": 3, "unit_price": 0.55, "order_ts": "2026-08-06T10:30:00"},
    {"order_id": "ORD-5001", "store_id": "S01", "product_id": "P002", "quantity": 2, "unit_price": 1.20, "order_ts": "2026-08-07T08:05:00"},
    {"order_id": "ORD-5002", "store_id": "S03", "product_id": "P001", "quantity": 4, "unit_price": 0.55, "order_ts": "2026-08-07T08:30:00"},
    {"order_id": "ORD-5003", "store_id": "S02", "product_id": "P004", "quantity": 1, "unit_price": 4.50, "order_ts": "2026-08-07T08:58:00"},
    {"order_id": "ORD-5004", "store_id": "S01", "product_id": "P003", "quantity": 2, "unit_price": 0.75, "order_ts": "2026-08-07T09:22:00"},
    {"order_id": "ORD-5005", "store_id": "S03", "product_id": "P002", "quantity": 3, "unit_price": 1.20, "order_ts": "2026-08-07T09:47:00"},
    {"order_id": "ORD-5006", "store_id": "S02", "product_id": "P001", "quantity": 5, "unit_price": 0.55, "order_ts": "2026-08-07T10:15:00"},
    {"order_id": "ORD-5007", "store_id": "S01", "product_id": "P001", "quantity": 2, "unit_price": 0.55, "order_ts": "2026-08-07T10:40:00"},
    {"order_id": "ORD-6001", "store_id": "S01", "product_id": "P001", "quantity": 6, "unit_price": 0.55, "order_ts": "2026-08-08T08:00:00"},
    {"order_id": "ORD-6002", "store_id": "S02", "product_id": "P002", "quantity": 3, "unit_price": 1.20, "order_ts": "2026-08-08T08:18:00"},
    {"order_id": "ORD-6003", "store_id": "S03", "product_id": "P001", "quantity": 4, "unit_price": 0.55, "order_ts": "2026-08-08T08:35:00"},
    {"order_id": "ORD-6004", "store_id": "S01", "product_id": "P004", "quantity": 2, "unit_price": 4.50, "order_ts": "2026-08-08T08:52:00"},
    {"order_id": "ORD-6005", "store_id": "S02", "product_id": "P003", "quantity": 3, "unit_price": 0.75, "order_ts": "2026-08-08T09:10:00"},
    {"order_id": "ORD-6006", "store_id": "S03", "product_id": "P002", "quantity": 2, "unit_price": 1.20, "order_ts": "2026-08-08T09:28:00"},
    {"order_id": "ORD-6007", "store_id": "S01", "product_id": "P003", "quantity": 1, "unit_price": 0.75, "order_ts": "2026-08-08T09:45:00"},
    {"order_id": "ORD-6008", "store_id": "S02", "product_id": "P001", "quantity": 7, "unit_price": 0.55, "order_ts": "2026-08-08T10:02:00"},
    {"order_id": "ORD-6009", "store_id": "S03", "product_id": "P004", "quantity": 1, "unit_price": 4.50, "order_ts": "2026-08-08T10:20:00"},
    {"order_id": "ORD-7001", "store_id": "S01", "product_id": "P001", "quantity": 2, "unit_price": 0.55, "order_ts": "2026-08-09T09:15:00"},
    {"order_id": "ORD-7002", "store_id": "S02", "product_id": "P002", "quantity": 1, "unit_price": 1.20, "order_ts": "2026-08-09T09:40:00"},
    {"order_id": "ORD-7003", "store_id": "S03", "product_id": "P001", "quantity": 3, "unit_price": 0.55, "order_ts": "2026-08-09T10:05:00"},
]


def generate_orders_at_scale(num_franchises: int) -> Iterator[Dict[str, Any]]:
    """Repeats KIOSKO_WEEK once per synthetic franchise. No random, no
    datetime.now(): franchise_id walks range(num_franchises) in fixed order,
    and the seven calendar dates never shift forward -- they get reused
    identically for every franchise."""
    for franchise_id in range(num_franchises):
        for row in KIOSKO_WEEK:
            yield {
                "order_id": f"F{franchise_id:06d}-{row['order_id']}",
                "franchise_id": franchise_id,
                "store_id": row["store_id"],
                "product_id": row["product_id"],
                "quantity": row["quantity"],
                "unit_price": row["unit_price"],
                "order_ts": row["order_ts"],
            }

Notice two deliberate design decisions. First, generate_orders_at_scale() is a generator (yield, not return of a list) — with ten million rows ahead, building the full list in memory before using it would waste memory for no reason; a generator produces one row at a time, on demand. Second, there's not a single import random or datetime.now() anywhere in the file — franchise_id walks range(num_franchises) in a fixed, predictable order, and the seven order_ts dates are literally the same ones from KIOSKO_WEEK, copied unmodified, in each of the 250,000 repetitions. Running this function twice, on two different machines, produces exactly the same data, row for row — the very definition of determinism.

Worked example, part 2: a small check, before the full scale

Before generating ten million rows, it's worth confirming the function does what it promises, with a small enough franchise count that you can review the output by hand.

# small_scale_check.py
from kiosko_scale import generate_orders_at_scale

rows = list(generate_orders_at_scale(3))
print(f"len(rows) = {len(rows)}")
assert len(rows) == 3 * 40 == 120

for r in rows[:3]:
    print(r)
print("...")
for r in rows[40:42]:
    print(r)

franchise_ids = sorted(set(r["franchise_id"] for r in rows))
print(f"franchise_ids present = {franchise_ids}")
assert franchise_ids == [0, 1, 2]
print("Verification: 3 franchises x 40 rows = 120, franchise_id in [0, 1, 2] -> OK")

What to expect. Running python3 small_scale_check.py, the output is exactly this (executed in this run):

len(rows) = 120
{'order_id': 'F000000-ORD-1001', 'franchise_id': 0, 'store_id': 'S01', 'product_id': 'P001', 'quantity': 3, 'unit_price': 0.55, 'order_ts': '2026-08-03T08:14:00'}
{'order_id': 'F000000-ORD-1002', 'franchise_id': 0, 'store_id': 'S01', 'product_id': 'P002', 'quantity': 1, 'unit_price': 1.2, 'order_ts': '2026-08-03T08:20:00'}
{'order_id': 'F000000-ORD-1003', 'franchise_id': 0, 'store_id': 'S02', 'product_id': 'P003', 'quantity': 2, 'unit_price': 0.75, 'order_ts': '2026-08-03T08:31:00'}
...
{'order_id': 'F000001-ORD-1001', 'franchise_id': 1, 'store_id': 'S01', 'product_id': 'P001', 'quantity': 3, 'unit_price': 0.55, 'order_ts': '2026-08-03T08:14:00'}
{'order_id': 'F000001-ORD-1002', 'franchise_id': 1, 'store_id': 'S01', 'product_id': 'P002', 'quantity': 1, 'unit_price': 1.2, 'order_ts': '2026-08-03T08:20:00'}
franchise_ids present = [0, 1, 2]
Verification: 3 franchises x 40 rows = 120, franchise_id in [0, 1, 2] -> OK

It confirms exactly what's expected: F000000-ORD-1001 is the first franchise's first order, F000001-ORD-1001 is the same order (same store_id, product_id, quantity, unit_price, order_ts), but from franchise number one — the order_id gets disambiguated with the F{franchise_id:06d}- prefix, so there are never two rows with the same order_id even though the rest of the columns repeat exactly.

Worked example, part 3: the full scale, with assert before trusting anything

# step1_generate.py
import csv

from kiosko_scale import generate_orders_at_scale

NUM_FRANCHISES = 250_000
OUT_PATH = "kiosko_orders_at_scale.csv"

row_count = 0
total_revenue = 0.0
by_store = {}

with open(OUT_PATH, "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["order_id", "franchise_id", "store_id", "product_id", "quantity", "unit_price", "order_ts"])
    for r in generate_orders_at_scale(NUM_FRANCHISES):
        writer.writerow([r["order_id"], r["franchise_id"], r["store_id"], r["product_id"], r["quantity"], r["unit_price"], r["order_ts"]])
        row_count += 1
        rev = r["quantity"] * r["unit_price"]
        total_revenue += rev
        by_store[r["store_id"]] = by_store.get(r["store_id"], 0.0) + rev

print(f"row_count = {row_count}")
print(f"total_revenue (rounded) = {round(total_revenue, 2)}")
print(f"by_store = { {k: round(v, 2) for k, v in sorted(by_store.items())} }")

assert row_count == NUM_FRANCHISES * 40 == 10_000_000
assert round(total_revenue, 2) == round(106.15 * NUM_FRANCHISES, 2) == 26_537_500.00
assert {k: round(v, 2) for k, v in by_store.items()} == {"S01": 9575000.0, "S02": 9700000.0, "S03": 7262500.0}
print("assert OK: 10,000,000 rows, revenue 26,537,500.00, exact breakdown by store")

What to expect. Running python3 step1_generate.py (executed in this run, generating the complete kiosko_orders_at_scale.csv file):

row_count = 10000000
total_revenue (rounded) = 26537500.0
by_store = {'S01': 9575000.0, 'S02': 9700000.0, 'S03': 7262500.0}
assert OK: 10,000,000 rows, revenue 26,537,500.00, exact breakdown by store

The three asserts confirm, with evidence — not anyone's word for it — this dataset's complete arithmetic: 250,000 franchises × 40 rows = 10,000,000 exact rows; 106.15 × 250,000 = 26,537,500.00 in total revenue, the exact same proportion as the real week; and the breakdown by store scales in the exact same proportion you already know from module 1 (S01=38.3, S02=38.8, S03=29.05 per franchise), multiplied by 250,000: S01 = 9,575,000.00, S02 = 9,700,000.00, S03 = 7,262,500.00. The resulting file on disk weighs approximately 573 MB — a single CSV file, not seven like real Kiosko's, a detail that shapes the partitions differently than lesson 5 is going to measure.

Worked example, part 4: reread with Spark

# read_at_scale.py
from pyspark.sql import SparkSession
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()

scale_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("franchise_id", IntegerType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
orders_at_scale_df = spark.read.csv(
    "kiosko_orders_at_scale.csv", schema=scale_schema, header=True, enforceSchema=False,
)

print(f"orders_at_scale_df.count() = {orders_at_scale_df.count()}")
assert orders_at_scale_df.count() == 10_000_000
print("Verification: Spark sees exactly the same 10,000,000 rows -> OK")

spark.stop()

What to expect (executed in this run):

orders_at_scale_df.count() = 10000000
Verification: Spark sees exactly the same 10,000,000 rows -> OK

Diagram: from one week to ten million rows

flowchart TD
    A["KIOSKO_WEEK\n40 fixed rows, 3 stores, 4 products,\n7 real dates"] --> B{"generate_orders_at_scale\n(250_000)"}
    B -->|"franchise_id = 0"| C0["40 rows,\norder_id with F000000- prefix"]
    B -->|"franchise_id = 1"| C1["40 rows,\norder_id with F000001- prefix"]
    B -->|"..."| C2["..."]
    B -->|"franchise_id = 249999"| C3["40 rows,\norder_id with F249999- prefix"]
    C0 --> D["kiosko_orders_at_scale.csv\n10,000,000 rows, ~573 MB"]
    C1 --> D
    C2 --> D
    C3 --> D
    D --> E["orders_at_scale_df\nspark.read.csv(...)\ncount() == 10,000,000"]

Going deeper: why this figure, and why reproducing it is the real guarantee

It's worth pausing on something easy to miss: this lesson's assert doesn't verify a made-up number — it verifies a formula anyone can recalculate without running a single line of Spark. 250,000 × 40 = 10,000,000 and 106.15 × 250,000 = 26,537,500.00 are simple arithmetic, and the fact that generate_orders_at_scale()'s real output matches that arithmetic exactly is what makes this dataset trustworthy: no matter how many times you regenerate it, on whatever machine, the result has to keep matching the same formula. This is the same cross-verification discipline you already saw in module 3 (comparing fact_orders against dict/DuckDB/Polars/SQL), applied here against a mathematical formula instead of against another engine.

It's also worth noting what does not change in this dataset relative to real Kiosko: the products are still the same four (P001 through P004, with the same unit_price), the stores are still the same three (S01, S02, S03), and the dates are still the same real calendar week (2026-08-03 through 2026-08-09) — no future dates ever get invented and the calendar never runs forward. The only genuinely new thing is franchise_id, a column that exists for a specific technical purpose: giving enough volume for the shuffle to be felt, and a partition key with 250,000 distinct values — very different from store_id, which only has three — that you're going to use in this module's and module 5's exercises to explore partitioning with finer granularity than the three stores offer.

Common mistakes

Using random "just to vary the data a little." What happens: someone, adapting generate_orders_at_scale() for their own experiment, adds import random to slightly vary quantity or unit_price between franchises, thinking it makes the dataset "more realistic." Why it happens: a dataset where every franchise is identical can feel artificial, and varying the values seems like a reasonable adjustment. How to spot it: if your version of generate_orders_at_scale() doesn't produce exactly the same 10,000,000 rows and the same 26,537,500.00 in revenue across two separate runs, something stopped being deterministic. How to fix it: this guide explicitly forbids random, precisely because this dataset's full pedagogical value — the asserts verifiable against an exact formula — depends on it being reproducible bit for bit; if you need variability for some other purpose, do it in a separate experiment, never in this guide's base dataset.

Forgetting generate_orders_at_scale() is a generator, and treating it as if it had a known length up front. What happens: someone writes len(generate_orders_at_scale(250_000)), expecting it to work as if it were a list. Why it happens: in many Python contexts, it's easy to forget the difference between a generator (which produces values on demand) and a materialized collection (which already has every value in memory). How to spot it: Python immediately throws TypeError: object of type 'generator' is not sized if you try this — an explicit error, not a silent incorrect result. How to fix it: if you need the row count before having consumed them, count them as you iterate (as step1_generate.py does, incrementing row_count inside the loop), or use the formula (num_franchises * 40) directly — never convert the whole generator into a list just to count its elements, because that forces all ten million rows to materialize in memory at once.

Confusing the kiosko_orders_at_scale.csv file with a partitioned file. What happens: someone, seeing the file weighs 573 MB and holds ten million rows, assumes it's already "partitioned" in Spark's sense, or that orders_at_scale_df's partition count is going to match some value related to franchise_id. Why it happens: the word "partition" in the context of files (partitioning a file into folders by column value, as you're going to see in module 7 with partitionBy) is different from "partition" in the context of an in-memory DataFrame (an executor's unit of work, this module's topic). How to spot it: kiosko_orders_at_scale.csv is a single flat file on disk, with no folder structure at all — lesson 5 measures how many partitions in memory Spark assigns it when reading it, and that number depends on spark.sql.files.maxPartitionBytes and available parallelism, not on franchise_id. How to fix it: always distinguish "file partition on disk" (a write decision, module 7's topic) from "in-memory DataFrame partition" (a read-and-execution decision, this module's topic) — they're related concepts sharing the same name, but not interchangeable.

Exercises

Exercise 1 — Generate the dataset with a different num_franchises and verify the formula. Using generate_orders_at_scale(1_000), count the rows and total revenue, and confirm with assert that both follow the same formula as the full dataset (1_000 × 40 rows, 106.15 × 1_000 in revenue).

See solution
from kiosko_scale import generate_orders_at_scale

NUM_FRANCHISES = 1_000
row_count = 0
total_revenue = 0.0
for r in generate_orders_at_scale(NUM_FRANCHISES):
    row_count += 1
    total_revenue += r["quantity"] * r["unit_price"]

print(f"row_count = {row_count}")
print(f"total_revenue (rounded) = {round(total_revenue, 2)}")
assert row_count == NUM_FRANCHISES * 40 == 40_000
assert round(total_revenue, 2) == round(106.15 * NUM_FRANCHISES, 2) == 106150.0
print("Verification: the formula holds for any num_franchises -> OK")

Expected output:

row_count = 40000
total_revenue (rounded) = 106150.0
Verification: the formula holds for any num_franchises -> OK

Confirmed: the formula num_franchises × 40 rows and 106.15 × num_franchises in revenue isn't a coincidence tied to the value 250,000 — it's a structural property of how generate_orders_at_scale() builds the data, valid for any num_franchises value you pass it.

Exercise 2 — Confirm two runs of generate_orders_at_scale(100) produce exactly the same output. Generate the list twice, in two separate calls to the function, and confirm with assert that they're identical — the direct proof of determinism.

See solution
from kiosko_scale import generate_orders_at_scale

first_run = list(generate_orders_at_scale(100))
second_run = list(generate_orders_at_scale(100))

print(f"len(first_run) = {len(first_run)}")
print(f"len(second_run) = {len(second_run)}")
assert first_run == second_run
print("Verification: two independent runs produce identical data, row for row -> OK")

Expected output:

len(first_run) = 4000
len(second_run) = 4000
Verification: two independent runs produce identical data, row for row -> OK

This is the direct proof of determinism: with no random or datetime.now() anywhere in the function, there's no source of variation at all between one run and the next. Compare this against what would happen if the function used random.uniform() to vary unit_price — that assert comparison would fail almost for certain.

Exercise 3 — Explain, without code, why the dates never run forward. In 2-3 sentences, explain why generate_orders_at_scale() reuses the same seven calendar dates (2026-08-03 through 2026-08-09) in each of the 250,000 franchises, instead of generating, say, 250,000 distinct, consecutive weeks.

See solution

Generating different dates for each franchise — for example, running the calendar forward one week per franchise — would produce absurd dates almost immediately: two hundred fifty thousand consecutive weeks amount to more than four thousand eight hundred years into the future, data with no business meaning at all, and it would also complicate deterministic verification (what exact date corresponds to franchise 137,842?). Reusing the same seven real dates for every franchise keeps the dataset simple to verify — always the same known dates from module 1 — and directs all the dataset's complexity toward the one dimension that actually matters for this module: row volume and the new partition key (franchise_id), not date variety.

Summary and next step

In this lesson you built this entire guide's first genuinely new piece of data: kiosko_orders_at_scale, 10,000,000 rows generated by generate_orders_at_scale(250_000), with no random used at any point, verified with three independent asserts against an exact formula — 250,000 × 40 = 10,000,000 rows, 106.15 × 250,000 = 26,537,500.00 in revenue, with the same proportional breakdown by store as always. You also confirmed Spark reads that file and sees exactly the same ten million rows. And you established, unambiguously, the declaration that accompanies this dataset in every lesson that uses it: it's synthetic, built to let you feel the cost of distributing, not a real projection of Kiosko's business.

Before moving on you should be able to: explain why generate_orders_at_scale() doesn't use random; recite from memory this dataset's three central numbers (10,000,000 rows, 26,537,500.00 in revenue, 250,000 franchises); and explain the difference between "file partition on disk" and "in-memory DataFrame partition."

Lesson 5 takes exactly this dataset and repeats lesson 3's same experiment — .explain() over a groupBy — but this time the Exchange you're going to see moves real data, measurable in bytes and records, not just a name in a plan.

Resources

  • Apache Spark — RDD Programming Guide, definition of lazy transformation applied to Python data generators (the same on-demand evaluation principle that justifies using yield in generate_orders_at_scale(), already seen in module 2 for Spark's own DataFrame). spark.apache.org/docs/latest/rdd-programming-guide.html.
  • Python — official typing.Iterator documentation (generate_orders_at_scale()'s declared return type). docs.python.org/3/library/typing.html#typing.Iterator.
  • This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — the exact specification for kiosko_orders_at_scale: 250,000 franchises, 10,000,000 rows, 26,537,500.00 in revenue.
  • data-engineering-foundations-guide DESIGN doc — the original source of KIOSKO_WEEK: Kiosko's forty real records this lesson multiplies unmodified. src/guides/data-engineering-foundations-guide/DISENO.md