Verifying correctness against the original `106.15`

Description

Lesson 3's pipeline finished with no error at all, and its own asserts already confirmed 26,537,500.00. But this lesson does something different, and more demanding: it builds the complete chain of evidence connecting that number back to the 106.15 that opened this guide, in module 1, and that opened Kiosko's whole story two guides back, in data-engineering-foundations-guide. It's not enough for the at-scale pipeline to give the right number — you need to prove, with simple math anyone can repeat without Spark, that number is exactly what the formula predicts, and that the real forty-row Kiosko and the ten-million-row synthetic Kiosko are, literally, the same data, replicated 250,000 times.

Connection to the module. This lesson resolves Deliverable 2 from lesson 2's brief: proof of correctness, not just execution. It reuses fact_orders (M3), generate_orders_at_scale() (M4), and the fact_orders_at_scale result you just assembled in lesson 3.

An analogy: the auditor who trusts no final balance, only the sum of its parts

A financial auditor reviewing a company's balance sheet never confirms an isolated final number — they rebuild the complete chain that produced it: each product line's revenue, summed by region, summed by quarter, up to the annual total. If the annual total "looks reasonable" but the auditor can't rebuild, step by step, how it was reached from individual revenues, that audit is worthless — it's just someone's word the number is correct. This lesson does exactly that audit work over fact_orders_at_scale: it doesn't trust 26,537,500.00 "looks reasonable" because the pipeline threw no error — it rebuilds the complete chain, from the real forty-row week, through the scaling formula, up to the full-scale final result.

Worked example: the complete correctness chain

Step 1 — Real fact_orders, the same one from module 3

# kiosko_correctness_chain.py
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)
from kiosko_scale import generate_orders_at_scale

print("=== Module 8, lesson 4: verifying correctness against the original 106.15 ===\n")

spark = (
    SparkSession.builder
    .appName("kiosko-spark")
    .master("local[*]")
    .config("spark.driver.memory", "4g")
    .getOrCreate()
)

orders_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
dim_store_schema = StructType([
    StructField("store_id", StringType(), False),
    StructField("store_name", StringType(), False),
    StructField("city", StringType(), False),
])
dim_product_schema = StructType([
    StructField("product_id", StringType(), False),
    StructField("product_name", StringType(), False),
    StructField("category", StringType(), False),
    StructField("unit_cost", DoubleType(), False),
])

print("Step 1 -- real fact_orders (40 rows), the same one from module 3")
orders_df = spark.read.csv("orders_2026-08-*.csv", schema=orders_schema, header=True, enforceSchema=False)
dim_store_df = spark.read.csv("dim_store.csv", schema=dim_store_schema, header=True, enforceSchema=False)
dim_product_df = spark.read.csv("dim_product.csv", schema=dim_product_schema, header=True, enforceSchema=False)

fact_orders_df = (
    orders_df.join(dim_store_df, "store_id").join(dim_product_df, "product_id")
    .withColumn("revenue", F.round(F.col("quantity") * F.col("unit_price"), 2))
)
real_count = fact_orders_df.count()
real_total = fact_orders_df.agg(F.round(F.sum("revenue"), 2).alias("t")).collect()[0][0]
real_by_store = {
    r["store_id"]: r["t"]
    for r in fact_orders_df.groupBy("store_id").agg(F.round(F.sum("revenue"), 2).alias("t")).collect()
}
print(f"real_count = {real_count}")
print(f"real_total = {real_total}")
print(f"real_by_store = {real_by_store}")
assert real_count == 40
assert real_total == 106.15
assert real_by_store == {"S01": 38.3, "S02": 38.8, "S03": 29.05}
print("Verification: 40 rows, 106.15, S01=38.3/S02=38.8/S03=29.05 -> OK\n")

Nothing new so far — it's, literally, module 3's same fact_orders, with the same output you already saw there. This lesson recalculates it on purpose, as the chain's first link.

Step 2 — generate_orders_at_scale(1): a single franchise should collapse to the real week

print("Step 2 -- generate_orders_at_scale(1): a single franchise, should collapse to the real week")
one_franchise = list(generate_orders_at_scale(1))
print(f"len(one_franchise) = {len(one_franchise)}")
assert len(one_franchise) == 40
one_franchise_revenue = round(sum(r["quantity"] * r["unit_price"] for r in one_franchise), 2)
print(f"revenue for 1 franchise = {one_franchise_revenue}")
assert one_franchise_revenue == 106.15
print("Verification: num_franchises=1 reproduces exactly the same 106.15 -> OK\n")

This step is the most direct mathematical proof that kiosko_orders_at_scale isn't a different dataset from Kiosko — it's Kiosko, repeated. With num_franchises=1, generate_orders_at_scale() produces exactly the same forty rows as always, and its revenue, computed in plain Python, with no Spark, gives the same 106.15 step 1 just confirmed with Spark.

Step 3 — fact_orders_at_scale, lesson 3's complete pipeline

print("Step 3 -- fact_orders_at_scale (10,000,000 rows), lesson 3's complete pipeline")
scale_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("franchise_id", IntegerType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
orders_at_scale_df = spark.read.csv("kiosko_orders_at_scale.csv", schema=scale_schema, header=True, enforceSchema=False)
fact_at_scale_df = (
    orders_at_scale_df.join(dim_store_df, "store_id").join(dim_product_df, "product_id")
    .withColumn("revenue", F.round(F.col("quantity") * F.col("unit_price"), 2))
)
scale_count = fact_at_scale_df.count()
scale_total = fact_at_scale_df.agg(F.round(F.sum("revenue"), 2).alias("t")).collect()[0][0]
scale_by_store = {
    r["store_id"]: r["t"]
    for r in fact_at_scale_df.groupBy("store_id").agg(F.round(F.sum("revenue"), 2).alias("t")).collect()
}
print(f"scale_count = {scale_count}")
print(f"scale_total = {scale_total}")
print(f"scale_by_store = {scale_by_store}")
assert scale_count == 10_000_000
assert scale_total == 26_537_500.00
assert scale_by_store == {"S01": 9_575_000.0, "S02": 9_700_000.0, "S03": 7_262_500.0}
print("Verification: 10,000,000 rows, 26,537,500.00, S01=9,575,000.00/S02=9,700,000.00/S03=7,262,500.00 -> OK\n")

Step 4 — The exact proportion: scale / 250,000 == original

print("Step 4 -- the exact proportion: scale / 250,000 == original, for the total and for each store")
NUM_FRANCHISES = 250_000
ratio_total = round(scale_total / NUM_FRANCHISES, 2)
print(f"scale_total / {NUM_FRANCHISES} = {ratio_total}")
assert ratio_total == real_total == 106.15

ratio_by_store = {k: round(v / NUM_FRANCHISES, 2) for k, v in scale_by_store.items()}
print(f"scale_by_store / {NUM_FRANCHISES} = {ratio_by_store}")
assert ratio_by_store == real_by_store == {"S01": 38.3, "S02": 38.8, "S03": 29.05}
print("Verification: the complete scale is exactly 250,000 copies of the same 106.15 -> OK\n")

print("=== Correctness chain closed: 106.15 (40 rows) -> 26,537,500.00 (10,000,000 rows) ===")
spark.stop()

What to expect. Running python3 kiosko_correctness_chain.py (executed in this run, PySpark 4.2.0):

=== Module 8, lesson 4: verifying correctness against the original 106.15 ===

Step 1 -- real fact_orders (40 rows), the same one from module 3
real_count = 40
real_total = 106.15
real_by_store = {'S02': 38.8, 'S01': 38.3, 'S03': 29.05}
Verification: 40 rows, 106.15, S01=38.3/S02=38.8/S03=29.05 -> OK

Step 2 -- generate_orders_at_scale(1): a single franchise, should collapse to the real week
len(one_franchise) = 40
revenue for 1 franchise = 106.15
Verification: num_franchises=1 reproduces exactly the same 106.15 -> OK

Step 3 -- fact_orders_at_scale (10,000,000 rows), lesson 3's complete pipeline
scale_count = 10000000
scale_total = 26537500.0
scale_by_store = {'S02': 9700000.0, 'S01': 9575000.0, 'S03': 7262500.0}
Verification: 10,000,000 rows, 26,537,500.00, S01=9,575,000.00/S02=9,700,000.00/S03=7,262,500.00 -> OK

Step 4 -- the exact proportion: scale / 250,000 == original, for the total and for each store
scale_total / 250000 = 106.15
scale_by_store / 250000 = {'S02': 38.8, 'S01': 38.3, 'S03': 29.05}
Verification: the complete scale is exactly 250,000 copies of the same 106.15 -> OK

=== Correctness chain closed: 106.15 (40 rows) -> 26,537,500.00 (10,000,000 rows) ===

(When reading with the wildcard pattern "orders_2026-08-*.csv", Spark prints, before the result, a benign log block with FileNotFoundException — the same warning module 1's lesson 6 already explained in detail. It isn't a real error; the block was omitted here since it's already documented.)

Four steps, four verifications, and a single conclusion: fact_orders_at_scale isn't a dataset "similar" to Kiosko at a larger scale — it's mathematically identical, multiplied by an exact, known constant. Every store scales by exactly the same proportion (250,000), the total scales by exactly the same proportion, and a single synthetic franchise reproduces the real week bit for bit.

Diagram: the complete correctness chain

flowchart LR
    A["real fact_orders\n40 rows, 106.15\nS01=38.3/S02=38.8/S03=29.05"] -->|"x 250,000"| B["fact_orders_at_scale\n10,000,000 rows, 26,537,500.00\nS01=9.575M/S02=9.7M/S03=7.2625M"]
    C["generate_orders_at_scale(1)\n40 rows, 106.15\n(plain Python, no Spark)"] -.->|"same chain,\nindependent path"| A
    B -->|"/ 250,000"| D["Returns to 106.15\nexact proportion confirmed"]

Going deeper: why verification with two independent paths matters more than a single assert

Notice something structural about this lesson: step 1 computes 106.15 with Spark, reading the seven real CSV files. Step 2 computes the same 106.15 without Spark, in plain Python, summing directly over the list generate_orders_at_scale(1) produces. Both paths — completely independent in their implementation — arriving at exactly the same number isn't decorative coincidence: it's the strongest kind of evidence this entire guide has used, the same cross-verification discipline module 3 applied when comparing Spark against the earlier guides' dict/DuckDB/Polars/SQL.

If there were only one path — only Spark, or only plain Python — and that path had a subtle bug (a rounding error, a mis-referenced column), that single path's assert could pass against an incorrect number, because the "incorrect" number would be self-consistent. Two independent paths matching exactly rules out that possibility: for both paths to fail the exact same way, they'd have to share the exact same bug, something far less likely when one path uses Spark's distributed DataFrame API and the other uses a direct sum over a list of Python dictionaries.

Common mistakes

Trusting scale_total == 26_537_500.00 is sufficient evidence, without verifying the per-store breakdown. What happens: someone runs only step 3, sees the total matches, and considers the correctness check closed without reviewing scale_by_store. Why it happens: the total is the most visible number, and "the total checks out" feels sufficient. How to spot it: a bug moving revenue from one store to another — a poorly conditioned JOIN assigning some S01 rows to S02, for example — could, in theory, leave the overall total intact while the per-store breakdown ends up completely wrong. How to fix it: this lesson, like every mini-project in this guide since module 3, always verifies the breakdown, not just the aggregate — this lesson's step 3 and step 4 never settle for a single number.

Assuming step 4 (the exact proportion) is redundant with step 3, because both use the same numbers. What happens: someone sees scale_total already got verified against 26_537_500.00 in step 3, and considers dividing by 250,000 in step 4 to compare against 106.15 again unnecessary. Why it happens: both steps use scale_total, so it can look like they're verifying the same thing twice. How to spot it: step 3 verifies scale_total matches a hardcoded number (26_537_500.00); step 4 verifies that same number, divided by the scaling constant, matches the number calculated independently in step 1 (real_total) — they're two distinct claims: one compares against a constant known ahead of time, the other compares against a result computed in this same run. How to fix it: keep both steps — step 3 confirms the at-scale pipeline gives the number this guide's DESIGN doc promised; step 4 confirms that number is mathematically consistent with the real result, computed in this same execution, not copied from an earlier lesson.

Reading the wildcard read's FileNotFoundException warning as a sign something went wrong in this specific lesson. What happens: someone, seeing the log block with the FileNotFoundException trace when reading "orders_2026-08-*.csv", interrupts execution thinking step 1 failed. Why it happens: an exception trace in terminal output, with no context, always looks like a fatal error. How to spot it: if the script continues after that block and produces real_count = 40 with the rest of the chain verified, the exception was just internal Spark log noise, not a real failure — exactly the same behavior module 1's lesson 6 already documented in complete detail. How to fix it: nothing to fix in the code; if you'd rather have output without that noise, use an explicit file list (sorted(glob.glob("orders_2026-08-*.csv"))) instead of passing the wildcard pattern as a text string.

Exercises

Exercise 1 — Repeat step 4 with a different NUM_FRANCHISES value, using generate_orders_at_scale(500) instead of the complete dataset. Generate 500 franchises with plain Python (no Spark), compute their total revenue, and confirm with assert that revenue / 500 == 106.15.

from kiosko_scale import generate_orders_at_scale

rows_500 = list(generate_orders_at_scale(500))
revenue_500 = round(sum(r["quantity"] * r["unit_price"] for r in rows_500), 2)
print(f"len(rows_500) = {len(rows_500)}")
print(f"revenue_500 = {revenue_500}")
assert len(rows_500) == 500 * 40 == 20_000
assert round(revenue_500 / 500, 2) == 106.15
print("Verification: the proportion holds for any num_franchises, not just 250,000 -> OK")

len(rows_500) = 20000
revenue_500 = 53075.0
Verification: the proportion holds for any num_franchises, not just 250,000 -> OK

Exercise 2 — Introduce a deliberate bug and confirm the verification chain catches it. Temporarily change step 3's withColumn("revenue", ...) to compute F.col("quantity") * F.col("unit_cost") instead of F.col("quantity") * F.col("unit_price") (the same unit_cost-versus-unit_price mistake module 3 already warned about). Which specific assert fails first?

Exercise 3 — Explain, without code, why this lesson doesn't trust lesson 3's result, even though it already had its own asserts. In 2-3 sentences, justify why recalculating fact_orders_at_scale from scratch in this lesson, instead of simply trusting the asserts that already passed in lesson 3, adds real value.

Summary and next step

In this lesson you built this capstone's complete correctness chain: real fact_orders (106.15, 40 rows, computed with Spark) matches generate_orders_at_scale(1) (106.15, computed in plain Python), and fact_orders_at_scale (26,537,500.00, 10,000,000 rows) is exactly 250,000 times that same number, for the total and for every store separately. Four steps, two independent calculation paths, a single conclusion verified with assert.

Before moving on you should be able to: explain why two independent calculation paths are stronger than a single assert; recite from memory the complete chain (106.15 → × 250,000 → 26,537,500.00); and explain why verifying only the total, without the per-store breakdown, isn't sufficient evidence of correctness.

Lesson 5 steps back from the numeric result and audits this pipeline's design decisions: why store_id and not franchise_id as the partition column, why BroadcastHashJoin and not SortMergeJoin, why cache this specific DataFrame and not another — each one, with the same measured-evidence discipline you already saw here.

Resources

• python-for-data-engineering-guide's DESIGN doc — the original source of Kiosko's 106.15 and star schema, which this correctness chain traces back to its origin. src/guides/python-for-data-engineering-guide/DISENO.md.
• This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — the exact specification for kiosko_orders_at_scale and the 106.15 × 250,000 = 26,537,500.00 formula this lesson verifies. src/guides/spark-and-distributed-processing-guide/DISENO.md.
• Apache Spark — SQL Getting Started (the read-and-aggregate pattern backing both step 1 and step 3 of this lesson). spark.apache.org/docs/latest/sql-getting-started.html.