The brief: Kiosko at scale

Description

Imagine someone from Kiosko's data team — not an engineer, but the person who decides the infrastructure budget — asks a direct question: "Before we consider paying for a managed Spark cluster, can you show me the distributed pipeline genuinely works, with evidence, over a serious data volume? And while you're at it, tell me honestly whether we actually need it." This lesson turns that question into a concrete brief — four deliverables, each with its own success criterion — that lessons 3 through 6 of this module are going to resolve, in order, with code and with evidence.

Connection to the module. The rest of this module exists to answer, one by one, this brief's four points. Lesson 3 delivers the assembled pipeline. Lesson 4 delivers proof of correctness. Lesson 5 delivers the justification for every design decision. And lesson 6 delivers the honest answer to the question that genuinely matters: was all of this needed?

An analogy: the engineer who gets asked "show me, don't tell me"

Anyone who's presented a technical project in front of whoever pays the bill knows the difference between two kinds of meeting. In the first, someone explains with slides why a technology is the right one, without running anything in front of the audience — a promise. In the second, someone opens a terminal, runs the complete pipeline in front of everyone, and lets the final number speak for itself — a demonstration. The second meeting is more uncomfortable to prepare — there's no room for something to "almost" work — but it's the only one that genuinely convinces whoever has to decide whether spending money on a real cluster is worth it. This module is that second meeting: not a series of slides about why Spark is powerful, but the complete pipeline, genuinely run over ten million rows, with every claim backed by a number anyone can verify by rerunning the same code.

The brief, in four deliverables

# the_brief.py
BRIEF = [
    ("Deliverable 1", "The complete distributed pipeline",
     "Reading kiosko_orders_at_scale (10M rows), broadcast joins, window functions, "
     "caching with real criteria, Parquet partitioned by store_id, and margin_category with pandas_udf -- "
     "all in a single SparkSession, end to end.",
     "Lesson 3"),
    ("Deliverable 2", "Proof of correctness, not just execution",
     "The pipeline can't just 'run with no errors' -- it has to produce, verified with assert, "
     "the same 26,537,500.00 the 106.15 x 250,000 formula predicts, and the same proportional "
     "per-store breakdown already known since module 1.",
     "Lesson 4"),
    ("Deliverable 3", "Every design decision, justified with measured evidence",
     "Why store_id and not franchise_id as the partition column. Why BroadcastHashJoin "
     "and not SortMergeJoin for the dimensions. Why cache fact_orders_at_scale_df and not "
     "any other DataFrame in the pipeline. None out of habit -- all with a number behind it.",
     "Lesson 5"),
    ("Deliverable 4", "The honest answer: does it genuinely need Spark?",
     "Applied to the real Kiosko (40 rows) and to a much larger hypothetical Kiosko, with the same "
     "cost criterion module 1 built -- without dodging the question, without selling itself "
     "as the only possible answer.",
     "Lesson 6"),
]

print("=== The Kiosko-at-scale brief: four deliverables ===\n")
for item, title, detail, owner in BRIEF:
    print(f"{item}: {title}")
    print(f"  {detail}")
    print(f"  Resolved in: {owner}\n")

What to expect. Running python3 the_brief.py, the output is exactly this:

=== The Kiosko-at-scale brief: four deliverables ===

Deliverable 1: The complete distributed pipeline
  Reading kiosko_orders_at_scale (10M rows), broadcast joins, window functions, caching with real criteria, Parquet partitioned by store_id, and margin_category with pandas_udf -- all in a single SparkSession, end to end.
  Resolved in: Lesson 3

Deliverable 2: Proof of correctness, not just execution
  The pipeline can't just 'run with no errors' -- it has to produce, verified with assert, the same 26,537,500.00 the 106.15 x 250,000 formula predicts, and the same proportional per-store breakdown already known since module 1.
  Resolved in: Lesson 4

Deliverable 3: Every design decision, justified with measured evidence
  Why store_id and not franchise_id as the partition column. Why BroadcastHashJoin and not SortMergeJoin for the dimensions. Why cache fact_orders_at_scale_df and not any other DataFrame in the pipeline. None out of habit -- all with a number behind it.
  Resolved in: Lesson 5

Deliverable 4: The honest answer: does it genuinely need Spark?
  Applied to the real Kiosko (40 rows) and to a much larger hypothetical Kiosko, with the same cost criterion module 1 built -- without dodging the question, without selling itself as the only possible answer.
  Resolved in: Lesson 6

Notice Deliverable 4: it's, quite deliberately, the only one of the four that can end in "you don't need it." A real brief — not one invented to sell a technology in advance — always leaves that possibility open. If this guide only showed Deliverables 1 through 3, and never honestly asked whether it was needed, it would be making exactly the mistake this guide's DESIGN doc's market warning flagged since module 1: jumping straight to "I need Spark" without first exhausting the question of whether a single node was already enough.

Diagram: from the brief to the four lessons

flowchart LR
    B["The brief:\nfour deliverables"]
    B --> E1["Deliverable 1:\ncomplete pipeline"] --> L3["Lesson 3"]
    B --> E2["Deliverable 2:\nproven correctness"] --> L4["Lesson 4"]
    B --> E3["Deliverable 3:\njustified decisions"] --> L5["Lesson 5"]
    B --> E4["Deliverable 4:\ndoes it need Spark?"] --> L6["Lesson 6"]
    L3 --> L8["Lesson 8:\nfinal mini-project"]
    L4 --> L8
    L5 --> L8
    L6 --> L8

Going deeper: why the brief asks for evidence, not just a result

It's worth noting none of this brief's four deliverables settles for "the pipeline finished without throwing an exception." That discipline isn't new to this module — it's the same one every mini-project in the previous seven modules held to: an explicit assert against a known number, never the absence of an error as proof of correctness. What this brief adds, for the first time in the guide, is a fourth layer of evidence no earlier module needed: it's not enough for the pipeline to be correct and well designed — it also needs to honestly justify whether building it was worth it. An engineer who only knows how to prove their code works, without knowing when that code was the right tool for the problem, has half the skill a senior data engineer needs. This module, and lesson 6 in particular, builds the other half.

Common mistakes

Treating Deliverable 4 as a formality that "obviously" is going to say Spark is needed. What happens: someone reads the brief, assumes that since this entire guide is dedicated to Spark, lesson 6 is going to find some way to justify Kiosko needing it, and takes that deliverable less seriously. Why it happens: it's natural to assume a technical guide dedicated to a tool is going to end up recommending that tool. How to spot it: if your expectation before reaching lesson 6 is "it's surely going to say yes, somehow," check module 1's market warning — the source this guide cites is explicit that "almost nobody in the world uses datasets larger than 100 TB" — and prepare for a verdict that can be, honestly, negative. How to fix it: lesson 6 applies the same runnable criterion (should_distribute()) you already built in module 1 — a criterion that doesn't know, and doesn't care, that it's being applied inside a guide about Spark; it just compares real bytes against measured thresholds.

Confusing "brief" with "exhaustive technical requirements document." What happens: someone expects this brief to include implementation details — which exact version of spark.sql.autoBroadcastJoinThreshold to use, how many executors to configure — instead of the four high-level deliverables that genuinely matter to whoever pays the bill. Why it happens: coming from seven very technical modules, it's easy to expect this brief to hold the same level of code detail. How to spot it: if you look, in this lesson, for a list of Spark configuration parameters, you won't find one — that detail lives in lessons 3 through 6, not in the brief motivating them. How to fix it: a real brief, aimed at whoever decides the budget, deliberately stays at the level of "what's going to get demonstrated," not "how" — the "how" is, precisely, the next four lessons' work.

Skipping this lesson because "it has no Spark code." What happens: someone, in a hurry to reach real execution, jumps straight to lesson 3 without reading the complete brief. Why it happens: this is the only lesson in this module, besides 7, that runs no PySpark. How to spot it: if you start lesson 3 unable to name the four deliverables from memory, you're missing the framework that gives meaning to why lesson 3 does what it does, in the order it does it. How to fix it: this lesson's four deliverables are the map for everything that follows — without them, lessons 3 through 6 look like four loose experiments, instead of the ordered answer to a single business question.

Exercises

Exercise 1 — Rewrite the brief in your own words, as if you had to present it in a five-minute meeting. Using the_brief.py's four deliverables, write a short paragraph (4-6 sentences) you could say out loud in front of someone who knows nothing about Spark, explaining what you're going to demonstrate and why they should care.

See solution

There's no single correct answer — the exercise asks for your own formulation — but a reasonable version would be: "We're going to run Kiosko's complete pipeline, but with a data volume ten million times larger than the real one, to prove the code works at scale. We're going to verify, with simple math anyone can repeat, that the final result is exactly what the formula predicts. We're going to justify every technical decision we made — how we organized the data on disk, when we saved results in memory, how we joined the tables — with a measured number, not an opinion. And, most importantly: we're going to honestly answer whether Kiosko, as it is today, genuinely needs this infrastructure, even if the answer is no."

Exercise 2 — Rank the four deliverables by the risk of them failing, from highest to lowest. Without running anything, think: which of the four deliverables is most likely to uncover a real problem in the pipeline, and which is more of a reporting formality? Justify your ranking in 2-3 sentences.

See solution

Deliverable 2 (proof of correctness) is, by far, the highest real risk: an assert comparing 26,537,500.00 against the computed result can fail because of a genuine bug in any of the previous seven modules — a JOIN losing rows, a different rounding, a badly partitioned window — and that failure would be a real signal of a problem, not just a reporting one. Deliverable 1 (the complete pipeline) has medium risk: it can fail for infrastructure reasons (insufficient memory, a missing file) without that implying a logic error. Deliverables 3 and 4 have the lowest "failure" risk in the strict sense — they're exercises in justification and criteria, not execution — although a poorly reasoned Deliverable 3 (justifying a decision with the wrong number) would be a silent failure, harder to catch than a Python exception.

Exercise 3 — Explain, without code, what would happen if Deliverable 2 failed but Deliverable 1 didn't. In 2-3 sentences, describe a concrete situation where lesson 3's complete pipeline runs start to finish, with no error at all, but lesson 4's correctness assert fails anyway.

See solution

This would happen, for example, if someone accidentally swapped F.sum("revenue") for F.avg("revenue") in one of the pipeline's aggregations: Spark would run the query with no error at all — AVG is just as valid a function as SUM — the complete pipeline would run end to end with no exception, and only comparing the final result against 26,537,500.00 would reveal the number doesn't match. This is precisely why Deliverable 2 exists as a piece separate from Deliverable 1: the absence of an error is never evidence the business logic is correct, the same discipline every mini-project across this entire guide has repeated since module 3.

Summary and next step

In this lesson you turned the business question behind this capstone — "show me, with evidence, whether Kiosko genuinely needs a distributed pipeline" — into a concrete four-deliverable brief: the complete pipeline, proof of correctness, the justification for every design decision, and the honest answer to the decision tree. Each of the four has a dedicated lesson, in the same order a real data engineer would approach them.

Before moving on you should be able to: name the four deliverables from memory; explain why Deliverable 4 can honestly end in "no"; and explain the difference between "the pipeline ran with no errors" and "the pipeline is correct."

Lesson 3 opens the first deliverable: Kiosko's complete distributed pipeline, assembled for the first time in a single SparkSession, genuinely run over kiosko_orders_at_scale's ten million rows.

Resources

• src/paths/data-engineering-ecosystem/VALIDACION.md — the source of the market warning backing why this brief's Deliverable 4 can honestly end in a negative verdict. Internal repository document, not a public URL.
• This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — module 8's "What actually gets run / verified" section, the exact source for this brief's four deliverables. src/guides/spark-and-distributed-processing-guide/DISENO.md.