Module 8: Project Kioskos Distributed Pipeline

Module introduction: Kiosko's distributed capstone

Next →

Why this module exists

You've gone through seven modules building, piece by piece, a complete mental model of Spark. Module 1 installed PySpark and Java, and confirmed Spark reads exactly the same forty Kiosko records you already knew by heart. Module 2 opened the black box: driver, executors, lazy evaluation, transformations versus actions. Module 3 rebuilt fact_orders with the DataFrame API — the same 106.15 as always, now computed by a fourth engine. Module 4 built, deterministically and with no random, this guide's only genuinely new piece of data: kiosko_orders_at_scale, ten million rows, and with it you felt for the first time what a real shuffle is. Module 5 taught the exact criterion behind BroadcastHashJoin versus SortMergeJoin, and window functions for running totals and rankings. Module 6 opened Catalyst through its phases, read Adaptive Query Execution actually acting, and established the criterion for when to cache. Module 7 closed the partitioned-Parquet debt and resolved, with execution-plan evidence, why a vectorized pandas_udf replaces a plain Python UDF.

Seven pieces, each proven separately, each with its own closing mini-project. This module, the capstone, adds no new concept — it's the day all seven start working together, end to end, over the complete ten million rows, and the result gets compared, number by number, against what you've known since this guide's module 1 and since data-engineering-foundations-guide, two guides back: 106.15 in real revenue, 26,537,500.00 at scale — exactly 106.15 × 250,000 — S01=9,575,000.00/S02=9,700,000.00/S03=7,262,500.00. And, closing out, this module asks the question that opened this entire guide in its first module, now with the complete pipeline in front of you to answer it with evidence, not intuition: does Kiosko genuinely need Spark?

Connection to the module. Every lesson in this module reuses, without changing a single line of its internal logic, the code you already built: KIOSKO_WEEK and generate_orders_at_scale() (M4), the three explicit StructTypes for orders/dim_store/dim_product (M3), the broadcast-join criterion and window functions (M5), the decision to cache with a real criterion (M6), the partitioned write by store_id and margin_category's pandas_udf (M7), and should_distribute(), the cost-criterion function you built in module 1, lesson 3, with the explicit promise you were going to reuse it here. This module doesn't rewrite any of that — it assembles it, runs it over the complete dataset, and audits it.

An analogy: the same shift, with the whole fleet working together for the first time

Every module in this guide taught, separately, one piece of a delivery truck fleet's operation: how boxes get split into partitions (module 4), how the small directory gets photocopied instead of loading the complete warehouse into every truck (module 5), how a GPS recalculates the route halfway through (module 6, AQE), how the warehouse's aisles get organized by destination before the first order arrives (module 7). Every lesson tested its piece in isolation, with a controlled experiment, almost always comparing two alternatives.

This module is the first complete shift where the whole fleet works together, at the same time, on the biggest real order this guide has built: kiosko_orders_at_scale's ten million rows. No single truck resolves the whole shift alone — the one deciding the JOIN strategy depends on the warehouse already being organized by aisle, the one reading Catalyst's plan depends on caching already being decided with a real criterion. And, at the end of the shift, someone has to ask the question no individual truck can answer: did this order genuinely need a complete fleet, or would one well-loaded truck have been enough? That question — lesson 6's decision tree — is this module's real close, not an appendix.

Worked example: the map of the seven pieces, before assembling anything

Before writing this module's first command, it's worth seeing the complete inventory — what each earlier module built, and exactly which piece of it you're going to reuse here, without changing a single line.

# capstone_inventory.py
PIECES = [
    ("M1", "SparkSession + Java 17", "local[*], spark.read.csv() over the 7 real files -- 40 rows"),
    ("M2", "Driver/executors, lazy evaluation", "transformations vs actions, the logical and physical DAG"),
    ("M3", "DataFrame API", "orders.join(dim_store).join(dim_product), revenue = quantity * unit_price -- 106.15"),
    ("M4", "kiosko_orders_at_scale, shuffle", "generate_orders_at_scale(250_000) -- 10,000,000 rows, no random"),
    ("M5", "Broadcast join, windows", "BroadcastHashJoin by default, Window.partitionBy/orderBy"),
    ("M6", "Catalyst, explain(), AQE, caching", ".cache() justified by 3 reuses, InMemoryTableScan"),
    ("M7", "Partitioned Parquet, pandas_udf", "partitionBy('store_id'), margin_category vectorized with Arrow"),
]

print("=== kiosko_orders_at_scale: seven pieces, assembled for the first time ===\n")
for module, tool, what in PIECES:
    print(f"{module}  {tool:32} -> {what}")

print("\nInput: kiosko_orders_at_scale.csv, 10,000,000 rows (generated in M4, unchanged).")
print("This module's output: the same pipeline, assembled end to end,")
print("verified against 26,537,500.00, and audited with the decision tree -- does it genuinely need Spark?")

What to expect. Running python3 capstone_inventory.py, the output is exactly this:

=== kiosko_orders_at_scale: seven pieces, assembled for the first time ===

M1  SparkSession + Java 17            -> local[*], spark.read.csv() over the 7 real files -- 40 rows
M2  Driver/executors, lazy evaluation -> transformations vs actions, the logical and physical DAG
M3  DataFrame API                     -> orders.join(dim_store).join(dim_product), revenue = quantity * unit_price -- 106.15
M4  kiosko_orders_at_scale, shuffle   -> generate_orders_at_scale(250_000) -- 10,000,000 rows, no random
M5  Broadcast join, windows           -> BroadcastHashJoin by default, Window.partitionBy/orderBy
M6  Catalyst, explain(), AQE, caching -> .cache() justified by 3 reuses, InMemoryTableScan
M7  Partitioned Parquet, pandas_udf   -> partitionBy('store_id'), margin_category vectorized with Arrow

Input: kiosko_orders_at_scale.csv, 10,000,000 rows (generated in M4, unchanged).
This module's output: the same pipeline, assembled end to end,
verified against 26,537,500.00, and audited with the decision tree -- does it genuinely need Spark?

Notice something this table doesn't show: none of the modules listed there built the "does it need Spark?" criterion — that criterion belongs specifically to module 1, lesson 3, and to this module's lesson 6, which reuses it. This table's seven pieces are how a computation gets distributed when it's needed; the criterion you're going to close out in lesson 6 is when it's genuinely needed — two related questions, but never the same one.

Diagram: seven modules converging into one

flowchart TD
    M1["M1: SparkSession + Java 17\n(reading 40 real rows)"]
    M2["M2: driver/executors\n(lazy evaluation)"]
    M3["M3: DataFrame API\n(fact_orders, 106.15)"]
    M4["M4: kiosko_orders_at_scale\n(10M rows, shuffle)"]
    M5["M5: broadcast join + windows\n(running total, ranking)"]
    M6["M6: Catalyst + explain() + AQE\n(caching with criteria)"]
    M7["M7: partitioned Parquet\n+ pandas_udf"]

    M1 --> M8
    M2 --> M8
    M3 --> M8
    M4 --> M8
    M5 --> M8
    M6 --> M8
    M7 --> M8

    M8["M8: CAPSTONE\nassemble, verify,\ndecide with criteria, close the ecosystem"]

    M8 --> L3["L3: assembling-the-distributed-pipeline-end-to-end"]
    M8 --> L4["L4: verifying-correctness-against-the-original-10615"]
    M8 --> L5["L5: choosing-partitioning-caching-and-join-strategy-with-criteria"]
    M8 --> L6["L6: the-decision-tree-does-kiosko-actually-need-spark"]
    M8 --> L7["L7: what-kiosko-still-needs"]

Notice the seven incoming arrows converge into a single node, and that node opens out into this module's concrete lessons — no arrow points back. Nothing you do in this module modifies kiosko_scale.py, the three StructTypes, the .join()/Window/.cache()/partitionBy() logic, or margin_category — each one stays exactly as its own module left it. This capstone only assembles, runs, and decides.

This module's map

Lesson    What it does
────────  ──────────────────────────────────────────────────────────────
L1        (this one) The seven pieces' inventory, before assembling
L2        The brief: why someone would ask for "Kiosko at scale" before investing in infrastructure
L3        Assembling the complete pipeline, end to end, over 10,000,000 rows
L4        Verifying correctness: the complete chain from 106.15 to 26,537,500.00
L5        Choosing partitioning, caching, and JOIN strategy -- with real criteria and measured evidence
L6        The decision tree: does Kiosko genuinely need Spark? (real answer: no)
L7        What Kiosko still needs -- the map of sibling guides
L8        Project: Kiosko's first distributed pipeline

Lessons 3 through 6 are the real execution, in the same order an engineer would approach it: assemble, verify the result is correct, justify every design decision with evidence, and — the step most "big data" tutorials skip — honestly ask whether all that work was needed. Lesson 7 traces the complete map toward this ecosystem's sibling guides. Lesson 8 closes with the final mini-project: the complete pipeline, run one last time, with this guide's definitive report.

Going deeper: why the "was it needed?" question closes the guide, not opens it

You might wonder why this guide didn't start by asking "do you need Spark?" and stop there, saving you seven modules if the answer for real Kiosko was always going to be no. The reason is the same one you already saw in module 1: knowing the answer is "no" for Kiosko, without ever having built the complete pipeline, would be a claim with no evidence — exactly the kind of unsupported opinion module 1 lesson 3's cost criterion rejects. This capstone exists so you give the final answer — "real Kiosko doesn't need Spark" — after having built, run, and verified the complete distributed pipeline, not before. Only then does the claim stop being an intuition and become a verdict backed by evidence: you genuinely used Spark, over ten million real rows, and the very criterion you built in module 1 — applied now to that same dataset, not just to the forty-row week — confirms not even this guide's synthetic dataset crossed the threshold that would justify paying a real cluster's cost.

That's, at bottom, the complete skill this guide delivers: not just knowing how to operate Spark — the DataFrame API, shuffle, joins, Catalyst, partitioned Parquet, pandas_udf — but knowing, with the same precision, when that operation is solving a real problem and when it's solving a practice problem. Both halves are the same skill, and this module brings them together.

Common mistakes

Expecting this module to teach an eighth tool, and feeling "you didn't learn anything" by the end. What happens: someone arrives at this module expecting a new technical piece — maybe something about real clusters or the cloud — and gets thrown off seeing the first lessons only assemble code that already existed. Why it happens: every earlier module in this guide introduced a new tool or concept (the SparkSession, the DataFrame API, shuffle, joins, Catalyst, partitioned Parquet), so it's reasonable, by pattern, to expect an eighth one. How to spot it: if you finish lesson 3 of this module thinking "so what new API do I learn now?", reread the fleet's-first-complete-shift analogy — this module's value isn't in a new concept, it's in the confirmation, with evidence executed over ten million rows, that the seven earlier pieces work together, and in the final criterion for whether using them was even needed. How to fix it: measure this module's success by two questions different from earlier modules': "can I assemble the complete pipeline from memory?" and "can I defend, with numbers, whether Kiosko needs Spark or not?" — those are the two skills this capstone certifies.

Skipping this module's real execution, assuming "you already ran it in modules 3 through 7, so it makes no difference." What happens: someone, seeing this module changes no file from kiosko_scale.py or the transformation logic, decides there's no need to rerun the complete pipeline over the ten million rows — after all, they already saw each piece work separately. Why it happens: each piece's code is identical to its source module's, so repeating the execution seems redundant. How to spot it: if you finish this module without having run, with your own hands, the complete pipeline in a single SparkSession — read, joins, windows, caching, partitioned Parquet, pandas_udf, and the decision tree applied — you didn't verify this capstone's real promise, you only recalled it from memory from earlier modules, a much lower bar. How to fix it: every lesson from 3 through 6 includes real commands with literal output — run them yourself, over module 4's same ten-million-row kiosko_orders_at_scale.csv, not sixteen loose fragments.

Confusing "real Kiosko doesn't need Spark" with "this guide was pointless." What happens: someone reaches lesson 6, sees the NO verdict applied to Kiosko, and wonders whether the eight modules they just studied were a waste, given the case study itself never justified the tool. Why it happens: it's tempting to measure a technical guide's value by whether its case study "needed" the technology it teaches. How to spot it: if your conclusion at the end of this guide is "so I learned Spark for nothing," check this lesson's "going deeper" section — the PySpark code you wrote runs, unchanged, on a real cluster once volume genuinely justifies it; what you learned isn't "how to solve Kiosko's problem," it's "how Spark operates, and how to recognize when it's needed." How to fix it: this module's lesson 6 applies the same criterion to a much larger hypothetical Kiosko, with concrete numbers, precisely to separate the two questions: you know how to use the tool, and you know when to use it — two different skills, both certified by this guide.

Exercises

Exercise 1 — Explain every piece without looking at the inventory. Without looking at capstone_inventory.py again, write from memory the table's seven rows (module, tool, what it contributes to this capstone) in your own words. Then compare against the original and note any piece you forgot or described imprecisely.

See solution

There's no single correct answer — the exercise asks for your own formulation — but every row should, at minimum, name: the module, the central concept or function, and which specific part of this capstone's pipeline depends on it (not just "shuffle" but "the mechanism kiosko_orders_at_scale, with its ten million rows, triggers every time there's a groupBy or a JOIN that can't resolve with broadcast"). If you struggled to precisely recall module 5 or 6, it's worth reviewing their closing lessons (08-project-...) before continuing with this capstone — this module assumes the seven pieces are already consolidated, it doesn't reteach them from scratch.

Exercise 2 — Predict what this module's pipeline would be missing if module 5 had never been written. Without running anything yet, explain in 2-3 sentences: if you'd never learned the BroadcastHashJoin versus SortMergeJoin criterion, which decision in this capstone would you make blindly, unable to justify it with evidence?

See solution

Without module 5, this capstone's lesson 3 .join() would still work — Spark chooses the JOIN strategy automatically, whether or not you understand the criterion — but you wouldn't be able to read the .explain() plan and explain why Spark chose BroadcastHashJoin instead of SortMergeJoin, nor could you justify, in this module's lesson 5, the decision to write dim_store.csv and dim_product.csv as small files instead of worrying about their size. You'd end up with a pipeline that works, but with no ability to defend any of its design decisions with evidence — exactly the difference between "knowing how to use Spark" and "knowing why Spark did what it did," the standard this entire guide has demanded since module 1.

Exercise 3 — Argue why the order of the seven modules mattered for this specific capstone. In 3-4 sentences, explain why this module couldn't have been written before module 4 (building kiosko_orders_at_scale), even though, in theory, module 5's .join()/Window logic doesn't directly depend on module 4.

See solution

Without kiosko_orders_at_scale — the ten-million-row synthetic dataset module 4 built — this capstone would have no real volume to assemble a distributed pipeline over: modules 5 through 7's joins, windows, caching, and partitioned Parquet each got tested precisely over that dataset, because only there do shuffle, partition pruning, and Arrow vectorization produce a measurable difference. Without that volume, this module would have to repeat, once more, module 3's same forty-row experiment — where none of those seven pieces changes anything observable — and lesson 6's decision tree would have no real intermediate point to evaluate between "forty rows" and "a much larger hypothetical Kiosko." This guide's order deliberately builds the evidence before the final criterion.

Summary and next step

In this lesson you saw the complete inventory of the seven pieces this module is going to assemble: SparkSession and reading with an explicit schema (M1), the lazy-evaluation model (M2), the DataFrame API rebuilding fact_orders (M3), kiosko_orders_at_scale and shuffle (M4), broadcast join and window functions (M5), Catalyst/explain()/AQE/caching with real criteria (M6), and partitioned Parquet with pandas_udf (M7). None of it is new — each is already built, documented, and proven in its own module. You confirmed, with the whole fleet's first shift together analogy, that this capstone's role is to assemble, verify, and decide — not carve a new piece.

Before moving on you should be able to: name the seven pieces and what each contributes to the final pipeline; explain why this module modifies no file from modules 1 through 7; and explain the difference between "knowing how to operate Spark" and "knowing when Spark is needed" — the exact distinction lesson 6 is going to close with evidence.

Lesson 2 puts this capstone in business context: the real brief that would motivate someone to specifically ask, "run Kiosko at scale and tell me whether this genuinely needs distributed infrastructure" — before spending a single dollar on a cluster.

Resources

  • PySpark — SQL Getting Started (the SparkSession.builder.appName(...).getOrCreate() pattern this capstone reuses unchanged since module 1). spark.apache.org/docs/latest/sql-getting-started.html.
  • Apache Spark — SQL Performance Tuning (Catalyst, .explain(), AQE, autoBroadcastJoinThreshold — the complete reference this module reuses from modules 5 and 6). spark.apache.org/docs/latest/sql-performance-tuning.html.
  • This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — the complete map of the eight modules, including this capstone. src/guides/spark-and-distributed-processing-guide/DISENO.md.
← PreviousProject: Kiosko's partitioned Parquet and a vectorized UDFNext →The brief: Kiosko at scale