Module 2: The Spark Execution Model

Module introduction: the Spark execution model

Why this module exists

Module 1 closed with a clear verdict: Kiosko, at its real scale of forty rows, doesn't need Spark — but it also closed with Spark for real installed, verified, and reading those same forty rows with an explicit schema. What that module 1 never did was explain how Spark does what it does. orders_df = spark.read.csv(...) and orders_df.count() worked, produced 40, and the module stopped there on purpose — install and confirm first, understand the architecture later.

This module 2 opens that box. You're not going to rebuild fact_orders with joins and aggregations yet (that starts in module 3) — you're going to understand, precisely and with executed evidence, what actually happens when you write a line of Spark code: who does the work (lesson 2, driver and executors), which API you should use and which one to leave in the past (lessons 3 and 4, RDDs versus the DataFrame API), and the single most important fact of all — the one that's going to save you more confusion than any other concept in this guide — that writing Spark code executes nothing until you explicitly call an action (lessons 5 through 7). Lesson 8's project brings the seven pieces together into a single chain of transformations, verified with real evidence that no work fires until you ask for it.

Connection to the module. This module doesn't touch fact_orders, dim_store, or dim_product yet — those joins and that aggregation arrive in module 3, on top of the mental model this module builds first. Without this module, any line of PySpark you write afterward is magic that happens to work; with it, every line has a precise explanation of what it does, when it does it, and why.

An analogy: the professional kitchen, before cooking anything

Picture a large professional kitchen, the kind with a head chef and several line cooks. The head chef never chops a single onion or flips a single pan — their job is to decide: what gets cooked, in what order, and which cook handles which part. The line cooks are the ones who actually have their hands in the food, each with their own station, their own set of pans, their own piece of the mise en place. No single line cook cooks the whole dish alone, and the head chef never cooks anything directly — they coordinate.

That's, with striking precision, Spark's architecture: your Python script is the driver — the head chef, the one who decides what to do and in what order, but who never touches a single byte of data directly — and the executors are the line cooks — the processes that actually read files, filter rows, and return results. Lesson 2 of this module builds this analogy with real evidence, not just as a metaphor.

And there's a second analogy, just as central to this module: the shopping list. Writing the list — "milk, eggs, bread" — doesn't spend a single dollar; you can cross things out, rewrite it, add ten more items, none of it costing your card anything. The real spending only happens at one precise moment: when you get to the checkout counter and pay. Spark builds a DataFrame's transformations exactly that way — lazily, at no cost, executing nothing — until it reaches the equivalent of the checkout counter: an action. Lessons 5 through 8 of this module build this second analogy with the same kind of evidence: counting real jobs, not guessing.

Worked example: this module's map, before writing anything

Before touching pyspark, it's worth seeing the full structure of what this module is going to build — the map before the territory, exactly as module 1 did with its cost checklist.

# execution_model_map.py
EXECUTION_MODEL = [
    (2, "The driver and the executors",
        "Who decides (driver) vs who executes (executors) -- with real evidence from Spark's API"),
    (3, "RDDs: the original abstraction, seen once",
        "How Spark distributed data BEFORE the DataFrame API -- for historical contrast, never again"),
    (4, "The DataFrame API as the modern interface",
        "Why almost all Spark code today uses DataFrame, not RDD -- with Catalyst as the central reason"),
    (5, "Lazy evaluation: transformations vs actions",
        "Why building a chain of .select()/.filter() executes absolutely nothing"),
    (6, "Building a chain without running it",
        "Verified with a count of real Spark jobs -- ZERO before the action"),
    (7, "Triggering execution with an action",
        "The same job count, now greater than zero -- the checkout counter"),
    (8, "Project: first transformation chain",
        "The previous seven lessons, integrated into a single script verified end to end"),
]

print("=== Spark's execution model, before building it ===\n")
for number, title, detail in EXECUTION_MODEL:
    print(f"L{number}. {title}")
    print(f"     {detail}\n")

print("No line in this module touches fact_orders, dim_store, or dim_product yet.")
print("That work starts in module 3, on top of the mental model this module builds first.")

What to expect. Running python3 execution_model_map.py, the output is exactly this:

=== Spark's execution model, before building it ===

L2. The driver and the executors
     Who decides (driver) vs who executes (executors) -- with real evidence from Spark's API

L3. RDDs: the original abstraction, seen once
     How Spark distributed data BEFORE the DataFrame API -- for historical contrast, never again

L4. The DataFrame API as the modern interface
     Why almost all Spark code today uses DataFrame, not RDD -- with Catalyst as the central reason

L5. Lazy evaluation: transformations vs actions
     Why building a chain of .select()/.filter() executes absolutely nothing

L6. Building a chain without running it
     Verified with a count of real Spark jobs -- ZERO before the action

L7. Triggering execution with an action
     The same job count, now greater than zero -- the checkout counter

L8. Project: first transformation chain
     The previous seven lessons, integrated into a single script verified end to end

No line in this module touches fact_orders, dim_store, or dim_product yet.
That work starts in module 3, on top of the mental model this module builds first.

Notice something deliberate in this map: no Kiosko number shows up yet — not 106.15, not 40, no revenue at all. This module works over the same orders_df you already read in module 1, but the goal isn't to calculate anything new about the business — it's to understand the machinery that makes calculating anything possible, with or without real business behind it.

Diagram: where you were, where you're headed

flowchart LR
    subgraph M1["Module 1 (already done)"]
        A["Spark installed,\nverified, JAVA_HOME OK"]
        B["orders_df read,\ncount() == 40"]
    end

    subgraph M2["This module (2 of 8)"]
        C["L2: driver vs executors\n-- who does what"]
        D["L3-L4: RDD (once)\nvs DataFrame API (the real vehicle)"]
        E["L5: lazy evaluation\n-- transformations vs actions"]
        F["L6-L7: chain built unrun,\nthen triggered with an action"]
        G["L8: project -- all\nintegrated and verified"]
    end

    subgraph Resto["Modules 3-8"]
        H["fact_orders rebuilt,\nshuffle, joins, Catalyst,\nParquet, UDFs, capstone"]
    end

    A --> B --> C --> D --> E --> F --> G --> H

This module's map

Lesson    What it builds
────────  ──────────────────────────────────────────────────────────────
L1        (this one) The map: the execution model before using it
L2        The driver and the executors -- who decides, who executes
L3        RDDs -- the original abstraction, historical contrast, once
L4        The DataFrame API -- the real vehicle for the whole guide, and why
L5        Lazy evaluation -- transformations vs actions, the central distinction
L6        Building a transformation chain without triggering it
L7        Triggering execution -- the first real action, with jobs counted
L8        Project: Kiosko's first transformation chain, end to end

Lessons 2 through 4 are architecture: who executes the work, and with which API. Lessons 5 through 7 are the module's single most important concept — lazy evaluation — split into three steps: the idea (L5), build without running (L6), and actually run it (L7). Lesson 8 pulls the previous seven together into a single script, verified with evidence that no job fired before the final action.

Going deeper: why this module's order matters

It would be faster, on the surface, to jump straight to writing .select().filter().show() and see that it "works" — in fact, you already saw code like that in module 1, lesson 6, when you read Kiosko's data. But "it works" and "you understand why it works that way" are different things, and the difference gets paid for later, not before. Someone who never understood lazy evaluation is surprised, later in this guide, when they see that building a ten-million-row DataFrame (module 4) takes no time at all — because nothing gets computed yet — and then surprised again when a single call to .count() does take time, because that's where the real work happens. Without this module's mental model, each of those surprises feels like erratic Spark behavior. With it, they're exactly what you'd expect.

The same goes for RDDs versus the DataFrame API. This guide uses RDDs exactly once, in lesson 3, and never again — but understanding why the industry migrated from one to the other (Catalyst, the optimizer that can only reason about data with a known schema) is what gives you the judgment to recognize, if you ever see old Spark code built on plain RDDs, why it's probably worth rewriting with the DataFrame API.

Common mistakes

Thinking this module is "theory you can skip." What happens: someone in a hurry to reach module 3 (where revenue finally gets calculated again) decides to skip this module, assuming Spark's internal architecture doesn't matter for writing code that works. Why it happens: module 1's code already "worked" without anyone explaining driver/executors or lazy evaluation, so it seems optional. How to spot it: if you can't explain, without looking back, why building a ten-million-row DataFrame takes no time at all while a .count() over that same DataFrame does, you're missing exactly what this module teaches. How to fix it: lessons 5 through 7 aren't a side note — they're the explanation for why every line of code in the rest of the guide behaves the way it does.

Confusing "learning about RDDs" with "you're going to program with RDDs." What happens: someone sees lesson 3 (RDDs) and assumes the rest of the guide is going to alternate between the RDD API and the DataFrame API, the way older Spark courses often do. Why it happens: a lot of Spark learning material, written before the DataFrame API matured, does build full pipelines on RDDs. How to spot it: if after lesson 3 you keep seeing .map()/.filter() on RDDs in your code instead of .select()/.filter() on DataFrames in the lessons that follow, something went wrong. How to fix it: lesson 3 is, literally, the only time in this module's eight lessons (and in the rest of this eight-module guide) that you're going to see the RDD API building something — everything else uses the DataFrame API without exception, by this guide's explicit design.

Expecting spark.read.csv() or .select() to print something or take time to run. What happens: someone runs a line like filtered_df = orders_df.filter(...) and expects to see some output, or notices the line runs "instantly" and wonders whether it actually did anything. Why it happens: in ordinary Python (and in pandas), every line that transforms data does the work immediately — it's the default mental model for anyone coming from sequential scripting. How to spot it: if you're surprised that a chain of ten .filter() calls stacked on a ten-million-row DataFrame runs in milliseconds, without having read a single byte of data yet, that surprise is exactly the sign you're missing the lazy-evaluation model. How to fix it: lessons 5 through 7 of this module, with evidence from counting real jobs (not anyone's word for it), show exactly at which line the real work starts — and it isn't the .filter() line.

Exercises

Exercise 1 — Recite this module's three analogies, without looking back. Without rereading this lesson, write from memory the three central analogies this module is going to use: (a) the head chef and the line cooks, (b) the shopping list and the checkout counter, and which technical concept each one corresponds to.

See solution

(a) The head chef who never cooks directly, only decides, corresponds to the driver — your Python script, which coordinates but never touches the data — and the line cooks who actually cook correspond to the executors — the processes that actually read, filter, and transform the data. (b) The shopping list that costs nothing while you write it, and only gets paid for at the checkout counter, corresponds to lazy evaluation: transformations (.select(), .filter()) are writing the list, at no cost, and actions (.count(), .show()) are the checkout counter, the moment the real work happens.

Exercise 2 — Explain why RDDs show up only once in this guide. In 2-3 sentences, explain the technical reason (not just "because the guide's design says so") why this guide uses RDDs in a single lesson and never builds anything on them again.

See solution

The technical reason is that RDDs carry no schema — Spark has no way of knowing, just by looking at an RDD, what type each element has or what columns exist — and without that information, the Catalyst optimizer has nothing to reason about: it can't reorder filters, can't push predicates down to the data source, can't pick a cheaper physical plan. The DataFrame API, by contrast, does carry an explicit schema, and that gives Catalyst all the information it needs to optimize. That's why lesson 4 of this module exists: to show, with executed evidence, that concrete difference between the two APIs.

Exercise 3 — Predict, without running any code yet, what's going to happen. Imagine you write this line in your own script: result = orders_df.filter(col("store_id") == "S01").select("order_id", "quantity"). Without running anything, predict: does this line read any data from disk? How many Spark "jobs" does it trigger when you run it?

See solution

No, this line doesn't read any data from disk yet, and it triggers zero Spark jobs. .filter() and .select() are both transformations — Spark only records, in an internal logical plan, what was requested, without executing anything. result ends up being a DataFrame object that describes the full plan (read, filter, select), but that plan doesn't run until an action like .count(), .show(), or .collect() gets called on result. Lessons 5 through 7 of this module verify exactly this prediction with real evidence, counting Spark jobs before and after.

Summary and next step

This module opens the box module 1 deliberately left closed: how Spark works underneath. You're going to understand the driver/executors architecture with real evidence from Spark's own API (lesson 2), see why the DataFrame API replaced RDDs as the main interface, with RDDs named exactly once for historical contrast (lessons 3 and 4), and — this module's single most important point — understand and verify, with a real job count, that Spark executes nothing until you explicitly ask it to with an action (lessons 5 through 7). Lesson 8's project pulls it all together into a single chain of transformations, verified end to end.

Before moving on you should be able to: explain the difference between driver and executors without looking back; say why this guide uses RDDs exactly once; and predict, for any line of Spark code you see, whether it's a transformation (lazy, no cost) or an action (triggers real work).

Lesson 2 starts with the model's first piece: who, exactly, does the work when you run a line of PySpark.

Resources

Apache Spark — Cluster Mode Overview (official definition of driver program and executors, the foundation for lesson 2). spark.apache.org/docs/latest/cluster-overview.html.
Apache Spark — RDD Programming Guide (definition of RDD, transformations vs actions, and the exact quote on lazy evaluation, the foundation for lessons 3 and 5). spark.apache.org/docs/latest/rdd-programming-guide.html.
Apache Spark — SQL Programming Guide (DataFrame as an evolution with "richer optimizations" over RDD, the foundation for lesson 4). spark.apache.org/docs/latest/sql-programming-guide.html.
This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — this module's full objective within the eight-module plan.

← PreviousProject: Kiosko's first Spark session Next →The driver and the executors