Lazy evaluation: transformations vs actions

Description

This is the single most important lesson in this whole module, and probably in this entire guide. Spark splits every operation you ask for into two completely different categories: transformations, which do no real work at all, and actions, which do. Confusing the two — or not knowing, faced with a new method, which category it falls into — is the most common source of surprises when learning Spark. This lesson doesn't ask you to memorize a list: it gives you the criterion to recognize the difference yourself, and verifies it with real evidence — a count of Spark jobs, before and after each operation — not anyone's word for it.

Connection to the module. Lessons 2 through 4 of this module gave you the "who" (driver/executors) and the "with what" (DataFrame API over RDDs). This lesson gives you the "when": exactly at what moment, within a chain of Spark code, real work happens. Lessons 6 and 7 split this same idea into two practical parts — building without executing, and triggering execution — and lesson 8 verifies it end to end in a single project.

An analogy: the shopping list and the checkout counter

Imagine you're about to do the week's grocery shopping. You write on your phone: "milk." That costs nothing — it's just text in a note. You add "eggs." Still costs nothing. You add "bread," "coffee," "ten oranges." You can keep adding items, cross some out, change your mind, for hours if you want, and your credit card never finds out about any of it. The list, however long, is free as long as it only exists as a list.

The real spending happens at one very specific, very different moment: when you get to the supermarket checkout counter, put the items on the belt, and the cashier charges you. There — and only there — the money leaves your account. Everything that happened before (writing, crossing out, reordering the list) was planning; what happens at the register is execution.

Spark splits work exactly like this. Transformations — .select(), .filter(), .withColumn(), .join(), .groupBy() (without .agg() yet) — are writing the list: they describe what you want, at no cost, without reading a single byte of real data. Actions — .count(), .show(), .collect(), .write.parquet(...) — are the checkout counter: the exact moment Spark finally reads the data, executes every accumulated transformation, and produces a real result.

Worked example: counting jobs, not guessing

The right way to verify this idea isn't "trust that it takes no time" — that's a stopwatch in disguise, and this guide doesn't use stopwatches to measure anything. The right way is to count, with Spark's own API, how many jobs (real jobs, the same ones that show up in the "Jobs" tab of the Spark UI at localhost:4040) have fired in the session, before and after each operation.

# lazy_jobcount.py
from pyspark.sql import SparkSession
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()

orders_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
orders_df = spark.read.csv(
    "orders_2026-08-*.csv", schema=orders_schema, header=True, enforceSchema=False,
)

status = spark.sparkContext.statusTracker()
print(f"Jobs before building the chain: {len(status.getJobIdsForGroup())}")

selected_df = orders_df.select("order_id", "store_id", "product_id", "quantity", "unit_price")
filtered_df = selected_df.filter(col("store_id") == "S01")
filtered_df = filtered_df.filter(col("quantity") > 1)

print(f"Jobs after select()+filter()+filter() (no action yet): {len(status.getJobIdsForGroup())}")
print(f"type(filtered_df) = {type(filtered_df)}")

result_count = filtered_df.count()

print(f"Jobs after .count() (one action): {len(status.getJobIdsForGroup())}")
print(f"filtered_df.count() = {result_count}")

spark.stop()

spark.sparkContext.statusTracker() is the entry point, from Python, into the same information that feeds the Spark UI's "Jobs" tab — per Spark's official monitoring documentation, the Spark UI and its REST API expose, among other things, /applications/[app-id]/jobs, "a list of all jobs for a given application." getJobIdsForGroup() returns the list of IDs for every job triggered so far in the session — counting how many there are, before and after each line, is a check as direct as opening the Spark UI in a browser, only programmatic and reproducible in a script.

What to expect. Running python3 lazy_jobcount.py, the output is exactly this (executed in this run):

Jobs before building the chain: 0
Jobs after select()+filter()+filter() (no action yet): 0
type(filtered_df) = <class 'pyspark.sql.classic.dataframe.DataFrame'>
Jobs after .count() (one action): 2
filtered_df.count() = 10

This is exactly the evidence this lesson promises. Three chained transformations — one .select() and two .filter() calls — leave the job count at zero, identical to where it was before even touching orders_df. filtered_df is a real DataFrame object, fully built, with its execution plan already defined — but that plan never ran. Only when .count() is called, an action, does the job count jump from 0 to 2, and filtered_df.count() finally produces the real number: 10 orders from S01 with quantity > 1, the exact same result you already verified in lessons 2 and 4 of this module from other angles.

(Why 2 jobs and not 1, for a single action? It's an honest observation that deserves an honest answer, not an oversimplification: since Spark 3.2, Adaptive Query Execution — AQE, on by default — can split even a seemingly simple operation like .count() into more than one job, because it replans parts of the work with real execution statistics instead of only prior estimates. Module 6 of this guide explains AQE in depth. For now, the only thing that matters — and what this lesson demonstrates with real evidence — is the binary distinction: zero jobs before the action, more than zero after. Exactly how many jobs a specific action triggers is an implementation detail that can vary; that transformations never trigger any, doesn't.)

Diagram: the exact moment real work starts

flowchart LR
    A["orders_df = spark.read.csv(...)\nTRANSFORMATION -- 0 jobs"] --> B["selected_df = orders_df.select(...)\nTRANSFORMATION -- 0 jobs"]
    B --> C["filtered_df = selected_df.filter(...)\nTRANSFORMATION -- 0 jobs"]
    C --> D["filtered_df = filtered_df.filter(...)\nTRANSFORMATION -- 0 jobs"]
    D -.->|"up to here: PLAN ONLY,\nnot a single byte read"| E(("ACTION:\n.count()"))
    E -->|"NOW:\nreads, filters, counts for real"| F["result = 10\njobs > 0"]

    style E fill:#f96,stroke:#333,stroke-width:3px

Going deeper: the official quote, and how to recognize a transformation from an action without memorizing lists

The RDD Programming Guide — the same source you already cited in lesson 3 for RDDs — describes lazy evaluation with a precision that applies equally well to DataFrames: "All transformations in Spark are lazy, in that they do not compute their results right away. Instead, they just remember the transformations applied to some base dataset (e.g. a file). The transformations are only computed when an action requires a result to be returned to the driver program."

That last sentence contains the criterion you need, with no list to memorize: an operation is an action if its result has to go back to the driver (a number, a list of rows, confirmation that a file got written) — and it's a transformation if its result is still, itself, another DataFrame or RDD describing more pending work. .filter(...) returns a DataFrame — it's still a plan, nothing went back to the driver yet. .count() returns a Python integer — something had to go back to the driver, so something had to actually execute to produce that integer.

With that criterion, you can classify almost any new method you run into in the rest of this guide without having to memorize it ahead of time:

You're going to use this same table, mentally, every time you see a new method in the rest of this guide — including .explain(), which you'll see in lesson 6 strictly belongs to neither category (it doesn't transform the DataFrame or execute it; it only describes it), a special case the next lesson clarifies with evidence.

Common mistakes

Expecting a transformation to "take time" if the DataFrame is large. What happens: someone builds a long chain of .filter()/.select()/.withColumn() on a DataFrame that, in theory, represents millions of rows (like module 4's kiosko_orders_at_scale), and expects the script to take time running that part, even though there's no action yet. Why it happens: the intuition of "more data, more time" is correct for any operation that genuinely processes data — but a transformation, no matter how many rows the DataFrame represents, processes none at all. How to spot it: if your script takes seconds or minutes on a line that only has .filter()/.select(), with no .count()/.show()/.collect() in between, something else is going on (maybe a hidden action, as the next mistake shows). How to fix it: remember this lesson's criterion — if the line's result is still a DataFrame, it's a transformation, and a transformation never executes anything over the real data, no matter what volume the DataFrame represents.

Not realizing that a plain Python print(df) CAN trigger work. What happens: someone writes print(orders_df) (instead of orders_df.show()) expecting to see a preview of the data, without knowing whether that counts as an action. Why it happens: print() on almost any Python object shows information about its contents, so it seems reasonable to expect the same from a DataFrame. How to spot it: try print(orders_df) yourself — the output is something like DataFrame[order_id: string, store_id: string, ...], just the schema, not the data. print() on a DataFrame calls its __repr__ method, which only describes the structure — it triggers no job at all, exactly like a transformation. How to fix it: to see real data, you always need an explicit action: .show() for a formatted console preview, .collect() to bring everything back into a Python list (with the memory risk that implies over large data), or .count() for just the row count.

Assuming two actions in a row automatically reuse the first one's work. What happens: someone calls .count() and then .show() on the same filtered DataFrame, and assumes the second action is "free" because the filter's result "already got computed" in the first one. Why it happens: it seems reasonable to think that, once Spark read and filtered the data for the .count(), that work stays available for the next action without repeating. How to spot it: this lesson's worked example, and lesson 8's project, show with the same job-counting technique that each action triggers its own fresh work — the job count keeps climbing with every additional action, it doesn't stop after the first one. How to fix it: by default, Spark doesn't remember intermediate results between actions — every action re-executes the whole plan from the data source, unless you explicitly use .cache()/.persist() to ask Spark to actually keep an intermediate result in memory. Module 6 of this guide (Catalyst, .explain(), and caching) teaches exactly when that decision is worth it and when it isn't.

Exercises

Exercise 1 — Classify five operations without running them. Without running any code, using only this lesson's criterion ("does the result go back to the driver, or is it still a DataFrame/RDD?"), classify these five operations as transformation or action: .distinct(), .limit(5), .toPandas(), .printSchema(), .repartition(4).

Exercise 2 — Verify your classification with a real job count. Using this lesson's worked example's statusTracker() pattern, verify with real code whether .distinct() and .toPandas() trigger jobs or not, confirming (or correcting) your answer from exercise 1.

status = spark.sparkContext.statusTracker()

print(f"Jobs before: {len(status.getJobIdsForGroup())}")
distinct_df = orders_df.select("store_id").distinct()
print(f"Jobs after .distinct() (no action): {len(status.getJobIdsForGroup())}")

pdf = distinct_df.toPandas()
print(f"Jobs after .toPandas(): {len(status.getJobIdsForGroup())}")
print(pdf)

Jobs before: 0
Jobs after .distinct() (no action): 0
Jobs after .toPandas(): 2
  store_id
0      S02
1      S01
2      S03

Exercise 3 — Explain, without code, the print(df) trap. In 2-3 sentences, explain why print(orders_df) doesn't show Kiosko's real data, even though print() normally shows the content of any other Python object.

Summary and next step

In this lesson you built this module's single most important concept: transformations (.select(), .filter(), .withColumn(), and everything that returns another DataFrame) are lazy and execute nothing, while actions (.count(), .show(), .collect(), and everything that returns a real result to the driver) are the ones that trigger real work. You verified it with a real count of Spark jobs — the same information that feeds the Spark UI's "Jobs" tab — not intuition: zero jobs after three chained transformations, real jobs only after the first action.

Before moving on you should be able to: classify any Spark method as transformation or action using the "does the result go back to the driver?" criterion; explain why print(df) executes nothing; and explain why two actions in a row on the same filtered DataFrame repeat the work, instead of reusing the first one's result.

Lessons 6 and 7 split this same idea into two longer, more practical steps: lesson 6 builds a more complete transformation chain — including .explain() to inspect the plan without executing it — and confirms, again with evidence, that nothing fired; lesson 7 takes that same chain and finally executes it with a real action.

Resources

• Apache Spark — RDD Programming Guide (the exact quote on lazy evaluation: "All transformations in Spark are lazy... The transformations are only computed when an action requires a result to be returned to the driver program"). spark.apache.org/docs/latest/rdd-programming-guide.html.
• Apache Spark — Monitoring and Instrumentation (the Spark UI and its REST API, including /applications/[app-id]/jobs, the source of the information statusTracker() exposes from Python). spark.apache.org/docs/latest/monitoring.html.
• Apache Spark — SQL Performance Tuning (Adaptive Query Execution, on by default since Spark 3.2, mentioned in this lesson as the reason a single action can trigger more than one job — covered in depth in module 6). spark.apache.org/docs/latest/sql-performance-tuning.html.