Building a transformation chain without running it

Description

Lesson 5 gave you the concept — transformations are lazy — with a short example. This lesson takes it further: it builds a longer chain of transformations over orders_df (four chained operations), confirms again with a real job count that nothing executed, and adds a new tool that's going to matter for the rest of this guide: .explain(), which lets you see the full plan Spark built — including the Catalyst optimizer's four internal phases — without executing a single byte of real data.

Connection to the module. This lesson is the "build" half of the build/trigger pair that organizes this module's core. Lesson 7 takes exactly the same chain you build here and executes it with a real action — so what you see here, with no real number yet, is the full plan of what lesson 7 is going to produce.

An analogy: keep adding items to the list, and look it over before paying

Pick back up with lesson 5's shopping list. You already know adding items costs nothing. This lesson adds an intermediate step to that analogy: before heading to the register, you can reread your entire list — see exactly what you're about to buy, in what order you wrote it, whether there's a duplicate you could drop — and rereading it costs you nothing either. Reviewing the list isn't the same as paying for it. .explain() is exactly that review: it lets you see, in full detail, the complete plan Spark built from your chain of transformations, without viewing it triggering a single cent of real spending.

Worked example: a four-transformation chain, inspected without running

# transformation_chain_no_run.py
from pyspark.sql import SparkSession
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()

orders_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
orders_df = spark.read.csv(
    "orders_2026-08-*.csv", schema=orders_schema, header=True, enforceSchema=False,
)
status = spark.sparkContext.statusTracker()

busy_orders_df = (
    orders_df
    .select("order_id", "store_id", "product_id", "quantity", "unit_price")
    .filter(col("store_id") == "S01")
    .filter(col("quantity") > 1)
)

print(f"Jobs before .explain(): {len(status.getJobIdsForGroup())}")
busy_orders_df.explain()
print(f"Jobs after .explain(): {len(status.getJobIdsForGroup())}")

What to expect (executed in this run). The job count, first:

Jobs before .explain(): 0
Jobs after .explain(): 0

Zero before, zero after — .explain(), as lesson 5 anticipated, is neither a transformation nor an action: it only describes the plan, without executing anything over the real data. And the plan itself, printed between those two lines:

== Physical Plan ==
*(1) Filter (((isnotnull(store_id#1) AND isnotnull(quantity#3)) AND (store_id#1 = S01)) AND (quantity#3 > 1))
+- FileScan csv [order_id#0,store_id#1,product_id#2,quantity#3,unit_price#4] Batched: false, DataFilters: [isnotnull(store_id#1), isnotnull(quantity#3), (store_id#1 = S01), (quantity#3 > 1)], Format: CSV, Location: InMemoryFileIndex(7 paths)[file:/private/tmp/.../scratchpad/kiosko-spark/orders_2026-08-03.csv, ... 6 entries], PartitionFilters: [], PushedFilters: [IsNotNull(store_id), IsNotNull(quantity), EqualTo(store_id,S01), GreaterThan(quantity,1)], ReadSchema: struct<order_id:string,store_id:string,product_id:string,quantity:int,unit_price:double>

(This run's Location was abbreviated with .../ for readability — it's the real absolute path of this run's working folder, and on your own machine it's going to show your own directory, just as you already saw with JAVA_HOME in module 1.)

By default, .explain() with no arguments shows only the physical plan — the final, already-optimized plan the executors are going to run. Notice it says *(1) Filter, not *(1) Filter followed by *(1) Filter again: the two .filter() calls you wrote separately in your code merged into a single combined condition (AND), and that combined condition shows up inside PushedFilters, pushed all the way down to the FileScan itself. You didn't do this merge by hand — it's Catalyst's work, visible here even though nothing has executed yet.

Going deeper: the plan's four phases, with explain(True)

.explain() with no arguments only shows you the final result. But Catalyst builds that result in four phases, and .explain(True) shows you all of them, one by one:

busy_orders_df.explain(True)

What to expect (executed in this run, all four phases in full):

== Parsed Logical Plan ==
'Filter '`>`('quantity, 1)
+- Filter (store_id#1 = S01)
   +- Project [order_id#0, store_id#1, product_id#2, quantity#3, unit_price#4]
      +- Relation [order_id#0,store_id#1,product_id#2,quantity#3,unit_price#4,order_ts#5] csv

== Analyzed Logical Plan ==
order_id: string, store_id: string, product_id: string, quantity: int, unit_price: double
Filter (quantity#3 > 1)
+- Filter (store_id#1 = S01)
   +- Project [order_id#0, store_id#1, product_id#2, quantity#3, unit_price#4]
      +- Relation [order_id#0,store_id#1,product_id#2,quantity#3,unit_price#4,order_ts#5] csv

== Optimized Logical Plan ==
Project [order_id#0, store_id#1, product_id#2, quantity#3, unit_price#4]
+- Filter ((isnotnull(store_id#1) AND isnotnull(quantity#3)) AND ((store_id#1 = S01) AND (quantity#3 > 1)))
   +- Relation [order_id#0,store_id#1,product_id#2,quantity#3,unit_price#4,order_ts#5] csv

== Physical Plan ==
*(1) Filter (((isnotnull(store_id#1) AND isnotnull(quantity#3)) AND (store_id#1 = S01)) AND (quantity#3 > 1))
+- FileScan csv [order_id#0,store_id#1,product_id#2,quantity#3,unit_price#4] Batched: false, DataFilters: [isnotnull(store_id#1), isnotnull(quantity#3), (store_id#1 = S01), (quantity#3 > 1)], Format: CSV, Location: InMemoryFileIndex(7 paths)[...], PartitionFilters: [], PushedFilters: [IsNotNull(store_id), IsNotNull(quantity), EqualTo(store_id,S01), GreaterThan(quantity,1)], ReadSchema: struct<order_id:string,store_id:string,product_id:string,quantity:int,unit_price:double>

These four phases are, literally, the "logical and physical DAG" this module promised from its introduction — and now you're seeing it with your own eyes, in your own terminal, not as an abstract diagram:

1. Parsed Logical Plan — the almost-direct translation of your Python code into a tree of operations. Notice the detail: it says 'Filter '>('quantity, 1) with single quotes in front of the names — those quotes mean that, at this phase, Spark hasn't confirmed yet that quantity is a real column or that the > comparison is valid. It's, literally, the first draft, before checking anything against the real schema.
2. Analyzed Logical Plan — Spark already resolved the column names against orders_df's real schema (quantity#3 now has a confirmed type: int), and confirmed the block's first line — the output schema, order_id: string, store_id: string, ... — is valid. This is the phase where a nonexistent-column error (col("quantitty"), misspelled) would get caught, if there were one.
3. Optimized Logical Plan — here's where you see, with your own eyes, the merge of the two separate .filter() calls into a single combined condition (AND), and the appearance of the automatic isnotnull(...) checks Spark adds before each comparison — a safety optimization your code never explicitly asked for, but that Catalyst adds because it knows comparing a null value produces ambiguous results.
4. Physical Plan — the final plan, the same one you already saw with .explain() with no arguments: it decides how, specifically, to execute the optimized version (FileScan with PushedFilters, instead of a full scan followed by a separate filter).

None of these four phases required reading a single byte of the CSV file. All the "thinking through the plan" work happens in the driver, over metadata and over the schema — completely separate from the "executing the plan" work, which only happens when an action arrives, as this same lesson's job count confirms: 0 before, 0 after the two calls to .explain().

Diagram: from your code to Catalyst's four phases

flowchart TD
    A["Your Python code:\n.select().filter().filter()"] --> B["Parsed Logical Plan\n('quantity not confirmed yet)"]
    B --> C["Analyzed Logical Plan\n(names and types already resolved\nagainst the real schema)"]
    C --> D["Optimized Logical Plan\n(Catalyst merges the two filter()\ninto an AND, adds isnotnull())"]
    D --> E["Physical Plan\n(FileScan + PushedFilters --\nthe plan that would actually run)"]
    E -.->|"all of this: ZERO jobs,\nZERO bytes read"| F(("Only an action\ntriggers real\nexecution -- Lesson 7"))

    style F fill:#f96,stroke:#333,stroke-width:3px

Common mistakes

Thinking explain(True) "executes" the plan in order to show it. What happens: someone sees explain(True)'s four detailed phases — with column names resolved, types confirmed, optimizations applied — and assumes that, to show that much detail, Spark must have read the real data. Why it happens: that much detail feels like the result of real work, not just "thinking" about metadata. How to spot it: this lesson's worked example's job count directly disproves it — 0 jobs before and after explain(True). How to fix it: remember that everything Catalyst needs to build and optimize these four phases is the schema (column names, types) and the file metadata (how many files there are, where they are) — neither requires reading the real content of a single row.

Confusing your code's order with the real execution order. What happens: someone writes .select(...).filter(A).filter(B) and assumes Spark executes, in that exact order, first the select, then filter A, then filter B, as separate sequential steps. Why it happens: in ordinary sequential programming, code order is execution order. How to spot it: look at this lesson's worked example's Optimized Logical Plan — the two .filter() calls show up merged into one single combined condition with AND, not as two separate steps. How to fix it: think of your transformation chain as a description of intent, not step-by-step instructions — Catalyst has total freedom to reorganize, merge, or even reorder operations (within the bounds of the final result being mathematically equivalent), and the optimized plan is the visible proof it exercises that freedom.

Ignoring PushedFilters because "I already know the filter works." What happens: someone checks that a query's final result is correct (the count, the values) but never checks whether the filters made it into PushedFilters in the physical plan — assuming that, if the result is correct, how it got there doesn't matter. Why it happens: a correct result feels like sufficient validation. How to spot it: on small data like Kiosko's forty records, the difference between a pushed filter and a non-pushed one is invisible in the result — both give the same number. How to fix it: get used to checking PushedFilters in the physical plan as part of your routine, not only when something goes wrong — at real volumes (module 4's ten-million-row synthetic dataset), a filter that doesn't get pushed down to the data source forces Spark to move into memory rows that were going to get discarded anyway — a real cost the .explain() plan reveals before the problem shows up in production.

Exercises

Exercise 1 — Add a third condition and confirm it merges too. Extend the worked example's chain with a third .filter(col("unit_price") < 1.0), and confirm with .explain() that the three conditions show up merged into a single AND expression in the Optimized Logical Plan and in PushedFilters.

three_filters_df = (
    orders_df
    .select("order_id", "store_id", "product_id", "quantity", "unit_price")
    .filter(col("store_id") == "S01")
    .filter(col("quantity") > 1)
    .filter(col("unit_price") < 1.0)
)
three_filters_df.explain()

== Physical Plan ==
*(1) Filter (((((isnotnull(store_id#1) AND isnotnull(quantity#3)) AND isnotnull(unit_price#4)) AND (store_id#1 = S01)) AND (quantity#3 > 1)) AND (unit_price#4 < 1.0))
+- FileScan csv [...] PushedFilters: [IsNotNull(store_id), IsNotNull(quantity), IsNotNull(unit_price), EqualTo(store_id,S01), GreaterThan(quantity,1), LessThan(unit_price,1.0)], ...

Exercise 2 — Confirm with a job count that adding an .orderBy() still executes nothing. Add .orderBy("order_id") to the end of the worked example's chain, and confirm with statusTracker() that the job count is still zero after adding it (despite .orderBy() being a more expensive operation than .filter(), as you're going to see in module 4 about shuffle).

status = spark.sparkContext.statusTracker()
print(f"Jobs before: {len(status.getJobIdsForGroup())}")

ordered_df = busy_orders_df.orderBy("order_id")
print(f"Jobs after adding .orderBy() (no action): {len(status.getJobIdsForGroup())}")

Jobs before: 0
Jobs after adding .orderBy() (no action): 0

Exercise 3 — Explain, without code, why the Parsed Logical Plan uses single quotes and the Analyzed Logical Plan doesn't. In 2-3 sentences, explain the exact difference between those first two plan phases, using this lesson's worked example's evidence.

Summary and next step

In this lesson you built a four-transformation chain over orders_df without executing anything — confirmed, again, with a real job count at 0 — and learned .explain(), both in its simple form (just the final physical plan) and its full form (explain(True), all four phases: Parsed, Analyzed, Optimized, Physical). You saw, with your own eyes and not a diagram, how Catalyst merges separate filters into a single condition and pushes them down to the data source (PushedFilters) — all of this without reading a single real byte of Kiosko data.

Before moving on you should be able to: call .explain() and .explain(True) from memory; name the plan's four phases in order; and explain why two separate .filter() calls in your code can show up merged as one in the optimized plan.

The chain you built in this lesson — busy_orders_df — still hasn't touched a single piece of real data. Lesson 7 takes exactly this same chain and finally triggers it with an action: you're going to see the same job count climb from zero to a real number, and you're going to see Kiosko's real result for the first time in this module.

Resources

• Apache Spark — SQL Programming Guide (the conceptual foundation for Catalyst as the optimizer that produces these four plan phases). spark.apache.org/docs/latest/sql-programming-guide.html.
• Apache Spark — SQL Performance Tuning (documentation for .explain(mode=...) and its different modes, a central reference for module 6 of this guide, where Catalyst gets covered in depth). spark.apache.org/docs/latest/sql-performance-tuning.html.
• Apache Spark — RDD Programming Guide (the lazy-evaluation definition this lesson extends with .explain(), already cited in lesson 5). spark.apache.org/docs/latest/rdd-programming-guide.html.