Triggering execution with an action

Description

It's time for the checkout counter. Lesson 6 built busy_orders_df — four chained transformations over Kiosko's forty orders — and confirmed, with a real job count, that nothing had executed. This lesson takes that exact same chain and triggers it with two different actions: .show() first, .count() after, showing in each case how the job count climbs and, finally, letting you see the real result.

Connection to the module. This lesson closes out the build/trigger pair lesson 6 started. With this lesson, you now have all three pieces of this module's execution model complete: who executes (lesson 2), which API to use (lessons 3-4), and when it executes (lessons 5-7). Lesson 8 pulls them all together into a single verified project.

An analogy: finally reaching the register

You already wrote the full list (busy_orders_df, with its four transformations) and already reviewed it at no cost (.explain(), in lesson 6). Now you walk up to the checkout counter, put the items on the belt, and the cashier starts scanning them one by one. This is the moment the real work happens: each item gets scanned, the price adds up, and at the end you get a total — a concrete result that didn't exist before. .show() and .count() are two different ways of "paying" for the same list: one shows you the contents of the bag (.show(), seeing the rows), the other just gives you the total (.count(), just the number).

Worked example: triggering the same chain with two different actions

# trigger_action.py
from pyspark.sql import SparkSession
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()
status = spark.sparkContext.statusTracker()

orders_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
orders_df = spark.read.csv(
    "orders_2026-08-*.csv", schema=orders_schema, header=True, enforceSchema=False,
)

busy_orders_df = (
    orders_df
    .select("order_id", "store_id", "product_id", "quantity", "unit_price")
    .filter(col("store_id") == "S01")
    .filter(col("quantity") > 1)
)

print(f"Jobs before show(): {len(status.getJobIdsForGroup())}")
busy_orders_df.show()
print(f"Jobs after show(): {len(status.getJobIdsForGroup())}")

print()
n = busy_orders_df.count()
print(f"busy_orders_df.count() = {n}")
print(f"Jobs after count(): {len(status.getJobIdsForGroup())}")

spark.stop()

What to expect. Running python3 trigger_action.py, the output is exactly this (executed in this run):

Jobs before show(): 0
+--------+--------+----------+--------+----------+
|order_id|store_id|product_id|quantity|unit_price|
+--------+--------+----------+--------+----------+
|ORD-6001|     S01|      P001|       6|      0.55|
|ORD-6004|     S01|      P004|       2|       4.5|
|ORD-1001|     S01|      P001|       3|      0.55|
|ORD-1008|     S01|      P001|       2|      0.55|
|ORD-5001|     S01|      P002|       2|       1.2|
|ORD-5004|     S01|      P003|       2|      0.75|
|ORD-5007|     S01|      P001|       2|      0.55|
|ORD-2004|     S01|      P003|       3|      0.75|
|ORD-4001|     S01|      P001|       4|      0.55|
|ORD-7001|     S01|      P001|       2|      0.55|
+--------+--------+----------+--------+----------+

Jobs after show(): 3

busy_orders_df.count() = 10
Jobs after count(): 5

This confirms exactly what you expected after lesson 5: 0 jobs before the first action, and the count climbs after each action — 3 after .show(), 5 after .count(). Notice something important: the count doesn't reset to zero between the two actions — it keeps accumulating, because getJobIdsForGroup() reports every job for the whole session, not just the last call. What matters isn't the exact number (which, as you already saw in lesson 5, can vary due to Adaptive Query Execution), but the direction: every new action adds new work, never zero.

And the result itself — the ten orders from S01 with quantity > 1 — is, finally, real Kiosko data, not just a plan. You can verify it by hand against data you already know from module 1: S01 has sixteen orders total (verified in the module 1 project); of those sixteen, the ones shown here are exactly the ones with quantity greater than one.

Diagram: the same plan, two ways of paying for it

flowchart TD
    A["busy_orders_df\n(full plan, 0 jobs)"] --> B{"Which action?"}
    B -->|".show()"| C["Executes the plan,\nformats the first 20 rows\nas a table in the console"]
    B -->|".count()"| D["Executes the plan (again,\nfrom the source -- doesn't reuse\nshow()'s work)"]
    C --> E["Visible result:\n10 rows on screen"]
    D --> F["Visible result:\na single integer: 10"]

    style B fill:#f96,stroke:#333,stroke-width:3px

Going deeper: why the result doesn't come sorted, and what that means

Look again at .show()'s result: the first row is ORD-6001 (from Saturday, August 8), not ORD-1001 (from Monday, August 3, the numerically lowest order_id). This isn't a bug — it's the direct consequence of something you already learned in module 1: Kiosko's seven files get read as separate partitions, and without an explicit .orderBy(), Spark doesn't guarantee any particular order for the resulting rows. The order you see here reflects the order in which the executors finished processing each partition (in this case, each of the seven files), which in turn depends on the order InMemoryFileIndex listed them in — an internal detail that can, in principle, vary between runs or between machines, though in practice, over static data like Kiosko's, it tends to be stable within the same machine.

This has a direct practical consequence for the rest of this guide: any time row order matters for a result — for example, showing "the first five orders of the day" — you're going to need an explicit .orderBy(), as you already saw module 1 do (orders_df.orderBy("order_id").show(5)). Without it, "the first rows .show() displays" and "the first rows by some business criterion" are two completely different things, and confusing them is a silent source of bugs.

It's also worth noting the cost of this comparison: adding an .orderBy("order_id") to this lesson's chain, though technically still a transformation (it triggers nothing on its own, as you already confirmed in lesson 6's exercise 2), does trigger a more expensive job once the action finally arrives — a shuffle, the central topic of module 4 of this guide. Over forty rows, that cost is invisible; over the ten-million-row synthetic dataset you're going to build in module 4, it isn't at all.

Common mistakes

Assuming .show() with no arguments always shows every row. What happens: someone runs .show() on a result bigger than twenty rows, and is surprised when the output cuts off with the line only showing top 20 rows (or whatever number applies), assuming something went wrong. Why it happens: in many data analysis tools (like pandas), the default behavior when printing a DataFrame varies, and there isn't always such an explicit limit. How to spot it: if your result has more than twenty rows and .show() doesn't show all of them, it isn't a bug — .show(), by default, shows only the first 20 rows (you can ask for more with .show(n), as in module 1's .show(5)). How to fix it: to see more rows, pass the explicit number (.show(50)); to confirm the total count without seeing every row, use .count() instead — exactly the pattern this lesson's worked example follows, showing both tools side by side.

Being surprised that .count() "repeats" the work .show() already did. What happens: someone notices, with this lesson's worked example's same job count, that .count() triggered more new jobs after .show() had already triggered its own, and it feels wasteful — "couldn't Spark remember it already read and filtered this data?" Why it happens: it seems reasonable to expect that, once a result gets computed, it stays available for the next operation. How to spot it: this lesson's job count confirms it directly — it climbed from 0 to 3 with .show(), and climbed again from 3 to 5 with .count(), with no drop in between. How to fix it: by default, this is how Spark works — every action re-executes the full plan from the source. If you know ahead of time you're going to call several actions on the same filtered DataFrame, .cache() (module 6 of this guide) is the tool designed exactly to avoid this repetition, at the cost of memory in exchange — a judgment call, not something Spark does automatically for you.

Reading .show()'s order as meaningful, with no .orderBy(). What happens: someone looks at this lesson's .show() result — ORD-6001 first — and incorrectly assumes it's "the first order" in some business sense (the oldest, the largest quantity, or any other criterion). Why it happens: when something shows up first in a list, it's natural to assume that order means something. How to spot it: check whether your transformation chain includes an explicit .orderBy() — if it doesn't, any order you see is an internal execution detail, not a guarantee. How to fix it: never assume any order without an explicit .orderBy() in your own code — this lesson demonstrates it with real evidence, showing the unsorted result exactly as Spark produced it.

Exercises

Exercise 1 — Add .orderBy("order_id") and compare the result. Repeat this lesson's worked example, but with .orderBy("order_id") added to the end of the chain, before .show(). Confirm that now ORD-1001 (or the lowest order_id in the filtered result) shows up first.

ordered_busy_df = busy_orders_df.orderBy("order_id")
ordered_busy_df.show()

+--------+--------+----------+--------+----------+
|order_id|store_id|product_id|quantity|unit_price|
+--------+--------+----------+--------+----------+
|ORD-1001|     S01|      P001|       3|      0.55|
|ORD-1008|     S01|      P001|       2|      0.55|
|ORD-2004|     S01|      P003|       3|      0.75|
|ORD-4001|     S01|      P001|       4|      0.55|
|ORD-5001|     S01|      P002|       2|       1.2|
|ORD-5004|     S01|      P003|       2|      0.75|
|ORD-5007|     S01|      P001|       2|      0.55|
|ORD-6001|     S01|      P001|       6|      0.55|
|ORD-6004|     S01|      P004|       2|       4.5|
|ORD-7001|     S01|      P001|       2|      0.55|
+--------+--------+----------+--------+----------+

Exercise 2 — Confirm .first() is also an action, with a job count. As a new script, independent of the worked example (a freshly opened SparkSession, at 0 jobs), use this lesson's statusTracker() pattern to verify whether .first() (which returns only the first row as a Row object) triggers new jobs, as lesson 5's table predicts.

status = spark.sparkContext.statusTracker()
print(f"Jobs before: {len(status.getJobIdsForGroup())}")

first_row = busy_orders_df.first()
print(f"Jobs after .first(): {len(status.getJobIdsForGroup())}")
print(f"First row: {first_row}")

Jobs before: 0
Jobs after .first(): 1
First row: Row(order_id='ORD-6001', store_id='S01', product_id='P001', quantity=6, unit_price=0.55)

Exercise 3 — Explain, without code, why .show() and .count() don't reuse work between each other. In 2-3 sentences, and without looking at the common mistakes section, explain why Spark, by default, re-executes the full plan for .count() after already running it for .show() on the same DataFrame.

Summary and next step

In this lesson you finally triggered the transformation chain you built in lesson 6, with two different actions: .show() and .count(). You confirmed, with the same real job count from the previous lessons, that every action triggers new work — climbing from 0 to 3, and from 3 to 5 — and got, for the first time in this module, a real Kiosko result: the ten orders from S01 with quantity > 1. You also learned that, without an explicit .orderBy(), row order isn't guaranteed — a detail worth remembering for the rest of this guide.

Before moving on you should be able to: explain why the job count climbs with every action, without resetting to zero in between; explain why .show()'s result doesn't come sorted without an explicit .orderBy(); and predict, for any new action you see (.first(), .take(n), .toPandas()), whether it's going to trigger real work using lesson 5's criterion.

With all three pieces of the execution model complete — driver/executors, the DataFrame API, and lazy evaluation with transformations vs actions — lesson 8 pulls them all together into a single project: a complete transformation chain, verified end to end with evidence that it triggers nothing until the exact moment you ask it to.

Resources

• Apache Spark — RDD Programming Guide (the definition of action already cited in lesson 5, now applied with executed evidence to .show() and .count()). spark.apache.org/docs/latest/rdd-programming-guide.html.
• Apache Spark — Monitoring and Instrumentation (the source of statusTracker() and the Spark UI's "Jobs" tab, used to verify the job count throughout this lesson). spark.apache.org/docs/latest/monitoring.html.
• Apache Spark — SQL Getting Started (the .show() pattern over Spark results, already used in module 1). spark.apache.org/docs/latest/sql-getting-started.html.