The driver and the executors

Description

Every time you write spark.read.csv(...) or orders_df.filter(...), that line runs in a specific process — the driver — but the heavy work (reading bytes from a file, evaluating a condition row by row) never happens there. It happens in one or more separate processes, the executors. This lesson doesn't ask you to take that claim on faith: you're going to verify it with Spark's own API, including the same REST API that powers the Spark UI at localhost:4040 mentioned in module 1.

Connection to the module. This is the module's first lesson that opens the box on Spark's internal architecture. Lessons 3 through 8 take this distinction as given — in particular, "who executes this?" is going to be the first question you ask yourself about any new line of Spark code in the rest of the guide.

An analogy: the head chef and the line cooks

In a large professional kitchen, the head chef — the executive chef — almost never chops an onion, flips a pan, or plates a dish with their own hands. Their job is to decide: what gets cooked, in what order, which line cook handles which station. The line cooks are the ones with their hands actually in the food, each at their own station, with their own set of pans and their own piece of the mise en place. If the restaurant has one small kitchen, the head chef sometimes ends up helping with their hands too — but conceptually it's still the same person playing two roles, not two different people.

Spark works with this same division of labor. The driver is your Python script: the process that runs your code, decides which operations to apply and in what order, coordinates everything — but, like the head chef, isn't the one moving data byte by byte. The executors are the processes that do: they read files, evaluate .filter() conditions, compute new columns, and return results. And, like the small restaurant where the chef ends up helping with their hands, when you run Spark on local[*] — as this whole guide does — the driver and the (single) executor end up living inside the same JVM process. You're going to verify this with evidence, not just the analogy, in this lesson's "going deeper" section.

Worked example: inspecting the driver from the inside

# driver_and_executors.py
from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("kiosko-spark")
    .master("local[*]")
    .getOrCreate()
)
sc = spark.sparkContext

print(f"Master: {sc.master}")
print(f"App name: {sc.appName}")
print(f"App ID: {sc.applicationId}")
print(f"Default parallelism (available task slots): {sc.defaultParallelism}")

status = sc.statusTracker()
print(f"Executor infos: {status.getExecutorInfos()}")

spark.stop()

What to expect. Running python3 driver_and_executors.py, the output is exactly this (executed in this run, PySpark 4.2.0, Java 17):

Master: local[*]
App name: kiosko-spark
App ID: local-1786655016713
Default parallelism (available task slots): 12
Executor infos: [SparkExecutorInfo(host='localhost', port=56666, cacheSize=0, numRunningTasks=0, usedOnHeapStorageMemory=0, usedOffHeapStorageMemory=0, totalOnHeapStorageMemory=455501414, totalOffHeapStorageMemory=0)]

Three things worth looking at closely in this output. First, App ID (local-1786655016713) is unique per run — the numeric suffix is an internal Spark timestamp, so yours is going to be different every time you run this script, even on the same machine; what matters is that it starts with local-, confirming this application runs in local mode. Second, Default parallelism (12 on this machine) is going to vary based on your own machine's logical cores — the same local[*] behavior you already saw in module 1. Third, and most important for this lesson: getExecutorInfos() returns a list with a single element — one single executor, running on localhost. That isn't an error or a demo simplification: it's exactly what the small-kitchen analogy predicts — in local[*], there's one process playing both roles.

Diagram: who does what, inside the same process

┌───────────────────────────────────────────────────────────────────┐
│                     A single JVM process                           │
│                                                                     │
│   ┌─────────────────────┐          ┌───────────────────────────┐  │
│   │       DRIVER          │          │        EXECUTOR           │  │
│   │  (your Python script,│  ─────>  │  (processes the data:     │  │
│   │   via SparkSession)  │  coordinates│  reads, filters, groups,│  │
│   │                       │          │   stores partial          │  │
│   │  - decides the plan   │  <─────  │   results)                 │  │
│   │  - builds the DAG     │  result  │  - 12 task threads         │  │
│   │  - never touches a    │          │    (defaultParallelism)   │  │
│   │    single byte of data│          │  - a single process in    │  │
│   │    directly            │          │    local[*], "id": driver │  │
│   └─────────────────────┘          └───────────────────────────┘  │
│                                                                     │
│   On a real cluster (outside this guide): the driver runs in its   │
│   own process, and each executor runs on its own separate node     │
└───────────────────────────────────────────────────────────────────┘

Going deeper: the definitive proof, with the Spark UI itself

Module 1 mentioned that Spark spins up a web interface at localhost:4040 while the SparkSession is active. That interface isn't just a visual panel — per Spark's official monitoring documentation, all the information it shows is also available as a REST API, at http://localhost:4040/api/v1/, with endpoints like /applications/[app-id]/executors. That API is the source of truth the Spark UI itself queries to draw its "Executors" tab — so querying it directly, while a SparkSession is still running, is the most direct possible evidence of which processes actually exist.

This is exactly what was done to verify this lesson: a SparkSession was left running in the background, and its REST API was queried with curl:

curl -s "http://localhost:4040/api/v1/applications/local-1786655109935/executors"

What to expect (verified in this run, full JSON response from Spark's REST API, formatted):

[ {
  "id" : "driver",
  "hostPort" : "localhost:56882",
  "isActive" : true,
  "totalCores" : 12,
  "maxTasks" : 12,
  "activeTasks" : 0,
  "completedTasks" : 0,
  "totalTasks" : 0,
  "maxMemory" : 455501414,
  "addTime" : "2026-08-13T21:05:09.959GMT"
} ]

Look closely at the "id" field: it literally says "driver". It isn't a generic placeholder name — it's the real identifier Spark assigns to this process within its own executor-tracking system, and it confirms exactly what the small-kitchen analogy predicted: in local[*] mode, the process that acts as the application's only executor is the driver process itself, registered under its own name in the list of executors. The fields "totalCores": 12 and "maxTasks": 12 match the defaultParallelism you already saw with sc.defaultParallelism — the Spark UI, the REST API, and statusTracker() are all reporting, from three different angles, exactly the same data.

This has an important practical consequence for the rest of this guide: when module 8 mentions, without building it, how this same code would run on a real managed cluster (EMR, Databricks), the central difference is exactly this — on a real cluster, the driver runs on its own node, and each executor runs on separate nodes, with independent JVM processes communicating over the network. The PySpark code you write doesn't change at all between local[*] and a real cluster; what changes is how many separate processes there are behind it, and across how many distinct physical machines they live.

Common mistakes

Assuming that, in local[*], "there are no executors" because everything runs in a single process. What happens: someone notices the driver and the executor coexist in the same JVM and incorrectly concludes that, in local mode, Spark has no real concept of an executor — that the parallelism is an illusion. Why it happens: the word "local" reasonably, but incorrectly, suggests "no real distribution." How to spot it: this lesson's evidence directly contradicts it — getExecutorInfos() and the REST API show a real executor, with its own id, its own totalCores and maxTasks. How to fix it: in local[*], there really is a real executor — it just lives in the same process as the driver, with 12 (or your machine's cores) real task threads running in parallel inside that single JVM. The parallelism is real; what doesn't exist in local mode is distribution across machines.

Confusing "task threads" (defaultParallelism) with "number of executors." What happens: someone sees defaultParallelism = 12 and assumes that means "twelve executors." Why it happens: both numbers sound like "parallelism," and without the explicit distinction, it's easy to mix them up. How to spot it: this lesson's output shows getExecutorInfos() returning a list with a single element (one executor), while that same executor reports totalCores: 12 — twelve worker cores, inside one single executor process. How to fix it: on a real cluster, the total number of parallel tasks is the sum of the cores across all executors combined — in local[*], since there's only one executor, that total happens to match your machine's logical cores, but conceptually they're two different numbers that, in this particular case, happen to be equal.

Expecting the driver to "know" the contents of the data without having read it. What happens: someone writes code that tries to inspect individual DataFrame values directly from Python variables in the driver, before calling an action like .collect() or .show(), and is surprised to find they have no access to any real data. Why it happens: since the driver writes the code that describes what to do with the data, it's easy to assume it also has direct access to it. How to spot it: if your code tries to use orders_df["order_id"] as if it were a regular Python list in the driver, instead of a Column describing a pending operation, you're going to get an error or an object that isn't what you expected. How to fix it: the driver only knows the DataFrame's structure (its schema, its execution plan) — never the real data values, which live exclusively on the executors, until an action explicitly brings them back to the driver (and, for large datasets, deliberately limited — .show() brings back only a few rows, .collect() brings back all of them, with the memory risk that implies).

Exercises

Exercise 1 — Verify sc.applicationId on your own machine, twice. Run this lesson's worked example twice in a row (two separate Python processes, not the same session). Confirm the App ID differs between the two runs, and explain why.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()
print(f"App ID: {spark.sparkContext.applicationId}")
spark.stop()

App ID: local-1786655016713
App ID: local-1786655891204

Exercise 2 — Count your own machine's cores a different way, and compare them. Without using Spark, use Python's os module to count your machine's logical cores (os.cpu_count()), and compare it against sc.defaultParallelism.

import os
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()
sc = spark.sparkContext

print(f"os.cpu_count() = {os.cpu_count()}")
print(f"sc.defaultParallelism = {sc.defaultParallelism}")
print(f"They match: {os.cpu_count() == sc.defaultParallelism}")

spark.stop()

os.cpu_count() = 12
sc.defaultParallelism = 12
They match: True

Exercise 3 — Explain, without code, what would change if this guide used a real cluster. In 2-3 sentences, explain what difference there would be, in terms of driver and executors, between running this lesson's example on local[*] (as this entire guide does) and running it on a real managed cluster (like EMR or Databricks, mentioned without being built in module 8).

Summary and next step

In this lesson you took apart the first piece of Spark's execution model: the driver (your Python script, coordinates but doesn't touch data) and the executors (processes that actually process the data). You verified, with Spark's own REST API (localhost:4040/api/v1/.../executors) that the Spark UI itself queries internally, that in local[*] a real executor exists — with its own id, literally "driver" — living inside the same JVM process as the driver, with as many task threads as your machine has logical cores (defaultParallelism).

Before moving on you should be able to: explain the difference between driver and executor without using the word "kitchen"; explain why, in local[*], the single executor is literally named "driver" in Spark's API; and tell defaultParallelism (task threads) apart from "number of executors" (processes).

With the driver/executors architecture now clear, lesson 3 looks at the first API Spark offered for describing work to executors — RDDs — not to build anything on them, but to understand, by contrast, why the DataFrame API replaced them.

Resources

• Apache Spark — Cluster Mode Overview (official definition of driver program: "The process running the main() function of the application and creating the SparkContext", and of executors: "A process launched for an application on a worker node, that runs tasks..."). spark.apache.org/docs/latest/cluster-overview.html.
• Apache Spark — Monitoring and Instrumentation (the Spark UI on port 4040 by default, and the REST API that powers it — /applications/[app-id]/executors, the source of this lesson's evidence). spark.apache.org/docs/latest/monitoring.html.
• Apache Spark — Submitting Applications, "Master URLs" table (the exact definition of local[*], already seen in module 1). spark.apache.org/docs/latest/submitting-applications.html.