Your first SparkSession

Description

With Java and PySpark installed, this lesson opens the front door to everything that follows: the SparkSession. Every operation you're going to write in the rest of this guide — reading a file, doing a join, grouping, writing Parquet — passes, one way or another, through this object. You're going to build it with the exact pattern Spark documents (SparkSession.builder.appName(...).master(...).getOrCreate()), run it for real, and understand what each piece means: why appName, what local[*] is, and why getOrCreate() creates the session only once, no matter how many times you call it.

Connection to the module. This is the first time in the whole guide that your code opens a real connection to the JVM you installed in lesson 4. Lessons 6 and 7 reuse the same SparkSession you build here to read and verify Kiosko's data.

An analogy: the central control panel

Think of a plant's control panel — the board with the master switches that have to be turned on before any machine on the production line can move. It doesn't matter how many machines the plant has: they all depend on that central panel being turned on first, and there's only one panel, not one per machine. The SparkSession is that panel: you turn it on once, at the start of your script, and every operation that follows — reading a file, transforming a column, writing a result — passes through it. Turning the panel off (spark.stop()) at the end matters just as much as turning it on at the start: a plant with the panel on and nobody working keeps consuming power for no benefit at all.

Worked example: opening, inspecting, and confirming your first session

# first_session.py
from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("kiosko-spark")
    .master("local[*]")
    .getOrCreate()
)

print(f"Spark version: {spark.version}")
print(f"App name: {spark.sparkContext.appName}")
print(f"Master: {spark.sparkContext.master}")
print(f"Default parallelism: {spark.sparkContext.defaultParallelism}")

spark.stop()

What to expect. Running python3 first_session.py, the output is exactly this (executed in this run, PySpark 4.2.0, Java 17):

Spark version: 4.2.0
App name: kiosko-spark
Master: local[*]
Default parallelism: 12

The first couple of lines — Spark version, App name, Master — are going to be identical on any machine running this same code with PySpark 4.2.0. The last line, Default parallelism, is not — this run's 12 reflects the logical cores available on this specific machine; on yours it might be 4, 8, 16, any number, because that's exactly how local[*] works: per Spark's official documentation, that string means "Run Spark locally with as many worker threads as logical cores on your machine." It isn't a fixed number — it's an instruction Spark resolves, at runtime, against the real hardware it's running on.

Diagram: what the SparkSession connects

┌──────────────────────┐        Py4J         ┌────────────────────────┐
│  Your Python script   │  <---- socket ---->  │  The JVM (Java process)│
│  (the "driver" in      │      local, on the    │  where Spark actually │
│   Python)              │      same machine      │  executes the work    │
│                        │                       │                        │
│  spark = SparkSession  │                       │  SparkContext          │
│    .builder            │ ───────────────────>  │  Spark's execution     │
│    .appName(...)       │                       │  engine                │
│    .master("local[*]") │                       │                        │
│    .getOrCreate()      │  <───────────────────  │  assigned port,        │
│                        │       confirms           │  Spark UI (M2 uses it)│
└──────────────────────┘        connection       └────────────────────────┘

Going deeper: appName, master("local[*]"), and how getOrCreate() behaves

appName("kiosko-spark") isn't decorative. It's the name that identifies this run in the Spark UI (which you're going to explore in depth in module 2) and in any log Spark produces. On a real cluster, with dozens of jobs running, a descriptive name is the difference between finding your job in seconds or losing minutes hunting for it among dozens of generic entries — a habit worth building from this very first lesson, even though in local mode, with a single job running, the benefit isn't as visible yet.

master("local[*]") tells Spark where to run the work. Per Spark documentation's official "Master URLs" table, local[*] runs Spark locally using as many threads as the machine has logical cores — the option this entire guide uses, without exception. Variants exist: local (a single thread, no real parallelism, useful only for debugging), local[4] (exactly four threads, no matter how many cores the machine has), and real cluster URLs (spark://..., yarn, k8s://...) which are out of scope for this guide by design — this guide is 100% local, and module 8 explicitly names, without building, how that same code would run unchanged on a managed cluster.

getOrCreate() has a behavior worth verifying with code, not just taking the documentation's word for it:

# getOrCreate idempotence
from pyspark.sql import SparkSession

s1 = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()
s2 = SparkSession.builder.appName("another-name-ignored").master("local[*]").getOrCreate()

print("s1 is s2:", s1 is s2)
print("s2.sparkContext.appName:", s2.sparkContext.appName)

s1.stop()

What to expect (executed in this run):

s1 is s2: True
s2.sparkContext.appName: kiosko-spark

Two real findings in this output. First, s1 is s2 gives True — these aren't two separate sessions that happen to agree on config; they're, literally, the same object in memory. Second, and more important: s2 was built asking for the name "another-name-ignored", but s2.sparkContext.appName still shows "kiosko-spark" — the first session's name. This confirms exactly what getOrCreate()'s documentation says: if a global active session already exists, it reuses it as-is, and any new configuration you request in a later call to getOrCreate() (like a different appName) is ignored for options already locked in when the original session was created. This has a practical consequence: if you run several cells or scripts in the same Python process without calling spark.stop() in between, you're going to keep working on the first session you opened, no matter which builder you use afterward.

Common mistakes

Creating a "new" SparkSession partway through a script, expecting different configuration. What happens: someone, within the same script or notebook, calls SparkSession.builder.config(...).getOrCreate() expecting to change something in the active configuration, and is surprised when the earlier configuration is still in effect. Why it happens: the name getOrCreate() suggests "get or create a new one," and not every configuration parameter can be changed on an already-active session. How to spot it: this lesson's exact evidence — s2.sparkContext.appName showing s1's name — is the tell: if your second configuration doesn't show up reflected, it's because getOrCreate() handed you back the existing session, not a new one. How to fix it: if you genuinely need a session with different configuration, call spark.stop() on the active session before building a new one with getOrCreate() — or, more simply, within a single script, decide the full configuration before the first call to getOrCreate() and don't touch it again.

Skipping .master(...) and relying on undocumented behavior. What happens: someone notices, maybe through experimentation, that their code works even without calling .master(...) explicitly. Why it happens: in some environments, PySpark can resolve a default master without throwing the MASTER_URL_NOT_SET error that, in theory, should show up if spark.master isn't set anywhere. How to spot it: if your code omits .master(...) and still runs, don't assume that behavior is guaranteed — it isn't documented as an official default, and it can vary between Spark versions, between ways of launching the script (python script.py versus spark-submit script.py), or between environments with different variables set. How to fix it: this guide, without exception, declares master("local[*]") explicitly in every SparkSession it builds — not out of caprice, but because relying on undocumented behavior is exactly the kind of fragility a real pipeline can't afford.

Forgetting spark.stop() at the end of the script. What happens: someone runs several Spark scripts back to back, in the same terminal or the same notebook, without closing the previous session, and the second script fails or behaves unexpectedly (for example, the Spark UI port, normally 4040, shows up taken and Spark silently jumps to the next available port). Why it happens: spark.stop() feels like an optional cleanup detail, compared with the rest of the script that actually produces visible results. How to spot it: if you notice Java processes piling up in your task manager or top/htop after running several Spark scripts, or if a new run's Spark UI shows up on a port other than 4040 without you having changed it, you probably left earlier sessions open. How to fix it: always close the session at the end of the script with spark.stop(), exactly as this lesson's worked example does — a habit as simple as closing a file you opened with open().

Exercises

Exercise 1 — Inspect one more configuration value. Using the SparkSession from the worked example, print spark.sparkContext.pythonVer (the Python version Spark is using internally) and compare it against your own python3 --version.

print(f"Python used by Spark: {spark.sparkContext.pythonVer}")

Python used by Spark: 3.14

Exercise 2 — Verify idempotence with three different builders, not just two. Extend this lesson's getOrCreate() example with a third call, s3, using yet another different appName. Verify that s1 is s2 is s3 gives True.

s1 = SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate()
s2 = SparkSession.builder.appName("second-name").master("local[*]").getOrCreate()
s3 = SparkSession.builder.appName("third-name").master("local[*]").getOrCreate()

print("s1 is s2 is s3:", s1 is s2 is s3)
print("final appName:", s3.sparkContext.appName)

s1 is s2 is s3: True
final appName: kiosko-spark

Exercise 3 — Explain, without code, the difference between local and local[*]. Using what you learned in the "going deeper" section, explain in 2-3 sentences what would happen, in terms of parallelism, if this guide used master("local") instead of master("local[*]") in every SparkSession.

Summary and next step

In this lesson you opened your first real SparkSession: SparkSession.builder.appName("kiosko-spark").master("local[*]").getOrCreate(), checked its version, name, and master, and confirmed with code — not just documentation — that getOrCreate() is idempotent: calling it several times in the same process always returns the same session, with the first call's configuration. You also learned what py4j is in practice (the bridge between your Python script and the JVM) and why spark.stop() matters just as much as opening the session.

Before moving on you should be able to: write the pattern SparkSession.builder.appName(...).master("local[*]").getOrCreate() from memory; explain precisely what local[*] means; and explain why a second call to getOrCreate() with a different appName doesn't change the active session's name.

With the SparkSession now mastered, lesson 6 uses it for the first time on something real: reading Kiosko's seven order files with an explicit schema.

Resources

• Apache Spark — SQL Getting Started (the exact SparkSession.builder.appName(...).getOrCreate() pattern this lesson uses). spark.apache.org/docs/latest/sql-getting-started.html.
• Apache Spark — Submitting Applications, "Master URLs" table (the exact definition of local[*]: "Run Spark locally with as many worker threads as logical cores on your machine"). spark.apache.org/docs/latest/submitting-applications.html.
• Apache Spark — RDD Programming Guide (historical context for SparkConf().setMaster(...), the older way of configuring the master, before the unified SparkSession). spark.apache.org/docs/latest/rdd-programming-guide.html.