Module 1: Why Distribute The Single Node Ceiling

Installing PySpark and Java locally

Next →

Description

With the criterion from lessons 2 and 3 already built, this lesson installs Spark for real, in local mode, on your own machine — no Docker, no cloud account, $0. pip install pyspark is a single command, but Spark runs on the JVM (Java Virtual Machine), which means that, before that command, you need Java 17 or newer installed and JAVA_HOME configured. This lesson doesn't take that for granted: it verifies it, step by step, and shows — with real evidence, not a generic warning — exactly what happens when that step gets skipped.

Connection to the module. This is the module's first lesson that installs real software. Lessons 5 through 7 assume pip install pyspark and this lesson's Java verification are already done.

An analogy: the industrial appliance and the correct outlet

Keep going with lesson 2's professional-kitchen analogy. Buying the industrial appliance — the high-power convection oven, say — is only half the work. That oven doesn't run on any household outlet: it requires a circuit with the correct voltage and amperage, installed before you plug it in. If you plug it into a regular outlet, nothing dramatic happens at first glance — the oven simply doesn't turn on, or the circuit breaker trips, and the error message (if there even is one) rarely says clearly "you need a 240V circuit." PySpark is exactly that oven: the Python package (pip install pyspark) is the visible half, but it runs on the JVM, a specific "circuit" that Java 17 or newer has to provide before Spark can start. This lesson installs the circuit first, then the oven.

Worked example: verify Java, install PySpark, confirm the version

Step 1 — Verify you have Java 17 or newer

Spark 4.x, according to PySpark's official installation documentation, requires Java 17 or later with JAVA_HOME correctly set. Start by confirming what version of Java you have, if any:

java -version

What to expect (verified in this run, macOS with Java 17 installed via Homebrew — brew install openjdk@17):

openjdk version "17.0.20" 2026-07-21
OpenJDK Runtime Environment Homebrew (build 17.0.20+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.20+0, mixed mode, sharing)

If your output shows a version lower than 17 (say, 1.8.0 or 11.0.x), or if the command doesn't exist at all, you need to install Java 17 or newer before continuing. On macOS, the simplest route is brew install openjdk@17; on Linux, your distribution's package manager (apt install openjdk-17-jdk, dnf install java-17-openjdk); on Windows, an Eclipse Temurin 17-or-newer installer. Any of these routes is valid — Spark's official downloads guide only requires the version, not the specific JDK vendor.

Step 2 — Configure JAVA_HOME

JAVA_HOME is an environment variable that tells any program — PySpark included — where the JDK is installed. It isn't optional: PySpark's official installation documentation says it plainly: "PySpark requires Java 17 or later with JAVA_HOME properly set". Check whether it's already configured:

echo $JAVA_HOME

What to expect (verified in this run, macOS/Homebrew):

/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home

That path is specific to this machine (macOS, installed with Homebrew) — on your setup it's going to look different: on Linux it's usually something like /usr/lib/jvm/java-17-openjdk-amd64, on Windows something like C:\Program Files\Java\jdk-17. What matters isn't the exact value, but that the variable points to the JDK's root directory — the one containing a bin/ folder with the java executable inside — and that it isn't empty. If echo $JAVA_HOME prints nothing, configure it before continuing (on macOS/Linux, typically by adding export JAVA_HOME=$(/usr/libexec/java_home -v 17) or your installation's equivalent path to your .zshrc/.bashrc).

Step 3 — Install PySpark

With Java 17 and JAVA_HOME confirmed, install PySpark with pip:

pip install pyspark

What to expect (executed in this run, clean environment):

Collecting pyspark
  Using cached pyspark-4.2.0-py2.py3-none-any.whl
Collecting py4j<0.10.9.10,>=0.10.9.7 (from pyspark)
  Using cached py4j-0.10.9.9-py2.py3-none-any.whl.metadata (1.3 kB)
Using cached py4j-0.10.9.9-py2.py3-none-any.whl (203 kB)
Installing collected packages: py4j, pyspark

Successfully installed py4j-0.10.9.9 pyspark-4.2.0

Two details in this output are worth noticing. First, pyspark installs py4j as an automatic dependency — it's the library that lets Python code (your script) talk to the JVM (where Spark actually runs) over a local socket; you'll see py4j again in lesson 5, when you open your first SparkSession. Second, this specific run shows Using cached because the package was already in this machine's local pip cache — the first time you install it in a fresh environment, you'll see Downloading with a progress bar instead, because the package is large (it includes Spark's JARs, not just Python code). The final result — Successfully installed py4j-0.10.9.9 pyspark-4.2.0 — is what confirms the install finished correctly, whether it came from cache or a fresh download.

You can install this inside a virtual environment (python3 -m venv .venv && source .venv/bin/activate, before the pip install) or straight into your system Python — this guide doesn't revisit packaging, because python-for-data-engineering-guide already covered it thoroughly with uv. All that matters here is that pyspark ends up installed in whichever Python interpreter you'll use in the lessons that follow.

Step 4 — Confirm the installed version

python3 -c "import pyspark; print(pyspark.__version__)"

What to expect (executed in this run):

4.2.0

PySpark 4.2.0 is, per the package's own PyPI page, the current version as of this guide's writing — published on July 14, 2026 — and it requires Python >=3.10. If your python3 --version shows something older than 3.10, you need to upgrade Python before pip install pyspark will work.

Diagram: the four layers you just installed

┌───────────────────────────────────────────────┐
│  Your Python script (lesson 5 onward)          │
├───────────────────────────────────────────────┤
│  pyspark (the package you installed, 4.2.0)    │
├───────────────────────────────────────────────┤
│  py4j (Python <-> JVM bridge, via local socket)│
├───────────────────────────────────────────────┤
│  The JVM (where Spark actually runs)           │
├───────────────────────────────────────────────┤
│  Java 17+ with JAVA_HOME configured            │
│  (the "circuit" that lets everything above     │
│   actually start)                              │
└───────────────────────────────────────────────┘

Going deeper: why Spark, written in Python, needs Java

It can seem strange that a guide promising "zero Scala" and "pure Python" requires installing Java before anything else. The reason is architectural, not a contradiction: Apache Spark itself — the execution engine, the Catalyst optimizer, distributed memory management — is written in Scala, and Scala compiles to JVM bytecode. When you write spark.read.csv(...) in Python, that code doesn't execute the CSV read in Python directly — it translates, through py4j, into a call to the real engine, which runs inside the JVM. Python, in this architecture, is a control layer: it describes what you want to do (the DataFrame API you'll use throughout this guide), and the JVM is what actually does it.

This also explains how this guide can keep its promise of "zero Scala in the code you write" without lying about what runs underneath: you're never going to write a single line of Scala, but every operation you write in Python does end up executing on a real JVM, and that's why Java — not Scala — is the installation requirement you can't skip. Spark's official downloads page confirms the other half of this architecture decision: Spark 4 ships pre-built with Scala 2.13, and Scala 2.12 support was dropped — a detail that doesn't affect you directly as a PySpark user, but confirms Scala is still the engine's language, even though you'll never touch it.

Common mistakes

Installing PySpark without having Java, and not understanding the resulting error. What happens: someone runs pip install pyspark (which works fine, because pip doesn't check for Java) and then, when trying to open a SparkSession, runs into a long, confusing error. What it looks like exactly, verified on this same machine with no Java installed:

The operation couldn't be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.
...
pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

Why it happens: pip install pyspark only installs the Python package and Spark's JARs — it doesn't install or check for Java, because pip has no way to install a JDK. The error only shows up later, when the code actually tries to start the JVM. How to spot it: the key message, regardless of operating system, is [JAVA_GATEWAY_EXITED] or Java gateway process exited before sending its port number — that's PySpark indirectly telling you the JVM never managed to start. The specific OS-level message before that line varies: on macOS without Java you'll see the "Unable to locate a Java Runtime" dialog shown above; on Linux, typically a more direct command not found: java. How to fix it: go back to Step 1 of this lesson — confirm java -version and JAVA_HOME before opening any SparkSession, not after it fails.

Installing a Java version lower than 17. What happens: someone has Java 8 or Java 11 installed from an earlier project (common versions in shops still running older applications) and assumes "any Java" works for PySpark. Why it happens: for years, several Spark versions supported Java 8 and 11, so the expectation of "any Java works" comes from real experience, just outdated. How to spot it: java -version shows a number lower than 17 — if your output starts with openjdk version "11... or "1.8..., you don't meet this guide's requirement, even though the java -version command runs without error. How to fix it: install Java 17 or newer specifically (you can have several Java versions installed at the same time on the same machine) and point JAVA_HOME at that specific installation, not whichever one you already had.

Confusing JAVA_HOME with the path to the java executable. What happens: someone configures JAVA_HOME pointing directly at the binary (.../bin/java) instead of at the JDK's root directory. Why it happens: it seems reasonable for the variable to point "at the java program" directly. How to spot it: if JAVA_HOME ends in /bin/java (a file, not a folder) instead of ending at the JDK's directory (which contains a bin/ folder inside it), it's misconfigured — tools like PySpark expect to be able to build the path $JAVA_HOME/bin/java themselves. How to fix it: JAVA_HOME should point at the installation's root directory (as in this lesson's example: .../openjdk.jdk/Contents/Home, which does contain bin/ inside it), never at the java binary directly.

Exercises

Exercise 1 — Verify your own installation with a single script. Write a Python script (not using pyspark yet) that reads the JAVA_HOME environment variable with os.environ.get("JAVA_HOME") and prints a clear message depending on whether it's empty or configured.

See solution 
import os

java_home = os.environ.get("JAVA_HOME")
if java_home:
    print(f"JAVA_HOME configured: {java_home}")
else:
    print("JAVA_HOME is NOT configured -- PySpark will fail with JAVA_GATEWAY_EXITED when trying to start")

Expected output, if JAVA_HOME is correctly configured (using this run's real value):

JAVA_HOME configured: /opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home

This small script is exactly the first step you're going to use inside the lesson 8 project, as an explicit check before opening the SparkSession.

Exercise 2 — Confirm the Python version requirement. Without installing anything new, verify your Python meets the >=3.10 requirement PySpark 4.2.0 demands.

See solution 
python3 --version

Any output of Python 3.10.x or newer meets the requirement. If your output shows Python 3.9.x or older, pip install pyspark will fail to resolve the dependency, with a message indicating it couldn't find a compatible version of pyspark for your interpreter — the fix is to upgrade your Python installation before installing PySpark, not to look for an older version of PySpark that does support it (this guide specifically uses 4.2.0).

Exercise 3 — Explain, without code, why pip install pyspark "works" even without Java. In 2-3 sentences, explain why the pip install pyspark command finishes successfully (Successfully installed) even without Java installed, and at exactly what point in the flow the problem finally shows up.

See solution

pip install pyspark only downloads and installs files: the Python package, its dependencies (py4j), and Spark's already-compiled JARs — none of those steps needs to run Java, just copy files to disk, so pip finishes with no error. The problem shows up later, at runtime: only when your code calls SparkSession.builder...getOrCreate() (lesson 5) does PySpark try to launch a real JVM process using java, and that's the moment — not before — when the absence of Java shows itself, with the JAVA_GATEWAY_EXITED error seen in this lesson.

Summary and next step

In this lesson you installed real Spark on your machine: you verified Java 17 with java -version, confirmed JAVA_HOME, installed PySpark 4.2.0 with pip install pyspark, and confirmed the version with python3 -c "import pyspark; print(pyspark.__version__)". You also saw, with the real error message, what happens when the Java step gets skipped — [JAVA_GATEWAY_EXITED] — and why that error only shows up when trying to start the JVM, not during installation.

Before moving on you should be able to: explain why Spark, even though you use it in Python, needs Java; verify JAVA_HOME on your own machine; and recognize the JAVA_GATEWAY_EXITED error if you ever see it.

With Java, JAVA_HOME, and PySpark now installed, lesson 5 opens your first real SparkSession — the entry point to everything Spark can do.

Resources

← PreviousA real cost criterion for when to distributeNext →Your first SparkSession