Project: Kiosko's first Spark session

Description

This project closes the module by pulling the previous seven lessons together into a single delivery: you verify the environment (Java, JAVA_HOME), open Kiosko's SparkSession, read the seven files with an explicit, strict schema, verify the result from four independent angles, and — the closing piece that gives the whole module its point — apply lesson 3's cost criterion to the very dataset you just read, confirming with numbers, not intuition, that real Kiosko still doesn't need anything the rest of this guide is about to teach.

Connection to the module. This project doesn't introduce any new concept — it's the final integration of lessons 4 through 7, with one closing piece: applying lesson 3's criterion to the real dataset lessons 4 through 7 just processed, closing the module's full circle.

An analogy: a new team's first day, with the simplest possible task

Pick back up with the lone accountant from lessons 1 and 3. Imagine that, after carefully measuring that the workload justifies it, they finally hire their first assistant. On day one, they don't hand over the most complex task of the month — they ask the assistant to repeat, exactly, the simplest register close-out the accountant themselves has already done a thousand times, to confirm the new assistant counts correctly before trusting them with anything bigger. That's, precisely, what this project does with Spark: it doesn't yet ask it to distribute anything complex — it asks it to repeat, exactly, Kiosko's simplest possible read and count, the same task dict, DuckDB, Polars, and SQL already did in the three previous guides. Before trusting Spark with anything bigger (the rest of this guide), you confirm it gets the simplest thing right.

The material: everything this module built, in a single flow

You need: Java 17+ installed with JAVA_HOME configured (lesson 4), PySpark 4.2.0 installed (lesson 4), and the seven files orders_2026-08-03.csv through orders_2026-08-09.csv in the same folder where you're going to run the script (lesson 6).

The reference solution, verified

Part 1 — Verify the environment before opening anything

# kiosko_first_spark_session.py
import csv
import glob
import os

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import (
    StructType, StructField, StringType, IntegerType, DoubleType, TimestampType,
)

print("=== Kiosko in Spark: first session, module 1 final delivery ===\n")

print("Part 1 -- verifying the environment")
java_home = os.environ.get("JAVA_HOME")
print(f"JAVA_HOME: {java_home}")
assert java_home, "JAVA_HOME is not configured -- PySpark cannot start the JVM"
print("Verification: JAVA_HOME configured -> OK\n")

This first part isn't a formality — it's exactly the step that, if skipped, produces the JAVA_GATEWAY_EXITED error you saw in lesson 4. Checking it with an assert, before opening the SparkSession, turns a confusing runtime error into a clear message at the right moment.

Part 2 — Open Kiosko's SparkSession

print("Part 2 -- opening the SparkSession")
spark = (
    SparkSession.builder
    .appName("kiosko-spark")
    .master("local[*]")
    .getOrCreate()
)
print(f"Spark version: {spark.version}")
print(f"App name: {spark.sparkContext.appName}")
print(f"Master: {spark.sparkContext.master}\n")

Part 3 — Read the seven files with an explicit, strict schema

print("Part 3 -- reading Kiosko's seven files with an explicit schema")
orders_schema = StructType([
    StructField("order_id", StringType(), False),
    StructField("store_id", StringType(), False),
    StructField("product_id", StringType(), False),
    StructField("quantity", IntegerType(), False),
    StructField("unit_price", DoubleType(), False),
    StructField("order_ts", TimestampType(), False),
])
orders_df = spark.read.csv(
    "orders_2026-08-*.csv", schema=orders_schema, header=True, enforceSchema=False,
)
spark_count = orders_df.count()
print(f"orders_df.count() = {spark_count}")
assert spark_count == 40
print("Verification: orders_df.count() == 40 -> OK\n")

Notice enforceSchema=False, carried over directly from lesson 6's common mistake — without this option, a file with misaligned columns would fail silently instead of with a clear error.

Part 4 — Cross-verification, with pure Python

print("Part 4 -- cross-verification against pure Python (csv.DictReader)")
python_count = 0
for path in sorted(glob.glob("orders_2026-08-*.csv")):
    with open(path, newline="") as f:
        python_count += sum(1 for _ in csv.DictReader(f))
print(f"python_count (csv.DictReader) = {python_count}")
assert spark_count == python_count == 40
print("Verification: spark_count == python_count == 40 -> OK\n")

Parts 5, 6, and 7 — Verification by file, by store, and of nulls

print("Part 5 -- rows by source file")
by_file = (
    orders_df
    .withColumn("source_file", F.element_at(F.split(F.input_file_name(), "/"), -1))
    .groupBy("source_file")
    .count()
    .orderBy("source_file")
)
by_file.show(truncate=False)

print("Part 6 -- orders by store")
orders_df.groupBy("store_id").count().orderBy("store_id").show()

print("Part 7 -- null verification by column")
null_counts = orders_df.select([
    F.sum(F.col(c).isNull().cast("int")).alias(c) for c in orders_df.columns
])
null_counts.show()

Part 8 — The cost criterion, closing the module

print("Part 8 -- the cost criterion, applied to this same dataset")
total_bytes = sum(os.path.getsize(f) for f in glob.glob("orders_2026-08-*.csv"))
size_gb = total_bytes / 1_000_000_000
print(f"Real size of Kiosko's week: {total_bytes} bytes ({size_gb:.9f} GB)")
if size_gb < 1000:
    print("Lesson 3 criterion verdict: NO -- far below 1 TB, DuckDB/Polars are more than enough")
print()

spark.stop()
print("=== spark.stop() -- session closed ===")

What to expect. Running python3 kiosko_first_spark_session.py in full (all eight parts together), the output is exactly this:

=== Kiosko in Spark: first session, module 1 final delivery ===

Part 1 -- verifying the environment
JAVA_HOME: /opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
Verification: JAVA_HOME configured -> OK

Part 2 -- opening the SparkSession
Spark version: 4.2.0
App name: kiosko-spark
Master: local[*]

Part 3 -- reading Kiosko's seven files with an explicit schema
orders_df.count() = 40
Verification: orders_df.count() == 40 -> OK

Part 4 -- cross-verification against pure Python (csv.DictReader)
python_count (csv.DictReader) = 40
Verification: spark_count == python_count == 40 -> OK

Part 5 -- rows by source file
+---------------------+-----+
|source_file          |count|
+---------------------+-----+
|orders_2026-08-03.csv|8    |
|orders_2026-08-04.csv|6    |
|orders_2026-08-05.csv|2    |
|orders_2026-08-06.csv|5    |
|orders_2026-08-07.csv|7    |
|orders_2026-08-08.csv|9    |
|orders_2026-08-09.csv|3    |
+---------------------+-----+

Part 6 -- orders by store
+--------+-----+
|store_id|count|
+--------+-----+
|     S01|   16|
|     S02|   13|
|     S03|   11|
+--------+-----+

Part 7 -- null verification by column
+--------+--------+----------+--------+----------+--------+
|order_id|store_id|product_id|quantity|unit_price|order_ts|
+--------+--------+----------+--------+----------+--------+
|       0|       0|         0|       0|         0|       0|
+--------+--------+----------+--------+----------+--------+

Part 8 -- the cost criterion, applied to this same dataset
Real size of Kiosko's week: 2236 bytes (0.000002236 GB)
Lesson 3 criterion verdict: NO -- far below 1 TB, DuckDB/Polars are more than enough

=== spark.stop() -- session closed ===

(JAVA_HOME in your output is going to show your own Java installation's real path, not this run's — what matters is that the line shows up, not that the value matches byte for byte.)

Stop on Part 8, because it's the real close of the whole module: Kiosko's first Spark session ends, literally, by applying to itself the criterion lesson 3 built, and the verdict is unambiguous — 2236 bytes are an astronomical distance from the 1 TB ceiling the market evidence uses as a reference. This isn't a contradiction with having spent seven lessons installing and learning Spark — it's, precisely, this guide's central pedagogical point, now said with the very tool you just learned: you know how to use Spark, and you know, with evidence, that Kiosko doesn't need it yet.

Diagram: the eight parts, closing out the whole module

flowchart TD
    A["Part 1 (L4): JAVA_HOME verified"] --> B
    B["Part 2 (L5): SparkSession opened\nkiosko-spark, local[*]"] --> C
    C["Part 3 (L6): orders_df read,\nexplicit schema, enforceSchema=False"] --> D
    D["Part 4 (L7): cross-count\nSpark == pure Python == 40"] --> E
    E["Parts 5-7 (L7): by file,\nby store, nulls -- all correct"] --> F
    F["Part 8 (L2-L3): the cost criterion\napplied to the real data -> NO"] --> G["Module 1 closed:\nSpark installed, Kiosko read,\nand the exact reason it\nwasn't needed -- all at once"]

Closing out this module's checklist, piece by piece

Two of the thirteen checklist rows are resolved — and they're, precisely, the two that needed to be resolved first: without Spark installed and without confirming it correctly reads the simplest possible data, none of the seven remaining pieces would have a reliable foundation to build on.

Common mistakes

Turning in the project without Parts 4 through 7's four checks. What happens: someone, in a hurry to reach Part 8 (the close with the cost criterion), only runs Parts 1 through 3 and jumps straight to the end. Why it happens: count() == 40 in Part 3 already feels like enough confirmation. How to spot it: if your final delivery doesn't include executed evidence from the four cross-checks (pure Python, by file, by store, nulls), you're trusting a single number — exactly the trap lesson 7 warned about in detail. How to fix it: Parts 4 through 7 aren't optional — they're the guarantee that makes Part 8 trustworthy; a cost criterion applied to data you didn't thoroughly verify is worth no more than the confidence you have in that data.

Reading Part 8's NO verdict as the end of the guide, not of the module. What happens: someone finishes this project, sees the cost criterion's NO verdict, and assumes there's no longer any reason to continue with modules 2 through 8. Why it happens: such a clear verdict feels like a final conclusion, not the close of a single introductory module. How to spot it: if your takeaway from this project is "so I don't need to learn the rest of Spark," check this lesson's checklist table — thirteen pieces total, eleven still pending. How to fix it: the NO verdict answers a question about Kiosko's real volume today, not about the value of learning Spark — the market evidence from lesson 1 (EMR, Databricks, DEA-C01, DP-700) is still real, and module 4 of this guide deliberately builds a ten-million-row synthetic dataset exactly so you genuinely feel what the real Kiosko never produces.

Copying this project's script without being able to explain each of the eight parts. What happens: someone reuses kiosko_first_spark_session.py in their own work, changing only the file names, without being able to explain what each part does or why it's in that order. Why it happens: a script that already works is faster to copy than to understand from scratch. How to spot it: if you can't explain, without looking at the code, why Part 1 comes before Part 2 (verifying the environment before opening the session), or why Part 4 uses a different tool than Spark to verify the same number, you need to reread lessons 4 through 7. How to fix it: every part of this project has a full lesson behind it that justifies it — before reusing this pattern in a project of your own, confirm you can explain each part in your own words.

Exercises

Exercise 1 — Extend the project with the count by product. Without using datetime.now() (forbidden in this guide to keep reproducibility), add a Part 9 to the script confirming the count by product_id (P001: 16, P002: 10, P003: 7, P004: 7) — the fifth independent check, closing out every possible dimension of the data with evidence.

print("Part 9 -- orders by product")
orders_df.groupBy("product_id").count().orderBy("product_id").show()

Part 9 -- orders by product
+----------+-----+
|product_id|count|
+----------+-----+
|      P001|   16|
|      P002|   10|
|      P003|    7|
|      P004|    7|
+----------+-----+

Exercise 2 — Apply the cost criterion to a Kiosko a hundred times bigger, without generating any new data. Using should_distribute() from lesson 3 (or reimplementing it if you don't have it handy), calculate the verdict for a hypothetical Kiosko whose disk size was a hundred times the real one (total_bytes * 100), with exhausted_single_node_tools false.

def should_distribute(dataset_size_gb, exhausted_single_node_tools):
    TB = 1000
    HUNDRED_TB = 100_000
    if dataset_size_gb < TB:
        return "NO"
    elif dataset_size_gb >= HUNDRED_TB:
        return "YES, PROBABLY"
    elif not exhausted_single_node_tools:
        return "NOT YET"
    else:
        return "IT DEPENDS -- MEASURE THE REAL COST"

size_gb_x100 = (total_bytes * 100) / 1_000_000_000
print(f"Hypothetical size (x100): {size_gb_x100:.9f} GB")
print(f"Verdict: {should_distribute(size_gb_x100, False)}")

Hypothetical size (x100): 0.000223600 GB
Verdict: NO

Exercise 3 — Explain, from memory, what this pipeline is missing to become the full module 8 capstone. Without looking at the guide's design, describe in a 4-6 sentence paragraph what transformations, checks, or decisions kiosko_first_spark_session.py is missing to become the full distributed pipeline you're going to build in module 8.

Summary and next step: the end of module 1

With this mini-project you close out module 1 in full. You verified the environment (Java 17, JAVA_HOME), opened Kiosko's first real SparkSession (appName="kiosko-spark", master("local[*]")), read the seven files with an explicit, strict schema (enforceSchema=False), and confirmed the result from four independent angles — all agreeing on the same forty rows you already knew from three previous guides. You closed out the module by applying lesson 3's cost criterion to that same data, with an unambiguous verdict: NO, real Kiosko doesn't need Spark.

You took the first step on an eight-module path: you have Spark installed, verified, and you know how to read a CSV with it — what you don't have yet is a single line of code that takes advantage of what makes Spark different from DuckDB or Polars. That starts in module 2.

Where you're headed. Module 2 — the-spark-execution-model — opens the box this guide deliberately left closed: the driver/executors architecture, the DataFrame API as the main interface, lazy evaluation, and the difference between transformations and actions — the mental model without which no line of Spark you write afterward makes real sense.

Resources

• Apache Spark — SQL Getting Started (the full SparkSession + read + verify pattern this project pulls together). spark.apache.org/docs/latest/sql-getting-started.html.
• PySpark — Installation (the Java 17 and JAVA_HOME requirement Part 1 of this project verifies before anything else). spark.apache.org/docs/latest/api/python/getting_started/install.html.
• src/paths/data-engineering-ecosystem/VALIDACION.md — the full source of the market evidence backing the cost criterion applied in Part 8 of this project. Internal repository document, no public URL.
• python-for-data-engineering-guide DESIGN doc and data-modeling-for-analytics-guide DESIGN doc — the sources of the counts by store and by product this project verifies again with Spark. src/guides/python-for-data-engineering-guide/DISENO.md and src/guides/data-modeling-for-analytics-guide/DISENO.md