Module 1: Why Distribute The Single Node Ceiling

Module introduction: why distribute — the single-node ceiling

Next →

Why this module exists

data-engineering-foundations-guide built Kiosko's pipeline with dict and sqlite3. python-for-data-engineering-guide took that same pipeline and pushed it, on purpose, right up to the limit of what a single node can do: DuckDB querying Parquet/CSV directly with SQL, with nothing loaded beforehand; Polars with its LazyFrame and an optimized query plan you read with .explain(). That guide didn't fall short for lack of ambition — it stopped there on purpose, backed by market evidence that most real day-to-day work lives exactly in that range. data-modeling-for-analytics-guide built, on top of that same single-node limit, a complete dimensional warehouse — star schema, historized dim_product, accumulating snapshot — and closed its scope with an explicit sentence: "Distributed computing, when to actually parallelize on a cluster → spark-and-distributed-processing-guide. Kiosko's warehouse still fits in DuckDB in-process, on purpose".

This guide collects that debt. But it collects it with the same brutal honesty the market evidence demands, and that honesty starts on the first line of this first module: Kiosko, at the scale it runs at across the four previous guides — forty rows, the week of August 3 to 9, 2026 — does not need Spark. Nobody needs Spark for forty rows. Not for four hundred thousand, either. The question this module answers isn't "how do you use Spark?" — that question starts in lesson 4 and takes up the rest of the guide. This module's question comes first and matters more: what, exactly, is the single-node ceiling, and with what evidence — not fashion — do you decide you've already hit it?

This isn't a guide that teaches Spark because Spark is trendy. A market evidence audit (src/paths/data-engineering-ecosystem/VALIDACION.md, high confidence) confirms Spark shows up in real job postings across LATAM and Spain — 2BRAINS asks for EMR, Logicalis Spain pays Databricks licenses of 30 to 45 thousand euros, it's an explicit task on the AWS DEA-C01 exam and a named block on Microsoft's DP-700 — so this guide exists for a reason. But the same audit carries a hard warning, quoting a 263-point Hacker News thread: "almost nobody in the world uses datasets bigger than 100TB", and that 650 GB "is a pretty reasonable approximation of the entire dataset most companies are even working with". One user, aleda145, sums it up this way: "In my network everyone is talking about DuckDB... under 1TB it will have everything you need." The same audit flags, as a HIGH-confidence evidence gap with no guide assigned to it before this one: "the list jumps straight from Python to Spark, which is the 2018 sequence. The most profitable skill according to practitioners isn't distributing — it's knowing NOT to distribute, and being able to justify it."

Connection to the module. This module 1 doesn't write a single line of Spark pipeline yet — that starts in module 3, when you rebuild fact_orders with the DataFrame API. What this module builds is the judgment: naming precisely what the single-node ceiling is (lesson 2), giving a real cost criterion for when to distribute — not a fashion-driven rule — (lesson 3), and installing and verifying Spark in local mode, confirming it reads exactly the same week of Kiosko data you've already seen with dict, DuckDB, Polars, and SQL over a star schema (lessons 4 through 8).

An analogy: the accountant working alone, and the real ceiling of their desk

Picture an accountant who handles the books for a single small business, at their own desk, with their own calculator and their own filing cabinet. That accountant can handle the bookkeeping for a neighborhood kiosk with no problem at all — and, if you think about it, could also handle a mid-sized chain of thirty stores, or even three hundred, with a faster calculator, a bigger filing cabinet, and enough hours in the day. The limit of "one accountant at one desk" isn't a fixed number of clients — it's elastic, and it can stretch a long way before it breaks.

But that limit does exist, at some point. No matter how fast they add, no matter how large their filing cabinet, there's a volume of transactions — thousands of branches, millions of movements per day — where a single desk, no matter how much you improve it, no longer keeps up within the time the business needs. That's the point, and only that point, where it makes sense to stop asking "how do I make this accountant work faster?" and start asking "how do I organize a team?" Hiring a team before you need one isn't prudence — it's paying the cost of coordinating several people (meetings, splitting up work, reconciling what everyone did) with no real need behind it. This module is exactly about knowing at what point one accountant working alone stops being enough — and about confirming, with evidence and not with the feeling of "this already feels big," that Kiosko, today, is still a one-desk job.

Worked example: the map of the criterion, before applying it

Before writing this module's first line, it's worth seeing the full checklist you're going to build — not to answer it yet, but to have a clear destination.

# decision_checklist_preview.py
CHECKLIST = [
    (1, "Have you already exhausted what a single node can do?",
        "DuckDB out-of-RAM, Polars lazy, columnar format -- not a plain Python loop"),
    (2, "Does the real volume -- measured in bytes, not 'it feels big' -- exceed hundreds of GB?",
        "650 GB approximates the complete dataset of the average company, per market evidence"),
    (3, "Are you close to or above 1 TB?",
        "Under 1 TB, the evidence says DuckDB alone already has everything you need"),
    (4, "Is the cost of operating a cluster (compute + people to run it) lower than the cost of a slower job on a bigger single node?",
        "This isn't a technical question -- it's a real, measured cost question, not fashion"),
]

print("=== This module's cost criterion, before applying it ===\n")
for number, question, detail in CHECKLIST:
    print(f"{number}. {question}")
    print(f"   {detail}\n")

print("This module 1 answers these four questions for Kiosko, with evidence -- not intuition.")
print("Modules 2 through 8 take the result as given and build real Spark on top of it.")

What to expect. Running python3 decision_checklist_preview.py, the output is exactly this:

=== This module's cost criterion, before applying it ===

1. Have you already exhausted what a single node can do?
   DuckDB out-of-RAM, Polars lazy, columnar format -- not a plain Python loop

2. Does the real volume -- measured in bytes, not 'it feels big' -- exceed hundreds of GB?
   650 GB approximates the complete dataset of the average company, per market evidence

3. Are you close to or above 1 TB?
   Under 1 TB, the evidence says DuckDB alone already has everything you need

4. Is the cost of operating a cluster (compute + people to run it) lower than the cost of a slower job on a bigger single node?
   This isn't a technical question -- it's a real, measured cost question, not fashion

This module 1 answers these four questions for Kiosko, with evidence -- not intuition.
Modules 2 through 8 take the result as given and build real Spark on top of it.

No Kiosko data yet — this is, deliberately, the map before the territory. Lesson 2 answers question 1 precisely (what "exhausting" a single node actually means). Lesson 3 formalizes questions 2 through 4 into an executable function, with the exact thresholds the market evidence provides. Lessons 4 through 8 stop talking about criteria and start installing and running real Spark — because once you decide you actually need to distribute (or, as with Kiosko, decide to learn it anyway because of its real weight in the job market), the rest of the guide teaches you to do it well.

Diagram: where you were, where you're headed

flowchart LR
    subgraph Anteriores["The three previous guides in the ecosystem"]
        A["foundations\ndict + sqlite3"]
        B["python-for-data-engineering\nDuckDB + Polars at the limit\nof a single node, on purpose"]
        C["data-modeling\nfull star schema,\nstill fits in DuckDB"]
    end

    subgraph M1["This module (1 of 8)"]
        D["L2: what, exactly,\nis a node's ceiling"]
        E["L3: the real cost\ncriterion -- executable function"]
        F["L4-L5: install and open\nthe first SparkSession"]
        G["L6-L7: read and verify\nthe same week of Kiosko"]
        H["L8: project -- Kiosko\nin Spark, 40 == 40"]
    end

    subgraph Resto["Modules 2-8 (the rest of the guide)"]
        I["Execution model, joins,\nshuffle, Catalyst, Parquet,\nUDFs, capstone at scale"]
    end

    A --> B --> C --> D --> E --> F --> G --> H --> I

This module's map

Lesson    What it builds
────────  ──────────────────────────────────────────────────────────────
L1        (this one) The map: why this guide starts by asking when NOT to distribute
L2        What, precisely, the single-node ceiling is -- what DuckDB/Polars already solved
L3        A real cost criterion -- executable function with the thresholds from the evidence
L4        Installing PySpark and Java locally -- EXECUTED
L5        Your first SparkSession -- EXECUTED
L6        Reading Kiosko's orders with Spark -- EXECUTED
L7        Verifying the same forty rows arrive -- EXECUTED
L8        Project: Kiosko's first Spark session, verified end to end

Lessons 2 and 3 are conceptual — they give you the vocabulary and the criterion before installing anything. Lessons 4 through 7 are the first time your code actually touches Spark: installation, session, reading, verification, each one executed and with literal output. Lesson 8 closes the module by pulling the previous seven together into a single delivery script.

Going deeper: why this guide doesn't start by writing Spark code

It's tempting, opening a guide that promises "Spark and distributed processing," to want to see spark.read.parquet(...) on the very first line of the very first lesson. This guide resists that temptation on purpose, and the reason isn't pedagogy for pedagogy's sake — it's real evidence of what happens when a team ignores it. The same market audit that confirms this guide's existence carries, from that same Hacker News thread, a concrete story: a user, patwolf, describes a company that "dumped Databricks once the first bill came". That isn't a technical failure of Spark — Spark did exactly what it was asked to do. It's a judgment failure: someone decided to distribute without first measuring whether the real volume justified it, and the cost of running the cluster ended up bigger than the problem it solved.

Learning Spark's API without the judgment for when to use it, too often, produces exactly that outcome: someone able to write a distributed groupBy().agg(), but without the instinct to first ask whether that groupBy().agg() would fit, with no drama at all, into a single DuckDB call. That's why this module — lesson 3 in particular — builds the criterion before the first line of PySpark. By the time you reach lesson 4 and write pip install pyspark, you'll already know, with evidence, that you're installing it for what it's worth in the job market and for what you're going to build across the rest of the guide — not because you assumed, without checking, that Kiosko needs it.

Common mistakes

Assuming that, because this is "the Spark guide," every lesson has to use Spark. What happens: someone opens this module expecting PySpark code starting in lesson 1, and on seeing two conceptual lessons without a single line of pyspark, assumes they're wasting time or skipped something. Why it happens: the guide's name creates a reasonable expectation of "Spark code in every lesson," but this guide, deliberately, invests in the criterion first. How to spot it: if you finish lesson 3 unable to explain, in your own words, why 650 GB and 1 TB are the numbers that matter (not "just because," but with the evidence behind them), you missed something more important than Spark's syntax. How to fix it: lessons 2 and 3 are the foundation for the rest of the guide — not a formality before "the interesting part." Lesson 4 is exactly where the code starts, and it arrives with the criterion already built.

Jumping straight to lesson 4 (installation), because "I already know the criterion." What happens: someone with prior Spark experience, or who's already heard the phrase "not everything needs Spark" somewhere, decides lessons 2 and 3 are repetition and skips them. Why it happens: the general idea — "you don't always need to distribute" — is popular and sounds obvious once you've heard it. How to spot it: if you can't recite, without looking, the three concrete numbers from the criterion (650 GB, 1 TB, 100 TB) and where they come from, your version of the criterion is a vague hunch, not the executable tool lesson 3 builds. How to fix it: lesson 3 doesn't repeat the general idea — it builds a real Python function, with cited thresholds, that you'll use again in the module 8 capstone against Kiosko's real data at scale. Skipping it leaves you without that tool when you need it.

Concluding, on seeing that Kiosko doesn't need Spark, that this guide is unnecessary. What happens: someone reads, in this very lesson, that Kiosko — forty rows — doesn't need Spark, and makes the leap to "then learning Spark isn't worth it either." Why it happens: it's easy to confuse "this specific case study doesn't need it" with "the tool isn't worth anything." How to spot it: if your conclusion from this lesson is "I'm not going to learn Spark," you misread the central point — the market evidence in this module's introduction (EMR, Databricks, DEA-C01, DP-700) confirms Spark is a real, in-demand skill. How to fix it: this module's point isn't "Spark is useless" — it's "learn first when you actually need it, then learn to use it well." The full guide does both: the rest of the eight modules build Spark in depth, with a synthetic dataset (module 4) designed exactly so you genuinely feel the need to distribute — something Kiosko's real scale never produces.

Exercises

Exercise 1 — Recite the criterion's three figures, without looking back. Without rereading this lesson, write from memory the three numbers the market evidence cited here anchors on (the approximate size of the average company's dataset, the ceiling under which DuckDB "has everything you need," and the volume almost no company exceeds). Then compare your answer against the introduction's text.

See solution

The three numbers are: 650 GB (a reasonable approximation of the average company's complete dataset, per the cited Hacker News thread), 1 TB (the ceiling under which, per aleda145, DuckDB alone "has everything you need"), and 100 TB (the volume "almost nobody in the world" exceeds, per the same thread). If you remembered all three, without needing the exact figures down to the last digit, you have a clear grasp of the numerical foundation lesson 3 builds an executable function on top of.

Exercise 2 — Explain patwolf's story in your own words. In 2-3 sentences, explain what happened to the company patwolf mentions in the Hacker News thread cited in the "Going deeper" section, and why that story is evidence of a judgment problem, not a technical problem with Spark or Databricks.

See solution

Per the quote, a company signed up for Databricks (managed Spark in the cloud) and dumped it as soon as the first bill arrived — meaning the real cost of running the platform exceeded the value it delivered. This isn't a technical failure: Databricks probably did exactly what it was asked to do, run distributed workloads. The failure was in the upfront decision: nobody measured, before signing up, whether the data volume and the real problem justified the cost of a managed cluster versus the alternative of a bigger or cheaper single node. It's concrete proof of why this guide insists on a cost criterion, not on "Spark is the trendy tool."

Exercise 3 — Tell apart "this guide exists" from "Kiosko needs it." In 2-3 sentences, explain why both statements are true at the same time without contradicting each other: (a) this guide covers a real, in-demand market skill, and (b) Kiosko, the forty-row case study, doesn't need Spark.

See solution

Both are true because they answer different questions. Statement (a) is about the job market in general: real companies, with real data volumes, do need people who know how to operate Spark — hence it showing up in job postings and certifications like the DEA-C01. Statement (b) is about a specific, deliberately small case study, chosen to teach the full pipeline without the complexity of a real-world volume. Kiosko not needing Spark doesn't invalidate the market demand for the skill — on the contrary, it's proof that this guide teaches you to apply that skill with judgment, not automatically every time someone opens a CSV file.

Summary and next step

This module closes the explicit debt data-modeling-for-analytics-guide left open: when volume stops fitting in a single node, and what changes — and what doesn't — when you distribute. You're not going to write a single Spark pipeline yet: first you're going to build the criterion for deciding whether you need one, and then you're going to install and verify that Spark actually runs on your machine, reading the same week of Kiosko data you already know from the three previous guides.

Before moving on you should be able to: explain, in your own words, why this guide doesn't start by writing Spark code; name the three central numbers from the market evidence (650 GB, 1 TB, 100 TB); and say from memory why Kiosko — at its real scale — doesn't need Spark, without that meaning the full guide is unnecessary.

Lesson 2 names, with technical precision and without exaggeration, exactly what the single-node ceiling is that DuckDB and Polars already pushed to its limit — the honest starting point before building the full criterion in lesson 3.

Resources

  • src/paths/data-engineering-ecosystem/VALIDACION.md — the internal market evidence audit (high confidence) backing this guide's verdict, the Hacker News thread quotes, and the "single-node engines first" gap this module addresses. Internal repository document, no public URL.
  • python-for-data-engineering-guide DESIGN doc — the source of the single-node limit with DuckDB/Polars this lesson names without repeating. src/guides/python-for-data-engineering-guide/DISENO.md
  • data-modeling-for-analytics-guide DESIGN doc — the source of the explicit boundary ("distributed computing... → spark-and-distributed-processing-guide") this guide closes. src/guides/data-modeling-for-analytics-guide/DISENO.md
  • PySpark — PyPI, the package page you'll install in lesson 4 (version 4.2.0, requires Python >=3.10). pypi.org/project/pyspark.
Next →The single-node ceiling: what DuckDB and Polars already solved