The single-node ceiling: what DuckDB and Polars already solved

Description

Before installing a single line of Spark, this lesson answers a question most "big data" courses never ask with precision: what, exactly, is a single node, and how far does it go before it hits a real ceiling? The answer isn't "a small computer" — it can be a machine with hundreds of cores and several terabytes of RAM — and it isn't "what fits in memory" — DuckDB and Polars process data that doesn't fit in RAM with no problem at all. The single-node ceiling is, precisely, the resource limit of a single machine (no matter how large), and this lesson measures, with Kiosko's real files, how far that ceiling actually is from the volume this case study produces.

Connection to the module. This lesson answers the first question from the checklist lesson 1 laid out as a map: "have you already exhausted what a single node can do?" Lesson 3 builds, on top of this foundation, the full executable criterion with the other three checklist points.

An analogy: the professional kitchen at home, and its maximum banquet

python-for-data-engineering-guide compared installing DuckDB to buying a professional kitchen for your own house instead of cooking by hand — industrial-grade pans, powerful burners, a huge countertop. That professional kitchen, compared with a two-burner home stove, cooks for far more people, far faster, without losing quality. You can improve that kitchen even further: add a second oven, a bigger exhaust hood, an industrial refrigerator. The limit of "one kitchen" is elastic — it stretches a long way before it breaks.

But it's still one kitchen, with a real physical ceiling: a maximum number of diners it can serve in one night, no matter how much you improve it, because there's a limit on space, on simultaneous burners, and on a single chef coordinating everything. That ceiling isn't a fixed "a hundred people" or "a thousand" — it depends on how good the kitchen is — but it exists. The single-node ceiling is exactly that: not a fixed number of rows, but the physical limit of a single machine, no matter how large you make it. This lesson measures how far Kiosko is from that limit, with real bytes, not with the feeling that "this already feels big."

Worked example: measuring the real distance between Kiosko and a node's ceiling

First, pull up Kiosko's seven real files — identical, byte for byte, to the ones foundations, python-for-data-engineering, and data-modeling all used: order_id,store_id,product_id,quantity,unit_price,order_ts, forty orders spread across August 3 to 9, 2026. If you don't have them handy, lesson 6 of this module rebuilds them in full; for now, you only need to know their real size on disk.

# ceiling_math.py
import glob
import os

files = sorted(glob.glob("orders_2026-08-*.csv"))
total_bytes = sum(os.path.getsize(f) for f in files)
total_rows = 40  # already verified by the three previous guides

bytes_per_row = total_bytes / total_rows
print(f"Files: {len(files)}")
print(f"Total bytes for Kiosko's real week: {total_bytes}")
print(f"Rows: {total_rows}")
print(f"Bytes per row (approx.): {bytes_per_row:.2f}")

GB = 1_000_000_000
TB = 1_000_000_000_000

# The three evidence anchors from lesson 1, cited from VALIDACION.md
targets = [
    ("650 GB (approx. the complete dataset of the average company, per HN)", 650 * GB),
    ("1 TB (DuckDB's stated ceiling in the same HN thread)", 1 * TB),
    ("100 TB (almost no company exceeds this, per the same thread)", 100 * TB),
]

WEEKS_PER_YEAR = 52
for label, target_bytes in targets:
    weeks_needed = target_bytes / total_bytes
    years_needed = weeks_needed / WEEKS_PER_YEAR
    print()
    print(f"To reach {label}:")
    print(f"  You would need {weeks_needed:,.0f} weeks identical to Kiosko's real week")
    print(f"  That's equivalent to {years_needed:,.0f} years of Kiosko history, with nothing deleted")

What to expect. Running python3 ceiling_math.py in the folder where you saved Kiosko's seven CSVs, the output is exactly this:

Files: 7
Total bytes for Kiosko's real week: 2236
Rows: 40
Bytes per row (approx.): 55.90

To reach 650 GB (approx. the complete dataset of the average company, per HN):
  You would need 290,697,674 weeks identical to Kiosko's real week
  That's equivalent to 5,590,340 years of Kiosko history, with nothing deleted

To reach 1 TB (DuckDB's stated ceiling in the same HN thread):
  You would need 447,227,191 weeks identical to Kiosko's real week
  That's equivalent to 8,600,523 years of Kiosko history, with nothing deleted

To reach 100 TB (almost no company exceeds this, per the same thread):
  You would need 44,722,719,141 weeks identical to Kiosko's real week
  That's equivalent to 860,052,291 years of Kiosko history, with nothing deleted

Stop on the 650 GB number, because it's the most relevant of the three: even the average dataset of a real company — according to the very market evidence that justifies this guide — would need Kiosko to generate that same week of forty orders, nonstop, for over five and a half million years. This isn't a serious projection of "when Kiosko is going to need Spark" — it's, deliberately, a concrete way to feel the real distance between this case study's volume and the volume the market evidence itself uses as a reference. Nobody expects the real Kiosko to ever get there; that's why module 4 of this guide later builds a dataset that's synthetic and declared as such (kiosko_orders_at_scale) — because the real Kiosko, the one with these forty rows, is never going to produce the volume needed for partitioning or distributing to make sense.

Diagram: a single node, elastic but bounded

┌─────────────────────────────────────────────────────────┐
│                      A SINGLE NODE                       │
│                                                            │
│   ┌──────────┐   ┌──────────┐   ┌────────────────────┐  │
│   │  CPU(s)  │   │   RAM    │   │  Disk (SSD/NVMe)    │  │
│   │ 1..N     │   │ elastic, │   │  elastic, TB        │  │
│   │ cores    │   │ up to TB │   │  possible           │  │
│   └──────────┘   └──────────┘   └────────────────────┘  │
│                                                            │
│           DuckDB / Polars run IN HERE                     │
│           (out of RAM if needed, columnar,                │
│           vectorized -- see python-for-data-engineering)  │
│                                                            │
│   Can be scaled up (more CPU, more RAM, faster disk)      │
│   but it's still ONE machine -- the ceiling exists,        │
│   it just moves further out.                               │
└─────────────────────────────────────────────────────────┘
                            │
                            │  when the real volume exceeds
                            │  what THIS node, scaled up to a
                            │  reasonable maximum, can sustain
                            ▼
┌───────────┐  ┌───────────┐  ┌───────────┐  ┌───────────┐
│  Node 1   │  │  Node 2   │  │  Node 3   │  │  Node N   │
│(coordinated│  │(coordinated│ │(coordinated│ │(coordinated│
│ by Spark) │  │ by Spark) │  │ by Spark) │  │ by Spark) │
└───────────┘  └───────────┘  └───────────┘  └───────────┘
      A CLUSTER -- the topic of the rest of this guide

Going deeper: why "doesn't fit in RAM" doesn't mean "I need a cluster"

The most common mistake when thinking about the single-node ceiling is confusing it with the size of available RAM. It's an understandable mistake — for years, many data analysis tools (pandas included) really did require the entire dataset to fit in memory before processing it. But DuckDB and Polars, the two tools python-for-data-engineering-guide covered in depth, don't have that limitation: both support out-of-core execution — they read and process data larger than the available RAM, in chunks, without loading the entire file at once. A Hacker News user, datadrivenangel, sums it up in a phrase worth remembering: "DuckDB is eating the query engines and catalogs" — a colloquial way of saying that a single, well-designed engine covers a far wider range of work than intuition suggests.

This has a direct practical consequence for the criterion you're going to build in lesson 3: the right question is never "does my data fit in my laptop's RAM?" — that question has an overly pessimistic answer, because it ignores out-of-core execution. The right question is "does my data fit in what a single, well-configured node — with fast disk, with a columnar engine that knows how to process in chunks — can sustain within the time my business needs?" That question has a far more generous answer, and it's the one the market evidence cited in lesson 1 answers with a concrete number: below 1 TB, according to the community that has already tested it in production, DuckDB "has everything you need."

Common mistakes

Measuring the "ceiling" in row count, not real bytes. What happens: someone concludes that "forty rows is small" and "a million rows is already big data," without measuring the real byte size of either case. Why it happens: counting rows is more intuitive than calculating bytes, and "a million" sounds big in the abstract. How to spot it: if your argument for "this needs Spark" is only a row count, with no calculation of disk or memory size, you don't have evidence yet — you have a feeling. A million rows of four short columns can weigh under 50 MB, well below any real ceiling. How to fix it: always convert the volume to bytes (or GB/TB) before drawing a conclusion, the way this lesson's worked example did — row count, on its own, says nothing about whether the dataset comfortably fits on a node.

Assuming that, if a dataset doesn't fit in RAM, you automatically need a cluster. What happens: someone sees that their file weighs more than their machine's RAM and concludes, without further thought, that they need Spark. Why it happens: the instinct of "loading everything into memory before working with it" comes from older tools that really did require it. How to spot it: if your reasoning is "it doesn't fit in RAM, so I need to distribute," you skipped a real intermediate option: out-of-core processing on a single node, exactly what DuckDB and Polars already do. How to fix it: before considering a cluster, confirm whether your single-node engine supports out-of-core execution (DuckDB and Polars do) — most cases that "don't fit in RAM" still fit, with no drama at all, on a single machine's disk.

Confusing "this guide teaches Spark" with "Spark is always the right answer from here on." What happens: someone, after installing Spark in lesson 4, starts using it as their default tool for any data task, even small ones. Why it happens: once you learn a new tool, it's tempting to use it for everything, especially if learning it took effort. How to spot it: if you find yourself writing spark.read.csv() for a file a few megabytes in size that DuckDB would read in one line, with no business reason to do it that way, you fell into this trap. How to fix it: lesson 3's criterion isn't a one-time exercise — it's a question you repeat every time you start a new project, including after you already know how to use Spark. Knowing how to use a tool and knowing when to use it are different skills, and this guide — starting with this module — insists on teaching both.

Exercises

Exercise 1 — Calculate the ceiling in "Kiosko-equivalents." Using ceiling_math.py, add a fourth target value: 10 GB (a small dataset, typical of an early-stage startup). How many weeks identical to Kiosko's would it take to reach that?

target_bytes = 10 * GB  # 10 GB in bytes
weeks_needed = target_bytes / total_bytes
print(f"To reach 10 GB: {weeks_needed:,.0f} weeks identical to Kiosko")

To reach 10 GB: 4,472,272 weeks identical to Kiosko

Exercise 2 — Explain, without code, why "out-of-core" changes the right question. In 2-3 sentences, explain the difference between asking "does my data fit in RAM?" and asking "does my data fit in what a single node can sustain?", and why DuckDB and Polars make the second question the right one.

Exercise 3 — Apply the ceiling to a scenario of your own. Think of some dataset you've worked with before (from a job, a course, a personal project) and estimate its approximate size in GB. Using this lesson's criterion (under 1 TB, DuckDB/Polars are more than enough), would that dataset need Spark? Justify your answer in 2-3 sentences.

Summary and next step

This lesson defined, with precision, what the single-node ceiling is: not a fixed row count and not the size of available RAM, but the combined capacity of CPU, RAM, and disk on a single machine — elastic, but bounded. By measuring the real bytes of Kiosko's week against the three anchors from the market evidence (650 GB, 1 TB, 100 TB), you confirmed with numbers — not intuition — that Kiosko is millions of years away from needing that ceiling. You also learned why "doesn't fit in RAM" isn't the same as "I need a cluster": DuckDB and Polars process out-of-core, and that changes the right question to ask.

Before moving on you should be able to: explain why a node's ceiling is measured in real bytes, not row count; explain the difference between "fits in RAM" and "fits in what a node can sustain"; and recite from memory the three numbers from the market evidence (650 GB, 1 TB, 100 TB).

Lesson 3 takes these numbers and turns them into an executable Python function: a real cost criterion, applicable to any dataset, that you'll use again in the module 8 capstone.

Resources

• src/paths/data-engineering-ecosystem/VALIDACION.md — the source of this lesson's three numbers and the literal quotes from the Hacker News thread (aleda145, datadrivenangel). Internal repository document, no public URL.
• python-for-data-engineering-guide DESIGN doc — the source of DuckDB/Polars out-of-core execution this lesson recalls without repeating in detail. src/guides/python-for-data-engineering-guide/DISENO.md
• DuckDB — official documentation for the Python client, the same tool python-for-data-engineering-guide covered in depth and that this lesson references as the engine that already solved the single-node ceiling. duckdb.org/docs/current/clients/python/overview.