The decision tree: does Kiosko actually need Spark?

Description

This is this entire guide's central lesson — not just this module's. You've made it here having installed Spark, rebuilt fact_orders with the DataFrame API, felt a real shuffle over ten million rows, chosen joins with real criteria, read Catalyst through its phases, written partitioned Parquet, and vectorized a UDF with Arrow. All of that work was real. This lesson takes should_distribute() — the cost-criterion function you built in module 1, lesson 3, with the explicit promise you were going to reuse it in this capstone — and applies it, with measured evidence and not intuition, to three different Kiosko scales: the real one (forty rows), the synthetic one this entire guide just built and ran (ten million rows), and a hypothetical one, much larger, with concrete numbers. The verdict for the first two is the same, and it's honest: no.

Connection to the module. This lesson resolves Deliverable 4 from lesson 2's brief — the question none of this module's earlier lessons dared dodge, because each one of them, deliberately, built the evidence this lesson needs to answer it with numbers, not an opinion.

An analogy: the engineer who audits their own work, with no favoritism

It's easy for someone who just spent eight modules building something to become biased toward their own work — after so much effort, it's tempting to conclude that, of course, it was needed. A real external auditor has no such bias: they apply the same criterion, regardless of who built the system or how much effort it cost. This lesson demands exactly that kind of honesty: should_distribute() doesn't know, and doesn't care, that it's being applied inside a complete guide dedicated to teaching Spark — it just compares real bytes against measured thresholds, and gives whatever verdict the numbers dictate, even if that verdict is "you didn't need it."

Worked example: the same criterion, three scales

The criterion, reused without changing a single line

# kiosko_decision_tree.py
import glob
import os


def should_distribute(dataset_size_gb: float, exhausted_single_node_tools: bool) -> dict:
    """The same criterion from module 1, lesson 3 (should_distribute()), reused
    without changing a single line of its internal logic. Thresholds anchored
    in market evidence (Hacker News, cited in VALIDACION.md):
      - "under 1TB [DuckDB] will have everything you need"
      - "[650GB] is a pretty reasonable approximation of the entire dataset
         most companies are even working with"
      - "almost nobody in the world uses datasets bigger than 100TB"
    """
    TB = 1000  # 1 TB expressed in GB
    HUNDRED_TB = 100_000  # 100 TB expressed in GB

    if dataset_size_gb < TB:
        verdict = "NO"
        reason = (
            f"{dataset_size_gb:,.6f} GB is below 1 TB -- the ceiling the "
            f"DuckDB community itself reports as 'everything you're going "
            f"to need.' A single node with DuckDB or Polars is enough."
        )
    elif dataset_size_gb >= HUNDRED_TB:
        verdict = "YES, PROBABLY"
        reason = (
            f"{dataset_size_gb:,.0f} GB exceeds 100 TB -- the threshold that, "
            f"per the same thread, 'almost nobody in the world' exceeds."
        )
    elif not exhausted_single_node_tools:
        verdict = "NOT YET"
        reason = (
            f"{dataset_size_gb:,.0f} GB is between 1 TB and 100 TB, but "
            f"single-node tools haven't been exhausted yet."
        )
    else:
        verdict = "IT DEPENDS -- MEASURE THE REAL COST"
        reason = (
            f"{dataset_size_gb:,.0f} GB is between 1 TB and 100 TB, and "
            f"single-node tools are already exhausted. Compare the cluster's "
            f"cost against a bigger node -- this isn't decided by fashion."
        )

    return {"dataset_size_gb": dataset_size_gb, "verdict": verdict, "reason": reason}

Scenario 1 — Real Kiosko: the forty-order week, measured in real bytes

print("=== Module 8, lesson 6: the decision tree applied to three Kiosko scales ===\n")

print("Scenario 1 -- real Kiosko: the 40-order week, measured in real bytes")
files = sorted(glob.glob("orders_2026-08-*.csv"))
real_bytes = sum(os.path.getsize(f) for f in files)
real_gb = real_bytes / 1_000_000_000
print(f"  files: {len(files)}, real bytes: {real_bytes}")
result1 = should_distribute(real_gb, exhausted_single_node_tools=False)
print(f"  dataset_size_gb = {result1['dataset_size_gb']:.9f}")
print(f"  Verdict: {result1['verdict']}")
print(f"  Reason: {result1['reason']}\n")
assert result1["verdict"] == "NO"

This is exactly the same calculation you already saw in module 1, lesson 2 (ceiling_math.py): seven files, 2236 bytes total, Kiosko's complete week. should_distribute() already gave this verdict back then, and gives it again here, with the same dataset, with no surprise — this decision tree's first link isn't new, it's confirmation the criterion stays consistent with itself.

Scenario 2 — this guide's SYNTHETIC dataset: the one you just processed

print("Scenario 2 -- kiosko_orders_at_scale: this guide's SYNTHETIC dataset, 10,000,000 rows")
scale_bytes = os.path.getsize("kiosko_orders_at_scale.csv")
scale_gb = scale_bytes / 1_000_000_000
print(f"  file's real bytes: {scale_bytes:,}")
result2 = should_distribute(scale_gb, exhausted_single_node_tools=False)
print(f"  dataset_size_gb = {result2['dataset_size_gb']:.6f}")
print(f"  Verdict: {result2['verdict']}")
print(f"  Reason: {result2['reason']}\n")
assert result2["verdict"] == "NO"

This is the scenario that genuinely matters, and the one a less honest course would probably avoid measuring. kiosko_orders_at_scale is the dataset you built in module 4 specifically so shuffle, partitioning, and joins would genuinely be felt — and it worked: you saw Exchange in the plan, measured real partitions, cached with real criteria. But the same cost criterion you built in module 1, applied to this dataset with its real size on disk, says NO — with the same clarity it gave the real forty-row week.

Scenario 3 — hypothetical enterprise-scale Kiosko: with concrete numbers

print("Scenario 3 -- hypothetical enterprise-scale Kiosko: 1,000 stores, 3 years, complete operational log")
NUM_STORES = 1_000
EVENTS_PER_STORE_PER_DAY = 50_000  # POS scans + app events, not just completed orders
DAYS = 365 * 3
BYTES_PER_EVENT = 300

total_events = NUM_STORES * EVENTS_PER_STORE_PER_DAY * DAYS
total_bytes_hypothetical = total_events * BYTES_PER_EVENT
hypothetical_gb = total_bytes_hypothetical / 1_000_000_000

print(f"  assumptions: {NUM_STORES:,} stores x {EVENTS_PER_STORE_PER_DAY:,} events/store/day x {DAYS:,} days")
print(f"  total_events = {total_events:,}")
print(f"  bytes_per_event = {BYTES_PER_EVENT}")
print(f"  total_bytes = {total_bytes_hypothetical:,}")
print(f"  dataset_size_gb = {hypothetical_gb:,.1f} GB ({hypothetical_gb / 1000:.3f} TB)")

result3 = should_distribute(hypothetical_gb, exhausted_single_node_tools=True)
print(f"  Verdict: {result3['verdict']}")
print(f"  Reason: {result3['reason']}\n")
assert result3["verdict"] == "IT DEPENDS -- MEASURE THE REAL COST"

print("=== Three scales, same criterion, three different verdicts for three different reasons ===")
print(f"  Real Kiosko (40 rows):              {result1['verdict']}")
print(f"  kiosko_orders_at_scale (10M rows):   {result2['verdict']}")
print(f"  Hypothetical Kiosko (16.4 TB):       {result3['verdict']}")

What to expect. Running python3 kiosko_decision_tree.py (executed in this run):

=== Module 8, lesson 6: the decision tree applied to three Kiosko scales ===

Scenario 1 -- real Kiosko: the 40-order week, measured in real bytes
  files: 7, real bytes: 2236
  dataset_size_gb = 0.000002236
  Verdict: NO
  Reason: 0.000002 GB is below 1 TB -- the ceiling the DuckDB community itself reports as 'everything you're going to need.' A single node with DuckDB or Polars is enough.

Scenario 2 -- kiosko_orders_at_scale: this guide's SYNTHETIC dataset, 10,000,000 rows
  file's real bytes: 601,305,672
  dataset_size_gb = 0.601306
  Verdict: NO
  Reason: 0.601306 GB is below 1 TB -- the ceiling the DuckDB community itself reports as 'everything you're going to need.' A single node with DuckDB or Polars is enough.

Scenario 3 -- hypothetical enterprise-scale Kiosko: 1,000 stores, 3 years, complete operational log
  assumptions: 1,000 stores x 50,000 events/store/day x 1,095 days
  total_events = 54,750,000,000
  bytes_per_event = 300
  total_bytes = 16,425,000,000,000
  dataset_size_gb = 16,425.0 GB (16.425 TB)
  Verdict: IT DEPENDS -- MEASURE THE REAL COST
  Reason: 16,425 GB is between 1 TB and 100 TB, and single-node tools are already exhausted. Compare the cluster's cost against a bigger node -- this isn't decided by fashion.

=== Three scales, same criterion, three different verdicts for three different reasons ===
  Real Kiosko (40 rows):              NO
  kiosko_orders_at_scale (10M rows):   NO
  Hypothetical Kiosko (16.4 TB):       IT DEPENDS -- MEASURE THE REAL COST

Pause on Scenario 2, because it's this entire guide's most honest — and most uncomfortable — claim: kiosko_orders_at_scale, the 601,305,672-byte file (roughly 573 MiB) you ran shuffles, broadcast joins, window functions, Catalyst, partitioned Parquet, and a pandas_udf over throughout seven complete modules, weighs 0.601306 GB — far below the 1 TB threshold where this guide's own criterion says a single node with DuckDB or Polars is already plenty. This guide used Spark over that dataset so you'd feel the mechanism — the shuffle, partition pruning, Arrow vectorization — not because the volume, measured with the same rigor demanded of any other dataset, required it. That's a distinction worth holding with complete clarity, not hiding.

Diagram: the complete decision tree, with the three scales placed on it

flowchart TD
    A["Dataset's real size,\nmeasured in GB"] --> B{"Less than 1 TB?"}
    B -->|"Yes"| C["NO -- a single node\nwith DuckDB/Polars is enough"]
    B -->|"No"| D{"More than 100 TB?"}
    D -->|"Yes"| E["YES, PROBABLY --\na real minority group"]
    D -->|"No, between 1TB and 100TB"| F{"Are single-node\ntools already\nexhausted?"}
    F -->|"No"| G["NOT YET -- exhaust\nDuckDB/Polars first"]
    F -->|"Yes"| H["IT DEPENDS -- measure\nthe cluster's real cost"]

    C -.->|"Real Kiosko\n2,236 bytes"| C
    C -.->|"kiosko_orders_at_scale\n0.60 GB"| C
    H -.->|"Hypothetical Kiosko\n16,425 GB / 16.4 TB"| H

Going deeper: why Scenario 3 uses "IT DEPENDS," not "YES"

It's worth pausing on why the hypothetical enterprise-scale Kiosko — 16.4 TB, much larger than anything else this guide has processed — falls under IT DEPENDS -- MEASURE THE REAL COST, and not directly into a categorical YES. The reason is in Scenario 3's explicit assumptions, and it's worth making them as visible as the final number: 1,000 stores — more than three hundred times Kiosko's real three — 50,000 operational events per store per day — not completed orders, but every point-of-sale scan, every inventory movement, every delivery-app event — sustained across 3 complete years, with a 300-byte record per event. That's a genuinely different data profile from what Kiosko produces today: it isn't "Kiosko with more stores," it's "a real regional chain with complete operational instrumentation."

And even with that aggressive assumption, the criterion doesn't jump straight to YES — it stops at IT DEPENDS, because 16.4 TB still sits far below the 100 TB threshold where "almost nobody in the world" operates, per the same market evidence module 1 cited. In that intermediate zone, module 1's lesson 3 already explained why the honest answer is never a fixed number: it depends on how much it costs, in your region and with your team, to operate a managed cluster against paying for one much larger single node with DuckDB or Polars. Notice, too, something connecting directly back to the rest of this guide: if someone from Kiosko honestly arrived at this scenario — 1,000 stores, complete instrumentation, 16 TB of operational data — the first step wouldn't be installing Spark immediately. It would be, following module 1's same criterion, first confirming whether one large single node, with DuckDB or Polars processing out of core, is already enough — exactly the same "exhaust the single node before paying the cost of coordinating a cluster" discipline that opened this entire guide.

Common mistakes

Reading Scenario 2's NO verdict as a criticism of this guide, or as evidence "Spark is useless." What happens: someone, seeing that not even the synthetic dataset built specifically for this guide justifies Spark by cost, concludes learning Spark was a waste of time. Why it happens: it's easy to confuse "this specific dataset didn't need it" with "this tool is useless." How to spot it: if your conclusion after this lesson is "so I shouldn't have learned Spark," reread this module's lesson 1's "going deeper" section — the PySpark code you wrote runs, unchanged, on a real cluster the day volume genuinely justifies it; what you learned is the mechanism, not a solution to a problem Kiosko never had. How to fix it: separate the two questions with complete clarity: "do I know how to operate Spark?" (yes, with seven modules of executed evidence) and "did Kiosko need Spark?" (no, with the same evidence). Both answers are true at the same time, and they don't contradict each other.

Tweaking Scenario 3's assumptions until forcing the verdict you want to see, instead of letting the numbers speak. What happens: someone, unsatisfied with IT DEPENDS, arbitrarily bumps up EVENTS_PER_STORE_PER_DAY or NUM_STORES until the result crosses into YES, PROBABLY, with no business justification for the new assumption. Why it happens: when a runnable criterion doesn't give the expected result, it's tempting to adjust its inputs until it does, instead of accepting the honest verdict. How to spot it: if you changed one of Scenario 3's assumptions and can't justify, in one sentence, why that new number is more realistic than the original — not just "because the result convinces me more this way" — you fell into this trap. How to fix it: any assumption in this scenario — events per store, number of stores, years of history, bytes per event — needs justifying with a concrete business reason, the same discipline any real capacity estimate demands; this lesson's criterion is only as honest as the assumptions fed into it, no more, no less.

Applying should_distribute() only once, at a real project's start, and never re-evaluating it as volume changes. What happens: someone uses this criterion to decide, today, that Kiosko doesn't need Spark, and never applies it again if Kiosko's business genuinely grew in the future. Why it happens: a decision made feels permanent, the same trap module 1's lesson 3 already warned about. How to spot it: if this lesson's criterion never gets re-evaluated as real data volume changes, you're operating on a potentially outdated verdict. How to fix it: treat should_distribute() as a recurring question, not a one-time decision — the same discipline module 1 already established, now applied to the same capstone you just built.

Exercises

Exercise 1 — Find the minimum number of stores that would push Scenario 3 from IT DEPENDS to YES, PROBABLY, keeping the other assumptions fixed. Using the same EVENTS_PER_STORE_PER_DAY=50_000, DAYS=1095, and BYTES_PER_EVENT=300, find the NUM_STORES value at which dataset_size_gb >= 100_000.

EVENTS_PER_STORE_PER_DAY = 50_000
DAYS = 1_095
BYTES_PER_EVENT = 300
HUNDRED_TB_GB = 100_000

bytes_per_store = EVENTS_PER_STORE_PER_DAY * DAYS * BYTES_PER_EVENT
gb_per_store = bytes_per_store / 1_000_000_000
stores_needed = HUNDRED_TB_GB / gb_per_store
print(f"GB per store (3 years): {gb_per_store:.4f}")
print(f"Stores needed to cross 100 TB: {stores_needed:,.0f}")

GB per store (3 years): 16.4250
Stores needed to cross 100 TB: 6,088

Exercise 2 — Apply should_distribute() to fact_orders_at_scale_m8.parquet, the partitioned file you wrote in lesson 3, instead of the original CSV. Use os.path.getsize() over every file inside the three partition folders (or, simpler, sum the whole directory's size with du/os.walk), and confirm whether the verdict changes relative to the original CSV.

import os

total_parquet_bytes = 0
for root, dirs, files in os.walk("fact_orders_at_scale_m8.parquet"):
    for f in files:
        total_parquet_bytes += os.path.getsize(os.path.join(root, f))

parquet_gb = total_parquet_bytes / 1_000_000_000
print(f"Partitioned Parquet's total size: {total_parquet_bytes:,} bytes ({parquet_gb:.4f} GB)")
result_parquet = should_distribute(parquet_gb, exhausted_single_node_tools=False)
print(f"Verdict: {result_parquet['verdict']}")

Partitioned Parquet's total size: ... bytes (0.1... GB)
Verdict: NO

Exercise 3 — Explain, without code, why this lesson measures Scenario 1 and Scenario 2 with exhausted_single_node_tools=False, but Scenario 3 with exhausted_single_node_tools=True. In 2-3 sentences, justify that difference using should_distribute()'s complete criterion.

Summary and next step

In this lesson you applied should_distribute() — the function you built in module 1 and promised to reuse here — to three different Kiosko scales, with the same honesty across all three: the real forty-row week (2,236 bytes, verdict NO), this guide's complete synthetic dataset (601,305,672 bytes, verdict NO, this entire capstone's most honest claim), and a hypothetical enterprise-scale Kiosko (16.4 TB, with explicit assumptions of 1,000 stores and three years of complete operational instrumentation, verdict IT DEPENDS -- MEASURE THE REAL COST). The same criterion, applied with no favoritism toward the tool this entire guide just taught you.

Before closing out this guide you should be able to: apply should_distribute() from memory to any dataset of your own; explain why this guide's own synthetic dataset doesn't cross the cost threshold; and explain the difference between "knowing how to operate Spark" and "knowing when Spark is needed" — the complete skill this guide, since its very first module, set out to teach.

Lesson 7 traces the complete map toward this ecosystem's sibling guides: what this distributed pipeline is still missing, and who resolves it.

Resources

• src/paths/data-engineering-ecosystem/VALIDACION.md — the source for should_distribute()'s three thresholds and the Hacker News thread quotes backing this lesson since module 1. Internal repository document, not a public URL.
• This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — the "Market warning" section integrating, from this guide's very design, the cost criterion as both the opening (M1) and closing (M8) framework. src/guides/spark-and-distributed-processing-guide/DISENO.md.
• This guide's module 1, lesson 3 (03-a-real-cost-criterion-for-when-to-distribute.md) — should_distribute()'s complete origin, with its own full set of examples and exercises.