A real cost criterion for when to distribute

Description

This is the central lesson of the whole module — the one that turns "not everything needs Spark," a phrase anyone can repeat without committing to anything, into an executable Python function with concrete, cited, verifiable thresholds. You're going to build should_distribute(): given a dataset's real size and whether you've already exhausted single-node tools, the function returns a reasoned verdict, not an opinion. You'll apply it to five different scenarios — including real Kiosko — and see, with evidence, why the most profitable skill according to market practitioners isn't knowing how to distribute: it's knowing when NOT to.

Connection to the module. This lesson formalizes questions 2, 3, and 4 from the checklist lesson 1 presented, using the numbers lesson 2 already measured against Kiosko. The result — should_distribute() — is a tool you'll use again, against Kiosko's real data at scale, in the module 8 capstone.

An analogy: hiring the accounting team, not just "feeling" that you need to

Pick back up with the lone accountant from the previous module. There comes a point where their workload grows, and someone at the company starts to feel — with no concrete number behind it — that "this is already too much for one person." That feeling can be well-founded or it can be a false alarm: maybe the accountant just needs a faster calculator (improve the node, don't distribute), or maybe the real volume has already exceeded what any one person could sustain, no matter how much help you give them. The difference between those two situations isn't resolved with a feeling — it's resolved by counting: how many transactions per day, really? How long does it take to close them all? How much would it cost to hire a team, in salaries and coordination time, compared to the cost of simply accepting that closing the books takes a little longer? This lesson builds, in code, exactly that calculation — so the decision "I need a team" (distributing) stops being a feeling and becomes an answer you can defend with numbers.

Worked example: should_distribute(), applied to five scenarios

The criterion has three thresholds, anchored directly in the market evidence cited in lessons 1 and 2: below 1 TB, a single node with DuckDB/Polars is more than enough; above 100 TB, you're in the real minority group that almost certainly needs to distribute; and in the middle zone — between 1 TB and 100 TB, where most real decisions live — the answer depends on two more things: whether you've already exhausted single-node tools, and whether you measured the real cost of running a cluster.

# cost_criterion.py
def should_distribute(dataset_size_gb: float, exhausted_single_node_tools: bool) -> dict:
    """
    A cost criterion, not a fashion one, for deciding whether a dataset
    needs distributed Spark. Thresholds anchored in market evidence
    (Hacker News, 263-point thread, cited in VALIDACION.md):
      - "under 1TB [DuckDB] will have everything you need" (aleda145)
      - "[650GB] is a pretty reasonable approximation of the entire dataset
         most companies are even working with"
      - "almost nobody in the world uses datasets bigger than 100TB"
    """
    TB = 1000  # 1 TB expressed in GB
    HUNDRED_TB = 100_000  # 100 TB expressed in GB

    if dataset_size_gb < TB:
        verdict = "NO"
        reason = (
            f"{dataset_size_gb:,.4f} GB is below 1 TB -- the ceiling that "
            f"the DuckDB community itself reports as 'everything you're "
            f"going to need.' A single node with DuckDB or Polars is enough."
        )
    elif dataset_size_gb >= HUNDRED_TB:
        verdict = "YES, PROBABLY"
        reason = (
            f"{dataset_size_gb:,.0f} GB exceeds 100 TB -- the threshold "
            f"that, per the same thread, 'almost nobody in the world' "
            f"exceeds. You're in the real minority group that justifies "
            f"a cluster."
        )
    elif not exhausted_single_node_tools:
        verdict = "NOT YET"
        reason = (
            f"{dataset_size_gb:,.0f} GB is between 1 TB and 100 TB, but "
            f"single-node tools haven't been exhausted yet (out-of-RAM "
            f"reads, disk partitioning, columnar format). That's the "
            f"step missing before paying the cost of coordinating a cluster."
        )
    else:
        verdict = "IT DEPENDS -- MEASURE THE REAL COST"
        reason = (
            f"{dataset_size_gb:,.0f} GB is between 1 TB and 100 TB, and "
            f"single-node tools have already been exhausted. Here the "
            f"question is no longer technical, it's operational cost: "
            f"compare the price of a cluster (compute + people to run it) "
            f"against the cost of a slower job on a bigger single node. "
            f"This isn't decided by fashion."
        )

    return {
        "dataset_size_gb": dataset_size_gb,
        "exhausted_single_node_tools": exhausted_single_node_tools,
        "verdict": verdict,
        "reason": reason,
    }


if __name__ == "__main__":
    scenarios = [
        ("Real Kiosko -- the full week of 40 orders, measured in lesson 2", 2236 / 1_000_000_000, False),
        ("A mid-size e-commerce -- 300 GB of event logs accumulated over 2 years", 300.0, True),
        ("An IoT sensor company -- 5 TB of accumulated readings", 5_000.0, True),
        ("A company with 5 TB, having not tried anything beyond pandas yet", 5_000.0, False),
        ("A global clickstream platform -- 500 TB accumulated", 500_000.0, True),
    ]

    print("=== The cost criterion applied to five scenarios ===\n")
    for label, size_gb, exhausted in scenarios:
        result = should_distribute(size_gb, exhausted)
        print(f"Scenario: {label}")
        print(f"  Size: {result['dataset_size_gb']:,.6f} GB")
        print(f"  Single-node tools exhausted: {result['exhausted_single_node_tools']}")
        print(f"  Verdict: {result['verdict']}")
        print(f"  Reason: {result['reason']}")
        print()

What to expect. Running python3 cost_criterion.py, the output is exactly this:

=== The cost criterion applied to five scenarios ===

Scenario: Real Kiosko -- the full week of 40 orders, measured in lesson 2
  Size: 0.000002 GB
  Single-node tools exhausted: False
  Verdict: NO
  Reason: 0.0000 GB is below 1 TB -- the ceiling that the DuckDB community itself reports as 'everything you're going to need.' A single node with DuckDB or Polars is enough.

Scenario: A mid-size e-commerce -- 300 GB of event logs accumulated over 2 years
  Size: 300.000000 GB
  Single-node tools exhausted: True
  Verdict: NO
  Reason: 300.0000 GB is below 1 TB -- the ceiling that the DuckDB community itself reports as 'everything you're going to need.' A single node with DuckDB or Polars is enough.

Scenario: An IoT sensor company -- 5 TB of accumulated readings
  Size: 5,000.000000 GB
  Single-node tools exhausted: True
  Verdict: IT DEPENDS -- MEASURE THE REAL COST
  Reason: 5,000 GB is between 1 TB and 100 TB, and single-node tools have already been exhausted. Here the question is no longer technical, it's operational cost: compare the price of a cluster (compute + people to run it) against the cost of a slower job on a bigger single node. This isn't decided by fashion.

Scenario: A company with 5 TB, having not tried anything beyond pandas yet
  Size: 5,000.000000 GB
  Single-node tools exhausted: False
  Verdict: NOT YET
  Reason: 5,000 GB is between 1 TB and 100 TB, but single-node tools haven't been exhausted yet (out-of-RAM reads, disk partitioning, columnar format). That's the step missing before paying the cost of coordinating a cluster.

Scenario: A global clickstream platform -- 500 TB accumulated
  Size: 500,000.000000 GB
  Single-node tools exhausted: True
  Verdict: YES, PROBABLY
  Reason: 500,000 GB exceeds 100 TB -- the threshold that, per the same thread, 'almost nobody in the world' exceeds. You're in the real minority group that justifies a cluster.

Notice something important: the third and fourth scenarios have exactly the same size (5 TB), and yet the criterion gives them different verdicts (IT DEPENDS versus NOT YET). Size alone isn't enough — the difference is whether single-node tools have already been exhausted. This isn't a minor detail of the function: it's precisely the point the market evidence flags as the most expensive gap in the typical learning sequence — jumping straight from "I have a lot of data" to "I need Spark," without first going through "have I already tried everything a single node, well used, can do?"

Diagram: this lesson's decision tree

flowchart TD
    A["Dataset's real size\nmeasured in GB, not rows"] --> B{"Less than 1 TB?"}
    B -->|"Yes"| C["NO -- a single node\nwith DuckDB/Polars is enough"]
    B -->|"No"| D{"More than 100 TB?"}
    D -->|"Yes"| E["YES, PROBABLY --\nyou're in the minority group"]
    D -->|"No, it's between 1TB and 100TB"| F{"Have you already\nexhausted single-node\ntools?"}
    F -->|"No"| G["NOT YET -- exhaust\nDuckDB/Polars out-of-RAM first"]
    F -->|"Yes"| H["IT DEPENDS -- measure the real cost\nof a cluster vs a bigger node"]

Going deeper: why the "IT DEPENDS" zone is the most honest part, not the weakest

It's tempting to see the IT DEPENDS -- MEASURE THE REAL COST verdict as an unsatisfying answer — you'd expect a "real" criterion to always give a definitive yes or no. But that middle zone is, entirely on purpose, the most honest part of the criterion, not the weakest. Between 1 TB and 100 TB — the range where the real decision for most companies that actually need to think about this lives — the correct answer genuinely does depend on factors a fixed size threshold can't capture: how many people on the team know how to operate a cluster, how much managed infrastructure costs in your region, how urgent the response time is, whether the team already has Spark experience from another project.

patwolf's story cited in lesson 1 — the company that dumped Databricks as soon as the first bill arrived — is exactly a case that fell into this middle zone and made the decision without measuring the real cost first. This lesson's criterion doesn't pretend to replace that measurement with a magic formula — it aims, honestly, to point to the exact moment where the question stops being technical ("does it fit on a node?") and becomes a business question ("is the operational cost worth it?"). Recognizing that boundary precisely is, in itself, the criterion — not a weakness of it.

Common mistakes

Using the criterion once and never reapplying it when volume changes. What happens: someone runs should_distribute() once, at the start of a project, and never evaluates it again even as the dataset grows over time. Why it happens: a decision made feels permanent, and revisiting it feels like unnecessary extra work. How to spot it: if your project has gone months or years without anyone asking again "is this still the right decision at today's volume?", you're probably operating on a stale decision. How to fix it: treat the criterion as a recurring question, not a one-time decision — especially in the middle range (1 TB to 100 TB), where the relative cost of a cluster versus a bigger node can shift over time (infrastructure pricing, new versions of DuckDB/Polars, real business growth).

Confusing exhausted_single_node_tools=True with "I tried it once and it didn't work." What happens: someone marks single-node tools as exhausted after a single failed attempt, without exploring the real options (out-of-core, columnar format, a faster disk, more RAM). Why it happens: a first failed attempt feels like enough evidence that "it's not enough." How to spot it: if you can't name, precisely, which specific DuckDB or Polars configurations you tried before concluding a single node isn't enough, your exhausted_single_node_tools=True isn't well grounded. How to fix it: before marking that field true, confirm you actually tried out-of-core reads, a columnar engine (not a plain Python loop or pandas loading everything into memory), and — if applicable — a machine with more resources. "I tried it with pandas and it ran out of memory" is not the same as "I exhausted what a single node can do."

Applying the criterion to "how many rows I expect to have in the future" instead of today's real volume. What happens: someone justifies using Spark from day one of a small project, reasoning that "someday we're going to have a lot more volume." Why it happens: planning for future growth sounds responsible, and it's a legitimate concern in other engineering contexts. How to spot it: if your dataset today weighs a few GB but you justify Spark by citing a hypothetical volume "in two years," you're paying the cost of coordinating a cluster for a problem that doesn't exist yet. How to fix it: this lesson's criterion applies to today's real, measured volume — the same PySpark code you learn in this guide runs unchanged on a real cluster once the volume actually justifies it; there's no need to pay that operational cost ahead of time. Module 8 of this guide, precisely, applies this same criterion to a much larger hypothetical Kiosko, with numbers, not a hunch about the future.

Exercises

Exercise 1 — Find the exact point where the verdict switches from NO to NOT YET/IT DEPENDS. Using should_distribute(), write a small loop that tests sizes from 900 to 1100 GB, in steps of 50, with exhausted_single_node_tools=True, and prints the verdict for each. At what exact value does the verdict change?

for size_gb in range(900, 1150, 50):
    result = should_distribute(size_gb, True)
    print(f"{size_gb} GB -> {result['verdict']}")

900 GB -> NO
950 GB -> NO
1000 GB -> IT DEPENDS -- MEASURE THE REAL COST
1050 GB -> IT DEPENDS -- MEASURE THE REAL COST
1100 GB -> IT DEPENDS -- MEASURE THE REAL COST

Exercise 2 — Add a fourth parameter: required latency. Extend should_distribute() with a max_latency_minutes: float parameter. If the natural verdict would be NO but max_latency_minutes is less than 5 (a need for near-immediate response), change the verdict to "IT DEPENDS -- STRICT LATENCY, EVALUATE ANYWAY", with a reason explaining it. Test it with real Kiosko (2236 / 1_000_000_000 GB) and max_latency_minutes=2.

def should_distribute_v2(dataset_size_gb: float, exhausted_single_node_tools: bool, max_latency_minutes: float) -> dict:
    result = should_distribute(dataset_size_gb, exhausted_single_node_tools)
    if result["verdict"] == "NO" and max_latency_minutes < 5:
        result["verdict"] = "IT DEPENDS -- STRICT LATENCY, EVALUATE ANYWAY"
        result["reason"] = (
            f"Size alone would say NO, but the required latency "
            f"({max_latency_minutes} min) is very strict -- it's worth "
            f"confirming a single node meets that time, not just that "
            f"the volume fits it."
        )
    return result

print(should_distribute_v2(2236 / 1_000_000_000, False, 2))

{'dataset_size_gb': 2.236e-06, 'exhausted_single_node_tools': False, 'verdict': 'IT DEPENDS -- STRICT LATENCY, EVALUATE ANYWAY', 'reason': "Size alone would say NO, but the required latency (2 min) is very strict -- it's worth confirming a single node meets that time, not just that the volume fits it."}

Exercise 3 — Explain, without code, why two 5 TB scenarios gave different verdicts. This lesson's worked example showed two scenarios with exactly the same size (5 TB) but different verdicts (IT DEPENDS and NOT YET). In 2-3 sentences, explain why that's correct and not an inconsistency in the criterion.

Summary and next step

This lesson turned the intuition "not everything needs Spark" into an executable function with concrete thresholds: below 1 TB, a single node is enough; above 100 TB, you probably need to distribute; in the middle zone, the answer depends on whether you've already exhausted single-node tools and whether you measured the real operational cost of a cluster. You applied the criterion to five scenarios, including real Kiosko, and confirmed — with evidence, not intuition — that Kiosko doesn't need Spark today.

Before moving on you should be able to: explain should_distribute()'s three thresholds from memory; explain why two datasets of the same size can get different verdicts; and apply the criterion, mentally, to a dataset of your own.

With the criterion now built, lessons 4 through 8 stop talking about "when" and start building "how": installing PySpark and Java locally, opening your first SparkSession, and confirming Spark reads exactly the same week of Kiosko data you already know.

Resources

• src/paths/data-engineering-ecosystem/VALIDACION.md — the source of should_distribute()'s three thresholds and of the patwolf quote about the real cost of running a managed cluster. Internal repository document, no public URL.
• data-modeling-for-analytics-guide DESIGN doc — the source of Kiosko's star schema this guide picks back up, and of the explicit boundary that reserves distributed computing for this guide. src/guides/data-modeling-for-analytics-guide/DISENO.md
• PySpark — PyPI, the package page you install in lesson 4, once this lesson's criterion is already built. pypi.org/project/pyspark.