The lie of the green checkmark

Description

Lesson 1 showed you, with a minimal example, that a status: SUCCESS can coexist with a completely wrong result. This lesson names that phenomenon and backs it with real evidence: it isn't a theoretical possibility this guide invented to justify its own existence — it's literally what data engineering practitioners report when they describe their worst incidents. You're going to read the exact quote that supports this guide's entire design, and you're going to understand why a pipeline that finishes red, even though it's bad news, is in a precise sense less dangerous than one that finishes green with bad data inside it.

Connection to the module. This lesson defines the central problem that the ecosystem's previous seven guides left unresolved, and that this module's seven remaining lessons begin to diagnose. It's, precisely, the "why" before the "how" — you still don't touch any S04 or validate_orders() code; that starts in lesson 4.

The quote that supports this entire guide

The market-evidence audit that validates this guide's design (src/paths/data-engineering-ecosystem/VALIDACION.md, confidence high) doesn't describe the data quality problem with abstract theory. It quotes, verbatim, the kind of incident practitioners report:

"The content should be what practitioners report, not theory: green failures ('The pipeline ran successfully — all green checkmarks' while 10% of transactions vanished), a source that changed from dollars to cents, seasonal thresholds, diff reports, reconciliation, WAP (write-audit-publish), and lineage."

Read that quoted sentence twice: "The pipeline ran successfully — all green checkmarks" — every visible indicator said everything was fine — while 10% of transactions vanished. Not 10% of rows with a minor typo. 10% of complete transactions, silently absent, while every checkmark in the system stayed green. Nobody got an alert. No dashboard turned red. The pipeline "worked," in the strictest and most useless sense of that word: it ran, it finished, it threw no exception — and yet it failed, in the only sense that matters to a business: the data that should have been there, wasn't.

The same quote names the second incident that supports this guide's central thread: "a source that changed from dollars to cents" — the same kind of bug you already saw, in miniature, in lesson 1's worked example (unit_price=60.00 instead of 0.60). You're going to run into that exact row, with that exact error, inside S04's first real file — lesson 6 of this module runs it for real, and lesson 7 confirms it slips through without a single warning.

An analogy: the disconnected smoke detector, not the absence of fire

Go back to lesson 1's analogy, but with a more precise detail. A disconnected smoke detector doesn't prevent a fire from existing — it only prevents someone from finding out it exists. From the outside, a building with a disconnected detector and a building with no fire at all look exactly the same: quiet, no alarms, apparently normal. The difference only shows up when it's too late for it to matter.

A pipeline with a green checkmark and broken data inside is that building with the disconnected detector. From the outside — a dashboard, a log, an automated Slack message that says "pipeline OK" — it looks exactly like a pipeline that actually finished well. The difference only shows up when someone, much later, tries to use that data for a real decision — a revenue report, an invoice, a sales commission — and discovers the number was wrong from the start. By then, the cost is no longer "fix a row" — it's "explain why a report that already went out was wrong."

Worked example: reconstructing the 10% incident

The incident quoted by VALIDACION.md doesn't come with source code — it's a synthesis of what practitioners report —, but it's worth reconstructing its exact mechanics with a minimal example, so that "10% of transactions vanished" stops being an abstract phrase and becomes something you can watch run:

# vanishing_transactions.py
def extract_and_load(source_rows: list[dict], destination: list[dict]) -> dict:
    """Naive version: a JOIN that drops rows without saying so."""
    known_customer_ids = {"C001", "C002", "C003"}  # the customer catalog, out of date

    loaded = 0
    for row in source_rows:
        if row["customer_id"] in known_customer_ids:  # implicit INNER JOIN
            destination.append(row)
            loaded += 1
        # rows with an unknown customer_id simply don't get in -- no log, no count

    return {"status": "SUCCESS", "rows_loaded": loaded}


source_rows = [
    {"order_id": "T-01", "customer_id": "C001", "amount": 42.00},
    {"order_id": "T-02", "customer_id": "C002", "amount": 18.50},
    {"order_id": "T-03", "customer_id": "C003", "amount": 9.99},
    {"order_id": "T-04", "customer_id": "C004", "amount": 120.00},  # new customer, not in the catalog
    {"order_id": "T-05", "customer_id": "C001", "amount": 33.25},
    {"order_id": "T-06", "customer_id": "C005", "amount": 61.10},  # new customer, not in the catalog
    {"order_id": "T-07", "customer_id": "C002", "amount": 15.00},
    {"order_id": "T-08", "customer_id": "C003", "amount": 88.00},
    {"order_id": "T-09", "customer_id": "C001", "amount": 24.00},
    {"order_id": "T-10", "customer_id": "C002", "amount": 50.00},
]

destination: list[dict] = []
result = extract_and_load(source_rows, destination)

print(f"status: {result['status']}")
print(f"rows_loaded: {result['rows_loaded']} out of {len(source_rows)} source rows")
print(f"Percentage that vanished: {round((1 - result['rows_loaded'] / len(source_rows)) * 100)}%")

What to expect. Running python3 vanishing_transactions.py, the output is exactly this:

status: SUCCESS
rows_loaded: 8 out of 10 source rows
Percentage that vanished: 20%

status: SUCCESS again — the for loop went through all ten rows, threw no exception, and extract_and_load() returned its result normally. But of the ten source transactions, only eight made it to destination — two vanished without a trace, because C004 and C005 are customers who exist in the real world (they bought something, generated a row) but not in the outdated catalog the code uses to filter. This example uses 20%, not the exact 10% from the quote — the real number depends on the specific incident each team lived through —, but the mechanism is identical: a silent filter, with no log of what it discarded, genuinely running — with no errors — while losing real data.

Diagram: why silence is more dangerous than an error

flowchart LR
    subgraph ROJO["Pipeline that finishes red"]
        A1["Real exception"] --> A2["Someone finds out\nimmediately"]
        A2 --> A3["It gets investigated,\nfixed,\ndocumented"]
    end

    subgraph VERDE["Pipeline that finishes green,\nwith bad data inside"]
        B1["Silent filter or bug"] --> B2["Nobody finds out"]
        B2 --> B3["The incorrect data\ngets used in reports,\ninvoices, decisions"]
        B3 --> B4["The error is discovered\nweeks or months later,\nif ever"]
    end

The diagram makes the real cost of each path explicit. The red path (A1 to A3) is uncomfortable in the moment, but cheap: the problem gets contained fast, precisely because it's visible. The green path (B1 to B4) looks more comfortable in the moment — nobody gets an alert, nobody has to investigate anything —, but it's far more expensive, because the cost doesn't disappear, it only gets postponed, and it grows while nobody knows about it. This is the concrete, not just rhetorical, reason why "a green pipeline with bad data is worse than a red one": red charges its cost immediately and in cash; green charges it later, with interest, and sometimes charges it to someone who didn't even know the problem existed.

Going deeper: why this isn't a "better programmers" problem

It's worth clarifying something that's easy to misread: the extract_and_load() code in the worked example has no syntax bug, no logic error in the traditional sense. if row["customer_id"] in known_customer_ids: does exactly what it says it does — it filters out rows whose customer_id isn't in the catalog. The problem isn't that the code is badly written. The problem is that nobody explicitly decided what should happen to rows that don't pass the filter — they should be rejected with a logged reason (as foundations' validate_orders() does), they should trigger an alert, or the catalog should be updated before running the pipeline. None of those three decisions got made; the code simply discarded them, silently, as a side effect of a filtering line that technically works fine.

This distinction matters because it changes where you look for the fix. It's not about writing "better" Python — the if in the example is perfectly idiomatic —; it's about building, apart from the transformation code, a separate system whose only job is to ask "how many rows came in, how many went out, and what happened to the difference?" — the question no pipeline answers on its own, unless someone makes it explicit. That question is, precisely, volume and consistency — two of the six dimensions that lesson 3 of this module is going to define exactly.

Common mistakes

Thinking "this won't happen to me" because the team is careful. What happens: someone reads the VALIDACION.md quote and mentally files it as "a disorganized-team problem," without seeing it as a real risk to their own work. Why it happens: it's more comfortable to think personal discipline is enough protection against this kind of failure. How to spot it: if your current pipeline has no automatic way to compare "how many rows came in" against "how many rows went out" at each step, you have exactly the same vulnerability the quote describes — no matter how careful you are writing each individual line. How to fix it: the 10% incident didn't happen because someone was careless writing a line of code — it happened because nobody built the separate system that asks "did anything vanish?" Building that system, not personal discipline, is what prevents the problem.

Confusing "zero exceptions" with "zero data loss." What happens: someone reviews a pipeline's logs, finds no ERROR or Traceback, and concludes the process was complete and correct. Why it happens: the absence of errors is the most visible and easiest signal to check, so it gets used as a substitute for a real verification. How to spot it: this lesson's worked example demonstrates the exact case — zero exceptions, and yet 20% real data loss. How to fix it: any pipeline step that filters, transforms, or joins data across sources needs, besides handling errors, to explicitly count how many rows came in and how many went out — and treat any unexpected difference as a signal to investigate, not as noise to ignore.

Expecting this problem to have a single-tool solution. What happens: someone, after reading this lesson, looks for "the tool" that solves the lie of the green checkmark once and for all, expecting to install something and be done. Why it happens: it's tempting to look for a single technical fix for a problem that's fundamentally about system design. How to spot it: if your mental plan after this lesson is "I'll install Pandera and that's it," you're missing the rest of this guide — Pandera (module 2) solves one part, referential consistency (module 3) another, contracts (module 4) another, anomaly detection (module 5) another, freshness and lineage (module 6) another, and governance (module 7) the last one. How to fix it: understand this entire guide as a system of complementary pieces, not a single tool — module 8's capstone is, precisely, the demonstration that the seven pieces together answer what none of them answers alone.

Exercises

Exercise 1 — Modify the worked example so it DOES log what it loses. Rewrite extract_and_load() so that, instead of silently discarding rows with an unknown customer_id, it counts them and reports them separately. You don't need to reject them entirely — just make visible what's invisible today.

def extract_and_load_honest(source_rows: list[dict], destination: list[dict]) -> dict:
    known_customer_ids = {"C001", "C002", "C003"}
    loaded = 0
    skipped_customer_ids = []

    for row in source_rows:
        if row["customer_id"] in known_customer_ids:
            destination.append(row)
            loaded += 1
        else:
            skipped_customer_ids.append(row["customer_id"])

    return {
        "status": "SUCCESS",
        "rows_loaded": loaded,
        "rows_skipped": len(skipped_customer_ids),
        "skipped_customer_ids": skipped_customer_ids,
    }


destination2: list[dict] = []
result2 = extract_and_load_honest(source_rows, destination2)
print(result2)

{'status': 'SUCCESS', 'rows_loaded': 8, 'rows_skipped': 2, 'skipped_customer_ids': ['C004', 'C005']}

Exercise 2 — Calculate the cost of the time gap. Suppose VALIDACION.md's 10% incident happened in a pipeline that runs once a day, and nobody noticed until a monthly revenue report didn't match another system's figures. In 2-3 sentences, estimate how many pipeline runs — and therefore how many days of data with the same problem — could have passed before someone detected it, and how likely it is to exactly recover the lost transactions from each of those days.

Exercise 3 — Argue why "fast red" beats "green that's slow to turn red." Using this lesson's diagram, explain in 2-3 sentences why a well-designed data system should, in many cases, prefer to fail loudly and fast, instead of trying to "keep running no matter what" in the face of suspicious data.

Summary and next step

In this lesson you named and backed with real evidence the problem that opened the module: the lie of the green checkmark, quoted verbatim from the market audit that supports this entire guide ("The pipeline ran successfully — all green checkmarks" while 10% of transactions vanished). You reconstructed, with an executed example, the exact mechanics of how a pipeline loses real data without throwing any exception, and saw why a silent failure is, in terms of real cost, more dangerous than a loud one.

Before moving on you should be able to: cite, in your own words, the market evidence behind this guide; explain why "zero exceptions" isn't synonymous with "zero data loss"; and argue why a loud, fast failure is preferable to a silent, slow one.

You have the problem named precisely. What you're still missing is the exact vocabulary to describe, unambiguously, what kind of problem a row of data has — a missing field isn't the same as a wrong price, even though both are "bad data." Lesson 3 builds that vocabulary: the six dimensions of data quality.

Resources

• src/paths/data-engineering-ecosystem/VALIDACION.md — the internal market audit (jul-19-2026, high confidence), the source of the literal quote this entire lesson rests on. Internal repo document, no public URL. In Spanish.
• Joe Reis & Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022) — the data quality chapter explicitly discusses the risk of silent failures versus loud failures. oreilly.com/library/view/fundamentals-of-data/9781098108298. In English.
• This guide's DESIGN — the full market warning, with the content mandate this lesson develops. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.