Module introduction: when green does not mean correct

Why this module exists

If you got here having gone through the eight previous guides in NIEVA's Data Engineering ecosystem, you've already seen Kiosko — the convenience store chain with a delivery app — finish, again and again, with a result painted green. data-engineering-foundations-guide built validate_orders() and a quality gate that separated good rows from broken ones. python-for-data-engineering-guide packaged that pipeline into a reproducible command. data-modeling-for-analytics-guide stood up a dimensional warehouse with dim_store and dim_product. dbt-analytics-engineering-guide versioned those transformations and put tests on them. airflow-and-declarative-orchestration-guide scheduled all of it to run on its own. spark-and-distributed-processing-guide scaled it to ten million rows. lakehouse-and-iceberg-guide gave those tables auditable history with Apache Iceberg. streaming-with-kafka-and-flink-guide made the data arrive in real time, not just in batches. Eight guides, eight pieces of real infrastructure, and all eight end the same way: the process runs, throws no exception, and something — a log, a terminal, a dashboard — turns green.

None of those eight guides answered the question that ultimately decides whether Kiosko can trust its own data: is the data that landed correct, or did only the process that produced it finish without error? They're different questions, and the distance between them is exactly where this guide lives. A dbt build that reports PASS says nothing about whether a row has the wrong price. An Airflow DAG that finishes green says nothing about whether a new store sent data with an empty field. An INFO line in validate_orders()'s log that says "Rejected: 0" says nothing about whether the rows that did pass are, in fact, correct — it only says none of them broke the rules someone thought to write. This guide — data-reliability-and-governance-guide, the ninth of the ecosystem's 17, and the one that closes its advanced level — teaches the discipline that closes that gap: data reliability engineering, with declarative quality tests, versioned data contracts, anomaly detection, lineage, and production-grade governance.

This module 1 doesn't build any new tool yet. It does something more fundamental and more urgent: it shows you, with evidence and with code you already know, exactly where the lie of the green checkmark lives, gives you the precise vocabulary to talk about data quality without loose adjectives, and uses the gate you already built in data-engineering-foundations-guide (module 5) to diagnose, live, what slips through it. It doesn't rewrite it. It doesn't improve it yet. It only puts it to the test against a real Kiosko file, and teaches you to read the result honestly.

The running case: Kiosko opens its fourth store

Kiosko doesn't change. Its three stores are still the same — S01 Kiosko Centro in Bogotá, S02 Kiosko Norte in Lima, S03 Kiosko Sur in Santiago —, its four products are still the same — P001 Bottled Water 600ml, P002 Energy Bar, P003 Instant Coffee Sachet, P004 Phone Charger Cable —, and the fixed week of forty orders (2026-08-03 through 2026-08-09) still anchors the same total revenue already verified by the eight previous guides: 106.15. This guide doesn't touch that number. Everything it introduces lives in a new incident, later and separate in time.

The thread of this guide is the opening of S04 Kiosko Reforma, in Mexico City — Kiosko opens its fourth store. And S04 doesn't come out of nowhere: it already had a first contact with Kiosko's pipeline, and that first contact went badly. Lesson 5 of this module tells that story precisely, citing the exact code from data-engineering-foundations-guide (module 7). For now it's enough to know this: S04 was already selling before its catalog was formally updated, and that desync made an entire pipeline finish red — not silently red, but with an explicit ValueError. This guide is, in a sense, the answer to that uncomfortable question: if a loud failure like that already happened once, what's happening right now, silently, inside the rows that do pass?

An analogy: the smoke alarm that was never installed

Picture two office buildings. The first has a fire alarm installed, wired, with batteries checked every month — and one day, a short circuit produces real smoke in the server room. The alarm sounds. It's a bad moment, but it's exactly the right moment for it to sound: someone reacts, the fire is contained before it spreads, and the building stays standing. The second building has no alarm installed at all. The same short circuit produces the same smoke, in the same server room — but nobody knows, because there's no sensor to detect it. The silence in that second building doesn't mean there's no fire. It means, precisely, that no one is going to find out until it's too big to put out with an extinguisher.

A data pipeline that finishes red — like foundations M7's, when S04 made transform_fact_orders() throw a ValueError — is the first building: the alarm sounded, someone noticed, the problem was contained. A pipeline that finishes green with bad data inside it is the second building: no error, no exception, no red line in the log — but that doesn't mean the data is fine. It means nobody installed the sensor that would detect the real problem. This entire guide — and this module in particular — is the installation of that alarm, starting by understanding, precisely, which sensors already exist (foundations M5's gate) and which are entirely missing.

Worked example: what a green pipeline can and cannot tell you

Before touching S04's real file, it's worth seeing, with a minimal example, the exact difference between "the process didn't throw any exception" and "the data is correct." This block doesn't need any Kiosko file — it's deliberately simple, so the point stays isolated from any other detail:

# green_checkmark_lie.py
def process_orders(rows: list[dict]) -> dict:
    """A deliberately naive version: it only confirms the code ran."""
    total_revenue = 0.0
    for row in rows:
        total_revenue += row["quantity"] * row["unit_price"]
    return {"status": "SUCCESS", "rows_processed": len(rows), "total_revenue": round(total_revenue, 2)}


# a row with a real pricing bug: someone typed 60.00 instead of 0.60
rows = [
    {"order_id": "ORD-A", "quantity": 2, "unit_price": 1.20},
    {"order_id": "ORD-B", "quantity": 1, "unit_price": 0.55},
    {"order_id": "ORD-C", "quantity": 1, "unit_price": 60.00},  # should be 0.60
]

result = process_orders(rows)
print(f"status: {result['status']}")
print(f"rows_processed: {result['rows_processed']}")
print(f"total_revenue: {result['total_revenue']}")

What to expect. Running python3 green_checkmark_lie.py, the output is exactly this:

status: SUCCESS
rows_processed: 3
total_revenue: 62.95

Read this carefully, because it's the entire point of this introductory lesson: status: SUCCESS isn't false — the code did, in fact, run without any exception, processed the three rows, and returned a number. But total_revenue: 62.95 is wrong by almost two orders of magnitude: if ORD-C had had the correct price (0.60, not 60.00), the real total would be 2.40 + 0.55 + 0.60 = 3.55, not 62.95. process_orders() has no way of knowing this — nobody gave it a rule that said "a unit_price of 60.00 is suspicious for this product." The green checkmark (SUCCESS) is honest about one thing (the process finished) and completely silent about the other (whether the result makes sense). That is, precisely, the lie: it's not that the system lies on purpose — it's that nobody asked it the right question.

Diagram: two different questions, one visible answer

flowchart TD
    A["Pipeline runs"] --> B{"Did it throw\nan exception?"}
    B -->|"yes"| C["status = FAILED\n(red, visible, someone reacts)"]
    B -->|"no"| D["status = SUCCESS\n(green, visible)"]
    D --> E{"Is the data\nthat landed\ncorrect?"}
    E -->|"nobody asked"| F["Nobody knows.\nGreen doesn't answer this."]
    E -->|"a quality system\nasked"| G["Real answer:\nyes or no, with evidence"]

The diagram points to the exact failure: the path on the left (status = FAILED) does answer a real question, even if it's bad news. The path on the right, as a pipeline runs without a dedicated quality system, stops at D — it never reaches E. status = SUCCESS answers "did it run without errors?", never "is the result correct?". Confusing those two questions — treating a SUCCESS as if it were also a "yes, the data is fine" — is, precisely, the lie of the green checkmark.

The map of this guide's 8 modules

data-reliability-and-governance-guide is the ninth guide in NIEVA's Data Engineering ecosystem, and it closes its advanced level. Eight modules make it up:

Notice the progression: this module 1 gives you the vocabulary and the diagnosis. Modules 2 and 3 give you the first declarative tools. Module 4 introduces the artifact that ties everything together — the contract. Module 5 solves the hardest of the six dimensions. Module 6 moves the focus from the row to the whole table. Module 7 closes with what to do when something really fails, and who can see what. And module 8 assembles all eight pieces into a single system.

The map of this module

Within module 1, eight lessons build the idea step by step:

Lesson    Question it answers
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Where we're coming from, and where this module goes
L2        What does it mean, with real market evidence, that a
          green pipeline can be worse than a red one?
L3        What, precisely, are the six dimensions of
          data quality?
L4        What does foundations M5's validate_orders() catch,
          and NOT catch?
L5        Who is S04, and why did it already collide once with
          Kiosko's pipeline?
L6        Actually run validate_orders() on S04's
          first real file
L7        What slips past that gate, in the rows that
          DID pass?
L8        Project: diagnose, without fixing anything yet,
          S04's silent failures

Lessons 2 and 3 build the vocabulary — the lie of green, and the six dimensions. Lesson 4 precisely recalls validate_orders()'s exact contract: what it promises, and what it never promised. Lessons 5, 6, and 7 are the real diagnosis: who S04 is, running the old gate on its first file, and honestly reading what it missed. And lesson 8 — the project — turns that diagnosis into a written report, the foundation the rest of the guide builds every solution on.

The boundary: what does NOT belong in this module (or in this guide)

This module does not rewrite validate_orders(). The local gate — schema, nulls and type, business rules, duplicates — was already taught in depth in data-engineering-foundations-guide (module 5), and here it's re-run exactly as it is, with no changes, with a single goal: diagnosing its limits. Pandera isn't installed yet either — that starts in module 2 —; in this module, the only tool that actually runs is the one you already know.

And at the level of the guide as a whole, the boundary with its sister guides in the ecosystem is already drawn:

• The basic quality gate (schema/nulls/type/range over a single table, with hand-written Python) → already taught by data-engineering-foundations-guide (M5). Here it deepens into declarative tests, versioned contracts, and observability — the lesson is never repeated from scratch.
• Declarative tests inside a dbt project (data_tests:, schema.yml) → dbt-analytics-engineering-guide. This guide's tests are reusable over any DataFrame, inside or outside dbt.
• Snapshots and auditable history at the table-format level (Apache Iceberg, time travel) → lakehouse-and-iceberg-guide. This guide builds the contract-and-quality system that would decide what's allowed to write to that table — the relationship is named, without requiring PyIceberg.
• Orchestrating checks as scheduled tasks (DAGs, sensors, retries) → airflow-and-declarative-orchestration-guide. Here everything runs by hand, from the terminal.
• Real streaming/CDC as the data source → streaming-with-kafka-and-flink-guide. S04's file arrives as a batch CSV, on purpose.
• Distributed compute → spark-and-distributed-processing-guide. Kiosko's volume stays toy-sized on purpose — the focus is trust, not scale.
• Infrastructure security (real IAM, KMS, VPC) → aws-core-services-guide / cloud-security-and-guardrails-guide. This guide's "governance" (module 7) is data governance — which column which role can see —, not infrastructure.
• Anomaly detection with Machine Learning → outside the scope of NIEVA Data Engineering. This guide's detection (module 5) is deterministic: rules and thresholds, always explainable in one sentence.

Common mistakes

Assuming this module is going to "fix" S04's file. What happens: someone, seeing that lessons 6 and 7 find problem rows in S04's file, expects the module to end by correcting those prices or filling in those empty fields. Why it happens: it's the natural instinct when facing a problem — solve it the moment you spot it. How to spot it: if you finished this module expecting a "clean" S04, re-read lesson 8's goal — it's a diagnosis, not a fix. How to fix it: this entire module exists so you understand, precisely and with evidence, what's broken and why the current tool doesn't catch it — fixing it is, literally, the job of the seven modules that follow, each with the right piece (Pandera, referential consistency, contracts, anomalies, freshness/lineage, governance).

Treating "rejected: 0" or "no exceptions" as synonymous with "everything is fine." What happens: someone runs a pipeline, sees a clean report or an error-free terminal, and considers the matter closed without asking anything else. Why it happens: it's the most visible, most immediate signal a system gives — and across the previous eight guides, that signal was, in fact, almost always enough. How to spot it: lesson 2 of this module, together with this very lesson's worked example, already showed you the exact case — a SUCCESS that's completely honest about a total that's completely wrong. How to fix it: learn to ask yourself, facing any green result, "what specific question did this green answer, and what question did it leave unanswered?" — it's the central habit this entire guide trains.

Thinking that foundations M5's gate was "badly built." What happens: seeing, in lesson 7, that validate_orders() lets through rows with real problems, someone concludes that function had a design flaw. Why it happens: it's tempting to judge a tool by what it doesn't do, instead of by what it actually promises to do. How to spot it: go back to lesson 4 of this module — validate_orders() fulfills exactly its original contract: schema, nulls, type, simple business rules, duplicates. It never promised referential integrity or price anomaly detection. How to fix it: foundations' gate isn't "badly built" — it's incomplete by design, because it solved a different, simpler problem. This guide's real progress isn't "fixing a bug," it's "extending the scope of what gets asked."

Exercises

Exercise 1 — Rewrite the worked example with a different bug. Using process_orders() from this lesson's worked example, build your own list of rows with a quality problem different from the pricing one (for example, a negative quantity) and confirm that process_orders() also reports status: SUCCESS with no warning at all.

rows_with_negative_quantity = [
    {"order_id": "ORD-X", "quantity": 2, "unit_price": 0.55},
    {"order_id": "ORD-Y", "quantity": -3, "unit_price": 0.75},  # impossible quantity
]
result = process_orders(rows_with_negative_quantity)
print(result)

{'status': 'SUCCESS', 'rows_processed': 2, 'total_revenue': -1.15}

Exercise 2 — Name the two questions. Without looking at this lesson's diagram, write from memory the two distinct questions a pipeline can answer, and explain which of the two a status: SUCCESS answers by itself.

Exercise 3 — Connect it to S04's backstory. Without having read lesson 5 of this module yet, and based only on what this introductory lesson says, write in 2-3 sentences your hypothesis for why S04's first contact with Kiosko's pipeline ended in a red ValueError, instead of passing silently like this lesson's green-checkmark example.

Summary and next step

In this lesson you saw the exact point where the previous eight guides in the ecosystem left you — processes that finish green, never answering whether the data that landed is correct —, and confirmed, with a minimal, executed example, that a completely honest status: SUCCESS can coexist with a completely wrong result. You walked through the full map of this guide's eight modules and this module's eight lessons, and drew the boundary with the ecosystem's sister guides.

Before moving on you should be able to: explain, in your own words, the difference between "the process finished without errors" and "the result is correct"; and name the unit_price=60.00 example as a concrete instance of that difference.

Lesson 2 earns the right to name the problem precisely: before touching any of S04's code, it revisits the real market evidence that motivates this entire guide, and defines exactly what "the lie of the green checkmark" means.

Resources

• Joe Reis & Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022) — the data quality framework this entire module rests on, already cited since data-engineering-foundations-guide. oreilly.com/library/view/fundamentals-of-data/9781098108298. In English.
• data-engineering-foundations-guide DESIGN — source of validate_orders() and of S04's rejection in module 7, the backstory this module cites precisely in lesson 5. src/guides/data-engineering-foundations-guide/DISENO.md. In Spanish.
• src/paths/data-engineering-ecosystem/VALIDACION.md — the internal market audit (jul-19-2026, high confidence) that supports this entire guide's content mandate, including the literal quote in lesson 2. Internal repo document. In Spanish.
• This guide's DESIGN — the full map of the eight modules, including the market warning on data reliability engineering. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.