Project: diagnosing S04's silent failures

Description

This project closes the module. You have the six-dimension vocabulary (lesson 3), the exact classification of what validate_orders() covers and what it doesn't (lesson 4), S04's full history (lesson 5), the real run of the old gate against its first file (lesson 6), and the analysis of what slips through and why (lesson 7). One step remains: pulling all of that together into a single diagnostic script, with a final report anyone at Kiosko — with no need to read Python code — can understand at a glance. And a constraint that defines this project as much as its content: nothing gets fixed yet.

Connection to the module. This project introduces no new concept — it's the complete synthesis of lessons 2 through 7, applied end to end to S04's real incident. It closes this module's thread exactly where module 2 picks it back up: the report this project produces is, literally, the list of problems Pandera — module 2's protagonist — is going to start solving declaratively and reusably.

An analogy: the doctor's report before surgery

Before operating, a surgeon doesn't start cutting. They order tests — blood work, X-rays, an MRI if needed — and with those results write a pre-op report: what's wrong, exactly, with what evidence, and how severe each finding is. That report doesn't cure anything by itself. Its value is that it turns a vague suspicion ("the patient doesn't feel well") into a precise, actionable list ("elevated blood pressure, an inflammatory marker out of range, a tissue area that needs a biopsy") — the foundation any subsequent treatment can be planned on with judgment, instead of blindly.

This project is that pre-op report, applied to S04's first file. It doesn't fix the 60.00 price, doesn't fill in the empty field, doesn't remove the duplicate row — it writes down, with executed, verifiable evidence, exactly what's wrong, which quality dimension each finding corresponds to, and how severe it is. That diagnosis is, precisely, what makes it possible for the seven modules that follow to build the right solution for each problem, instead of generic patches applied blindly.

The material: everything this module built, in one place

You need, in a new working directory:

kiosko_diagnosis/
├── kiosko.py                     (lesson 4: validate_orders() complete, no changes)
├── orders_2026-08-14.csv         (lesson 6: S04's 12 real lines)
└── diagnose_s04.py                (this project: pulls it all together into a report)

The reference solution, verified

# diagnose_s04.py -- module 1 closing project
# diagnoses S04's first file, without fixing anything yet
import csv
from datetime import datetime

from kiosko import PRODUCTS, validate_orders, print_validation_report

KNOWN_PRODUCT_IDS = {p["product_id"] for p in PRODUCTS}
CANONICAL_WEEK_PRICE_P002 = 1.20  # confirmed across the 40 rows of the canonical week (foundations M1-M2)
PIPELINE_RUN_AT = "2026-08-16T09:00:00"  # this guide's fixed "now" -- never datetime.now()
SLA_DEADLINE = "2026-08-15T00:00:00"     # EXPECTED_ARRIVAL (2026-08-14) + 24-hour SLA


def find_silent_failures(valid_rows: list[dict]) -> list[tuple[dict, str, str]]:
    """Finds, among the rows that ALREADY passed validate_orders(), the ones
    that have a real problem that gate isn't designed to see."""
    failures = []
    for row in valid_rows:
        if row["product_id"] not in KNOWN_PRODUCT_IDS:
            failures.append((row, "consistency", f"product_id '{row['product_id']}' doesn't exist in the catalog"))

        price = float(row["unit_price"])
        if row["product_id"] == "P002" and abs(price - CANONICAL_WEEK_PRICE_P002) / CANONICAL_WEEK_PRICE_P002 > 0.5:
            ratio = price / CANONICAL_WEEK_PRICE_P002
            failures.append((row, "accuracy", f"unit_price={price} is {ratio:.1f}x off the canonical price ({CANONICAL_WEEK_PRICE_P002})"))
    return failures


def check_file_freshness(run_at: str, deadline: str) -> float:
    """Returns how many hours the quality apparatus is past the SLA. Positive = violated."""
    return (datetime.fromisoformat(run_at) - datetime.fromisoformat(deadline)).total_seconds() / 3600


def main() -> None:
    print("=== S04 diagnosis: orders_2026-08-14.csv ===\n")

    with open("orders_2026-08-14.csv", newline="") as f:
        rows = list(csv.DictReader(f))
    print(f"Rows read: {len(rows)}\n")

    valid, rejected = validate_orders(rows)
    print_validation_report(valid, rejected)

    print("\n=== 'valid' rows with problems validate_orders() didn't detect ===")
    silent_failures = find_silent_failures(valid)
    for row, dimension, reason in silent_failures:
        print(f"  {row['order_id']} [{dimension}]: {reason}")

    hours_late = check_file_freshness(PIPELINE_RUN_AT, SLA_DEADLINE)
    print(f"\n=== Freshness (a property of the whole file, not of any row) ===")
    print(f"SLA expired on: {SLA_DEADLINE}")
    print(f"Reviewed on: {PIPELINE_RUN_AT}")
    print(f"Hours late past the SLA: {hours_late}")

    print("\n=== Final summary ===")
    print(f"Total rows: {len(rows)}")
    print(f"Caught by validate_orders(): {len(rejected)} (completeness, validity, uniqueness)")
    print(f"Silent within 'valid': {len(silent_failures)} (consistency, accuracy)")
    print(f"File freshness: VIOLATED ({hours_late}h past the SLA)")
    print(f"Quality dimensions touched by this incident: 6 of 6")


if __name__ == "__main__":
    main()

What to expect (verified by actually running python3 diagnose_s04.py, with kiosko.py and orders_2026-08-14.csv in the same directory):

=== S04 diagnosis: orders_2026-08-14.csv ===

Rows read: 12

=== Kiosko: validation report ===
Total rows: 12
Valid: 9
Rejected: 3

=== Rejected row details ===
ORD-9503:
  - field 'unit_price' is null or empty
ORD-9507:
  - quantity must be > 0, got -1
ORD-9502:
  - duplicate order_id 'ORD-9502'

=== 'valid' rows with problems validate_orders() didn't detect ===
  ORD-9508 [consistency]: product_id 'P099' doesn't exist in the catalog
  ORD-9509 [accuracy]: unit_price=60.0 is 50.0x off the canonical price (1.2)

=== Freshness (a property of the whole file, not of any row) ===
SLA expired on: 2026-08-15T00:00:00
Reviewed on: 2026-08-16T09:00:00
Hours late past the SLA: 33.0

=== Final summary ===
Total rows: 12
Caught by validate_orders(): 3 (completeness, validity, uniqueness)
Silent within 'valid': 2 (consistency, accuracy)
File freshness: VIOLATED (33.0h past the SLA)
Quality dimensions touched by this incident: 6 of 6

Read the final summary carefully, because it's this entire module's complete result, condensed into five lines: of the twelve rows in S04's first file, three were correctly rejected (completeness, validity, uniqueness), two passed through with no warning at all despite having real problems (consistency, accuracy), and the whole file — independent of any individual row — violated the freshness SLA by 33 hours. The six quality dimensions lesson 3 defined, all six, are represented in this single incident — this isn't a pedagogical-design coincidence detached from reality: it's exactly the kind of file a real data engineer encounters when a new producer, without a formal contract yet, starts sending data.

Diagram: where you came from, where you landed

flowchart LR
    A["Lessons 1-3:\nthe problem\nand the vocabulary"] --> B["Lesson 4:\nwhat validate_orders()\ncovers and doesn't"]
    B --> C["Lesson 5:\nwho S04 is,\nthe M7 backstory"]
    C --> D["Lesson 6:\nvalidate_orders()\nactually run:\n9 valid, 3 rejected"]
    D --> E["Lesson 7:\nP099 and 60.00,\nsilent inside\n'valid'"]
    E --> F["This project:\ncomplete report,\n6 of 6 dimensions"]
    F --> G["Module 2:\nPandera -- the first\ndeclarative tool"]

Closing lesson 1's promise, point by point

No row of orders_2026-08-14.csv got fixed yet — not ORD-9509's price, not ORD-9503's empty field, not ORD-9508's broken reference. That's exactly correct at this point in the guide: this module delivers the complete diagnosis, with executed evidence. Fixing each problem, with the right tool for each dimension, is the job of the seven modules that follow.

Common mistakes

Adding fix-up code "while we're at it" inside diagnose_s04.py. What happens: someone, seeing ORD-9509's anomalous price in the report, adds a line that corrects it to 0.60 directly inside the diagnostic script, "so the final report looks clean." Why it happens: after identifying a problem so clearly, fixing it feels like the natural next step. How to spot it: if your diagnose_s04.py ends up modifying orders_2026-08-14.csv or building a "corrected" version of the rows, it's no longer a diagnosis — it mixes two responsibilities this guide deliberately keeps separate. How to fix it: a diagnostic script describes, never fixes — it's the same principle foundations M5 already established about print_validation_report() in its own Exercise 3 ("the gate never invents values, it only separates, counts, and explains"), applied here at a broader level.

Treating find_silent_failures() as if it were already the solution for modules 3 and 5. What happens: someone, satisfied with this project's report, assumes there's no longer any need to build validate_referential_integrity() (module 3) or check_price_baseline() (module 5), because "I already have a function that finds those problems." Why it happens: find_silent_failures() does find the real problems, with code that actually runs. How to spot it: check find_silent_failures()'s signature — it receives valid_rows, not a reusable DataFrame; its consistency logic compares against KNOWN_PRODUCT_IDS, a hand-coded constant, not against dim_product read from the warehouse; its accuracy logic uses a single fixed price (CANONICAL_WEEK_PRICE_P002 = 1.20), not a baseline genuinely computed over the canonical week's forty rows. How to fix it: understand this function for what it is — the manual diagnosis that closes this module —, not the reusable architecture modules 3 and 5 are going to build with data read live from the warehouse.

Forgetting that "6 of 6 dimensions" is a pedagogical-design coincidence, not a general rule. What happens: someone generalizes from this single case — where all six dimensions appeared exactly once each — to the idea that any file of broken data always has that perfect distribution. Why it happens: this project's clean, tidy result feels like a natural pattern, not like an example deliberately built to teach. How to spot it: if you expect the next problem file you encounter in your real work to have "one broken row per dimension," you're going to be surprised — real data rarely arrives that neatly organized. How to fix it: remember that orders_2026-08-14.csv was deliberately built with that exact structure — twelve lines, six clean and six broken, one per dimension — precisely so this module could teach all six dimensions with a single file, with none left without an example. A real production file could have twenty completeness violations and zero accuracy ones, or the other way around.

Exercises

Exercise 1 — Run the whole project yourself, from scratch. In a new directory, with kiosko.py (lesson 4), orders_2026-08-14.csv (lesson 6), and this project's diagnose_s04.py, run python3 diagnose_s04.py. Confirm you see the final summary with exactly 3 caught rows, 2 silent rows, and 33.0 hours late.

Exercise 2 — Add a "dimensions touched" counter that computes itself, not by hand. This project's script prints "Quality dimensions touched by this incident: 6 of 6" as fixed text. Modify main() so that number gets computed dynamically, counting how many distinct dimensions appear across rejected (grouping by reason type) and silent_failures, plus freshness if hours_late > 0.

def count_dimensions_touched(rejected: list[dict], silent_failures: list[tuple], hours_late: float) -> set[str]:
    dimensions: set[str] = set()
    for item in rejected:
        for reason in item["reasons"]:
            if "null or empty" in reason:
                dimensions.add("completeness")
            elif "duplicate" in reason:
                dimensions.add("uniqueness")
            elif "must be" in reason or "not a valid" in reason:
                dimensions.add("validity")
    for _row, dimension, _reason in silent_failures:
        dimensions.add(dimension)
    if hours_late > 0:
        dimensions.add("freshness")
    return dimensions


dimensions_touched = count_dimensions_touched(rejected, silent_failures, hours_late)
print(f"Quality dimensions touched by this incident: {len(dimensions_touched)} of 6 ({sorted(dimensions_touched)})")

Quality dimensions touched by this incident: 6 of 6 (['accuracy', 'completeness', 'consistency', 'freshness', 'uniqueness', 'validity'])

Exercise 3 — Argue what this report is missing to become a contract. This project produces a text report, printed to the terminal, that exists only while the script runs. In 3-4 sentences, describe what this result would need to let Kiosko use it as the basis for a formal, versioned agreement with S04 about what's expected from its future files — without writing code yet, just reasoning about the gap.

Summary and next step: closing this module

With this project you close module 1. You diagnosed, with end-to-end executed evidence, S04's first real file: twelve rows, three correctly caught by the gate inherited from foundations (completeness, validity, uniqueness), two silently incorrect despite passing that very gate (consistency, accuracy), and the whole file violating the freshness SLA by 33 hours. The six data quality dimensions you defined in lesson 3, all six, ended up represented with real evidence, not just abstract examples.

And, along the way, you closed the question that opened this entire module: a pipeline that finishes green — validate_orders() reporting Valid: 9 with no exception at all — can, with total apparent calm, have incorrect data inside it. It's no longer a theoretical claim quoted from a market audit. It's a result you ran yourself, with your own hands, against real Kiosko data.

Where you go next. Module 2 — Declarative data quality tests with Pandera — takes exactly the problem this module diagnosed and starts building the first reusable piece of the solution: instead of hand-written imperative functions like check_schema() or check_business_rules(), you're going to declare a Pandera DataFrameModel — a direct statement about what the data should look like, not a scattered if — and run it against S04's same file, confirming it catches exactly the same three dimensions you already know (completeness, uniqueness, validity), now with a declarative syntax you'll reuse in every module that follows in this guide.

Resources

• data-engineering-foundations-guide, module 5 (data-quality-gates) — the source of validate_orders(), integrated with no changes in this project. src/guides/data-engineering-foundations-guide/workbook/module-05-data-quality-gates/es/. In Spanish.
• Joe Reis & Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022) — the data quality framework this entire module rests on. oreilly.com/library/view/fundamentals-of-data/9781098108298. In English.
• src/paths/data-engineering-ecosystem/VALIDACION.md — the internal market audit motivating this entire guide, quoted in this module's lesson 2. Internal repo document. In Spanish.
• This guide's DESIGN — the full map of the eight modules, including the module 2 that follows. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.