What happens when a check fails in production

Description

This lesson answers, with precision and with evidence already seen in this guide, a question no earlier module needed to solve: when one of the six data quality tools you already built finds a problem, what happens next? Not what it reports — you already know that well —, but what action follows. You're going to review the two bad answers, both with real evidence from this guide, before naming the third — the one this module's lessons 3 and 4 build. And you're going to write this module's first new code: a function that unites the three reusable tools from modules 2, 3, and 5 into a single failure report, the exact input quarantine() needs in the next lesson.

Connection to the module. Lesson 1 previewed the hospital idea: diagnosing isn't the same as treating. This lesson builds the complete argument, naming the three possible answers one by one, and leaves ready the piece of data — the unified failure report — lesson 3 is going to separate into clean and quarantined.

The three possible answers, with real evidence from this guide

When a pipeline finds a row — or a whole file — that doesn't comply with a rule, there are, generally speaking, exactly three ways to respond. This guide already showed, with executed evidence, the first two — and both fail, for opposite reasons.

Answer 1 — Reject everything. If something is wrong, discard the entire set, with no distinction of which specific part failed. data-engineering-foundations-guide (module 7) showed this answer with S04's first file, orders_2026-08-11.csv: transform_fact_orders() found a single row with an unknown store_id (ORD-8102, S04 still wasn't in DIM_STORE), raised ValueError: unknown store_id: S04, and the entire run ended with status='failed', rows_loaded=0. Not a single row of that file — including the ones with absolutely no problem at all — reached the warehouse. The cost of this answer is that it treats good and bad exactly the same: a single error, in a single row, cancels the entire job.

Answer 2 — Pass in silence. If something is wrong but nobody wrote an explicit rule to detect it, or if the rule that got written discards the problematic row with no trace left, the entire process ends up reported as successful. This guide's module 1 (lesson 2) showed this answer with the extract_and_load() example: rows with an unknown customer_id got silently filtered — neither counted nor logged — and the function returned {'status': 'SUCCESS'}, with no hint at all that two out of every ten rows in the batch had been lost along the way. The cost of this answer is the exact reverse of the first: nothing gets lost visibly, but the problem doesn't get resolved either — it simply disappears from view of anyone only looking at the final status.

Answer 3 — Quarantine, alert, runbook. The one this module's lessons 3 and 4 build: problematic rows get separated from clean ones, with neither part discarded; a structured alert documents exactly what happened, with no dependence on someone deciding, on their own, to review the report; and a written runbook says, step by step, what to do with that incident, with no dependence on the right person being on shift that day and remembering the procedure by heart. This answer doesn't prevent broken rows from existing — that's already impossible to prevent, S04 is a new store connecting its first real system — but it avoids both earlier answers' costs at once: no good row gets lost, and the problem never stays invisible.

flowchart TD
    A["A check finds a problem"] --> B{"How does the system respond?"}
    B -->|"Answer 1"| C["Reject EVERYTHING\n(foundations M7:\nrows_loaded=0)"]
    B -->|"Answer 2"| D["Pass in silence\n(module 1 L2:\nstatus=SUCCESS,\nhidden loss)"]
    B -->|"Answer 3"| E["Quarantine + alert + runbook\n(this module:\nnothing gets lost,\nnothing stays invisible)"]
    C --> F["Cost: what WAS good\nalso gets lost"]
    D --> G["Cost: the problem\nbecomes invisible"]
    E --> H["Without that double cost --\nlessons 3 and 4's\ntopic"]

An analogy: a food factory's quality control

Think of a bottling line that checks every bottle before packaging it. A poorly designed factory could react to a bottle with a badly sealed cap in two ways: stopping the entire line every time a defective bottle shows up — losing hours of production of perfectly good bottles over one loose cap — or letting the defective bottle through with nobody finding out, trusting that "it probably doesn't matter." A well-designed factory does something different: the defective bottle gets set aside into a separate container, with no stop to the rest of the line; a counter logs how many bottles got set aside and why; and there's a written procedure — who checks the set-aside container, when, and what they do with it — that doesn't depend on a specific supervisor being present that shift. That's, precisely, what this module's lessons 3 and 4 build, applied to orders_2026-08-14.csv's six broken rows.

Worked example: uniting three tools into a single failure report

Before you can separate good rows from bad ones (lesson 3), you need a single place gathering every already-known problem in S04, instead of three loose reports — one from OrdersSchema (module 2), one from validate_referential_integrity() (module 3), one from check_price_baseline() (module 5). This lesson builds that union: build_failure_report(), this module's first new function.

# build_failure_report.py -- module 7's first step: uniting M2 + M3 + M5
import duckdb
import pandera
import pandera.polars as pa
import polars as pl

REFERENCE_PRICES = {"P001": 0.55, "P002": 1.2, "P003": 0.75, "P004": 4.5}


# --- the three reusable tools, with no change at all (modules 2, 3, and 5) ---
class OrdersSchema(pa.DataFrameModel):
    order_id: str = pa.Field(unique=True)
    unit_price: float = pa.Field(nullable=False, ge=0)
    quantity: int = pa.Field(gt=0)


CHECK_TO_DIMENSION = {
    "not_nullable": "completeness",
    "field_uniqueness": "uniqueness",
    "greater_than(0)": "validity",
}


def validate_referential_integrity(orders_df: pl.DataFrame, dim_product_df: pl.DataFrame) -> pl.DataFrame:
    """Rows in orders_df whose product_id does NOT exist in dim_product_df (module 3, no changes)."""
    return orders_df.join(dim_product_df, on="product_id", how="anti")


def check_price_baseline(df: pl.DataFrame, reference_prices: dict[str, float], tolerance: float = 0.5) -> pl.DataFrame:
    """Rows in df whose unit_price deviates from the baseline beyond tolerance (module 5, no changes)."""
    return (
        df.with_columns(
            pl.col("product_id").replace_strict(reference_prices, default=None).alias("reference_price")
        )
        .filter(pl.col("unit_price").is_not_null() & pl.col("reference_price").is_not_null())
        .with_columns(
            ((pl.col("unit_price") - pl.col("reference_price")).abs() / pl.col("reference_price")).alias("deviation")
        )
        .filter(pl.col("deviation") > tolerance)
    )


# --- new in this lesson: unite the three into a single report, at the physical row level ---
def build_failure_report(
    df: pl.DataFrame, dim_product_df: pl.DataFrame, reference_prices: dict[str, float]
) -> pl.DataFrame:
    """A DataFrame with one row per broken physical occurrence: row_idx, order_id, dimension, detail."""
    rows: list[dict] = []

    try:
        OrdersSchema.validate(df, lazy=True)
    except pa.errors.SchemaErrors as exc:
        for r in exc.failure_cases.iter_rows(named=True):
            rows.append({
                "row_idx": r["index"],
                "order_id": df["order_id"][r["index"]],
                "dimension": CHECK_TO_DIMENSION[r["check"]],
                "detail": f"{r['column']}={r['failure_case']}",
            })

    indexed = df.with_row_index("row_idx")

    orphans = validate_referential_integrity(indexed, dim_product_df)
    for r in orphans.iter_rows(named=True):
        rows.append({
            "row_idx": r["row_idx"],
            "order_id": r["order_id"],
            "dimension": "consistency",
            "detail": f"product_id={r['product_id']} does not exist in dim_product",
        })

    anomalies = check_price_baseline(indexed, reference_prices, tolerance=0.5)
    for r in anomalies.iter_rows(named=True):
        rows.append({
            "row_idx": r["row_idx"],
            "order_id": r["order_id"],
            "dimension": "accuracy",
            "detail": f"unit_price={r['unit_price']} is {round(r['deviation'], 1)}x away from the reference price ({r['reference_price']})",
        })

    return pl.DataFrame(rows).sort(["row_idx", "dimension"])


def main() -> None:
    pl.Config.set_fmt_str_lengths(60)

    con = duckdb.connect("kiosko.duckdb")
    con.execute("""
        CREATE OR REPLACE TABLE dim_product (
            product_id VARCHAR, product_name VARCHAR, category VARCHAR, unit_cost DOUBLE
        )
    """)
    con.execute("""
        INSERT INTO dim_product VALUES
            ('P001', 'Bottled Water 600ml', 'beverages', 0.40),
            ('P002', 'Energy Bar', 'snacks', 0.60),
            ('P003', 'Instant Coffee Sachet', 'beverages', 0.35),
            ('P004', 'Phone Charger Cable', 'electronics', 2.10)
    """)

    df = con.sql("SELECT * FROM orders_s04").pl()
    dim_product_df = con.sql("SELECT * FROM dim_product").pl()

    print(f"pandera version: {pandera.__version__}")
    print(f"orders_s04: {df.height} rows\n")

    failures = build_failure_report(df, dim_product_df, REFERENCE_PRICES)
    print("=== build_failure_report(df, dim_product_df, REFERENCE_PRICES) ===")
    print(failures)

    print(f"\nPhysical rows with at least one problem: {failures['row_idx'].n_unique()} of {df.height}")
    print(f"Dimensions represented: {sorted(failures['dimension'].unique().to_list())}")


if __name__ == "__main__":
    main()

What to expect (verified by actually running python3 build_failure_report.py, with kiosko.duckdb containing module 2's orders_s04, pandera==0.32.1, polars==1.43.2, duckdb==1.5.5):

pandera version: 0.32.1
orders_s04: 12 rows

=== build_failure_report(df, dim_product_df, REFERENCE_PRICES) ===
shape: (6, 4)
┌─────────┬──────────┬──────────────┬─────────────────────────────────────────────────────────┐
│ row_idx ┆ order_id ┆ dimension    ┆ detail                                                     │
│ ---     ┆ ---      ┆ ---          ┆ ---                                                        │
│ i64     ┆ str      ┆ str          ┆ str                                                        │
╞═════════╪══════════╪══════════════╪═════════════════════════════════════════════════════════╡
│ 1       ┆ ORD-9502 ┆ uniqueness   ┆ order_id=ORD-9502                                          │
│ 2       ┆ ORD-9503 ┆ completeness ┆ unit_price=None                                            │
│ 6       ┆ ORD-9507 ┆ validity     ┆ quantity=-1                                                │
│ 7       ┆ ORD-9508 ┆ consistency  ┆ product_id=P099 does not exist in dim_product              │
│ 8       ┆ ORD-9509 ┆ accuracy     ┆ unit_price=60.0 is 49.0x away from the reference price ... │
│ 9       ┆ ORD-9502 ┆ uniqueness   ┆ order_id=ORD-9502                                          │
└─────────┴──────────┴──────────────┴─────────────────────────────────────────────────────────┘

Physical rows with at least one problem: 6 of 12
Dimensions represented: ['accuracy', 'completeness', 'consistency', 'uniqueness', 'validity']

Read this table carefully, because it differs from any earlier report in this guide in one important detail: it's the first time modules 2, 3, and 5's three tools appear combined into a single structure, at the physical row level — not distinct order_id, but exact position within the file. Notice row_idx 1 and row_idx 9: both say ORD-9502, both say uniqueness, but they're two different physical rows — the duplicate's first and second appearance, at positions 1 and 9 in the file (0-indexed). This granularity matters a lot for lesson 3: quarantine() needs to know exactly which physical row to set aside, not just which order_id has a problem, because ORD-9502's first appearance is, on its own, perfectly well-formed.

dimension lists only five names, not six — freshness doesn't appear, because build_failure_report() only gathers row-level checks, and freshness, since module 6, is a property of the whole file, not of any individual row. This module's lesson 4 picks freshness back up separately, with its own alert.

Diagram: where each row of the report comes from

flowchart LR
    A["OrdersSchema.validate()\n(module 2)"] -->|"completeness,\nuniqueness x2,\nvalidity"| F["build_failure_report()"]
    B["validate_referential_integrity()\n(module 3)"] -->|"consistency"| F
    C["check_price_baseline()\n(module 5)"] -->|"accuracy"| F
    F --> D["6 physical rows,\na single report,\nrow_idx + order_id +\ndimension + detail"]
    D --> E["Lesson 3:\nquarantine(df, failures)"]

Going deeper: why unite, instead of running each check separately at the moment of the incident

You might wonder why it's worth building build_failure_report() as an independent function, instead of simply calling the three tools separately every time you need to act on an incident. The answer has to do with what quarantine() needs in the next lesson: a single source of truth about which physical rows have some problem, no matter which of the three tools detected it. If quarantine() had to call, itself, OrdersSchema.validate(), validate_referential_integrity(), and check_price_baseline() separately, and combine their three results every time it gets invoked, you'd be repeating the same union logic in every place in the system that needs that answer — this module's report, a future alert, a future dashboard. build_failure_report() calculates that union once, and any function that needs it — quarantine() included — receives it already ready, as a parameter.

This is also, without having been explicitly named until now, the same separation of responsibilities module 5 already practiced between check_price_baseline() and validate_referential_integrity(): each function does one single thing — detecting one kind of problem —, and a higher-level function combines them for a specific purpose. build_failure_report() is that higher-level function, built for the first time in this guide because it's the first time a mechanism (quarantine()) really needs more than one tool's combined result at once.

Common mistakes

Expecting build_failure_report() to also include freshness. What happens: someone, seeing that module 6 already closed all six dimensions, expects to see freshness in this lesson's report's dimension column. Why it happens: after six modules getting used to "six dimensions, one complete report," it seems natural for any new report to include all of them. How to spot it: review build_failure_report()'s signature — it receives df, dim_product_df, and reference_prices, but never run_at or sla_hours, the parameters check_freshness() needs. How to fix it: this function gathers exactly the three tools that work row by row — OrdersSchema, referential consistency, the price baseline; freshness remains, as module 6 established, a whole-file question, and that's why this module's lesson 4 handles it with its own call to raise_alert(), separate from this report.

Assuming row_idx is the same as counting how many distinct order_id have a problem. What happens: someone reads "6 of 12" in this lesson's output and compares it, without thinking, against the "5 of 6 dimensions" they already saw in earlier projects, as if they were the same kind of number. Why it happens: both numbers appear in similar contexts — S04 quality reports — and it's easy to mix different counting units. How to spot it: carefully count how many distinct order_id values appear in failures — there are five (ORD-9502, ORD-9503, ORD-9507, ORD-9508, ORD-9509), not six, because ORD-9502 appears twice. How to fix it: row_idx counts physical rows (exact positions within the file), not distinct order_id — the same kind of distinction module 3's lesson 4's common mistake and module 5's project's exercise 2 already demanded. Always check which unit a number is counting before comparing it against another.

Exercises

Exercise 1 — Run build_failure_report() yourself, from scratch. In a new folder, with kiosko.duckdb containing the orders_s04 table (module 2), run python3 build_failure_report.py. Confirm you see exactly 6 physical rows with a problem, spread across five distinct dimensions.

Exercise 2 — Extend build_failure_report() with a severity column calculated per dimension. Add a dimension_severity(dimension: str) -> str function that returns "high" for completeness, uniqueness, and accuracy (problems directly affecting a business number: missing rows, duplicates, or incorrect prices), and "medium" for validity and consistency. Apply it as a new report column.

def dimension_severity(dimension: str) -> str:
    high_severity = {"completeness", "uniqueness", "accuracy"}
    return "high" if dimension in high_severity else "medium"

failures_with_severity = failures.with_columns(
    pl.col("dimension").map_elements(dimension_severity, return_dtype=pl.String).alias("severity")
)
print(failures_with_severity.select(["order_id", "dimension", "severity"]))

shape: (6, 3)
┌──────────┬──────────────┬──────────┐
│ order_id ┆ dimension    ┆ severity │
│ ---      ┆ ---          ┆ ---      │
│ str      ┆ str          ┆ str      │
╞══════════╪══════════════╪══════════╡
│ ORD-9502 ┆ uniqueness   ┆ high     │
│ ORD-9503 ┆ completeness ┆ high     │
│ ORD-9507 ┆ validity     ┆ medium   │
│ ORD-9508 ┆ consistency  ┆ medium   │
│ ORD-9509 ┆ accuracy     ┆ high     │
│ ORD-9502 ┆ uniqueness   ┆ high     │
└──────────┴──────────────┴──────────┘

Exercise 3 — Argue whether build_failure_report() should include rows check_price_baseline() silently excludes (the ones with an unknown product_id, like ORD-9508). You already know, from module 5, that check_price_baseline() excludes rows whose product_id doesn't appear in reference_prices, before calculating any deviation. In 2-3 sentences, explain whether that means ORD-9508 could, in theory, end up completely outside failures if validate_referential_integrity() hadn't caught it first.

Summary and next step

In this lesson you named, with real evidence already seen in this guide, the two insufficient answers for "what to do when a check fails in production" — reject everything, or pass in silence — and you previewed the third, the one the next two lessons build. And you wrote this module's first new function: build_failure_report(), which unites OrdersSchema (module 2), validate_referential_integrity() (module 3), and check_price_baseline() (module 5) into a single report of six physical rows, at the exact-position-within-the-file level — the exact input the next lesson needs.

Before moving on you should be able to: name the three possible answers to a failing check, with a real example from this guide for each of the two bad ones; explain why build_failure_report() works at the row_idx level and not the order_id level; and reproduce, by running the code yourself, the exact six-physical-row report.

Lesson 3 uses this same report to build quarantine(): the function that finally physically separates the six clean rows from the six broken ones — with no entire file discarded, and no problem left invisible.

Resources

• Pandera — official documentation (SchemaErrors.failure_cases, the source of this report's first three dimensions). pandera.readthedocs.io. In English.
• Polars — official documentation (with_row_index, join with how="anti", build_failure_report()'s technical foundation). docs.pola.rs. In English.
• data-engineering-foundations-guide, module 7 — the literal source of S04's total rejection (ValueError: unknown store_id: S04, rows_loaded=0), this lesson's Answer 1. src/guides/data-engineering-foundations-guide/workbook/module-07-partitioning-and-orchestration/en/. In English.
• Module 1, lesson 2, of this same guide — the literal source of the green-checkmark example, this lesson's Answer 2. src/guides/data-reliability-and-governance-guide/workbook/module-01-when-green-does-not-mean-correct/en/02-the-lie-of-the-green-checkmark.md. In English.
• This guide's DESIGN. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.