Project: testing S04 with Pandera

Description

This project closes the module. You have the tool chosen with evidence (lesson 3), installed and connected to kiosko.duckdb through Polars (lesson 4), the first DataFrameModel written (lesson 5), and the three rules that complete OrdersSchema — completeness, uniqueness, validity — built and read in depth (lessons 6 and 7). One step remains: pulling it all together into a single script, run end to end against S04's real file, with a final report that confirms, with executed evidence, the exact promise this module opened with — that Pandera catches the same three dimensions foundations' validate_orders() already caught in module 1, now declared instead of programmed step by step.

Connection to the module. This project introduces no new concept — it's the synthesis of lessons 3 through 7, applied end to end to the same incident module 1 diagnosed. It closes this module's thread exactly where module 3 picks it back up: the two rows OrdersSchema still lets through with no warning — ORD-9508 and ORD-9509 — are, literally, the next module's starting point.

An analogy: the inspector's complete checklist, not an eyeballed review

A restaurant inspector doesn't walk into a place and say "looks clean, approved" — they go through a fixed list of concrete, verifiable items, one by one: refrigerator temperature, each product's expiration date, prep-surface hygiene, kitchen ventilation. Each item on that list is a specific question, with an objective answer — meets it or doesn't —, and the inspector's final result is the sum of those individual answers, never a general impression.

OrdersSchema, as it stands at the end of lesson 7, is exactly that list, applied to orders_s04: three concrete items — unique order_id, present and non-negative unit_price, positive quantity —, each checked independently, with a structured result that says exactly which item failed, on which row. This project is the complete inspection: running the whole list, all at once, against the real file, and reading the result with the same rigor a real inspector signs their report with.

The material you need

You need, in module 2's same working folder:

modulo_2_pandera/
├── kiosko.duckdb              (lesson 4: contains the orders_s04 table)
└── validate_s04_pandera.py    (this project)

If you don't have kiosko.duckdb with the orders_s04 table yet, repeat lesson 4's steps 2 and 3 before continuing — the CREATE OR REPLACE TABLE orders_s04 AS SELECT * FROM read_csv(...) block. This project doesn't explain that step again, it assumes it's done.

The reference solution, verified

# validate_s04_pandera.py -- module 2 closing project
# runs the complete OrdersSchema against S04, with lazy=True, and compares against M1
import duckdb
import pandera
import pandera.polars as pa
import polars as pl


class OrdersSchema(pa.DataFrameModel):
    order_id: str = pa.Field(unique=True)
    unit_price: float = pa.Field(nullable=False, ge=0)
    quantity: int = pa.Field(gt=0)


CHECK_TO_DIMENSION = {
    "not_nullable": "completeness",
    "field_uniqueness": "uniqueness",
    "greater_than(0)": "validity",
}


def main() -> None:
    print("=== Kiosko: OrdersSchema (Pandera) over orders_s04 ===")
    print(f"pandera version: {pandera.__version__}\n")

    con = duckdb.connect("kiosko.duckdb")
    df = con.sql("SELECT * FROM orders_s04").pl()
    print(f"Rows read from orders_s04: {df.height}\n")

    try:
        OrdersSchema.validate(df, lazy=True)
        print("All rows passed OrdersSchema. (not expected for S04)")
        return
    except pa.errors.SchemaErrors as exc:
        fc = exc.failure_cases

    readable = fc.with_columns(
        pl.col("index").map_elements(lambda i: df["order_id"][i], return_dtype=pl.String).alias("order_id")
    ).select(["order_id", "column", "check", "failure_case"])

    print("=== OrdersSchema.validate(df, lazy=True): SchemaErrors ===")
    print(readable)

    failing_indices = sorted(fc["index"].unique().to_list())
    dimensions = sorted({CHECK_TO_DIMENSION[c] for c in fc["check"].to_list()})

    print(f"\nPhysical rows with at least one error: {len(failing_indices)} of {df.height}")
    print(f"Quality dimensions caught: {len(dimensions)} of 6 ({dimensions})")

    passing = df.with_row_index("idx").filter(~pl.col("idx").is_in(failing_indices))
    print(f"\n=== Rows OrdersSchema lets through with no error ({passing.height} of {df.height}) ===")
    for row in passing.select(["order_id", "product_id", "unit_price"]).iter_rows(named=True):
        print(f"  {row['order_id']} | product_id={row['product_id']} | unit_price={row['unit_price']}")

    print("\n=== Comparison against module 1's validate_orders() ===")
    print("validate_orders() (imperative, module 1): 9 valid, 3 rejected")
    print("  Rejected: ORD-9503 (completeness), ORD-9507 (validity), ORD-9502 2nd appearance (uniqueness)")
    print(f"OrdersSchema (declarative, this module): {passing.height} pass, {len(failing_indices)} rows with an error")
    print("  With an error: ORD-9503 (completeness), ORD-9507 (validity), ORD-9502 x2 (uniqueness)")
    print("Difference: Pandera flags BOTH appearances of the duplicate; validate_orders() flags only the second.")
    print("Both tools agree on the SAME 3 dimensions: completeness, uniqueness, validity.")


if __name__ == "__main__":
    main()

What to expect (verified by actually running python3 validate_s04_pandera.py, with kiosko.duckdb and lesson 4's orders_s04 table in the same folder, pandera==0.32.1):

=== Kiosko: OrdersSchema (Pandera) over orders_s04 ===
pandera version: 0.32.1

Rows read from orders_s04: 12

=== OrdersSchema.validate(df, lazy=True): SchemaErrors ===
shape: (4, 4)
┌──────────┬────────────┬──────────────────┬──────────────┐
│ order_id ┆ column     ┆ check            ┆ failure_case │
│ ---      ┆ ---        ┆ ---              ┆ ---          │
│ str      ┆ str        ┆ str              ┆ str          │
╞══════════╪════════════╪══════════════════╪══════════════╡
│ ORD-9502 ┆ order_id   ┆ field_uniqueness ┆ ORD-9502     │
│ ORD-9502 ┆ order_id   ┆ field_uniqueness ┆ ORD-9502     │
│ ORD-9503 ┆ unit_price ┆ not_nullable     ┆ null         │
│ ORD-9507 ┆ quantity   ┆ greater_than(0)  ┆ -1           │
└──────────┴────────────┴──────────────────┴──────────────┘

Physical rows with at least one error: 4 of 12
Quality dimensions caught: 3 of 6 (['completeness', 'uniqueness', 'validity'])

=== Rows OrdersSchema lets through with no error (8 of 12) ===
  ORD-9501 | product_id=P001 | unit_price=0.55
  ORD-9504 | product_id=P004 | unit_price=4.5
  ORD-9505 | product_id=P001 | unit_price=0.55
  ORD-9506 | product_id=P003 | unit_price=0.75
  ORD-9508 | product_id=P099 | unit_price=1.0
  ORD-9509 | product_id=P002 | unit_price=60.0
  ORD-9510 | product_id=P004 | unit_price=4.5
  ORD-9511 | product_id=P002 | unit_price=1.2

=== Comparison against module 1's validate_orders() ===
validate_orders() (imperative, module 1): 9 valid, 3 rejected
  Rejected: ORD-9503 (completeness), ORD-9507 (validity), ORD-9502 2nd appearance (uniqueness)
OrdersSchema (declarative, this module): 8 pass, 4 rows with an error
  With an error: ORD-9503 (completeness), ORD-9507 (validity), ORD-9502 x2 (uniqueness)
Difference: Pandera flags BOTH appearances of the duplicate; validate_orders() flags only the second.
Both tools agree on the SAME 3 dimensions: completeness, uniqueness, validity.

Read the complete result carefully, because it confirms, with executed evidence, this module's exact opening promise. OrdersSchema catches the same three dimensions validate_orders() already caught — completeness, uniqueness, validity —, with the only difference being that Pandera reports both appearances of the duplicate pair (a design decision already explained in lessons 6 and 7, not a disagreement between tools). And, with the same clarity module 1 already showed, ORD-9508 (product_id=P099, consistency) and ORD-9509 (unit_price=60.0, accuracy) still pass with no warning at all — declaring the rules as a schema, instead of imperative code, didn't make them appear out of nowhere. Those two rows are still waiting on this guide's module 3 and module 5.

Diagram: where you came from, where you landed

flowchart LR
    A["Module 1:\nvalidate_orders()\nimperative\n9 valid, 3 rejected"] --> B["Lesson 3:\nPandera chosen\nwith evidence"]
    B --> C["Lesson 4:\nCSV -> DuckDB -> Polars\nbridge built"]
    C --> D["Lesson 5:\ntypes-only\nDataFrameModel: 12/12 pass"]
    D --> E["Lessons 6-7:\nnullable, unique, gt\nadded one at a time"]
    E --> F["This project:\ncomplete OrdersSchema,\n8 pass, 4 with an error,\n3 of 6 dimensions"]
    F --> G["Module 3:\nreferential consistency --\nORD-9508 and ORD-9509\nstill not caught"]

Closing the module's promise, point by point

OrdersSchema, as it stands at this module's close, is a reusable piece — a class you can import and run against any Polars DataFrame with order_id, unit_price, quantity columns —, unlike validate_orders(), which was born specific to orders's exact schema. That reusability is, precisely, what makes it possible for this guide's module 4 to regenerate this same schema from a YAML contract, without rewriting any rule by hand.

Common mistakes

Considering S04's entire incident resolved because OrdersSchema "already runs." What happens: someone, satisfied with this project's report, concludes Kiosko already has a complete data quality system for S04. Why it happens: OrdersSchema really runs, with a clear, professional report — it feels like a finished solution. How to spot it: check this project's "What to expect" "Rows OrdersSchema lets through with no error" block — ORD-9508 and ORD-9509 are still there, with no flag. How to fix it: this project closes exactly three of the six data quality dimensions module 1 defined — completeness, uniqueness, validity. Consistency (module 3), accuracy (module 5), and freshness (module 6) still have no check at all; not a single line of this project solves them.

Modifying OrdersSchema to add a product_id rule "while we're at it." What happens: someone, seeing ORD-9508 still unflagged, adds product_id: str = pa.Field(isin=["P001", "P002", "P003", "P004"]) directly to OrdersSchema, as a quick fix. Why it happens: Pandera does have a parameter (isin) that could technically solve this specific case. How to spot it: if your list of valid products is hand-written inside the schema, you have the same problem module 1's lesson 4 already warned about with check_business_rules() — it works today, with four products, but goes out of sync with the warehouse's dim_product when Kiosko adds a fifth product anywhere else in the system. How to fix it: this guide's module 3 builds consistency as a real anti-join against dim_product, read live from kiosko.duckdb — not as a hand-coded list. That's the right path, even though isin exists as a tempting shortcut.

Comparing Pandera's 8 pass number against validate_orders()'s 9 valid with no complete explanation. What happens: someone reports, with no further context, that "Pandera found one more error than validate_orders()," as if one tool were stricter or better than the other. Why it happens: the numbers 8 and 9 are visibly different, and without lessons 6 and 7's explanation, the difference looks like a real disagreement about the data. How to spot it: if your report doesn't explain why the numbers differ, you left an open question anyone at Kiosko is going to ask immediately. How to fix it: the difference is entirely attributable to how each tool counts a duplicate pair — validate_orders() flags only the second appearance, Pandera flags both —, not to either one finding a problem the other misses. The three dimensions caught are identical; only the physical row count changes, for an already-documented reason.

Exercises

Exercise 1 — Run the whole project yourself, from scratch. In a new folder, with kiosko.duckdb (with lesson 4's orders_s04 table) and this project's validate_s04_pandera.py, run python3 validate_s04_pandera.py. Confirm you see exactly 4 physical rows with an error, 3 dimensions, and 8 rows that pass.

Exercise 2 — Add a dynamic count of "dimensions NOT covered by this module." This project's script prints the caught dimensions (3 of 6). Modify main() so it also prints, computed (not hand-written), which of the six dimensions are not in that list.

ALL_DIMENSIONS = {"completeness", "uniqueness", "validity", "consistency", "accuracy", "freshness"}

missing_dimensions = sorted(ALL_DIMENSIONS - set(dimensions))
print(f"\nDimensions this module does NOT cover yet: {missing_dimensions}")

Dimensions this module does NOT cover yet: ['accuracy', 'consistency', 'freshness']

Exercise 3 — Argue what OrdersSchema would need to be reusable at another Kiosko store. OrdersSchema, as written, doesn't mention S04 anywhere — not the store's name, not any specific filter. In 2-3 sentences, explain why that means it's already reusable for any Kiosko store, and what would need to happen for it to also be reusable for a completely different business, not just another Kiosko store.

Summary and next step: closing this module

With this project you close module 2. You installed Pandera with evidence of why, against the market's other two alternatives; built the DuckDB → Polars bridge that holds up the rest of this guide; wrote OrdersSchema bit by bit, first seeing how little a types-only schema protects, then how each new rule — nullable, unique, gt — catches exactly the row it's supposed to catch. And you confirmed, with executed evidence, the complete promise: the same three dimensions validate_orders() already caught in module 1, now declared as a readable, reusable schema, not programmed step by step.

And, along the way, you made clear the exact limit of what this module solves: eight of S04's twelve rows pass OrdersSchema with no error, but two of those eight — ORD-9508 and ORD-9509 — still have real problems no Field in this module can express, because they need to look outside the row itself: another table, in one case; a historical baseline, in the other.

Where you go next. This guide's module 3 — Consistency and referential checks — takes exactly that silent row, ORD-9508, and builds the piece OrdersSchema was missing: a real anti-join against dim_product, read live from kiosko.duckdb, able to answer the question no single-table schema can answer on its own — does this product_id really exist in Kiosko's catalog?

Resources

• Pandera — official documentation (complete DataFrameModel, Field, Check, lazy, SchemaErrors reference). pandera.readthedocs.io. In English.
• data-engineering-foundations-guide, module 5 (data-quality-gates) — the source of validate_orders(), this entire module's basis for comparison. src/guides/data-engineering-foundations-guide/workbook/module-05-data-quality-gates/es/. In Spanish.
• Module 1, project (lesson 8), of this same guide — S04's original diagnosis, which this project confirms with a different tool. src/guides/data-reliability-and-governance-guide/workbook/module-01-when-green-does-not-mean-correct/es/08-project-diagnosing-s04s-silent-failures.md. In Spanish.
• This guide's DESIGN — the full map of the eight modules, including the module 3 that follows. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.