Module introduction: declarative data quality tests with Pandera

Why this module exists

Module 1 ended on an uncomfortable number: of the twelve lines in orders_2026-08-14.csv — S04 Kiosko Reforma's first real file —, data-engineering-foundations-guide's validate_orders() caught three broken rows (completeness, uniqueness, validity) and let through, with no warning at all, two rows with real business problems (ORD-9508 with a product_id that doesn't exist, ORD-9509 with a price fifty times higher than normal). The full diagnosis — six quality dimensions, all six touched by a single incident — got written, verified, executed. Nothing got fixed yet. That was module 1's explicit deal: diagnose, don't fix.

This module 2 starts fixing. But not by rewriting validate_orders() with more ifs — that function already fulfilled exactly what it promised, and continuing to add rules to it would turn it into an unmaintainable tangle —, but with a complete change of approach: instead of imperative code that checks row by row with hand-written logic, you're going to declare what the data should look like, and let a tool — Pandera — handle comparing reality against that declaration. The result, on S04's same file, is going to be the same: the same three broken rows, precisely identified. But the path to get there — and what that path lets you do afterward — is completely different.

Connection to the previous module. This module picks up exactly where module 1's project closed: the same orders_2026-08-14.csv, the same twelve lines, the same kiosko.duckdb that data-modeling-for-analytics-guide and dbt-analytics-engineering-guide already opened. The difference is the tool. Where module 1 ran a Python function written line by line, this module declares a schema — a class with a name and a type per column — and lets Pandera do the comparison work. You're going to see, with executed evidence, that declaring a schema and running validate_orders() aren't conceptually distant alternatives: they're two ways of expressing the same intent, with very different consequences for how easy that intent is to maintain, read, and reuse over time.

The running case: the same S04, the same new tool

Nothing about the case changes. S04 Kiosko Reforma, Kiosko's fourth store in Mexico City, is still the same store that got onboarded by hand into dim_store after foundations M7's rejection. orders_2026-08-14.csv still has the same twelve lines — six clean, six broken, one for each quality dimension — that module 1 already opened and classified. Kiosko's canonical week (2026-08-03 through 2026-08-09, total revenue 106.15) stays intact, untouched by this module.

What changes is the question this module asks about that same file: not "what slips past the old gate?" — you already answered that —, but "how do I build a new gate that declares its rules instead of executing them step by step, and that can also be reused without copy-pasting code every time a new table shows up?"

An analogy: the requirements list taped to the door, not a guard improvising

Picture two ways of controlling who enters an event with limited capacity. In the first, a guard at the door remembers the rules from memory — "only over eighteen," "with an invitation," "no large backpacks" — and applies them, person by person, according to their own judgment in the moment. If two different guards work different shifts, each may apply the rules with slightly different nuances, because the rules only live inside each one's head. In the second, someone wrote those same rules on a sign, taped to the door, visible to anyone: "Entry requirements: over 18 years old (verify ID), valid invitation, no backpacks over 40cm." The sign doesn't decide anything by itself — you still need someone to read it and compare it against each person —, but the rule itself is a fixed, written object that any new guard can read and apply exactly the same way, without having to ask the previous guard "how did you decide this?"

Foundations M5's validate_orders() is the first guard: the rules — quantity <= 0 is invalid, order_id can't repeat — live inside the function's logic, woven together with the code that applies them. They're correct, well-written, but to know exactly what the function requires, you have to read its entire body, instruction by instruction. A declarative quality test is the sign on the door: a direct statement — "the order_id column must be unique," "the unit_price column can't be null" — that exists as an object separate from the logic that applies it. Pandera is the one who reads that sign and does the comparison work; you only write down what should be true, not how to verify it step by step.

Worked example: the same rule, two ways of writing it

Before installing anything, it's worth seeing the contrast with a minimal example — no Pandera yet, just to isolate the difference in approach. This is the check_business_rules() rule from foundations M5 you already know, rewritten twice: first as imperative code, then as a declarative statement expressed in a simple dictionary:

# declarative_vs_imperative.py
# form 1: imperative -- HOW it's checked, step by step
def check_quantity_imperative(rows: list[dict]) -> list[str]:
    errors = []
    for row in rows:
        quantity = int(row["quantity"])
        if quantity <= 0:
            errors.append(f"{row['order_id']}: quantity must be > 0, is {quantity}")
    return errors


# form 2: declarative -- WHAT should be true, without saying how to check it
QUANTITY_RULE = {"column": "quantity", "rule": "greater_than", "value": 0}


def apply_declarative_rule(rows: list[dict], rule: dict) -> list[str]:
    """A generic interpreter that knows how to read ANY rule with this shape,
    without anyone having to write a new loop for each column."""
    errors = []
    for row in rows:
        value = int(row[rule["column"]])
        if rule["rule"] == "greater_than" and not (value > rule["value"]):
            errors.append(f"{row['order_id']}: {rule['column']} must be > {rule['value']}, is {value}")
    return errors


rows = [
    {"order_id": "ORD-A", "quantity": 3},
    {"order_id": "ORD-B", "quantity": -1},
]

print("Imperative form:", check_quantity_imperative(rows))
print("Declarative form:", apply_declarative_rule(rows, QUANTITY_RULE))

What to expect. Running python3 declarative_vs_imperative.py, the output is exactly this:

Imperative form: ["ORD-B: quantity must be > 0, is -1"]
Declarative form: ["ORD-B: quantity must be > 0, is -1"]

Both paths reach the same result — both catch ORD-B, with the same message —, but notice the structural difference. check_quantity_imperative() mixes two things into one function: what gets checked (quantity > 0) and how each row gets traversed and compared. If tomorrow you need the same rule for unit_price >= 0, you'd have to write a new function, almost identical, with the same loop repeated. QUANTITY_RULE, on the other hand, separates those two things: the rule is data — a dictionary, something you could save to a file, version, read from another system —, and apply_declarative_rule() is a generic interpreter that knows how to apply any rule with that shape, without anyone writing a new loop per column. Pandera is, in essence, a much more complete and mature version of apply_declarative_rule(): an interpreter that knows how to read rules declared as Python classes, and apply them against an entire DataFrame, with a much richer vocabulary than "greater than."

Diagram: where the rule lives in each approach

flowchart TB
    subgraph IMPERATIVO["Imperative approach -- validate_orders()"]
        A["The rule lives INSIDE\nthe logic that applies it"] --> B["To know what it requires,\nyou have to read the whole code"]
        B --> C["Adding a new rule =\nwriting a new function"]
    end

    subgraph DECLARATIVO["Declarative approach -- Pandera"]
        D["The rule is a separate\nobject: a class, a Field"] --> E["To know what it requires,\nyou read the declaration directly"]
        E --> F["Adding a new rule =\nadding a line to the class"]
    end

The diagram points to the difference this entire module develops: in the imperative approach, the rule and the logic that applies it are the same thing, woven together. In the declarative approach, they're two separate things — the declaration (what should be true) and the engine that interprets it (Pandera) —, and that separation is what makes it possible, later in this guide (module 4), for a YAML data contract to automatically generate the same schema this module writes by hand: if the rule is already an object separate from the logic, generating that object from another format stops being a big conceptual leap.

The map of this guide's 8 modules (reminder)

The map of this module

Lesson    Question it answers
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Where we're coming from, and what this module builds
L2        What, precisely, is a declarative quality test?
L3        Why Pandera, and not Great Expectations or Soda?
L4        How do you install Pandera, and how does DuckDB
          bridge to Polars?
L5        Write the first DataFrameModel: OrdersSchema
L6        Completeness and uniqueness declared, run against S04
L7        Validity declared, and how to read a complete
          failure report (SchemaErrors.failure_cases)
L8        Project: S04's full file validated with
          Pandera, end to end

Lessons 2 and 3 build the criteria: what a declarative test is, and why this guide picks Pandera among three real market tools, with license and maturity evidence for each. Lesson 4 installs the tool and builds the technical bridge — DuckDB as the SQL source, Polars as the only DataFrame format Pandera needs to work here. Lessons 5, 6, and 7 build OrdersSchema step by step, one dimension at a time, until reaching a complete schema. And lesson 8 — the project — runs that complete schema against S04's real file, confirming it catches exactly the same three dimensions you already knew from module 1, now declared instead of programmed step by step.

The boundary: what does NOT belong in this module

This module installs and uses Pandera for three of the six quality dimensions: completeness, uniqueness, validity. The other three dimensions have their own module, with their own tool: consistency needs an anti-join against dim_product (module 3, because Pandera validates one table at a time, not relationships across tables); accuracy needs a price baseline computed over historical data (module 5, because no static schema can know, on its own, how much something "should" cost); freshness is a property of the whole file, not of any column (module 6).

And an important boundary with a sister guide: dbt's data_tests: (unique, not_null, accepted_values, relationships in schema.yml) already solved, inside a dbt project, a version of exactly what this module builds. dbt-analytics-engineering-guide (module 4) is where that gets taught in depth. The difference isn't vocabulary — "unique," "not null" are the same idea on both sides — but scope: a dbt test lives and dies with the dbt project that declares it, runs only when dbt build or dbt test runs, and only makes sense over models that already live inside that graph. A Pandera DataFrameModel is a completely independent layer: it runs over any Polars DataFrame, whether that DataFrame lives inside a dbt project, was just read from a CSV, or was built by hand in a notebook. This guide uses Pandera precisely because the trust system it builds — contracts, anomalies, freshness, governance — needs to work over raw data, before it reaches any dbt project, not just over already-transformed models.

Common mistakes

Thinking "declarative" means "simpler" in the sense of "less code." What happens: someone expects declaring a schema with Pandera to be, in lines of code, dramatically shorter than validate_orders(). Why it happens: "declarative" sounds like "less work," and sometimes it is, but that isn't this module's central promise. How to spot it: if your success metric for this module is "did I write fewer lines?", you're measuring the wrong thing — a complete OrdersSchema, with five columns and their rules, can end up a similar length to foundations M5's equivalent functions. How to fix it: a declarative test's real payoff isn't the amount of code, it's that the rule itself — not the logic that applies it — is exposed as a readable, reusable object, and later in this guide (module 4), one generatable from an external artifact like a YAML contract.

Assuming Pandera completely replaces validate_orders(). What happens: someone, finishing this module, concludes foundations' function is no longer needed, and deletes it from their project. Why it happens: Pandera's OrdersSchema catches the same three dimensions (completeness, uniqueness, validity) validate_orders() already caught, so they seem interchangeable. How to spot it: if your plan is to never look at validate_orders() again for the rest of this guide, check what it does differently: validate_orders() splits rows into valid/rejected with a function designed specifically for orders's schema, while OrdersSchema.validate() is a reusable declaration Pandera knows how to interpret without anyone writing traversal logic by hand. How to fix it: there's no need to choose between the two — this guide builds Pandera's full apparatus precisely because it scales better as more tables and more rules show up, but this module's goal is learning the new tool, not deleting the previous one.

Writing the first DataFrameModel without first deciding whether the engine is pandas or Polars. What happens: someone copies an example from Pandera's documentation that uses import pandera as pa with pandas, and gets surprised when some things — like Field's checks= parameter — don't work the same with Polars. Why it happens: Pandera supports several DataFrame engines (pandas, Polars, PySpark), and its API varies in details depending on which one you use. How to spot it: if your code imports pandera as pa (not pandera.polars as pa) and fails with an error about unrecognized checks, you mixed the two APIs. How to fix it: this guide always uses import pandera.polars as pa (lesson 4 confirms it with executed code) — it's the only correct way to work with Polars, the only DataFrame engine allowed in this guide (pandas is exclusive territory for python-for-data-engineering-guide).

Exercises

Exercise 1 — Extend the declarative example with a second rule. Using apply_declarative_rule() from this lesson's worked example, add a new rule for the unit_price column (unit_price >= 0) and run apply_declarative_rule() against these rows: [{"order_id": "ORD-C", "unit_price": 1.20}, {"order_id": "ORD-D", "unit_price": -0.50}]. You don't need to modify apply_declarative_rule() — you just need a new rule; using "rule": "greater_than" with "value": -1 wouldn't work; think about what new operator you'd need to add to the interpreter.

def apply_declarative_rule_v2(rows: list[dict], rule: dict) -> list[str]:
    errors = []
    for row in rows:
        value = float(row[rule["column"]])
        if rule["rule"] == "greater_than" and not (value > rule["value"]):
            errors.append(f"{row['order_id']}: {rule['column']} must be > {rule['value']}, is {value}")
        elif rule["rule"] == "greater_or_equal" and not (value >= rule["value"]):
            errors.append(f"{row['order_id']}: {rule['column']} must be >= {rule['value']}, is {value}")
    return errors


PRICE_RULE = {"column": "unit_price", "rule": "greater_or_equal", "value": 0}
rows = [{"order_id": "ORD-C", "unit_price": 1.20}, {"order_id": "ORD-D", "unit_price": -0.50}]
print(apply_declarative_rule_v2(rows, PRICE_RULE))

['ORD-D: unit_price must be >= 0, is -0.5']

Exercise 2 — Name the difference between "rule" and "engine that applies it." Without looking at this lesson's diagram, explain in your own words, in 2-3 sentences, the difference between QUANTITY_RULE (the dictionary) and apply_declarative_rule() (the function), and why that separation is the central idea of a declarative test.

Exercise 3 — Predict what a third engine would need for the six dimensions. Based on this lesson's diagram and what you already know from module 1, why do you think a single Pandera DataFrameModel — this module's engine — can't, on its own, solve the entire guide's six quality dimensions? Justify your answer in 2-3 sentences, without looking at lessons 3 through 8 of this module yet.

Summary and next step

In this lesson you saw, with a minimal, executed example, the structural difference between an imperative test — where the rule and the logic that applies it are woven together — and a declarative test — where the rule is a separate object, and a generic engine handles interpreting it. You walked through the full map of this guide's eight modules and this module's eight lessons, and drew the exact boundary with dbt-analytics-engineering-guide: dbt's tests live inside a dbt project, this guide's work over any DataFrame.

Before moving on you should be able to: explain, in your own words, the difference between "the rule" and "the engine that applies it"; and name why completeness, uniqueness, and validity can be expressed as a single-table schema, while consistency, accuracy, and freshness need something more.

Lesson 2 puts a precise name to what you just saw with the declarative example, and lesson 3 chooses, with real license and maturity evidence, which market tool is going to interpret those rules for the rest of this guide: Pandera.

Resources

• Pandera — official documentation. The foundation for this entire module: DataFrameSchema/DataFrameModel, Field, Check, Polars integration. pandera.readthedocs.io. In English.
• dbt-analytics-engineering-guide DESIGN — the source of data_tests:/schema.yml, the exact boundary this lesson draws with declarative tests inside a dbt project. src/guides/dbt-analytics-engineering-guide/DISENO.md. In Spanish.
• data-engineering-foundations-guide, module 5 — the source of validate_orders() and check_business_rules(), the imperative starting point this module rewrites as a declaration. src/guides/data-engineering-foundations-guide/workbook/module-05-data-quality-gates/es/. In Spanish.
• This guide's DESIGN — the full map of the eight modules, including this module 2's exact mandate. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.