What, precisely, a declarative quality test is

Description

"Declarative" is a word that gets used a lot and defined rarely. This lesson gives it an exact, unambiguous definition: a declarative quality test is a statement about what the data should look like, expressed as an object separate from the logic that verifies it — never a scattered if, hidden inside a function that's also doing other things. Lesson 1 already showed the contrast with a minimal example (QUANTITY_RULE against check_quantity_imperative()); this lesson builds that contrast up into a complete schema, with several columns and several rules at once, still without installing Pandera — so that when lesson 4 finally installs it, you immediately recognize exactly which problem it solves.

Connection to the module. This lesson is the conceptual foundation for everything that follows: lesson 3 uses this same definition to compare three real market tools (Pandera, Great Expectations, Soda), and lessons 5 through 8 are, literally, this same lesson's idea, with Pandera's real syntax instead of the homemade interpreter you build here.

An analogy: the recipe with exact measurements, not "a bit of salt"

Two people cook the same dish, each with a different recipe. The first recipe says: "add salt, not too much, until it tastes right." It's a real instruction, and someone with experience can follow it — but the decision of "how much is enough" lives entirely in the cook's judgment, in the moment, with no written criterion anyone else could verify later. If two different cooks follow that same recipe, it's perfectly possible for one dish to turn out salty and the other bland, and neither would be "doing the recipe wrong" — the recipe itself was never precise.

The second recipe says: "add 6 grams of salt per 500 grams of dough." It's a verifiable statement: anyone with a scale can confirm, unambiguously, whether that instruction was followed or not. The recipe doesn't cook the dish by itself — you still need someone, or a machine, to execute it —, but the instruction itself stopped depending on the judgment of whoever follows it. A declarative quality test is that second recipe: order_id must be unique, unit_price can't be null, quantity must be greater than zero — each is a verifiable statement, with a binary answer, that doesn't depend on the judgment of whoever reads it. validate_orders(), though perfectly correct, resembles the first recipe in one precise sense: its rules exist, but they live mixed in with the logic that applies them, so confirming exactly what it requires means reading code, not a list of statements.

Worked example: a schema declared by hand, still no Pandera

This example extends lesson 1's idea — a single rule — into a complete schema: several columns, each with its own statement, interpreted by a single generic engine that knows nothing specific about orders:

# a_declared_schema.py -- still no Pandera: a minimal, hand-written version
# of what it means to "declare" a schema instead of writing it step by step
ORDERS_SCHEMA = [
    {"column": "order_id", "unique": True},
    {"column": "unit_price", "nullable": False},
    {"column": "quantity", "rule": "greater_than", "value": 0},
]

rows = [
    {"order_id": "ORD-T1", "unit_price": 1.20, "quantity": 2},
    {"order_id": "ORD-T2", "unit_price": None, "quantity": 1},   # violates nullable=False
    {"order_id": "ORD-T3", "unit_price": 0.55, "quantity": -1},  # violates greater_than 0
    {"order_id": "ORD-T1", "unit_price": 0.75, "quantity": 3},   # violates unique=True (repeats ORD-T1)
]


def validate_declared_schema(rows: list[dict], schema: list[dict]) -> list[str]:
    """A generic interpreter: it knows nothing about 'orders' specifically,
    it only knows how to read a list of declared rules and apply them."""
    violations = []
    seen_values: dict[str, set] = {}

    for rule in schema:
        column = rule["column"]
        seen_values.setdefault(column, set())

    for i, row in enumerate(rows):
        for rule in schema:
            column = rule["column"]
            value = row[column]

            if rule.get("nullable") is False and value is None:
                violations.append(f"row {i} ({row['order_id']}): '{column}' cannot be null")

            if rule.get("unique") and value in seen_values[column]:
                violations.append(f"row {i} ({row['order_id']}): '{column}'={value} already appeared before")
            seen_values[column].add(value)

            if rule.get("rule") == "greater_than" and value is not None and not (value > rule["value"]):
                violations.append(f"row {i} ({row['order_id']}): '{column}'={value} must be > {rule['value']}")

    return violations


for v in validate_declared_schema(rows, ORDERS_SCHEMA):
    print(v)

What to expect. Running python3 a_declared_schema.py, the output is exactly this:

row 1 (ORD-T2): 'unit_price' cannot be null
row 2 (ORD-T3): 'quantity'=-1 must be > 0
row 3 (ORD-T1): 'order_id'=ORD-T1 already appeared before

Three violations, each precisely identified: which row, which column, which rule broke. Notice something important about ORDERS_SCHEMA: it's a simple list of dictionaries — no logic inside it, no if, no loop. If Kiosko adds a new column tomorrow (say, store_id, with its own "not null" rule), you add it as one more entry to that list, without touching a single line of validate_declared_schema(). That interpreter already knows how to read nullable, unique, and greater_than for any column it's asked about — you didn't have to write a new function per column, the way you would have had to with check_business_rules()'s imperative approach.

Diagram: the anatomy of a declarative test

flowchart LR
    subgraph DECLARACION["The declaration (the data)"]
        A["ORDERS_SCHEMA:\na list of rules,\nwith no logic"]
    end

    subgraph MOTOR["The engine (the logic, generic)"]
        B["validate_declared_schema():\nknows how to read ANY rule\nwith this shape"]
    end

    subgraph RESULTADO["The result"]
        C["A list of violations,\neach with row + column\n+ broken rule"]
    end

    A --> B
    D["rows: the real data\nto check"] --> B
    B --> C

The diagram isolates the three pieces that are going to reappear, with different names, in every real declarative tool that exists: a declaration (what should be true), an engine (that knows how to read that declaration and apply it against real data), and a structured result (what got violated, where). Pandera — which you install in lesson 4 — is exactly this same anatomy, with a much richer rule vocabulary (Field(unique=True), Field(nullable=False), Field(gt=0), and many more you're going to learn) and a much more battle-tested, faster engine than validate_declared_schema(), written and maintained by an open-source project instead of by you, by hand, in a single lesson.

Going deeper: three properties that separate a declarative test from "just moving the code somewhere else"

Not just any code reorganization counts as "declarative." For a statement to be, precisely, a declarative quality test, it needs to satisfy three properties at once:

It's data, not a control-flow instruction. ORDERS_SCHEMA is a list of dictionaries — you could save it to a JSON file, send it over an API, print it and read it without running any Python code. check_business_rules(), on the other hand, is a function: to know what it requires, you have to execute it mentally (or for real) instruction by instruction. This property is what makes it possible, later in this guide (module 4), for a YAML file — something that isn't even Python — to generate the same schema you write by hand in this module: if the rule is already data, any format that can represent data can represent it.

It's interpreted by a generic engine, not by case-specific logic. validate_declared_schema() knows nothing about "orders" or "Kiosko" — it knows how to read unique, nullable, and greater_than, and apply them against any column of any list of dictionaries handed to it. Compare that to foundations M5's check_business_rules(), which has, hand-written, if order.quantity <= 0 — a line that only makes sense for that specific column, in that specific function.

Its result is structured information about what got violated, not just a boolean. validate_declared_schema() doesn't return True/False — it returns a list with the exact detail of each violation. A declarative test that only said "passed" or "didn't pass," without saying what failed, would be much less useful: the real value of declaring rules separately is, precisely, that the engine can report precisely which rule, on which data, broke — the exact foundation for what Pandera does with SchemaErrors.failure_cases, which you're going to read in detail in lesson 7.

Common mistakes

Confusing "declarative" with "no code." What happens: someone thinks a declarative test requires writing no Python at all, as if the declaration ran itself. Why it happens: the word "declarative" sounds like "just say what you want, no programming needed," and that's true only for whoever uses the declaration, not for whoever builds the engine that interprets it. How to spot it: if you expect ORDERS_SCHEMA (or, later, Pandera's OrdersSchema) to "just work" with no engine reading it, you're forgetting half the system. How to fix it: always remember the two pieces — the declaration (easy to write, almost no logic) and the engine (with all the logic inside, already written for you by Pandera starting in lesson 4).

Thinking any function with parameters already counts as "declarative." What happens: someone sees lesson 1's apply_declarative_rule(rows, rule) and concludes any function that takes a config dictionary already counts as declarative. Why it happens: receiving parameters feels similar to "separating the rule from the logic." How to spot it: ask yourself whether the dictionary you're passing describes what should be true (a verifiable statement) or is just a different way of passing values to a function that still has case-specific logic inside it. How to fix it: the real test is this lesson's "generic engine" property — if your function needs a new if every time a new kind of rule shows up, you still have case-specific logic hidden, even if it's organized as parameters.

Underestimating how much real work a well-built generic engine does. What happens: after seeing validate_declared_schema() come in under twenty lines, someone assumes writing Pandera's engine "can't be that hard," and considers rewriting it by hand instead of installing it. Why it happens: this lesson's example is deliberately minimal, to teach the concept without noise — it doesn't handle wrong types, doesn't generate structured reports like a DataFrame, doesn't support thousands of different rules, isn't optimized for large tables. How to fix it: this example's goal was never "build a Pandera competitor" — it was isolating the central idea enough that you'd immediately recognize it in the real syntax. Lesson 4 installs the real, mature version, maintained by an open-source community since 2018.

Exercises

Exercise 1 — Add a range rule with two bounds. Extend ORDERS_SCHEMA with a new rule for quantity that, besides greater_than: 0, requires quantity not exceed 10 (a reasonable upper bound for a single Kiosko order). You'll need to add a new rule type ("less_or_equal") both to the list and to the interpreter.

ORDERS_SCHEMA_V2 = [
    {"column": "order_id", "unique": True},
    {"column": "unit_price", "nullable": False},
    {"column": "quantity", "rule": "greater_than", "value": 0},
    {"column": "quantity", "rule": "less_or_equal", "value": 10},
]

rows_v2 = [
    {"order_id": "ORD-T5", "unit_price": 1.20, "quantity": 25},  # violates less_or_equal 10
]


def validate_declared_schema_v2(rows: list[dict], schema: list[dict]) -> list[str]:
    violations = []
    seen_values: dict[str, set] = {}
    for rule in schema:
        seen_values.setdefault(rule["column"], set())

    for i, row in enumerate(rows):
        for rule in schema:
            column = rule["column"]
            value = row[column]
            if rule.get("nullable") is False and value is None:
                violations.append(f"row {i} ({row['order_id']}): '{column}' cannot be null")
            if rule.get("unique") and value in seen_values[column]:
                violations.append(f"row {i} ({row['order_id']}): '{column}'={value} already appeared before")
            seen_values[column].add(value)
            if rule.get("rule") == "greater_than" and value is not None and not (value > rule["value"]):
                violations.append(f"row {i} ({row['order_id']}): '{column}'={value} must be > {rule['value']}")
            if rule.get("rule") == "less_or_equal" and value is not None and not (value <= rule["value"]):
                violations.append(f"row {i} ({row['order_id']}): '{column}'={value} must be <= {rule['value']}")
    return violations


for v in validate_declared_schema_v2(rows_v2, ORDERS_SCHEMA_V2):
    print(v)

row 0 (ORD-T5): 'quantity'=25 must be <= 10

Exercise 2 — Name the three properties without looking at Going deeper. From memory, write down the three properties this lesson requires for something to count, precisely, as a declarative test — not just "reorganized code."

Exercise 3 — Argue whether foundations M5's check_schema() is, in some sense, already "partially declarative." Revisit check_schema() from data-engineering-foundations-guide M5 (quoted in this guide's module 1 lesson 4): it loops over REQUIRED_FIELDS, a list of column names, and checks each one is present. In 2-3 sentences, argue whether that function already satisfies, even partially, any of this lesson's three properties.

Summary and next step

In this lesson you precisely and unambiguously defined what a declarative quality test is: a statement — data, not logic — about what the data should look like, interpreted by a generic engine that produces a structured result about what got violated. You built, by hand and with no external library, a complete schema with three rules across three different columns, and confirmed, with executed code, that adding a new rule never requires touching existing rules.

Before moving on you should be able to: explain, without looking at the text, the three properties that separate a declarative test from just-reorganized code; and recognize, in any new code snippet you see, whether the rule lives as a separate piece of data or is woven together with the logic that applies it.

You have the complete concept, built by hand. Lesson 3 uses it as a measuring stick to compare, with real license and maturity evidence, three market tools that already solved — much better than this lesson's validate_declared_schema() — the problem of interpreting data quality declarations: Pandera, Great Expectations, and Soda.

Resources

• Pandera — official documentation, introduction and project motivation section. pandera.readthedocs.io. In English.
• Joe Reis & Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022) — the data quality framework that supports module 1 and continues here. oreilly.com/library/view/fundamentals-of-data/9781098108298. In English.
• data-engineering-foundations-guide, module 5 — the source of check_schema(), analyzed in this lesson's Exercise 3. src/guides/data-engineering-foundations-guide/workbook/module-05-data-quality-gates/es/. In Spanish.
• This guide's DESIGN — the exact definition of a declarative test this module develops. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.