Module 4: Data Contracts As Versioned Artifacts
From contract to Pandera schema
Description
This is the module's central lesson — the one that turns everything built so far into the claim that gives this guide its title as a source of truth: the contract generates the tests, the tests don't live separately from the contract. contract_to_pandera_schema(), the function this lesson builds, takes lesson 4's already-validated DataContract object and returns an executable pa.DataFrameSchema — and the demonstration doesn't stay theoretical: it runs against S04's real file and gets compared, row by row, against module 2's OrdersSchema, written entirely by hand.
Connection to the module. Lessons 2 through 4 built the contract as an artifact — definition, YAML, parsing. This lesson is where that artifact stops being just a document and starts doing something: generating executable validation code, with nobody having to rewrite OrdersSchema by hand ever again.
An analogy: the mold, not the finished part
Picture a metal mold for manufacturing identical plastic parts. The mold isn't the part — it's the shape that, every time plastic gets poured into it, produces a part identical to the previous one. If the manufacturer needs to change the part's design, they don't modify every already-manufactured part one by one: they modify the mold, and the next entire batch comes out with the new design, with no extra manual work at all. OrdersSchema, as you wrote it by hand in module 2, is like an already-manufactured part — it works, but if something changes (a new rule, a bound adjustment), someone has to edit that part directly. contract_to_pandera_schema() is the mold: the contract declares the shape, and this function manufactures the Pandera schema every time it's needed, from that shape — never the other way around.
Worked example: contract_to_pandera_schema(contract)
# gen_schema.py
import pandera.polars as pa
from contract import DataContract
TYPE_MAP = {"string": str, "float": float, "integer": int}
def contract_to_pandera_schema(contract: DataContract) -> pa.DataFrameSchema:
"""Converts an already-parsed DataContract into an executable pa.DataFrameSchema."""
columns = {}
for col in contract.schema_:
cast = int if col.type == "integer" else float
checks = []
if col.minimum is not None:
checks.append(pa.Check.ge(cast(col.minimum)))
if col.exclusive_minimum is not None:
checks.append(pa.Check.gt(cast(col.exclusive_minimum)))
columns[col.name] = pa.Column(
TYPE_MAP[col.type],
checks=checks,
nullable=col.nullable,
unique=col.unique,
)
return pa.DataFrameSchema(columns)Read the function with the same care you already trained in the earlier lessons. TYPE_MAP translates the three strings ColumnContract.type accepts ("string", "float", "integer") into the real Python types Pandera needs (str, float, int) — the contract stores types as text because YAML has no native concept of "Python type," so this translation is the exact point where the contract connects with Pandera's executable world. The cast in the loop's first line exists for a concrete reason: col.minimum and col.exclusive_minimum always arrive as float from ColumnContract (that's how lesson 4 declared them), but quantity is an integer column — passing pa.Check.gt(0.0) to an int column would produce a slightly different check name (greater_than(0.0) instead of greater_than(0)) than OrdersSchema's, even though the validation result would be identical. cast fixes that, so the result is, byte for byte, the same as the hand-written schema's.
pa.DataFrameSchema(columns) — unlike pa.DataFrameModel, the class module 2's OrdersSchema used — is Pandera's form designed exactly for this case: building a schema dynamically, from data that isn't known until the program runs, instead of writing a fixed class ahead of time. The two forms — DataFrameModel (a class) and DataFrameSchema (an object built at runtime) — expose the same .validate(df, lazy=True) method, so everything you already learned about reading SchemaErrors.failure_cases in module 2 applies with no change.
Worked example: the comparison that proves the module's promise
# gen_schema.py -- continued
import duckdb
import pandera
import polars as pl
import yaml
class OrdersSchema(pa.DataFrameModel):
order_id: str = pa.Field(unique=True)
unit_price: float = pa.Field(nullable=False, ge=0)
quantity: int = pa.Field(gt=0)
if __name__ == "__main__":
with open("orders_contract.yaml") as f:
raw = yaml.safe_load(f)
contract = DataContract.model_validate(raw)
generated_schema = contract_to_pandera_schema(contract)
con = duckdb.connect("kiosko.duckdb")
df = con.sql("SELECT * FROM orders_s04").pl()
print(f"Rows read from orders_s04: {df.height}\n")
def run(schema, label):
try:
schema.validate(df, lazy=True)
print(f"{label}: all rows passed (not expected)")
return None
except pa.errors.SchemaErrors as exc:
fc = exc.failure_cases
readable = fc.with_columns(
pl.col("index").map_elements(lambda i: df["order_id"][i], return_dtype=pl.String).alias("order_id")
).select(["order_id", "column", "check", "failure_case"])
print(f"=== {label} ===")
print(readable)
return readable
r1 = run(OrdersSchema, "OrdersSchema (module 2, hand-written)")
print()
r2 = run(generated_schema, "contract_to_pandera_schema(contract) (this module, generated)")
print(f"\nBoth failure_cases reports are identical: {r1.equals(r2)}")What to expect. Running python3 gen_schema.py, with orders_contract.yaml (lesson 3) and kiosko.duckdb (with module 2's orders_s04) in the same folder, the output is exactly this:
Rows read from orders_s04: 12
=== OrdersSchema (module 2, hand-written) ===
shape: (4, 4)
┌──────────┬────────────┬──────────────────┬──────────────┐
│ order_id ┆ column ┆ check ┆ failure_case │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str │
╞══════════╪════════════╪══════════════════╪══════════════╡
│ ORD-9502 ┆ order_id ┆ field_uniqueness ┆ ORD-9502 │
│ ORD-9502 ┆ order_id ┆ field_uniqueness ┆ ORD-9502 │
│ ORD-9503 ┆ unit_price ┆ not_nullable ┆ null │
│ ORD-9507 ┆ quantity ┆ greater_than(0) ┆ -1 │
└──────────┴────────────┴──────────────────┴──────────────┘
=== contract_to_pandera_schema(contract) (this module, generated) ===
shape: (4, 4)
┌──────────┬────────────┬──────────────────┬──────────────┐
│ order_id ┆ column ┆ check ┆ failure_case │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ str │
╞══════════╪════════════╪══════════════════╪══════════════╡
│ ORD-9502 ┆ order_id ┆ field_uniqueness ┆ ORD-9502 │
│ ORD-9502 ┆ order_id ┆ field_uniqueness ┆ ORD-9502 │
│ ORD-9503 ┆ unit_price ┆ not_nullable ┆ null │
│ ORD-9507 ┆ quantity ┆ greater_than(0) ┆ -1 │
└──────────┴────────────┴──────────────────┴──────────────┘
Both failure_cases reports are identical: True
Stop on the last line, because it's the complete proof of this module's promise: True. Not "similar." Not "the same dimensions, with a different count" (as happened, with an explained reason, between validate_orders() and OrdersSchema in module 2). Identical — same order_id, same column, same check, same failure_case, row by row, compared with Polars's .equals(). ORD-9502 flagged twice (both appearances of the duplicate), ORD-9503 for a null unit_price, ORD-9507 for a negative quantity — the same four physical rows, the same three dimensions (completeness, uniqueness, validity), you already knew from module 2. The difference isn't in the result — it's in where the schema that produced that result came from: one, typed directly into Python; the other, automatically generated from a versionable YAML file any system can read.
Diagram: two paths, one same destination
flowchart LR
A["orders_contract.yaml"] --> B["DataContract\n(pydantic, lesson 4)"]
B --> C["contract_to_pandera_schema()"]
C --> D["pa.DataFrameSchema\n(generated)"]
E["Module 2, hand-written"] --> F["OrdersSchema\n(pa.DataFrameModel)"]
D --> G{"validate(df, lazy=True)\nagainst orders_s04"}
F --> G
G --> H["The exact same result:\n4 rows, 3 dimensions"]
Two arrows with completely different origins — a parsed YAML file, a hand-written class — converge on the same result. That is, in a single image, this entire module's argument: the path doesn't matter, what matters is that from now on only one of the two paths needs to be maintained by hand.
Going deeper: what Kiosko gains by eliminating the duplication
Before this lesson, Kiosko had two potential sources of truth about orders_s04's rules: OrdersSchema in a .py file, and — starting in lesson 3 — orders_contract.yaml. Two sources of truth about the same fact are, in practice, a recipe for going out of sync: someone changes unit_price's bound in the YAML because Kiosko decided to accept negative prices for credit-note cases, but forgets to update OrdersSchema in the script running in production — now the contract says one thing and the code does another, and nobody notices until a real case exposes it. contract_to_pandera_schema() eliminates that possibility by design: if OrdersSchema never gets hand-written again, and always gets generated from the contract, it becomes structurally impossible for the two sources to say different things — because there aren't two sources anymore, there's one, and an automatic derivative of it.
Common mistakes
Comparing the DataFrameSchemas instead of the failure_cases. What happens: someone tries to confirm the two schemas are "equal" by comparing the generated_schema and OrdersSchema.to_schema() objects directly with ==, and runs into an unexpected result or an error. Why it happens: it seems like the most direct way to confirm equivalence, but Pandera's schema objects aren't designed to be compared that way — they can differ in internal representation details (say, whether a rule was built as part of a DataFrameModel class versus a hand-built DataFrameSchema) with no real difference in behavior at all. How to spot it: if your schema comparison fails while validation behavior looks identical, you're comparing the wrong object. How to fix it: the correct equivalence test, the one this lesson uses, is comparing the result of running both schemas against the same data — r1.equals(r2) over the failure_cases —, not the schema objects against each other. Two molds with a slightly different internal shape can produce identical outer parts; what matters is the part, not the mold.
Forgetting the cast based on the column's type, and getting a differently named check. What happens: someone simplifies contract_to_pandera_schema() by removing the cast = int if col.type == "integer" else float line, and passes col.exclusive_minimum directly to pa.Check.gt(...) with no conversion. The result still catches the same rows, but the check's name in failure_cases comes out as greater_than(0.0) instead of greater_than(0). Why it happens: ColumnContract.exclusive_minimum is typed as float | None in pydantic — a deliberate lesson 4 decision, so the same field serves both integer and floating-point column bounds —, so with no explicit conversion, the incoming value is always float, even if the target column is int. How to spot it: compare the exact text of the check column between the two reports — if one says greater_than(0) and the other greater_than(0.0), you lost the type conversion. How to fix it: keep the explicit cast based on col.type, exactly as this lesson does — it's a small line, but it's the one holding up the "identical result, not just similar" promise.
Thinking contract_to_pandera_schema() needs to know the name S04 or orders_s04. What happens: someone searches, inside the function, for some specific mention of S04 or of the orders_s04 table, and gets surprised finding none. Why it happens: after three modules always working over the same concrete data, it's easy to expect any new function to also mention it. How to spot it: revisit the function's signature — it receives a generic contract: DataContract, and no store or file name appears in its body. How to fix it: that absence is exactly the point — contract_to_pandera_schema() doesn't know or care whether the contract it receives describes orders_s04, orders_s01, or any future Kiosko dataset. All the specificity lives in the YAML file passed to it, not in the function — the same reusability OrdersSchema already had in module 2, now extended one level up, to the function that generates complete schemas from any well-formed contract.
Exercises
Exercise 1 — Run the complete example yourself, and confirm the final True. In a folder with orders_contract.yaml, kiosko.duckdb (with orders_s04 loaded), and gen_schema.py, run python3 gen_schema.py. Confirm the last line printed is exactly Both failure_cases reports are identical: True.
See solution
If orders_contract.yaml wasn't modified since lesson 3, and orders_s04 has module 2's exact twelve rows, the output should reproduce exactly this lesson's: four physical rows flagged in both reports, with the same order_id, column, check, and failure_case in each row, and the final comparison at True. If your result is False, first check whether the type cast inside contract_to_pandera_schema() is still present — it's the most common cause of a tiny difference in some check's text.
Exercise 2 — Add a fourth column to the contract (order_ts, with no rule beyond the type) and confirm it doesn't break the equivalence. Modify orders_contract.yaml by adding a schema entry for order_ts with type: string (no nullable, unique, or bounds — just the name and the type). Run gen_schema.py again (note: OrdersSchema, the hand-written class, doesn't change — that's on purpose).
See solution
schema:
- name: order_id
type: string
nullable: false
unique: true
- name: unit_price
type: float
nullable: false
minimum: 0
- name: quantity
type: integer
nullable: false
exclusive_minimum: 0
- name: order_ts
type: stringWith this fourth column added to the contract, generated_schema now validates one more column than OrdersSchema — but since order_ts has no additional rule (nullable by ColumnContract's default, no bounds), and orders_s04's twelve real rows do have that field present in every row, generated_schema's resulting failure_cases remains identical to OrdersSchema's across the four rows you already knew — the new column adds no new error, because it had no rule to violate. This exercise confirms something important: the contract can already grow beyond what OrdersSchema declared by hand, with no break in the comparison — that's exactly the advantage that motivates Kiosko to stop maintaining OrdersSchema as an independent source.
Exercise 3 — Explain why pa.DataFrameSchema (not pa.DataFrameModel) is the right choice for a generator function. In 2-3 sentences, based on the difference between a class written ahead of time and an object built at runtime, explain why contract_to_pandera_schema() couldn't as easily generate and return a pa.DataFrameModel class dynamically.
See solution
pa.DataFrameModel is meant to be defined as a Python class written directly in the source code — with type annotations (order_id: str) Python evaluates when the file gets imported, not when a function runs —, while pa.DataFrameSchema accepts its columns as normal constructor arguments, something that naturally fits a for loop building a dictionary from data only known at runtime (contract.schema_'s content, which varies depending on which YAML got loaded). It would be technically possible to dynamically generate a DataFrameModel class using advanced Python mechanisms (like type() to create classes at runtime), but that would add unnecessary complexity for the same result — pa.DataFrameSchema already solves the problem directly, with no need for any additional mechanism.
Summary and next step
In this lesson you built contract_to_pandera_schema(), the function that closes this module's complete loop: it takes a DataContract already validated by pydantic (lesson 4) and returns an executable pa.DataFrameSchema. You ran it, for real, against orders_s04's twelve real rows, and confirmed — with an exact comparison, .equals(), not a visual glance — that it produces the same failure_cases as module 2's OrdersSchema, written entirely by hand: the same four physical rows, the same three dimensions.
Before moving on you should be able to: explain, without looking at the code again, why pa.DataFrameSchema (and not pa.DataFrameModel) is the right choice for a generator function; and reproduce the True report comparison result yourself, from scratch.
You have the contract generating real tests, with evidence. Lesson 6 faces a question every versioned artifact confronts sooner or later: what happens when the contract itself needs to change?
Resources
- Pandera — official documentation,
DataFrameSchemaversusDataFrameModel(the two ways of declaring a schema, and when to use each). pandera.readthedocs.io/en/stable/dataframe_schemas.html. In English. - Pandera — official documentation,
Check(the complete reference forCheck.ge,Check.gt, and the rest of the built-in checks). pandera.readthedocs.io/en/stable/checks.html. In English. - Polars — official documentation,
DataFrame.equals(the method used to confirm, exactly, that the twofailure_casesreports are identical). docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.equals.html. In English. - Module 2, project (lesson 8), of this same guide — the exact source of
OrdersSchemaand itsfailure_casesreport, this entire lesson's basis for comparison.src/guides/data-reliability-and-governance-guide/workbook/module-02-declarative-data-quality-tests-with-pandera/es/08-project-testing-s04-with-pandera.md. In Spanish. - This guide's DESIGN.
src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.