Parsing the contract with pydantic

Description

Lesson 3 closed with a warning: yaml.safe_load() converts the YAML into a Python dictionary, but guarantees absolutely nothing about its structure — if someone forgets a field, or writes on_violation: "delete_everything" instead of a recognized value, that broken dictionary goes through with no warning at all. This lesson closes that gap: it builds DataContract and ColumnContract, two pydantic classes that define, with type precision, what a valid contract looks like — and demonstrates, with a deliberately broken YAML, exactly what happens when someone doesn't respect it.

Connection to the module. This lesson turns lesson 3's guarantee-free dictionary into a Python object with real types and validation. Lesson 5 takes that already-validated object and turns it into an executable Pandera schema.

An analogy: the form that rejects an impossible date before filing it

Picture two paperwork forms. The first is a blank sheet of paper: you can write anything in any box, and the clerk who receives it files it without reviewing it — the error, if there is one, gets discovered weeks later, when someone tries to use that information and it doesn't make sense. The second is a digital form with validation: if you type 32 in the "day of the month" field, the system rejects the submission on the spot, with a clear message about which box is wrong and why. yaml.safe_load(), on its own, is the blank sheet of paper — it reads whatever structure the YAML has, with no opinion on whether it makes sense. pydantic is the digital form: it defines, ahead of time, the exact shape every field must have, and rejects — with a precise, field-by-field message — any document that doesn't have it.

Worked example: DataContract and ColumnContract

# contract.py
from typing import Literal
from pydantic import BaseModel, Field


class ColumnContract(BaseModel):
    name: str
    type: Literal["string", "float", "integer"]
    nullable: bool = True
    unique: bool = False
    minimum: float | None = None
    exclusive_minimum: float | None = None


class RowCountRange(BaseModel):
    min: int
    max: int


class SLAContract(BaseModel):
    freshness_hours: int
    row_count: RowCountRange


class DataContract(BaseModel):
    contract_version: str
    dataset: str
    owner: str
    description: str
    schema_: list[ColumnContract] = Field(alias="schema")
    sla: SLAContract
    on_violation: Literal["quarantine", "reject", "alert"]

Four classes, each responsible for a different level of the contract. ColumnContract describes one single entry in the schema: list — notice type: Literal["string", "float", "integer"]: it isn't just any str, it's a type that only accepts those three exact values, so type: "boolean" (something this contract never declared support for) would fail immediately. minimum and exclusive_minimum are float | None = None — optional, because not every column needs a numeric bound (order_id, of type string, never uses them). RowCountRange and SLAContract nest one level deeper: sla.row_count.min and sla.row_count.max, mirroring the YAML's exact nested structure. And DataContract brings it all together, with a syntax detail that deserves separate explanation: schema_: list[ColumnContract] = Field(alias="schema").

Why schema_ with an underscore, and not schema plain

schema is a reserved word in pydantic's vocabulary — the framework itself used that name internally in earlier versions of its API —, so naming the attribute schema plain can clash with that internal reservation. This lesson's solution uses the mechanism pydantic offers exactly for this case: the Python attribute is named schema_ (with an underscore, so it doesn't clash with anything reserved), but Field(alias="schema") tells pydantic that, when reading the input dictionary, it should look for the schema key (no underscore, the one that really exists in the YAML) and assign it to that attribute. model_validate(), the method that parses the dictionary, respects that alias with no need for you to rename anything in the YAML file itself — orders_contract.yaml still says schema:, exactly as you wrote it in lesson 3.

Worked example: parsing the real contract

# contract.py -- continuation, executable block
if __name__ == "__main__":
    import yaml

    with open("orders_contract.yaml") as f:
        raw = yaml.safe_load(f)

    contract = DataContract.model_validate(raw)
    print(f"contract_version: {contract.contract_version}")
    print(f"dataset: {contract.dataset}")
    print(f"owner: {contract.owner}")
    print(f"columns: {[c.name for c in contract.schema_]}")
    for c in contract.schema_:
        print(f"  - {c.name}: type={c.type}, nullable={c.nullable}, unique={c.unique}, "
              f"minimum={c.minimum}, exclusive_minimum={c.exclusive_minimum}")
    print(f"sla.freshness_hours: {contract.sla.freshness_hours}")
    print(f"sla.row_count: min={contract.sla.row_count.min}, max={contract.sla.row_count.max}")
    print(f"on_violation: {contract.on_violation}")

What to expect. Running python3 contract.py, with lesson 3's orders_contract.yaml in the same folder, the output is exactly this:

contract_version: 1.0.0
dataset: orders_s04
owner: kiosko-data-team
columns: ['order_id', 'unit_price', 'quantity']
  - order_id: type=string, nullable=False, unique=True, minimum=None, exclusive_minimum=None
  - unit_price: type=float, nullable=False, unique=False, minimum=0.0, exclusive_minimum=None
  - quantity: type=integer, nullable=False, unique=False, minimum=None, exclusive_minimum=0.0
sla.freshness_hours: 24
sla.row_count: min=5, max=20
on_violation: quarantine

DataContract.model_validate(raw) — pydantic v2's method that replaces the old DataContract(**raw) when you want to explicitly pass an already-built dictionary — walks the entire nested structure, validates every type, applies the aliases, and returns a Python object with typed attributes: contract.sla.row_count.min is already a real int, not a string you'd need to convert by hand. Compare this against lesson 3's raw["sla"]["row_count"]["min"] — the same information, but now with the guarantee that, if this line ran with no exception thrown, the entire contract respects the shape DataContract requires.

Worked example: breaking the contract on purpose

A contract is only useful if it fails loudly when someone breaks it — not if it silently accepts anything. This file, saved as orders_contract_broken.yaml, has two deliberate errors: it's missing the owner field, and on_violation carries a value the class never declared it accepts:

# orders_contract_broken.yaml
contract_version: "1.0.0"
dataset: orders_s04
description: >
  Deliberately broken version: it's missing 'owner' and 'on_violation'
  carries a value the contract doesn't recognize.
schema:
  - name: order_id
    type: string
    nullable: false
    unique: true
  - name: unit_price
    type: float
    nullable: false
    minimum: 0
  - name: quantity
    type: integer
    nullable: false
    exclusive_minimum: 0
sla:
  freshness_hours: 24
  row_count:
    min: 5
    max: 20
on_violation: delete_silently

# parse_broken.py
import yaml
from pydantic import ValidationError
from contract import DataContract

with open("orders_contract_broken.yaml") as f:
    raw = yaml.safe_load(f)

try:
    contract = DataContract.model_validate(raw)
    print("The broken contract passed validation (not expected).")
except ValidationError as exc:
    print(f"ValidationError: {exc.error_count()} errors\n")
    print(exc)

What to expect. Running python3 parse_broken.py, the output is exactly this:

ValidationError: 2 errors

2 validation errors for DataContract
owner
  Field required [type=missing, input_value={'contract_version': '1.0...ion': 'delete_silently'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.13/v/missing
on_violation
  Input should be 'quarantine', 'reject' or 'alert' [type=literal_error, input_value='delete_silently', input_type=str]
    For further information visit https://errors.pydantic.dev/2.13/v/literal_error

Read this result with the same care you already trained with Pandera's SchemaErrors.failure_cases in module 2 — it's the same design philosophy, applied to a different problem. ValidationError doesn't stop at the first error it finds (owner missing) and call it done; it accumulates both, exactly like Pandera's lazy=True accumulated S04's three failures into a single report, instead of stopping at the first. Each error names the exact field (owner, on_violation), the kind of problem (missing, literal_error), and in the second case, even tells you which are the only values it does accept ('quarantine', 'reject' or 'alert') — enough information to fix the contract with no guessing.

Diagram: two possible paths for the same YAML

flowchart TD
    A["orders_contract.yaml"] --> B["yaml.safe_load()"]
    B --> C{"DataContract.model_validate(raw)"}
    C -->|"correct structure"| D["typed and\nvalidated DataContract object"]
    C -->|"missing field,\nor an invalid value"| E["ValidationError\nwith the exact fields"]

Going deeper: why this matters more for a contract than for any other config file

It's worth asking why this guide devotes a whole lesson to type-validating a YAML file, when other Kiosko configs in earlier guides in the ecosystem never needed anything like this. The answer lies in what a contract does with its own content: contract_to_pandera_schema(), in lesson 5, is going to read contract.schema_[i].minimum and hand it straight to pa.Check.ge(...) — if that value turned out to be, by accident, the string "zero" instead of the number 0, the error wouldn't show up when reading the contract, but much later, deep inside Pandera's logic, with a far less clear message about the real cause. Validating the contract with pydantic, at the moment it gets read, moves the failure point as close as possible to the real error — the same "fail fast, with a clear message" principle Pandera's lazy=True already applied in module 2, now applied to the layer that sits before Pandera.

Common mistakes

Writing schema as the attribute name, with no alias. What happens: someone, copying this lesson's structure, defines schema: list[ColumnContract] directly, with no Field(alias="schema") or underscore, and runs into confusing behavior or an error defining the class. Why it happens: schema looks, at first glance, like a perfectly reasonable attribute name — there's no obvious reason to avoid it, until it collides with pydantic's internal use. How to spot it: if your DataContract class doesn't behave as expected when accessing .schema, or pydantic complains about a reserved name, check whether you named the attribute schema plain. How to fix it: follow this lesson's exact pattern — name the attribute schema_ (with an underscore) and use Field(alias="schema") so reading the YAML (which does say schema:, no underscore) keeps working with no change to the file.

Using DataContract(**raw) and assuming it behaves exactly like DataContract.model_validate(raw) in every case. What happens: someone, used to building Python objects by unpacking a dictionary with **, uses DataContract(**raw) instead of model_validate(raw) — and for a well-formed dictionary, like this lesson's raw, the result is, in fact, identical: same attributes, same values, including schema_'s alias correctly resolved. The problem shows up at the edge case, not the happy path. Why it happens: **raw is native Python syntax — unpacking a dictionary as keyword arguments —, evaluated by the interpreter before pydantic even gets a chance to step in. If raw isn't, for some reason, a valid dictionary (say, if yaml.safe_load() returned None because the file is empty), **raw fails with a plain Python TypeError, not a pydantic ValidationError. How to spot it: compare the two error messages for the same case — DataContract(**None) throws TypeError: contract.DataContract() argument after ** must be a mapping, not NoneType; DataContract.model_validate(None) throws pydantic.ValidationError: Input should be a valid dictionary or instance of DataContract, a much clearer error and, above all, catchable with the same except ValidationError the rest of this lesson already uses. How to fix it: use model_validate() whenever the dictionary's origin isn't guaranteed — like a YAML file someone else could accidentally leave empty —, so any problem, including an empty or badly read file, ends up as the same kind of controlled error you already know how to handle.

Catching Exception instead of ValidationError specifically. What happens: someone writes except Exception as exc: instead of except ValidationError as exc: when parsing a contract, and ends up also catching (and hiding) completely different errors — a file that doesn't exist, a YAML with broken syntax — as if they were the same kind of problem. Why it happens: except Exception feels like a "safe" way to keep anything from crashing the program, but it hides valuable information about what kind of error happened. How to spot it: if your error handling can't tell "the YAML has bad syntax" apart from "the contract doesn't respect DataContract's structure" apart from "the file doesn't exist," your except is too broad. How to fix it: catch ValidationError specifically for contract structure errors (as this lesson does), and let other kinds of errors — FileNotFoundError, yaml.YAMLError — propagate or get handled separately, with their own specific message.

Exercises

Exercise 1 — Break the contract a third way: an invalid column type. Modify orders_contract_broken.yaml (or create it again) by changing unit_price's type: float to type: number (a value Literal["string", "float", "integer"] doesn't accept). Run parse_broken.py again and confirm the exact error message.

schema.1.type
  Input should be 'string', 'float' or 'integer' [type=literal_error, input_value='number', input_type=str]

Exercise 2 — Confirm a contract with only extra, undeclared fields still passes. Add a new field, not declared in any class — say, extra_note: "undeclared field" — at orders_contract.yaml's top level, and confirm whether DataContract.model_validate(raw) accepts or rejects it.

Exercise 3 — Argue why ValidationError accumulates every error instead of stopping at the first. In 2-3 sentences, connect this pydantic behavior to the same pattern you already saw in Pandera's lazy=True, in module 2.

Summary and next step

In this lesson you built DataContract and ColumnContract, two pydantic classes that give lesson 3's loose dictionary a real structural guarantee: correct types, required fields present, values restricted to what the contract declares it accepts. You parsed S04's real contract with model_validate(), and confirmed, with a YAML deliberately broken in two different places, that pydantic reports both errors together, with the exact field and exact reason for each.

Before moving on you should be able to: explain why schema_ uses an alias instead of the direct schema name; and reproduce the two-error ValidationError by running parse_broken.py yourself.

You have a fully validated DataContract object, with typed attributes accessible with no need to touch the raw dictionary again. Lesson 5 — this module's central moment — takes that object and turns it back into an executable Pandera schema, confirming it catches exactly the same rows module 2's OrdersSchema did.

Resources

• Pydantic — official documentation, BaseModel (the base class for DataContract/ColumnContract, and the field validation mechanism). docs.pydantic.dev/latest/concepts/models. In English.
• Pydantic — official documentation, model_validate (the method that parses an already-built dictionary, used throughout this lesson). docs.pydantic.dev/latest/concepts/models/#validating-data. In English.
• Pydantic — official documentation, field aliases (Field(alias=...), the solution to the name clash with schema). docs.pydantic.dev/latest/concepts/alias. In English.
• Pydantic — official documentation, error handling (ValidationError, the accumulated-errors structure). docs.pydantic.dev/latest/errors/errors. In English.
• Module 2, lesson 7, of this same guide — the source of the lazy=True / accumulated-error-report pattern, the same design principle this lesson applies with pydantic. src/guides/data-reliability-and-governance-guide/workbook/module-02-declarative-data-quality-tests-with-pandera/es/07-validity-checks-and-reading-failure-cases.md. In Spanish.
• This guide's DESIGN. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.