Module introduction: data contracts as versioned artifacts

Why this module exists

After three modules, Kiosko already knows how to detect four of the six data quality dimensions module 1 defined. validate_orders() and OrdersSchema (modules 1 and 2) catch completeness, uniqueness, and validity. validate_referential_integrity() (module 3) catches consistency, with ORD-9508 as evidence. Module 3's final report said it with the same honesty as always: four order_ids with some known problem, seven with none — and, among those seven, ORD-9509 remains, silently, waiting on module 5.

This module does not add a fifth dimension to that list. Accuracy is module 5. Freshness is module 6. What this module changes is something different, and in a sense more fundamental: where the rules that already exist live, and who can read them.

Look carefully at where every rule you've already built currently lives. order_id must be unique: one line inside an OrdersSchema class, in a .py file. unit_price can't be null or negative: another line, in the same file. The 24-hour SLA module 1's lesson 5 mentioned — "a sales file must be available for review within 24 hours of the day it bills" — never got written anywhere executable; that same lesson said so explicitly: "not yet a formal contract, that's module 4". And what to do if a file violates any of these rules — does it get rejected whole, like it did in foundations M7? Do the good rows get separated from the bad ones? — isn't written anywhere either yet; it's a decision that exists only in the head of whoever built the pipeline.

Three kinds of knowledge — what shape the data should have, with what SLA it should arrive, and what to do if it doesn't comply — scattered across Python code only someone who knows how to read pandera.polars can interpret, and informal agreements nobody wrote down. That is, precisely, the problem a data contract solves: a single artifact, in a format any system can read (YAML, not Python), version-controlled like any other important file, that declares all three things together — and that also generates the OrdersSchema you already built, instead of keeping it hand-written in two places that can drift out of sync.

Connection to the previous module. Module 3 closed by naming exactly this missing piece: "nobody has yet written what's expected of the data S04 is going to send... this guide's module 4 is, precisely, where that gap gets closed with a real, versioned artifact." This module doesn't replace any tool from the previous three modules — it rewrites them as the output of a more trustworthy process: a parsed contract, not a hand-typed schema.

A bit of history: this isn't a new idea from this guide

The term "data contract," in the sense this guide uses it, wasn't born in an academic paper or in a tool's documentation — it was born from real data teams, solving the same problem you just read about. Andrew Jones, on GoCardless's data team (a British payments fintech), published "Improving Data Quality with Data Contracts" in December 2021, documenting how they started explicitly and versionedly declaring what data consumers expected from each event a service produced — instead of discovering a broken schema change in production. Chad Sanderson, leading data at Convoy, published "The Rise of Data Contracts" in August 2022, the article that gave the concept the name the industry widely knows it by today and connected it to the broader data governance problem. And in May 2023, PayPal open-sourced (Apache-2.0 license) its data-contract-template, a concrete, tool-agnostic template for writing contracts like the one this module builds — the signal that it stopped being an idea from a handful of early-adopter teams and became a practice large companies document and share.

It's worth saying, with the same honesty this guide's market warning already demanded: no audited competitor curriculum teaches this topic. It isn't an oversight — it's a real gap, documented with market evidence, that this guide closes.

An analogy: the fixed menu a supplier signs with the restaurant

A restaurant that buys fresh fish every day has two ways of working with its supplier. The first: an informal phone call every week, "send me the usual," and every time the order arrives, the chef checks what came in and decides, on the spot, whether it's good or not. The second: a fixed menu, signed by both parties, that states exactly which species get delivered, in what weight range, how often, and what happens if a batch doesn't comply — it gets returned, accepted at a discount, the contract gets canceled. The first way works, until the day the supplier sends different fish, of lower quality, and the chef finds out by inspecting the goods at the back door, with diners already seated. The second way doesn't prevent a bad batch from ever arriving — but when it does, both parties already know, in writing, exactly what was expected and what should happen.

validate_orders() (module 1) and OrdersSchema (module 2) are the chef inspecting the fish at the back door: they do their job well, but only the chef — whoever wrote that code — knows which rules they're applying. orders_contract.yaml, the artifact this module builds, is the signed menu: the same quality requirements, now written somewhere the supplier, the manager, and any new system Kiosko connects tomorrow can read with no need to open a single .py file.

Worked example: the rules that already exist, but that nobody outside Python can read

Before writing a single line of YAML, it's worth confirming with evidence the exact problem this module solves. OrdersSchema, as module 2 left it, already contains the rules — they just live exclusively as a Python object in memory:

# implicit_contract_intro.py
import pandera.polars as pa


class OrdersSchema(pa.DataFrameModel):
    order_id: str = pa.Field(unique=True)
    unit_price: float = pa.Field(nullable=False, ge=0)
    quantity: int = pa.Field(gt=0)


schema = OrdersSchema.to_schema()

print("OrdersSchema's rules, as they already exist in module 2 (but only in Python):\n")
for name, col in schema.columns.items():
    checks = [str(c) for c in col.checks]
    print(f"  {name}: nullable={col.nullable}, unique={col.unique}, checks={checks}")

print("\nQuestion: if someone at Kiosko who NEVER saw this .py file asked you")
print("'what exactly is expected of an orders file?', could you answer without opening the code?")

What to expect. Running python3 implicit_contract_intro.py, the output is exactly this:

OrdersSchema's rules, as they already exist in module 2 (but only in Python):

  order_id: nullable=False, unique=True, checks=[]
  unit_price: nullable=False, unique=False, checks=['<Check greater_than_or_equal_to: greater_than_or_equal_to(0)>']
  quantity: nullable=False, unique=False, checks=['<Check greater_than: greater_than(0)>']

Question: if someone at Kiosko who NEVER saw this .py file asked you
'what exactly is expected of an orders file?', could you answer without opening the code?

OrdersSchema.to_schema() — a method that converts the DataFrameModel class into its internal schema representation — confirms the three rules are there, complete and correct. And, at the same time, it confirms the problem: that information only exists if someone knows this .py file exists, knows how to read Pandera's syntax, and has access to the code repository. S04's operations manager, who just wants to know "what's expected of my sales file?", has no reasonable way to reach that answer. This module closes exactly that distance.

Diagram: from three scattered places to a single artifact

flowchart TD
    A["Schema:\nOrdersSchema in\na .py file"] --> D["orders_contract.yaml\n(a single, versioned\nartifact)"]
    B["24h SLA:\nmentioned in prose,\nmodule 1 lesson 5"] --> D
    C["Violation policy:\ndoesn't exist\nanywhere yet"] --> D
    D --> E["DataContract / ColumnContract\n(pydantic, this module)"]
    E --> F["contract_to_pandera_schema()\nregenerates OrdersSchema"]

Three different origins — Python code, a sentence in an earlier lesson, a decision nobody wrote down — converge into a single file. And that file isn't a passive document: pydantic parses it, and a function turns it back into the same executable OrdersSchema you already know, closing the loop between "what Kiosko declared" and "what the code actually checks."

The map of this guide's 8 modules (reminder)

#	Module	What it's about
1	When green does not mean correct	The lie of the green checkmark; the six dimensions; diagnosing S04 without fixing anything.
2	Declarative data quality tests with Pandera	OrdersSchema, catching completeness/uniqueness/validity — three of six dimensions.
3	Consistency and referential checks	Referential integrity across tables; an anti-join that catches S04's orphan product_id.
4	Data contracts as versioned artifacts (you are here)	What a data contract is; orders_contract.yaml; the contract generates module 2's tests.
5	Accuracy and deterministic anomaly detection	Why a valid row can still be wrong; a price baseline; catching the dollars-to-cents bug.
6	Freshness, volume, and lineage	Freshness and volume as file-level properties; lineage traced by hand.
7	The incident and data governance	Quarantine, alerting, runbook; role-based access; PII masking.
8	Project: Kiosko's trust system	The capstone, run against S04 and against a clean day.

The map of this module

Lesson    Question it answers
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Why the rules that already exist need an artifact
L2        What, precisely, is a data contract? Schema + SLA +
          violation policy, not just a schema with another name.
L3        Write orders_contract.yaml, section by section, with the
          reason behind every field.
L4        Parse the contract with pydantic: DataContract/ColumnContract,
          and what happens when the YAML is malformed.
L5        contract_to_pandera_schema(contract): the contract GENERATES
          the same OrdersSchema from module 2, with evidence.
L6        What happens when the contract changes: version 1.1.0
          (compatible) and version 2.0.0 (breaks compatibility).
L7        The governance question: should S04 be allowed to write
          WITHOUT a contract? Admission, not just after-the-fact
          validation.
L8        Project: Kiosko's first data contract, end to end.

Lessons 2 and 3 build the vocabulary and the artifact itself: what makes something a data contract — not just any config file — and what S04's looks like, section by section. Lesson 4 parses it with pydantic, and demonstrates what happens when someone accidentally breaks the YAML. Lesson 5 is the module's central moment: it turns the contract back into an executable OrdersSchema, and confirms, with evidence, it catches exactly the same rows module 2 caught by hand. Lesson 6 faces a question every versioned artifact confronts sooner or later: what happens when the contract itself has to change? Lesson 7 opens the governance question that gives the module depth — a contract isn't just a tidier way to validate after the data has already arrived, it can also decide whether writing is allowed at all, before it arrives. And lesson 8, the project, assembles everything into a single closing script.

The boundary: what does NOT belong in this module

There's an important boundary with a sister guide, worth drawing right away. dbt has its own contract mechanism: schema.yml, with data_tests: (unique, not_null, accepted_values, relationships) and, since more recent dbt versions, even an explicit model-level contract: section — already taught in depth by dbt-analytics-engineering-guide. The difference isn't in intent — both declare what's expected of a table —, it's in scope: a dbt contract lives inside a dbt project, gets interpreted by dbt's compiler, and only makes sense to someone already working inside that model graph. The contract this module builds is an independent artifact: a YAML file that needs no dbt project to exist, that any system — a Python script, an Airflow pipeline, a service in another language — can read and enforce. orders_contract.yaml could, in theory, end up generating a dbt schema.yml too as one of its many possible outputs — but that isn't part of this guide, and the relationship gets named, not built.

Common mistakes

Thinking "contract" is just a fancier synonym for "schema." What happens: someone, hearing "data contract" for the first time, assumes it's exactly the same as OrdersSchema, with a trendier name. Why it happens: the two artifacts do share a real part — the column schema —, so the overlap is genuine, not a complete misunderstanding. How to spot it: if your mental definition of "contract" includes neither the SLA nor the violation policy, you're missing two-thirds of the complete definition. How to fix it: lesson 2 formalizes this precisely — a contract is schema + SLA + violation policy, all three together, versioned as a single artifact. A file that only has the schema is a schema, not a complete contract.

Believing this module replaces modules 1 through 3's work. What happens: someone, seeing this module regenerates OrdersSchema from YAML, concludes the earlier modules' Python code "no longer matters" or was wasted work. Why it happens: it's tempting to think of a newer version of something as a complete replacement of the previous one. How to spot it: if you can't explain which Pandera function keeps running exactly the same way after this module, you lost the thread. How to fix it: OrdersSchema.validate(df, lazy=True) remains, letter for letter, the same validation mechanism — the only thing that changes is where the OrdersSchema object comes from: before, hand-written; after this module, generated from a parsed contract. The engine doesn't change, its source does.

Expecting the contract to solve accuracy or freshness in this module. What happens: someone, seeing the contract declares a freshness_hours SLA, assumes this module already implements the check that compares that SLA against the file's real arrival time. Why it happens: the field is right there, in the YAML, since lesson 3 — it seems natural for it to also be used right away. How to spot it: revisit this module's map — no lesson mentions PIPELINE_RUN_AT or runs an hour comparison. How to fix it: this module declares the SLA as part of the contract; module 6 is the one that enforces it, with check_freshness(df, run_at=PIPELINE_RUN_AT, sla_hours=24), run against this guide's fixed hour. Declaring a rule and enforcing it are two different steps, and this module only builds the first.

Exercises

Exercise 1 — Reproduce the worked example, and add a fourth, imaginary rule. Run this lesson's implicit_contract_intro.py, then figure out, mentally (without running it), what line of pa.Field(...) you'd need if Kiosko decided order_id must also always start with the prefix "ORD-".

See solution

Pandera lets you express that rule with a text-type Check, for example pa.Field(unique=True, str_startswith="ORD-"). The exercise's point isn't memorizing that exact parameter, but noticing something more important: every new rule someone comes up with has to get translated, by hand, into Pandera's specific syntax — and it still doesn't end up written anywhere other than that .py file. That is, precisely, the problem this entire module is going to solve: declaring the rule once, somewhere readable, and letting the code translate it automatically.

Exercise 2 — Verify module 1's lesson 5's exact quote about the informal SLA. Open workbook/module-01-when-green-does-not-mean-correct/es/05-meet-s04-kioskos-fourth-store.md and find the sentence mentioning module 4. Copy it verbatim. Why do you think that lesson decided not to build the contract right there, instead of leaving it for later?

See solution

The exact sentence is: "The informal agreement with S04 — not yet a formal contract, that's module 4 — is that a sales file must be available for review within 24 hours of the day it bills." It makes sense that lesson didn't build the contract right there because, at that point in the guide, neither OrdersSchema (module 2) nor referential consistency (module 3) existed yet — building a contract before knowing clearly which rules it needed to capture would have meant guessing its content, instead of deriving it from already-built and tested tools. This guide consistently follows the same order: first build the rules concretely and executably, and only afterward formalize them as a versioned artifact — never the other way around.

Exercise 3 — Argue, in your own words, why a YAML contract is more useful than a README.md describing the same rules in prose. A README.md file could also, in theory, describe "order_id must be unique, the price can't be negative..." In 2-3 sentences, explain what this module gains by using YAML parsed with pydantic, instead of free text a human would have to read and interpret.

See solution

A README.md in prose communicates the same information to a person, but no program can reliably read it and act on it — translating "the price can't be negative" from a sentence into code is still manual work, prone to someone reading it differently than the original author wrote it. A YAML parsed with pydantic, on the other hand, is structure, not prose: contract.schema_[1].minimum is an exact value any function can read unambiguously, and contract_to_pandera_schema() demonstrates, in lesson 5, that you can generate executable code directly from it. The gain isn't just about format — it's that the contract stops being documentation someone has to manually keep in sync with the code, and becomes the source the code automatically derives from.

Summary and next step

In this lesson you saw, with executed evidence, the exact problem this module solves: S04's quality rules already exist — three modules built them —, but they live scattered across Python code only someone who reads pandera.polars can interpret, and a loose sentence about a 24-hour SLA no earlier lesson ever formalized. You learned the concept's real history — GoCardless in 2021, Chad Sanderson in 2022, PayPal's open template in 2023 — and confirmed no audited competitor curriculum teaches it. You walked through the full map of this guide's eight modules and this module's eight lessons, and drew the boundary with a contract scoped to a dbt project.

Before moving on you should be able to: explain why OrdersSchema, as it stands in module 2, isn't yet a complete data contract; and name the three components that would turn it into one.

Lesson 2 gives this idea a precise, complete definition, with the three components named one by one.

Resources

• Andrew Jones (GoCardless) — "Improving Data Quality with Data Contracts" (December 2021, the article documenting one of the earliest real uses of the term in a production data team). medium.com/gocardless-tech/improving-data-quality-with-data-contracts-238041e35698. In English.
• Chad Sanderson — "The Rise of Data Contracts" (August 2022, the article that popularized the term industry-wide). dataproducts.substack.com/p/the-rise-of-data-contracts. In English.
• PayPal — official data-contract-template repository (May 2023, Apache-2.0 license, the open template that helped standardize the format). github.com/paypal/data-contract-template. In English.
• Pydantic — official documentation (BaseModel, the foundation for the DataContract/ColumnContract this module builds starting in lesson 4). docs.pydantic.dev/latest. In English.
• dbt-analytics-engineering-guide — the source of schema.yml and data_tests:, the exact boundary this lesson draws. src/guides/dbt-analytics-engineering-guide/DISENO.md. In Spanish.
• Module 3, project (lesson 8), of this same guide — this module's exact starting point: the rules already built, still with no artifact tying them together. src/guides/data-reliability-and-governance-guide/workbook/module-03-consistency-and-referential-checks/es/08-project-s04s-full-consistency-report.md. In Spanish.
• This guide's DESIGN — the full map of the eight modules, including this module 4's exact mandate. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.