Module introduction: Kiosko's complete trust system

Why this module exists

Seven modules, seven pieces, each tested separately. Module 1 diagnosed, with foundations' validate_orders(), what slipped through a local gate: ORD-9508 (product_id="P099") and ORD-9509 (unit_price=60.00) sailed through clean. Module 2 declared the first three rules as a Pandera schema, OrdersSchema, and confirmed the same three dimensions foundations already caught — completeness, uniqueness, validity — now declaratively. Module 3 wrote validate_referential_integrity(), a real anti-join against dim_product, and finally caught ORD-9508. Module 4 took those same rules and wrote them as a versioned artifact, contracts/orders_contract.yaml, capable of generating the same OrdersSchema instead of keeping it hand-written in two places. Module 5 built reference_prices from the clean canonical week and check_price_baseline(), and finally caught ORD-9509 — the dollars-to-cents bug that gave this guide its name. Module 6 wrote check_freshness() and check_volume(), the first file-level, not row-level, checks, and mapped LINEAGE_MAP, this entire ecosystem's first lineage map. Module 7 closed the diagnosis with action: quarantine() instead of rejecting everything or passing in silence, raise_alert() and a written runbook.md, and data governance — ACCESS_POLICY, mask_pii(), generate_catalog() — over this guide's first table with a real person's information, customers.

Each of those seven modules lived in its own working directory, with its own closing script, testing one piece at a time. None of the seven showed you the complete system: the seven pieces — contract, declarative tests, consistency, anomalies, freshness, volume, quarantine, alert, access, masking, catalog — working together, in a single entry point, capable of receiving any day's Kiosko file and answering, in a single run, the question that gives this entire guide its name: can Kiosko trust this data, yes or no?

This module is that assembly. It introduces no new data quality tool — every piece you're going to use, you already built, with your own hands, in an earlier module. What it does is exactly what data-modeling-for-analytics-guide, dbt-analytics-engineering-guide, and lakehouse-and-iceberg-guide's capstones already did before: bring scattered pieces together into a single system, verified end to end, run twice — once on broken data, once on clean data — to demonstrate, with executed evidence, the complete system does both things: it catches what's wrong, and it lets what's right through with no friction.

Connection to the previous module. Module 7 closed with this exact sentence: "Module 8 — this entire guide's capstone — assembles everything, from module 4's contract to this module's governance, into a single system, run twice." This module fulfills that promise to the letter, adding no new business rule at all — everything you need already exists, scattered across seven modules, and this module brings it together.

The case that accompanies us: two runs, one system

By this module's close, Kiosko has a single entry point — run_full_gate() — capable of receiving any orders file and returning, in a single call, the verdict of the seven checks this guide built: completeness, uniqueness, and validity (generated from module 4's contract), consistency (module 3), accuracy (module 5), freshness and volume (module 6). This module runs that system twice, on two completely different files:

• orders_2026-08-14.csv — the same S04 file that opened this guide in module 1: twelve lines, six clean and six broken, one per quality dimension except volume. The complete system, run on this file, reports six failures out of seven checks — the exact same number you already know, now produced by a single integrated system, not seven loose scripts.
• A clean day rebuilt from the S01-S03 canonical week — a new file, orders_2026-08-15.csv, built with data already confirmed reliable by eight earlier guides in this ecosystem: same stores, same products, same reference prices. The complete system, run on this file, reports zero failures out of seven checks — the proof that "the system detects what's broken" and "the system is too sensitive and flags anything" are completely different claims, and only the first is true.

The difference between those two numbers — 6 and 0 — is this module's, and this entire guide's, central evidence: a real trust system isn't one that rejects everything just in case, nor one that approves everything to avoid friction — it's one that distinguishes, with executed evidence, between a file that really has problems and one that really doesn't.

An analogy: the central alarm panel, not seven loose detectors

Imagine a building with seven different smoke detectors, each installed and tested separately over seven weeks of construction: one in the kitchen, tested with real smoke from burnt toast; one in the electrical room, tested with a controlled overload; one in storage, tested with a candle near the sensor. Every individual test worked — the kitchen detector went off when it should have. But if a real fire broke out today in the entire building, no resident would know, with a single glance, what's happening: they'd have to walk to each of the seven detectors, one by one, to find out which ones are going off and which aren't.

This module installs the central panel. It changes not a single detector — each one stays exactly the one you already tested, with the exact same sensitivity — but it connects them to a single dashboard a resident can look at once and understand immediately: seven lights, each with its own name, green or red, and a single general alarm that goes off if any of the seven is red. That panel is run_full_gate(). This module's two runs are the proof the panel works in both directions: when there's a real fire (S04), six of the seven lights turn red, exactly the correct ones; when there's no fire at all (the clean day), all seven stay green, with no false alarm.

The map of this guide's 8 modules (reminder)

#	Module	What it's about
1	When green doesn't mean correct	The green checkmark lie; the six dimensions; diagnosing S04 without fixing anything.
2	Declarative data quality tests with Pandera	OrdersSchema, catching completeness/uniqueness/validity — three of six dimensions.
3	Consistency and referential checks	Referential integrity between tables; an anti-join that catches S04's orphan product_id.
4	Data contracts as versioned artifacts	What a data contract is; orders_contract.yaml; the contract generates module 2's tests.
5	Accuracy and deterministic anomaly detection	A price baseline; the dollars-to-cents bug, finally caught.
6	Freshness, volume, and lineage	Freshness and volume as file-level properties; lineage mapped by hand.
7	The incident and data governance	Quarantine, alert, runbook; role-based access; deterministic PII masking; a minimal catalog.
8	Project: Kiosko's trust system (you are here)	The capstone, run against S04 and against a clean day. Closes the guide.

The map of this module

Lesson    What it solves
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Why this module exists, the two runs, the map.
L2        The brief -- why Kiosko needs ONE system, not seven
          loose scripts nobody runs together.
L3        run_full_gate() -- really assembled, tested first
          on a toy DataFrame (1 failure of 7).
L4        The real run on S04: 6 failures of 7, quarantine +
          alert triggered by the contract itself (on_violation).
L5        A clean day rebuilt from S01-S03: 0 failures of 7 --
          the proof the system doesn't flag just anything.
L6        LINEAGE_MAP + generate_catalog() + ACCESS_POLICY + mask_pii()
          published as the complete system's governance layer.
L7        Ecosystem close: the sibling guides and what each one
          solves about what this system leaves pending.
L8        Final project: both runs, one script, automatic
          assertions -- CLOSES the whole guide.

Lessons 2 through 5 build the module's technical core: the brief that justifies why a single system is needed (lesson 2), run_full_gate()'s real assembly (lesson 3), and the two runs that confirm its behavior in both directions (lessons 4 and 5). Lessons 6 and 7 close the module with what a trust system needs beyond detection itself: published governance (lesson 6) and the honest map of what remains pending (lesson 7). Lesson 8, the project, brings both runs together into a single script with automatic assertions — this entire guide's close.

What this module does NOT teach again

Every mechanism you're going to see run in lessons 3 through 6 was already taught, step by step, with its own analogy and its own common mistakes, in an earlier module. This module doesn't repeat those explanations — it uses them. If any code step confuses you, the exact reference to where it was first taught is flagged in every lesson: contract_to_pandera_schema() got explained in depth in module 4, validate_referential_integrity() in module 3, check_price_baseline() in module 5, check_freshness()/check_volume()/LINEAGE_MAP in module 6, and quarantine()/raise_alert()/ACCESS_POLICY/mask_pii()/generate_catalog() in module 7. This capstone assumes you already know them, and focuses on something no earlier module could show in isolation: how they behave, together, in a single system, against two completely different files.

Going deeper: why "6 failures" and "0 failures" are equally important

It's tempting, after seven modules dedicated almost entirely to finding problems, to think this capstone's job is just confirming, once more, that S04 has six failures. But a system that only knows how to say no — that flags any file as suspicious, including the ones that are perfectly fine — is, in practice, just as useless as one that never says no. A security guard who sounds the alarm every time someone walks near the door, regardless of whether they have a legitimate key, trains the whole building to ignore the alarm. This module's second run — the clean day, zero failures — isn't an anticlimax after S04's incident; it's half the evidence a real trust system needs: not just that it catches what's bad, but that it doesn't bother what's good. Without that second run, nobody at Kiosko would have any reason to trust that a file passing all seven checks is really clean, instead of the system simply failing to detect the problem.

Common mistakes

Expecting this module to teach a data quality technique the seven earlier ones didn't cover. What happens: someone arrives at this module expecting to learn something new about Pandera, contracts, or anomaly detection — an API they haven't seen before. Why it happens: "capstone," in some courses, sounds like "the most advanced module," and it's easy to assume "advanced" means "technically new." How to spot it: if you finish lesson 3 without recognizing any function you already used before, check: contract_to_pandera_schema(), validate_referential_integrity(), check_price_baseline(), check_freshness(), check_volume(), quarantine() — all come from the seven earlier modules. How to fix it: this module is about integration, not new technique — the value is in seeing the seven pieces coexist inside a single entry point, verified in both directions on real data.

Thinking lesson 5's clean day is an "invented" file that proves nothing real. What happens: someone notices orders_2026-08-15.csv isn't a file any earlier guide in this ecosystem explicitly built, and assumes it's less "real" than orders_2026-08-14.csv. Why it happens: S04 and its incident have seven modules of history behind them; the clean day appears for the first time in this module. How to spot it: review where its data comes from — stores S01-S03, products P001-P004, prices identical to reference_prices, the baseline already built and verified in module 5 over Kiosko's real canonical week — no value in that file gets invented at random. How to fix it: understand the clean day for what it is — a deliberate reconstruction, with the same determinism discipline as every earlier file in this guide, designed specifically to test the side of the system S04 could never test on its own: what happens when everything really is fine?

Exercises

Exercise 1 — Recall, from memory, which function from which module solves each of run_full_gate()'s seven checks. Without going back to the earlier modules, try to complete this list: what function catches completeness/uniqueness/validity? Which catches consistency? Which catches accuracy? Which two catch freshness and volume?

See solution

Completeness, uniqueness, and validity come from contract_to_pandera_schema(contract).validate(df, lazy=True) (module 4, generating the same result as module 2's OrdersSchema). Consistency comes from validate_referential_integrity(orders_df, dim_product_df) (module 3, an anti-join against dim_product). Accuracy comes from check_price_baseline(df, reference_prices, tolerance=0.5) (module 5, comparing against the canonical week's baseline). Freshness comes from check_freshness(df, run_at=PIPELINE_RUN_AT, sla_hours=24) and volume from check_volume(df, min_rows=5, max_rows=20) — both from module 6. None of the seven is new in this module; run_full_gate(), the only genuinely new piece, just orders and gathers them into a single results list.

Exercise 2 — Predict, before lesson 4, how many of the seven checks you're going to see PASS when the system runs on S04. Based on what you already know from modules 1 through 7, how many of the seven checks do you expect to pass, and which one(s)?

See solution

A single check out of the seven should pass on S04: volume. Module 6 already confirmed, with executed evidence, that check_volume(df, min_rows=5, max_rows=20) reports PASS for S04's file's twelve rows — within the expected range for a new store's first day — while the other six checks (completeness, uniqueness, validity, consistency, accuracy, freshness) already each failed, in some earlier module of this guide. The expected result is, then, 6 failures and 1 PASS out of 7 total checks — not 6 of 6, a detail worth confirming carefully before reading lesson 4.

Exercise 3 — Explain, in 2-3 sentences, why this module's lesson 8 is called "project" and not "summary." Considering this module already runs both complete runs in lessons 4 and 5, what does lesson 8's project specifically add?

See solution

A summary would repeat, in prose, the numbers already seen in lessons 4 and 5 — a claim with no new evidence. A runnable project, by contrast, rebuilds the complete system from scratch, in a new folder, and verifies with assert that the result is still exactly the same: not "trusting S04 still gives 6 failures," but testing it again, in the same "verify, don't trust" style modules 1 through 7's closing projects already applied in this guide, and data-modeling-for-analytics-guide, dbt-analytics-engineering-guide, and lakehouse-and-iceberg-guide's capstones before that. The difference between "summary" and "project" is, exactly, the difference between remembering a result and reproducing it with code that really runs.

Summary and next step

In this lesson you saw why this module exists: seven earlier modules each built a specific piece of Kiosko's trust system over S04 — this module brings them together into a single entry point, run_full_gate(), and runs it twice to demonstrate it works in both directions. You learned the complete map of the eight lessons, and the central idea organizing them: 6 failures on broken data and 0 failures on clean data aren't two loose results, they're the two halves of the same reliability proof.

Before moving on you should be able to: name the seven checks making up the complete system and which module each comes from; and explain why a system that only detects problems, with no confirmation that it also lets what's correct through, isn't yet a complete trust system.

Lesson 2 translates this need into a concrete brief: what, in its own words, Kiosko's data team asks of itself, and why seven loose scripts, each perfect on its own, are no longer enough.

Resources

• Module 7, project (lesson 8), of this same guide — the source of the exact state this module picks S04 back up in (quarantine(), raise_alert(), ACCESS_POLICY, mask_pii(), generate_catalog(), all verified). src/guides/data-reliability-and-governance-guide/workbook/module-07-the-incident-and-data-governance/en/08-project-s04s-incident-response-and-access-policy.md. In English.
• Module 4, project (lesson 8), of this same guide — the source of contract_to_pandera_schema(), the piece this module reuses as the completeness/uniqueness/validity generator. src/guides/data-reliability-and-governance-guide/workbook/module-04-data-contracts-as-versioned-artifacts/en/08-project-kioskos-first-data-contract.md. In English.
• lakehouse-and-iceberg-guide, module 8 (capstone) — the exact "assemble seven pieces into a single system, verified with assert" pattern this module imitates. src/guides/lakehouse-and-iceberg-guide/workbook/module-08-project-kioskos-lakehouse/. In English.
• This guide's DESIGN — this module 8's exact mandate, this guide's last. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.