Module 8: Project Kioskos Reliability And Governance System
The brief: Kiosko needs a trust system, not just a pipeline
Description
Before writing a single line of run_full_gate(), it's worth writing the brief that justifies it — the same discipline lakehouse-and-iceberg-guide already used before assembling its capstone. This lesson has no code yet: it translates, in Kiosko's own data team's voice, why seven loose scripts — each perfect on its own — stop being enough at the exact moment someone needs to answer, with a single question, "can I trust the file that arrived today?".
Connection to the module. Lesson 1 showed the complete map. This lesson builds the business argument behind that map: what Kiosko is missing today, with seven modules already built, and what shape the piece uniting them needs to have. Lesson 3 builds that piece for real.
An analogy: seven specialists, no primary care doctor
A patient with seven different medical tests — blood work, an X-ray, an electrocardiogram, each done by a different specialist, in different offices, in different weeks — technically has all the information they need to know whether they're healthy. But if nobody brings those seven results together in one place, with a single "all good" or "something needs attention" criterion, the patient ends up carrying seven loose papers, with no fast way to answer the question that actually matters: am I okay, yes or no? A primary care doctor repeats no test — they read the seven already-done results, and give a single verdict, with a clear list of what needs follow-up if something came back wrong.
Kiosko, at module 7's close, has exactly that problem: seven excellent results, none gathered together. This lesson writes the explicit request for the "primary care doctor" Kiosko is missing.
The material: Kiosko's real state at module 7's close
Before requesting anything new, it's worth confirming, precisely, what exists today. Review this table with no earlier module open — it's exactly the state module 7's closing project left:
| Piece | Where it lives | What it does |
|---|---|---|
contract_to_pandera_schema(contract) | Module 4, lesson 5 | Generates a pa.DataFrameSchema from orders_contract.yaml — completeness, uniqueness, validity |
validate_referential_integrity(orders_df, dim_product_df) | Module 3, lesson 4 | Anti-join against dim_product — consistency |
check_price_baseline(df, reference_prices, tolerance) | Module 5, lesson 5 | Compares against REFERENCE_PRICES — accuracy |
check_freshness(df, run_at, sla_hours) | Module 6, lesson 4 | Compares the most recent order_ts against PIPELINE_RUN_AT — freshness |
check_volume(df, min_rows, max_rows) | Module 6, lesson 5 | Confirms df.height falls within range — volume |
build_failure_report(df, ...) | Module 7, lesson 2 | Unites row-level failures into a single table |
quarantine(df, failures) | Module 7, lesson 3 | Separates clean_df from quarantined_df |
raise_alert(check_name, failure_count, sample) | Module 7, lesson 4 | Structures a notification |
ACCESS_POLICY / mask_pii() | Module 7, lessons 5-6 | Column-level governance by role |
generate_catalog(tables) | Module 7, lesson 7 | catalog.yaml for Kiosko's tables |
Ten pieces, each really tested, each in its own closing script. And yet, if someone at Kiosko — not necessarily whoever wrote the code — asked today "is the file that arrived this morning okay?", the honest answer would require opening seven different folders, importing ten functions from files that might not even coexist in the same directory, and assembling the result by hand, every time.
Worked example: the brief, in Kiosko's voice
Here's how someone from Kiosko's data team would request it, in their own words — not a specialist in each of the seven pieces, but the person who has to answer for the complete pipeline's reliability to the rest of the company:
"We have seven different ways to review an orders file, each built and tested separately. What I need isn't an eighth tool — I need a single function I can pass a file to, and it tells me, with no need to remember to run seven scripts in the right order: did it pass or not? If it didn't pass, which of the seven checks failed, and with what evidence? And if it failed, I want the file to neither get lost nor loaded halfway — I want the good rows separated from the bad ones, and an alert that notifies someone, with no need for me to be watching the terminal at the exact moment the file arrives."
Translated into concrete requirements, the brief asks for four things, none of them a new tool:
- A single entry point. A function that takes an orders DataFrame (and whatever reference tables it needs:
dim_product,reference_prices) and returns the result of all seven checks, not one at a time across seven loose calls. - A structured result, not a pile of
print(). Every check needs a name, a status (PASS/FAIL), and enough detail for someone to understand what failed with no need to read each function's source code. - A clear decision about what to do if something fails. Module 4's contract already declared this —
on_violation: quarantine— the complete system needs to honor that declaration, not just report it. - The same confidence in both directions. The system has to demonstrate, with evidence, both that it catches what's broken and that it doesn't bother what's fine — a system that only does the first isn't, yet, a complete trust system.
What THIS module builds, point by point
| Brief requirement | What this module builds | In which lesson |
|---|---|---|
| A single entry point | run_full_gate(df, dim_product_df, reference_prices, schema, run_at=...) | Lesson 3 |
Structured result, no loose print() | List of dict with check/status/detail, one per check | Lesson 3 |
Honoring the contract's on_violation: quarantine | build_failure_report() + quarantine() triggered when the gate reports any FAIL | Lesson 4 |
| Confidence in both directions | Two runs: S04 (6 failures) and a clean day (0 failures) | Lessons 4 and 5 |
| Accessible governance alongside the gate's result | LINEAGE_MAP, generate_catalog(), ACCESS_POLICY, mask_pii() published alongside the system | Lesson 6 |
Diagram: from seven loose pieces to one system
flowchart LR
subgraph Before["Before this module: 7 pieces, not united"]
A1["contract_to_pandera_schema()\nM4"]
A2["validate_referential_integrity()\nM3"]
A3["check_price_baseline()\nM5"]
A4["check_freshness()\nM6"]
A5["check_volume()\nM6"]
A6["quarantine() + raise_alert()\nM7"]
A7["ACCESS_POLICY + mask_pii()\nM7"]
end
subgraph Now["This module: one system"]
G["run_full_gate()\n7 checks, 1 call"]
end
A1 --> G
A2 --> G
A3 --> G
A4 --> G
A5 --> G
G -->|"if any FAIL"| A6
G -.->|"available alongside the result"| A7
Going deeper: why this isn't "just another wrapper"
It's tempting to read run_full_gate() as a cosmetic layer — a function that simply calls the other seven in order, adding no real value. It's worth resisting that reading. What this lesson's brief asks for isn't just "fewer lines of code for the user" — it asks for a new behavior contract none of the seven pieces separately guarantees: that all seven run always together, in the same order, on the same data, and that the result is comparable across different runs. Without run_full_gate(), nothing stops someone from running check_freshness() on one file and check_volume() on another by mistake, or forgetting to run validate_referential_integrity() that week because "there was no time." The integrated system doesn't just save lines — it eliminates an entire category of human error: running the correct checks, but not all of them, or not on the same data.
This same logic is, underneath, the reason any real "quality gate" exists in the industry — not any individual function's technical novelty, but the guarantee that a file never advances without passing through the complete set of checks the team already decided matter.
Common mistakes
Thinking this brief asks for rewriting one of the seven pieces. What happens: someone, reading "I need a single function," starts imagining a rewrite of check_freshness() or validate_referential_integrity() with a different interface. Why it happens: "system" sounds like "new architecture," and it's easy to overestimate how much change is needed. How to spot it: review this lesson's "What THIS module builds" table — no row says "rewrite." How to fix it: the brief asks for composition, not reconstruction — the seven existing functions get imported with no change (with a single naming exception in lesson 3: build_failure_report() is going to receive the schema generated by the contract instead of the hand-written OrdersSchema class, the natural evolution module 4 already promised), and run_full_gate() is, literally, the function that orchestrates them.
Confusing "trust system" with "system that never fails." What happens: someone interprets the module's goal as building something that guarantees no future Kiosko file will ever have a problem. Why it happens: the word "trust," in everyday language, sounds like an absolute guarantee. How to spot it: review what lessons 4 and 5's two runs actually prove — that the system correctly detects what's wrong and doesn't falsely flag what's right, never that no future file can ever have a problem. How to fix it: a trust system doesn't eliminate the risk of bad data — it makes it visible, with evidence, every time it occurs, instead of letting it slip through in silence like the green checkmark that opened this guide in module 1.
Exercises
Exercise 1 — Write, in your own words, Kiosko's brief in a single sentence. Without copying this lesson's text, condense Kiosko's data team's complete request into a single sentence capturing all four parts: an entry point, a structured result, an action on failure, and evidence in both directions.
See solution
A reasonable sentence: "We need a single function that takes any Kiosko orders file, runs the seven checks we already built, clearly reports which passed and which didn't, automatically quarantines what fails, and demonstrates — with at least one broken file and one clean one — that it does both things correctly." There's no single correct wording, but any valid answer should touch all four parts of the brief, not just problem detection.
Exercise 2 — Identify which of the ten pieces in "The material" table does NOT form part of the seven checks run_full_gate() runs, and explain why. Review "The material: Kiosko's real state at module 7's close" section's table and determine which pieces are "checks" (contribute to a PASS/FAIL) and which are "actions" (respond to an already-calculated result).
See solution
build_failure_report(), quarantine(), raise_alert(), ACCESS_POLICY/mask_pii(), and generate_catalog() aren't checks — they're actions depending on the seven checks' result, not checks in themselves. build_failure_report() and quarantine() act after the gate has already determined which rows have problems; raise_alert() communicates that result; ACCESS_POLICY/mask_pii()/generate_catalog() are governance, a layer completely orthogonal to whether a specific file passed the seven checks or not (they apply the same over S04 or over the clean day). The seven real checks — the ones run_full_gate() runs and reports as PASS/FAIL — are: completeness, uniqueness, validity (the contract's three), consistency, accuracy, freshness, and volume.
Exercise 3 — Argue whether run_full_gate() should live in the same file as the seven functions it orchestrates, or in a separate one. In 2-3 sentences, considering the reusability pattern every earlier module of this guide already followed (one function, one purpose, importable from anywhere), give your opinion.
See solution
There's no single correct answer, but a good argument recognizes the tension: keeping run_full_gate() alongside the seven functions it orchestrates (in a single module, as this capstone does) makes it easy to see the whole system at a glance and avoids seven separate imports; splitting it into its own file (say, gate.py importing from contracts.py, consistency.py, accuracy.py, etc.) better reflects the real boundary between "individual business rules" and "the orchestration that unites them" — useful if someone someday wanted to reuse check_price_baseline() with no need to drag along the rest of the system. For Kiosko's size in this guide, a single file (lesson 3's choice) is reasonable; in a real production system with many more checks, splitting by responsibility becomes a more defensible maintainability decision.
Summary and next step
In this lesson you wrote the brief justifying this entire module, with no new line of code: Kiosko already has seven excellent checks, each tested separately, but no way to run them together with a single structured result, no automatic action when something fails, and no evidence yet that the system generates no false alarms. You translated that request into four concrete requirements, and saw, in a table, exactly which lesson of this module solves each one.
Before moving on you should be able to: name the seven pieces run_full_gate() is going to orchestrate, and which module each comes from; and explain the difference between a "check" (PASS/FAIL) and an "action" (quarantine(), raise_alert()) within the complete system.
Lesson 3 builds run_full_gate() for real: the function's signature, the order of the seven checks, and the first run — on a toy DataFrame, before touching S04's real incident — confirming the assembly works.
Resources
- Module 7, project (lesson 8), of this same guide — the exact source of the ten pieces this brief would invent, if they didn't already exist.
src/guides/data-reliability-and-governance-guide/workbook/module-07-the-incident-and-data-governance/en/08-project-s04s-incident-response-and-access-policy.md. In English. lakehouse-and-iceberg-guide, module 8, lesson 2 (that capstone's brief) — the same "translate a business need into concrete requirements before writing code" pattern, applied there to a complete lakehouse.src/guides/lakehouse-and-iceberg-guide/workbook/module-08-project-kioskos-lakehouse/en/02-the-brief-kiosko-needs-a-lakehouse.md. In English.- This guide's DESIGN — this module 8's complete mandate.
src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.