Module introduction: the incident and data governance

Why this module exists

Module 6's closing project ended with a line that didn't exist in any earlier module of this guide: DIMENSIONS_STILL_OPEN: []. Empty. The six data quality dimensions module 1 defined — completeness, uniqueness, validity, consistency, accuracy, freshness — each have their own runnable check, really run on orders_2026-08-14.csv. On top of that, S04 has a volume check (PASS) and a complete lineage map of fifteen columns of Kiosko's warehouse. Six modules of work, and the detection system is, finally, complete.

And yet, if you asked today "what does Kiosko do with orders_2026-08-14.csv's six broken rows?", the honest answer would be: nothing yet. Every module in this guide built a tool that identifies a problem — ORD-9503 has an empty unit_price, ORD-9508 references a product that doesn't exist, the whole file arrived 47.58 hours late — but none of those tools moves a single row, notifies a single person, or says what to do a minute after printing its report. Detecting a fire isn't the same as having an evacuation plan. This module closes that final gap: it turns six modules of diagnosis into a system that acts when something fails, and that also decides, explicitly, who can see what once the data is already inside.

Two new questions, related but different, organize this module. The first: what happens, in practice, when a check fails in production? You already know two bad answers, both seen in this guide with real evidence: data-engineering-foundations-guide (module 7) showed what happens when the answer is "reject everything" — ValueError: unknown store_id: S04, status='failed', rows_loaded=0, even the file's first perfectly valid row got lost —; this same guide's module 1 showed what happens when the answer is "say nothing" — ORD-9508 and ORD-9509 passing the old gate with no flag at all, the green checkmark that opened this entire guide. This module builds the third answer, the one no earlier module had the chance to build: quarantine — separating good from bad, with no loss of either part — plus a structured alert and a written runbook, so "someone finds out" stops depending on someone, by chance, rerunning the diagnosis script.

The second question is entirely different: once Kiosko's data is inside, correct or not, who can see what? None of the six earlier modules needed to ask that question, because none worked with data identifying a real person. This module introduces this entire guide's first table with contact information — customers, Kiosko's delivery app's directory — and with it, the governance problem: which column an analyst can see, which column finance can see, which column support needs to do its job, and how Kiosko can keep analyzing its customers' behavior without anyone with access to a table being able to read, in plain text, someone's email or phone number.

Connection to the previous module. Module 6 said it in its final line: "What's still missing isn't more detection — it's decision: what to do, in practice, when any of these checks fails in production." This module fulfills that complete promise, with the same six broken rows and the same late file the six earlier modules already diagnosed — no new orders_s04 data, just the missing piece: what to do with what's already known.

An analogy ahead of time: the hospital, not the diagnosis

You're going to build each piece's complete analogy in its own lesson, but it's worth previewing the idea that organizes the whole module. A doctor who only diagnoses — "you have this condition" — and then does nothing else, isn't yet a hospital. A hospital has an emergency room that sorts cases by severity instead of treating people randomly (triage), a system that notifies the right staff when something goes out of normal range (an alarm, not silence), a written protocol for what to do in each type of emergency (so the response doesn't depend on the right person being on shift that day), and a record of who can see each patient's medical file (not everyone in the building has access to anyone else's complete history). This guide's six earlier modules built the diagnosis. This module builds the complete hospital around that diagnosis: quarantine is the emergency room, the alert is the alarm, the runbook is the written protocol, and data governance is the record of who can see which file.

The map of this guide's 8 modules (reminder)

#	Module	What it's about
1	When green doesn't mean correct	The green checkmark lie; the six dimensions; diagnosing S04 without fixing anything.
2	Declarative data quality tests with Pandera	OrdersSchema, catching completeness/uniqueness/validity — three of six dimensions.
3	Consistency and referential checks	Referential integrity between tables; an anti-join that catches S04's orphan product_id.
4	Data contracts as versioned artifacts	What a data contract is; orders_contract.yaml; the contract generates module 2's tests.
5	Accuracy and deterministic anomaly detection	A price baseline; the dollars-to-cents bug, finally caught.
6	Freshness, volume, and lineage	Freshness and volume as file-level properties; lineage mapped by hand.
7	The incident and data governance (you are here)	Quarantine, alert, runbook; role-based access; deterministic PII masking; a minimal catalog.
8	Project: Kiosko's trust system	The capstone, run against S04 and against a clean day.

The map of this module

Lesson    Question it answers
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Why "detecting" and "acting" are different problems,
          and why data governance shares a module with the incident.
L2        What happens when a check fails in production -- the three
          possible answers, with evidence of the two bad ones already
          seen in this guide.
L3        quarantine(df, failures) -- built, run on S04:
          6 clean rows, 6 in quarantine.
L4        raise_alert() structured; runbook.md with S04's incident's
          six steps, written end to end.
L5        The customers table (new); ACCESS_POLICY -- who sees which
          column, by role.
L6        mask_pii() with hashlib.sha256 + MASK_SALT -- deterministic,
          verified twice.
L7        generate_catalog() -- catalog.yaml for Kiosko's four
          tables.
L8        Project: S04's incident and governance, end to end,
          in a single script.

Lessons 2 through 4 build the answer to the first question — what to do when something fails: first the conceptual framework with the three possible answers (lesson 2), then the real mechanism that separates good rows from bad ones (lesson 3), and finally communicating that incident to the people who need to act (lesson 4). Lessons 5 through 7 change topic — from "what to do about an incident?" to "who can see what?" — building data governance end to end: per-column access (lesson 5), deterministic PII masking (lesson 6), and a catalog documenting everything above in a single place (lesson 7). Lesson 8, the project, brings both halves of the module together into a single incident-response script.

Going deeper: why the incident and governance share a module

It's worth asking, before moving on, why "what to do when something fails" and "who can see what" — two questions that at first glance belong to different projects — close this same guide together. The answer has to do with what, at bottom, each of this guide's six already-built tools is. validate_orders(), OrdersSchema, validate_referential_integrity(), the contract, check_price_baseline(), check_freshness(), and check_volume() all answer the same underlying question: is this data correct? None of them, until now, had to answer a completely different question: who's allowed to see it? — a row can be perfectly clean, pass all six dimensions with no problem, and still be information not everyone at Kiosko should be able to read in plain text.

This module's two questions — what to do with what's incorrect, and who can see what's correct — are, really, the two halves of the same broader discipline that gives this guide its name: data reliability and governance. A data system you can trust isn't just one that detects its own errors — Kiosko already has that since module 6 —; it's one that also acts on those errors predictably, and that protects the sensitive information that same system processes, with neither depending on a specific person remembering to do it right every time. That's why the module closing this guide's technical apparatus needs to cover both pieces — not because they share an implementation (quarantine() and mask_pii() share no code at all), but because together they complete the promise this guide's title made from its first line.

The boundary: what does NOT enter this module

One concrete boundary, worth naming from the start because it's going to reappear in each of the governance lessons. This module teaches data governance: which column of which table which business role can see, with a Python dictionary (ACCESS_POLICY) any script can read and apply. This module does not teach infrastructure security: there are no AWS accounts, no real IAM roles, no KMS keys, no VPC to isolate, no CloudTrail to audit access. aws-core-services-guide and cloud-security-and-guardrails-guide cover that complete layer — the one that decides who can connect to a system and with what credentials. This module assumes the connection already exists, and answers a question that comes after: once inside, with a legitimate connection, which columns should a query return depending on the role of whoever runs it? They're complementary layers, not a substitute for one another — a real production system needs both, and this guide teaches only one of the two, the one that deals directly with the data's shape.

Neither does a real integration with an alerting system enter this module — Slack, PagerDuty, an automated email. raise_alert(), in lesson 4, builds and returns an alert's structure: a dictionary with all the information a notification system would need to act. It never makes a real HTTP call to any external service. That real integration is, exactly like the rest of this guide, an orchestrator's territory (airflow-and-declarative-orchestration-guide, which can schedule retries, on_failure_callback, and managed notifications) — it gets named in module 8's close, without being implemented here.

Common mistakes

Thinking this module "fixes" S04's six broken rows. What happens: someone, after seeing quarantine() work, expects the quarantined rows to end up, somehow, corrected — ORD-9503's empty unit_price filled in, ORD-9509's price divided by a hundred. Why it happens: "quarantine," in everyday language, sounds like an intermediate step toward a cure. How to spot it: review exactly what quarantine() does in lesson 3 — it separates rows into two groups, it never modifies any column's value. How to fix it: quarantine is, quite deliberately, a containment mechanism, not a repair one — the same principle module 1 already established about validate_orders() ("a diagnosis describes, it never corrects"), applied here to a system that does act, but that acts by moving rows to a safe place, not by inventing new values for them.

Confusing ACCESS_POLICY with an authentication system. What happens: someone expects this module's functions to verify passwords, tokens, or user sessions, as if apply_access_policy() were a login system. Why it happens: the word "access" in ACCESS_POLICY sounds, at first glance, like complete access control. How to spot it: check what the function receives as a parameter — an already-loaded DataFrame and a role's name as a text string ("analyst", "finance", "support"), never a password or a token. How to fix it: ACCESS_POLICY answers a single, very specific question: given that someone has already authenticated as a known role (a real identity system would resolve that, outside this guide's scope), which columns should they see? It's the column-level authorization layer, never authentication.

Exercises

Exercise 1 — Recall, from memory, the exact state module 6 left S04 in. Without going back to that lesson, try to remember: how many of the six data quality dimensions already have a runnable check? How many of the file's twelve physical rows have some known problem? What's check_freshness()'s and check_volume()'s status?

See solution

All six dimensions have a runnable check (DIMENSIONS_STILL_OPEN: [], the line that closed module 6's project). Six of the file's twelve physical rows have at least one known problem: ORD-9502 (two appearances, uniqueness), ORD-9503 (completeness), ORD-9507 (validity), ORD-9508 (consistency), ORD-9509 (accuracy). check_freshness() reported FAIL (47.58 hours over a 24-hour SLA); check_volume() reported PASS (12 rows within the [5, 20] range). This module doesn't recalculate any of these numbers — it takes them exactly as module 6 left them, and builds what's missing: what to do with them.

Exercise 2 — Name the two answers already seen in this guide for "what to do when a check fails," and why neither is enough. From memory, with no looking back: what did data-engineering-foundations-guide (module 7) do when S04 still didn't exist in DIM_STORE? What did this guide's module 1's green-checkmark example do with customers not in its catalog?

See solution

Foundations M7 rejected the entire file: transform_fact_orders() raised ValueError: unknown store_id: S04, and the run ended with status='failed', rows_loaded=0 — even the file's first perfectly valid row got lost, because the policy was "all or nothing." The green-checkmark example (module 1, lesson 2) did exactly the opposite: extract_and_load() silently filtered out rows with an unknown customer_id, with no record of how many got lost or which ones — the process ended in SUCCESS, hiding the problem instead of solving it. Neither is enough because both lose information: the first loses good data along with the bad; the second loses visibility into the problem itself. This module builds the third path: separate with no loss of either part, and make what got separated visible.

Exercise 3 — Predict, before lesson 5, why none of the six already-built data quality tools needed to ask "who can see this?". In 1-2 sentences, without having read lesson 5 yet, try to anticipate what's different about the customers table you're going to meet in that lesson, compared to orders_s04, dim_product, or dim_store.

See solution

There's no single "correct" answer before reading lesson 5, but a reasonable answer points to this: no column in orders_s04, dim_product, or dim_store identifies a real person — order_id, product_name, city describe transactions and catalogs, not individuals. customers, by contrast, is going to have columns that do identify a specific person (an email, a phone number), and that kind of data — personally identifying information, PII — brings with it a question no purely transactional data needs to ask: not just whether the value is correct, but who should be allowed to read it.

Summary and next step

In this lesson you learned module 7's complete map: why "detecting" a problem and "acting" on it are different capabilities this guide deliberately split into two stages, and why data governance — who can see what — shares a module with the incident, even though it shares not one line of code. You saw the complete map of the seven lessons that follow, and you drew, from the start, the boundary with real infrastructure security (aws-core-services-guide / cloud-security-and-guardrails-guide) and with real alert integration (airflow-and-declarative-orchestration-guide).

Before moving on you should be able to: recall the exact state module 6 left S04 in (six dimensions covered, six broken rows, freshness at FAIL); name the two insufficient answers this guide already showed for "what to do when something fails"; and anticipate, in your own words, why the customers table you're going to meet in lesson 5 needs a question no earlier table needed.

Lesson 2 starts the real work: it builds, with executed evidence, the complete argument for why "quarantine + alert" is the third answer this guide was missing, before writing a single line of quarantine().

Resources

• Module 6, project (lesson 8), of this same guide — the source of the exact state this module picks S04 back up in (DIMENSIONS_STILL_OPEN: [], the six broken rows, freshness: FAIL). src/guides/data-reliability-and-governance-guide/workbook/module-06-freshness-volume-and-lineage/en/08-project-s04s-freshness-volume-and-lineage-report.md. In English.
• Module 1, lesson 5, of this same guide ("Meet S04") — the literal source of S04's rejection in foundations M7 (ValueError: unknown store_id: S04), cited in this lesson's Exercise 2. src/guides/data-reliability-and-governance-guide/workbook/module-01-when-green-does-not-mean-correct/en/05-meet-s04-kioskos-fourth-store.md. In English.
• AWS — "AWS Certified Data Engineer - Associate (DEA-C01)" exam guide (Domain 4, "Data Security and Governance," 18% of the exam — the figure that anchors this module's boundary with infrastructure security). docs.aws.amazon.com/aws-certification. In English.
• This guide's DESIGN — this module 7's exact mandate. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.