Writing an alert and a runbook

Description

quarantine(), in lesson 3, separated S04's broken rows into a safe place. But a quarantined_df nobody reviews is, in practice, almost as invisible as the green checkmark that opened this guide — the difference is the problem is now stored somewhere, instead of lost, but it still reaches nobody. This lesson closes that cycle with two pieces: raise_alert(), a function that structures a complete notification about the incident — never, ever calling any real messaging system —, and a runbook.md written end to end, with the six steps anyone at Kiosko should be able to follow when that alert goes off, needing neither memory nor prior experience with this specific incident.

Connection to the module. Lessons 2 and 3 built S04's incident's detection and containment. This lesson builds communication and response: the part that turns "the system already knows something's wrong" into "a specific person already knows something's wrong, and knows exactly what to do."

An analogy: the fire alarm, not just the smoke detector

A smoke detector that activates, but isn't connected to any audible alarm, does only half its job: it knows there's smoke, but nobody else knows until someone happens to walk by and notices the detector blinking. A complete fire alarm has three parts, not one: the sensor that detects the problem (already built: build_failure_report(), check_freshness()), the siren that makes anyone in the building find out immediately, with no need to actively check the sensor (raise_alert(), this lesson), and the evacuation protocol posted on the wall — which door to use, where to gather, who counts people — that nobody has to invent in a moment of panic, because it's already written down beforehand (runbook.md, this same lesson). All three parts are necessary: a sensor with no siren is useless to anyone not looking at it; a siren with no protocol generates panic with no direction.

Worked example: raise_alert(), structured and deterministic

raise_alert() sends nothing anywhere — it builds and returns a Python dictionary with all the information a real notification system (Slack, PagerDuty, an automated email) would need to act. This guide names that real integration, but never implements it — the same pattern OpenLineage and Marquez already used in module 6.

# alert_and_runbook.py -- module 7, lesson 4
import json

import duckdb
import polars as pl
from datetime import datetime

PIPELINE_RUN_AT = "2026-08-16T09:00:00"


def check_freshness(df: pl.DataFrame, run_at: str, sla_hours: int, timestamp_col: str = "order_ts") -> dict:
    """Module 6, no changes."""
    latest_ts = df.select(pl.col(timestamp_col).max()).item()
    run_at_dt = datetime.fromisoformat(run_at)
    hours_since_latest = (run_at_dt - latest_ts).total_seconds() / 3600
    return {
        "check": "freshness",
        "latest_row_ts": str(latest_ts),
        "run_at": run_at,
        "sla_hours": sla_hours,
        "hours_since_latest": round(hours_since_latest, 2),
        "status": "PASS" if hours_since_latest <= sla_hours else "FAIL",
    }


def raise_alert(check_name: str, failure_count: int, sample: list[dict]) -> dict:
    """Structures an alert. NEVER calls a real system (Slack/PagerDuty/email) -- only the structure."""
    return {
        "alert": "data_quality_incident",
        "pipeline": "kiosko_orders_s04",
        "check_name": check_name,
        "run_at": PIPELINE_RUN_AT,
        "severity": "high" if failure_count >= 5 else "medium",
        "failure_count": failure_count,
        "sample": sample[:3],
    }


def main() -> None:
    con = duckdb.connect("kiosko.duckdb")
    df = con.sql("SELECT * FROM orders_s04").pl()

    # --- Alert 1: the row-level incident, summarized from lessons 2-3 ---
    # (failures and quarantined_df were already calculated in earlier lessons;
    #  here we only pick back up the summary raise_alert() needs)
    quarantine_sample = [
        {"order_id": "ORD-9502", "dimension": "uniqueness", "detail": "order_id=ORD-9502"},
        {"order_id": "ORD-9503", "dimension": "completeness", "detail": "unit_price=None"},
        {"order_id": "ORD-9507", "dimension": "validity", "detail": "quantity=-1"},
        {"order_id": "ORD-9508", "dimension": "consistency", "detail": "product_id=P099 does not exist in dim_product"},
        {"order_id": "ORD-9509", "dimension": "accuracy", "detail": "unit_price=60.0 is 49.0x away from the reference price (1.2)"},
        {"order_id": "ORD-9502", "dimension": "uniqueness", "detail": "order_id=ORD-9502"},
    ]
    row_alert = raise_alert(check_name="s04_full_gate", failure_count=6, sample=quarantine_sample)

    print("=== Alert 1: row-level incident (quarantine, lessons 2-3) ===")
    print(json.dumps(row_alert, indent=2, ensure_ascii=False))

    # --- Alert 2: the file-level incident (freshness, module 6) ---
    freshness_result = check_freshness(df, run_at=PIPELINE_RUN_AT, sla_hours=24)
    file_alert = raise_alert(check_name="freshness", failure_count=1, sample=[freshness_result])

    print("\n=== Alert 2: file-level incident (freshness, module 6) ===")
    print(json.dumps(file_alert, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

What to expect (verified by actually running python3 alert_and_runbook.py, with kiosko.duckdb containing orders_s04, polars==1.43.2):

=== Alert 1: row-level incident (quarantine, lessons 2-3) ===
{
  "alert": "data_quality_incident",
  "pipeline": "kiosko_orders_s04",
  "check_name": "s04_full_gate",
  "run_at": "2026-08-16T09:00:00",
  "severity": "high",
  "failure_count": 6,
  "sample": [
    {
      "order_id": "ORD-9502",
      "dimension": "uniqueness",
      "detail": "order_id=ORD-9502"
    },
    {
      "order_id": "ORD-9503",
      "dimension": "completeness",
      "detail": "unit_price=None"
    },
    {
      "order_id": "ORD-9507",
      "dimension": "validity",
      "detail": "quantity=-1"
    }
  ]
}

=== Alert 2: file-level incident (freshness, module 6) ===
{
  "alert": "data_quality_incident",
  "pipeline": "kiosko_orders_s04",
  "check_name": "freshness",
  "run_at": "2026-08-16T09:00:00",
  "severity": "medium",
  "failure_count": 1,
  "sample": [
    {
      "check": "freshness",
      "latest_row_ts": "2026-08-14 09:25:00",
      "run_at": "2026-08-16T09:00:00",
      "sla_hours": 24,
      "hours_since_latest": 47.58,
      "status": "FAIL"
    }
  ]
}

Read these two alerts carefully, because they confirm something important about raise_alert()'s design: it's a generic function, capable of structuring both a row-level incident (six broken rows, check_name="s04_full_gate") and a whole-file incident (check_name="freshness", a single element in sample, check_freshness()'s complete result). Notice severity: the first alert is "high" because failure_count=6 exceeds the 5 threshold; the second is "medium" because failure_count=1 doesn't exceed it — even though, in terms of real business impact, a whole file 47.58 hours late could well be considered just as serious as six broken rows. This is a deliberate simplification in this guide, and this lesson's Going deeper section discusses it in depth. sample[:3] caps the sample size at a maximum of three elements — in the first alert, of six total failures only the first three get included —, a common decision in real alerting systems: the alert needs to give enough context to act, without turning into a complete data dump nobody's going to read in full.

The runbook: what a human does when the alarm goes off

A runbook isn't general system documentation — this guide's DESIGN and each module's README already have that. It's an operational document, written to be read during an incident, by someone who may never have seen this specific problem before. Kiosko uses a six-step structure, a variation of the incident lifecycle both PagerDuty and Google's SRE book document (cited at the end of this lesson), applied here, step by step, to orders_2026-08-14.csv's exact incident.

# Runbook: data quality incident in `orders_s04`

## 1. Detect
`build_failure_report()`, run on `2026-08-16T09:00:00` (`PIPELINE_RUN_AT`) over
`orders_2026-08-14.csv`, reported 6 physical rows with at least one known problem.
`check_freshness()` reported `FAIL` on the whole file (`47.58` hours over a
`24`-hour SLA). `raise_alert()` structured both findings as two independent
alerts: `check_name="s04_full_gate"` (severity=high) and
`check_name="freshness"` (severity=medium).

## 2. Triage
Classify scope and impact before acting. Dataset: `orders_s04`. Pipeline:
`kiosko_orders_s04`. Does it block anything downstream? Not yet -- `fact_orders`
from `data-modeling-for-analytics-guide` hasn't been recalculated with these
rows yet; the incident is contained before touching the ecosystem's shared
warehouse. Priority: review the high-severity rows first (completeness,
uniqueness, accuracy -- see lesson 2's Exercise 2), then the medium-severity
ones (validity, consistency).

## 3. Contain
`quarantine(df, failures)` already separated the 6 clean rows from the 6 broken
ones. The clean ones (`ORD-9501`, `ORD-9504`, `ORD-9505`, `ORD-9506`,
`ORD-9510`, `ORD-9511`) can continue the normal pipeline without waiting for
the incident's resolution. The 6 broken ones stay in `quarantined_df`, outside
any business table, with no whole-file block -- the exact lesson foundations
M7 left with its total rejection.

## 4. Root cause
Per row, the specific cause behind each broken dimension:
- `ORD-9503` (completeness): empty `unit_price` -- probable capture error at
  S04's point of sale, its first day operating with Kiosko's real system.
- `ORD-9502` x2 (uniqueness): duplicate retransmission -- a network retry from
  the delivery app resent the same sale, seven minutes later.
- `ORD-9507` (validity): `quantity=-1` -- probable return logged as a negative
  sale, instead of through a dedicated returns process.
- `ORD-9508` (consistency): `product_id=P099` doesn't exist -- S04's local
  catalog still isn't synced with the central warehouse's `dim_product`.
- `ORD-9509` (accuracy): `unit_price=60.00` vs. reference `1.20` -- the
  dollars-to-cents bug: S04's system probably records the price in a different
  unit (cents) than the one Kiosko's contract expects (dollars).

## 5. Fix
Short term: the rows stay in quarantine. No value gets invented or corrected
by hand -- the same principle module 1 already established about
`validate_orders()` ("a diagnosis describes, it never corrects"), extended
here to a system that also acts, but acts by moving rows, not by guessing
values. Medium term (outside this guide's technical scope, named in module 8's
close): S04 needs (a) deduplication at the source before resending a sale,
(b) module 4's data contract applied at the point of origin, not only on
arrival at Kiosko, and (c) an explicit currency unit conversion in S04's
capture system, before the file goes out to Kiosko.

## 6. Postmortem
Blameless -- the standard documented by Google's SRE book (cited below): the
goal isn't pointing at S04 as "the store that sent bad data," it's identifying
the systemic cause. And the systemic cause, in this case, is clear: module 4's
data contract got written *after* S04 started selling, not before -- S04
never had the chance to validate its first file against a contract, because
that contract didn't exist yet when it sent it. Follow-up action: every new
data producer at Kiosko should have a contract and an active quality gate
*before* its first real file, not after -- exactly the question module 4,
lesson 7 already previewed ("should S04 be allowed to write without a
contract?"), now answered with this incident's complete evidence.

What to expect. This runbook.md doesn't get "run" — it gets read, during a real incident. Its verification isn't a terminal output, it's that every step answers, with evidence this module already produced, the question it's responsible for: Detect cites build_failure_report()'s and check_freshness()'s exact numbers; Triage classifies with no invented new information; Contain cites quarantine()'s literal result; Root cause names the five specific causes, one per dimension; Fix precisely distinguishes what's this guide's responsibility from what's left to the ecosystem; Postmortem closes with a concrete follow-up action, not a generic "we need to improve" line.

Diagram: from detection to complete response

flowchart TD
    A["build_failure_report()\n+ check_freshness()"] --> B["raise_alert()\nx2: row + file"]
    B --> C["runbook.md"]
    C --> D["1. Detect"]
    D --> E["2. Triage"]
    E --> F["3. Contain\n(quarantine, already done)"]
    F --> G["4. Root cause"]
    G --> H["5. Fix\n(short and medium term)"]
    H --> I["6. Postmortem\nblameless"]
    I --> J["Follow-up action:\ncontract BEFORE the\nfirst file, not after"]

Going deeper: why raise_alert()'s severity is a conscious simplification

It's worth being honest about a limitation in raise_alert()'s design in this lesson: severity gets calculated with a single rule — "high" if failure_count >= 5, "medium" in any other case —, with no distinction of what kind of failure occurred or how much business impact it has. This lesson's Alert 2 is a good example of that rule's limit: a whole file arriving almost two days late (47.58 hours over a 24-hour SLA) gets severity="medium", exactly the same label a single row with a minor problem would get, only because failure_count=1 in both cases.

A real production alerting system almost never uses a single number as its severity criterion — it typically combines the number of affected rows, the percentage they represent of the total, whether the problem touches a financial column (like unit_price, in ORD-9509's case) or a purely descriptive one, and whether the whole file, not just one row, is at risk of not reaching the warehouse on time. This guide deliberately chooses the simplest version, for the same reason module 5 already explained about check_price_baseline(): every rule in this guide has to be explainable in a single sentence, without turning into a complex scoring system nobody can audit at a glance. A more sophisticated severity system — one that did treat the freshness case as critical — is a reasonable extension, and this lesson's Exercise 2 builds an improved version.

Common mistakes

Expecting raise_alert() to make a real network call, or trying to "complete" it with a Slack integration. What happens: someone, after seeing this lesson's structured JSON, looks for adding requests.post(SLACK_WEBHOOK_URL, json=alert) so the alert really reaches somewhere. Why it happens: an alert that doesn't really notify anyone feels incomplete. How to spot it: check whether your version of raise_alert() imports requests, slack_sdk, or any networking library. How to fix it: this guide names the real integration as a possibility, never builds it — the exact same pattern OpenLineage/Marquez (module 6) and Great Expectations/Soda (module 2) already practiced: naming with precision, without installing. Connecting raise_alert() to a real notification system — Slack, PagerDuty, a transactional email — is a legitimate extension for your own project, but it falls outside this guide's scope on purpose, to keep it at $0 and with no third-party credentials.

Thinking a generic runbook.md — with no exact incident numbers — is enough. What happens: someone writes a runbook with abstract steps ("Detect: check the pipeline logs," "Contain: isolate the problematic data"), with no concrete figure or order_id at all. Why it happens: a "reusable" runbook for any future incident feels more efficient than one written for a specific case. How to spot it: if your runbook could apply, with no word changed, to a completely different incident at another Kiosko store, it's probably too generic to be useful in a real emergency's moment. How to fix it: this lesson's runbook cites exact figures (47.58 hours, 6 rows, ORD-9502 specifically) because a living runbook, written during or immediately after a real incident, serves as a concrete reference for the next similar incident — the general template (the six headers: Detect, Triage, Contain, Root cause, Fix, Postmortem) is indeed reusable; each section's content is not.

Exercises

Exercise 1 — Run alert_and_runbook.py yourself, from scratch. In a new folder, with kiosko.duckdb containing orders_s04 (module 2), run python3 alert_and_runbook.py. Confirm you see exactly the two alerts, with severity="high" for the first and severity="medium" for the second.

Exercise 2 — Improve raise_alert() so Alert 2 (freshness) gets severity="high". Based on this lesson's Going deeper section, modify the severity logic so a "freshness"-type check is always "high", no matter the failure_count — arguing that a whole file being late is, almost always, more serious than a few broken rows.

def raise_alert(check_name: str, failure_count: int, sample: list[dict]) -> dict:
    if check_name == "freshness":
        severity = "high"
    else:
        severity = "high" if failure_count >= 5 else "medium"
    return {
        "alert": "data_quality_incident",
        "pipeline": "kiosko_orders_s04",
        "check_name": check_name,
        "run_at": PIPELINE_RUN_AT,
        "severity": severity,
        "failure_count": failure_count,
        "sample": sample[:3],
    }

file_alert = raise_alert(check_name="freshness", failure_count=1, sample=[freshness_result])
print(file_alert["severity"])

high

Exercise 3 — Argue whether the runbook's Postmortem step should be written immediately after Contain, or only after the incident is fully resolved. This lesson's runbook lists Postmortem as the sixth and last step. In 2-3 sentences, considering Google's SRE book's definition of "blameless postmortem" (cited below), argue when that step should actually be written.

Summary and next step

In this lesson you closed module 7's first half: raise_alert(), a generic function capable of structuring both a row-level incident and a whole-file one, really run on S04's two real alerts — never calling an external system, naming that real integration as a legitimate extension outside this guide's scope. And you wrote a complete runbook.md, with all six steps — Detect, Triage, Contain, Root cause, Fix, Postmortem — applied, word for word, with S04's real incident's exact figures, closing with a blameless postmortem that identifies the systemic cause, not a culprit.

Before moving on you should be able to: explain why raise_alert() never makes a real network call; recite the runbook's six steps from memory, with at least one concrete S04 data point per step; and argue when the Postmortem step should be written relative to the rest of the incident.

With this, Kiosko now knows what to do when a check fails in production. This module's second half remains, a completely different question: once the data is inside, who can see it? Lesson 5 introduces customers, this guide's first table with information identifying a real person, and ACCESS_POLICY, the dictionary that decides which column each Kiosko role sees.

Resources

• PagerDuty — "Incident Response Documentation" (PagerDuty's public guide on an incident's lifecycle: before, during, and after — the basis for this runbook's six-step structure, adapted to this guide's data quality vocabulary). response.pagerduty.com. In English.
• Google — "Site Reliability Engineering," chapter 15, "Postmortem Culture: Learning from Failure" (the exact blameless postmortem definition this runbook's step 6 cites: "a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions"). sre.google/sre-book/postmortem-culture. In English.
• Python — official documentation, json module (json.dumps, used to print this lesson's alerts in a readable, deterministic way). docs.python.org/3/library/json.html. In English.
• Module 4, lesson 7, of this same guide ("Should S04 be allowed to write without a contract?") — the source of the question this lesson's Postmortem step answers with complete evidence. src/guides/data-reliability-and-governance-guide/workbook/module-04-data-contracts-as-versioned-artifacts/en/07-should-s04-be-allowed-to-write-without-a-contract.md. In English.
• This guide's DESIGN. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.