Module 8: Capstone The Andes Cargo Reliability Package

4. End-to-end: the alert fires, the incident gets declared

Description

This is the lesson where the complete machine really gets put to the test. Lesson 3's two numbers — 400.00x burn rate in the short window, 25.00x in the long one, on the synthetic_incident scenario — feed into Module 4's three engines without any of the three changing a single line of their own logic. If all three agree this fires, and none gets it wrong on normal (which still doesn't fire), the machine generalizes. With that confirmation, this lesson formally declares the incident, with severity classified by the real matrix and roles assigned from a rotation week no earlier module in this guide used yet.

Connection to the module

This lesson extends observability/burn_rate_exporter.py (Module 4, lesson 4) with a third scenario value, without touching observability/alert_rules.yml or observability.tf — this lesson's central proof is, precisely, that neither one needs to change. The incident declaration follows the exact pattern Module 6, lesson 4 already established for the Claude Code incident, applied here, for the first time, to a case where SEV1 severity is justified by the matrix's burn rate criterion — not the independent data-loss criterion the Claude Code incident needed, because there no burn rate was measurable.

Step 1 — Extending the exporter with a third scenario, without touching anything else

# burn_rate_exporter.py (Module 4, lesson 4) -- extended with a third scenario.
# ONLY new lines below: two new .labels(...).set(...) calls, copy-pasted literally
# from Module 8, lesson 3's own computed output. No change to the Gauge
# definition, no change to the HTTP server, no change to any other line.

import time
from prometheus_client import Gauge, start_http_server

manifest_burn_rate = Gauge(
    "manifest_burn_rate",
    "Burn rate of process-shipment-manifest's error budget, per scenario and window",
    ["scenario", "window"],
)

# From scripts/burn_rate_evaluator.py (M4.3), literal output:
manifest_burn_rate.labels(scenario="bad_week", window="short").set(17.54)
manifest_burn_rate.labels(scenario="bad_week", window="long").set(43.41)
manifest_burn_rate.labels(scenario="normal", window="short").set(0.00)
manifest_burn_rate.labels(scenario="normal", window="long").set(0.90)

# NEW in this lesson -- from Module 8, lesson 3's own computed output, Step 4:
manifest_burn_rate.labels(scenario="synthetic_incident", window="short").set(400.00)
manifest_burn_rate.labels(scenario="synthetic_incident", window="long").set(25.00)

if __name__ == "__main__":
    start_http_server(8001)
    print("burn_rate_exporter listening on :8001/metrics")
    print("bad_week: short=17.54x long=43.41x | normal: short=0.00x long=0.90x")
    print("synthetic_incident: short=400.00x long=25.00x")
    while True:
        time.sleep(3600)

Two new lines, nothing else. observability/alert_rules.yml doesn't appear in this step because it doesn't change — its three rules already select by window, never by a fixed scenario value, so any new scenario label the exporter exposes automatically feeds into the same calculation, with nobody needing to anticipate it.

# Stop the previous exporter (Ctrl+C in its terminal) and start the extended version
python3 observability/burn_rate_exporter.py &

What to expect (literal):

burn_rate_exporter listening on :8001/metrics
bad_week: short=17.54x long=43.41x | normal: short=0.00x long=0.90x
synthetic_incident: short=400.00x long=25.00x

Prometheus, with scrape_interval: 15s already configured since Module 4, lesson 4, picks up the new scenario on its next scraping cycle — with no container restarted, no prometheus.yml edited.

Step 2 — Confirming the scraping, with all six series now visible

curl -s 'http://localhost:9090/api/v1/query?query=manifest_burn_rate'

What to expect (literal — six series now, two for each of the three scenarios; the Unix timestamp in each value is your variable value, the labels and numbers are literal):

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {"metric": {"scenario": "bad_week", "window": "short"}, "value": [1789084804.113, "17.54"]},
      {"metric": {"scenario": "bad_week", "window": "long"},  "value": [1789084804.113, "43.41"]},
      {"metric": {"scenario": "normal",   "window": "short"}, "value": [1789084804.113, "0"]},
      {"metric": {"scenario": "normal",   "window": "long"},  "value": [1789084804.113, "0.9"]},
      {"metric": {"scenario": "synthetic_incident", "window": "short"}, "value": [1789084804.113, "400"]},
      {"metric": {"scenario": "synthetic_incident", "window": "long"},  "value": [1789084804.113, "25"]}
    ]
  }
}

Step 3 — The three engines, run again, with no code change

Engine 1 — scripts/burn_rate_evaluator.py (Module 4, lesson 3), with a third call added to its __main__ block:

# Added at the end of the if __name__ == "__main__": block from Module 4, lesson 3.
# No change to evaluate(), evaluate_scenario(), or TIERS.
evaluate_scenario("synthetic incident (M8.3)", short_burn_rate=400.00, long_burn_rate=25.00)

python3 scripts/burn_rate_evaluator.py

What to expect (literal — the first two sections, unchanged, you already saw in Module 4; the third is new):

--- bad week (M2.7) (short=17.54x, long=43.41x) ---
  Page (fast)  >= 14.4x (1h/5m): FIRES
  Page (slow)  >=  6.0x (6h/30m): FIRES
  Ticket       >=  1.0x (3d/6h): FIRES

--- normal (M2.4) (short=0.00x, long=0.90x) ---
  Page (fast)  >= 14.4x (1h/5m): does not fire
  Page (slow)  >=  6.0x (6h/30m): does not fire
  Ticket       >=  1.0x (3d/6h): does not fire

--- synthetic incident (M8.3) (short=400.00x, long=25.00x) ---
  Page (fast)  >= 14.4x (1h/5m): FIRES
  Page (slow)  >=  6.0x (6h/30m): FIRES
  Ticket       >=  1.0x (3d/6h): FIRES

Engine 2 — Prometheus + Alertmanager, with alert_rules.yml unmodified, not a single line:

curl -s http://localhost:9090/api/v1/rules

What to expect (literal, after the first evaluation cycle following the new scraping — ~1 minute):

ManifestErrorBudgetBurnRatePageFast  state=firing  alerts=[('bad_week', 'firing'), ('synthetic_incident', 'firing')]
ManifestErrorBudgetBurnRatePageSlow  state=firing  alerts=[('bad_week', 'firing'), ('synthetic_incident', 'firing')]
ManifestErrorBudgetBurnRateTicket    state=firing  alerts=[('bad_week', 'firing'), ('synthetic_incident', 'firing')]

curl -s http://localhost:9093/api/v2/alerts

What to expect (literal — six active alerts, two scenarios, three severities each):

ManifestErrorBudgetBurnRatePageFast bad_week            page   active
ManifestErrorBudgetBurnRatePageFast synthetic_incident  page   active
ManifestErrorBudgetBurnRatePageSlow bad_week            page   active
ManifestErrorBudgetBurnRatePageSlow synthetic_incident  page   active
ManifestErrorBudgetBurnRateTicket   bad_week            ticket active
ManifestErrorBudgetBurnRateTicket   synthetic_incident  ticket active

cat observability/webhook.log

What to expect (literal — the receiver's complete log, now with synthetic_incident present):

alert_webhook_receiver listening on :9099/alerts
[alert_webhook_receiver] status=firing   alertname=ManifestErrorBudgetBurnRatePageFast scenario=bad_week severity=page
[alert_webhook_receiver] status=firing   alertname=ManifestErrorBudgetBurnRateTicket scenario=bad_week severity=ticket
[alert_webhook_receiver] status=firing   alertname=ManifestErrorBudgetBurnRatePageSlow scenario=bad_week severity=page
[alert_webhook_receiver] status=firing   alertname=ManifestErrorBudgetBurnRatePageFast scenario=synthetic_incident severity=page
[alert_webhook_receiver] status=firing   alertname=ManifestErrorBudgetBurnRateTicket scenario=synthetic_incident severity=ticket
[alert_webhook_receiver] status=firing   alertname=ManifestErrorBudgetBurnRatePageSlow scenario=synthetic_incident severity=page

At no point in this step did scenario=normal appear — not in /api/v1/rules, not in /api/v2/alerts, not in the receiver's log. The and ignoring(window) rule, written in Module 4 before synthetic_incident existed, evaluates it correctly anyway.

Engine 3 — CloudWatch Alarm, with no change to observability.tf:

awslocal cloudwatch describe-alarms \
  --alarm-names andes-cargo-manifest-error-budget-burn-rate \
  --query 'MetricAlarms[0].{State:StateValue,Reason:StateReason}'

What to expect (representative — same reason declared since Module 3: with no LOCALSTACK_AUTH_TOKEN, the LocalStack container doesn't start in this authoring environment; reconstructed with lesson 3, Step 4's error_ratio = 0.025, against observability.tf's real threshold = 0.001):

{
    "State": "ALARM",
    "Reason": "Threshold Crossed: 1 datapoint [0.025 (17/03/26 15:00:00)] was greater than or equal to the threshold (0.001)."
}

   THREE ENGINES, THE SAME NEW SCENARIO, THE SAME VERDICT

   Python (M4.3, extended)        Alertmanager (M4.4,           CloudWatch (M4.5,
                                    unchanged)                    unchanged)
   ──────────────────────         ───────────────────           ─────────────────
   synthetic_incident               synthetic_incident             error_ratio 2.5%
   -> FIRES x3                      -> firing x3, delivered         -> ALARM
                                     to the webhook

   normal -> still doesn't fire in ANY of the three, exactly as in M4.4/M4.5

Step 4 — Classifying the severity: the other path to SEV1

INCIDENT-RESPONSE-PLAN.md defines SEV1 with two criteria, joined by "OR": Page (fast), ≥ 14.4x — OR irreversible data loss, regardless of measured burn rate. The Claude Code incident (Module 6) was SEV1 by the second criterion — there was no measurable burn rate, because the entire infrastructure that would have generated it no longer existed. This synthetic incident is this whole guide's first case that's SEV1 by the first criterion, the one the matrix measures directly:

burn_rate (short window) = 400.00x  >=  14.4x (Page fast threshold)  ->  SEV1

Unambiguously, with no need for the exception criterion — the matrix classifies this case with its main mechanism, the one ALERTING-POLICY.md built since Module 4 to be the normal path toward a severity, not the exception.

Step 5 — The roles: a different week of the rotation, with no calendar discrepancy

Module 6 had to resolve a real discrepancy: the Claude Code incident happened on February 26, 2026, six days before INCIDENT-RESPONSE-PLAN.md's formal rotation started (March 2). This synthetic incident doesn't have that problem — its date (March 17, 2026, a Tuesday) genuinely falls within Week 3 of the rotation oncall/schedule.py already generated:

Week  Starts      Primary   Secondary
3     2026-03-16  Carla     Diego          <- week used for this incident

With the same INCIDENT-RESPONSE-PLAN.md hard rule for a SEV1/SEV2 (IC and OL are never the same person) and the same two-person pattern Module 6 already applied:

Role	Person	Why
Incident Commander (IC)	Carla	Primary on-call this week; coordinates, doesn't execute technical changes
Communications Lead (CL)	Carla (dual role with IC)	Both are coordination functions, not modifying the system
Operations Lead (OL)	Diego	Secondary on-call this week; the only one executing mitigation

Step 6 — The declaration message

# INCIDENT DECLARED — SEV1

**Time:** T+~3min (relative to the alert crossing `Page (fast)` threshold — see Step 3 above)
**Declared by:** Carla (Incident Commander)

## Status

`process-shipment-manifest`'s error ratio crossed 40% in a short burst (10 of 25 recent
invocations failing with `missing required field: carrier`), and 2.5% sustained over the full
hour. All three alerting engines (Python evaluator, Alertmanager, CloudWatch) confirm: burn
rate 400.00x (short window) / 25.00x (long window), both above every tier in
`ALERTING-POLICY.md`'s Table 5-8.

## Severity

**SEV1** — burn rate 400.00x >= 14.4x, `INCIDENT-RESPONSE-PLAN.md`'s primary severity criterion
(not the independent data-loss exception the Claude Code incident required — this is the first
incident in this guide classified through the matrix's main mechanism).

## Roles

- **Incident Commander:** Carla — coordinates the response, owns this document, does not run
  mitigation commands directly.
- **Operations Lead:** Diego — the only person taking mitigation actions during this incident.
- **Communications Lead:** Carla (dual role with IC, per `INCIDENT-RESPONSE-PLAN.md`'s
  two-person guidance for a SEV1/SEV2).

## What we know right now (first known state)

- The failure pattern is concentrated, not scattered: all 10 failed invocations share the exact
  same validation error (`missing required field: carrier`), starting abruptly at a specific
  point in the batch — consistent with a recent change upstream, not isolated bad data.
- The alarm and all three alerting engines agree; no engine shows a conflicting result.
- Next update: within 15 minutes, or as soon as Module 8, lesson 5's runbook execution
  identifies the specific branch of the decision tree that applies.

## What this declaration does not claim

Root cause is not analyzed here — Module 8, lesson 5 runs `runbooks/manifest-processor-error-
rate.md`'s decision tree against this exact data to classify the branch, then mitigates and
closes with a second postmortem. This declaration exists only to make the incident official,
assign roles, and record the first known state.

Common mistakes

Assuming that, because the three engines were already tested once in Module 4, there's no need to reconfirm they fire on the new scenario (taking generalization for granted without verifying it). What happens: someone, reaching this lesson, assumes synthetic_incident is going to fire "because the thresholds are the same as always," without actually running Step 3's commands. How to spot it: if your evidence for this lesson is a prose statement instead of the literal output of /api/v1/rules, /api/v2/alerts, and describe-alarms. How to fix it: this module's value is precisely in the verification — actually running the commands, on data no engine saw before, and confirming the result matches what's expected; a prediction with nothing run isn't the same evidence as real output, even if the result turns out the same.

Classifying this incident as SEV1 "by the same criterion as the Claude Code incident" (losing Step 4's distinction between the two paths to SEV1). What happens: someone, writing the declaration's "Severity" section, cites "irreversible data loss" as the reason, copying Module 6's declaration pattern without checking which criterion applies here. How to spot it: if your declaration for this incident mentions data loss anywhere. How to fix it: this incident lost no data — ten manifests were rejected by validation, no infrastructure was destroyed. SEV1 severity here comes, unambiguously, from the matrix's first criterion (Page (fast), ≥14.4x), measured directly. Confusing the two paths to SEV1 would be losing exactly the distinction this lesson's Step 4 exists to point out.

Merging Incident Commander with Operations Lead again, "because that's what was done in Module 6 and it worked" (treating a two-person exception as if it were the general rule). What happens: someone, assigning roles for this incident, mechanically copies Module 6's Ana/Bruno pattern without re-checking the hard rule against INCIDENT-RESPONSE-PLAN.md. How to spot it: if your role assignment doesn't distinguish between "combining IC and CL" (allowed) and "combining IC and OL" (prohibited in SEV1/SEV2). How to fix it: Step 5's two-person pattern is still valid here, but for the exact same reason as in Module 6 — both combined roles (IC+CL) are coordination, never execution — not because "it already worked before." With a four-person team, any week of the rotation has exactly this same constraint: two people on call, three roles, IC and OL always different.

Exercises

Exercise 1 — A classmate proposes adding the synthetic_incident scenario directly to alert_rules.yml, with a fourth rule dedicated only to that scenario, "to make it more explicit." Why does that proposal contradict this lesson's central point?

See solution

This lesson's central point is that the three already-existing rules — written before synthetic_incident existed — correctly fire on that scenario with no change at all, because they select by the window label, never by a fixed scenario value. Adding a fourth dedicated rule would break exactly that property: it would turn a system that automatically generalizes to any new scenario into one that needs a new rule every time a different case shows up — the same maintenance problem a static threshold (Module 4, lesson 1) has against burn rate, now applied to the rules' own structure instead of the threshold.

Exercise 2 — Calculate, without looking at Step 6, whether Carla and Diego could have been assigned exactly backward (Diego as IC/CL, Carla as OL), using only Step 5's rotation table information.

See solution

Technically yes, it would be possible without breaking the hard rule — the only thing INCIDENT-RESPONSE-PLAN.md demands is that IC and OL be different people, not that the primary on-call (Primary) necessarily be the IC. But the pattern this lesson follows, same as Module 6 with Ana/Bruno, assigns the higher-coordination role (IC) to the primary (Primary, Carla) and the execution role (OL) to the secondary (Secondary, Diego) — a reasonable convention, not a mandatory rule: the primary on-call is, by definition of the on-call role, who first sees the alert and who has the freshest context to coordinate, while the secondary is available to execute without having been the first point of contact. Any real team could document the opposite convention if it served them better, as long as the hard rule (IC ≠ OL) holds.

Exercise 3 — Explain why Step 6's declaration message, in its "What this declaration does not claim" section, refers to lesson 5 instead of trying to classify the runbook's decision-tree Branch in this same lesson.

See solution

Classifying the runbook's decision tree's Branch (A, B, or C) requires the complete diagnosis from runbooks/manifest-processor-error-rate.md's Steps 3 and 4 — grouping the failure reasons with jq and confirming whether one dominates or they're scattered — work this lesson hasn't done yet. Declaring a Branch without having run that diagnosis would be exactly the error Module 7, lesson 6 already warned about: "following Branch A [...] without first confirming [...] that the reasons really cluster into a single dominant pattern." This lesson's declaration, quite intentionally, limits itself to what's known in the first moment — the burn rate, the severity, the roles — and explicitly points the cause diagnosis to the lesson where it's actually done, with evidence.

Summary and next step

This lesson ran the complete alerting machine against lesson 3's synthetic incident, without changing any already-built logic: you extended burn_rate_exporter.py with two new lines, and the three engines — the Python evaluator, Prometheus/Alertmanager, and the CloudWatch alarm — correctly fired on synthetic_incident, with no false positive on normal. You classified the severity as SEV1 by the matrix's primary criterion (burn rate ≥14.4x) — the first time in this guide that path, not the data-loss exception, decides the severity — and formally declared the incident, with Carla as Incident Commander/Communications Lead and Diego as Operations Lead, Week 3 of the real rotation, with no calendar discrepancy to resolve.

Before moving on you should be able to: explain why extending the exporter required no change to alert_rules.yml or observability.tf; distinguish the severity matrix's two paths to SEV1; and defend why this lesson's declaration doesn't yet include any runbook Branch classification.

Lesson 5 runs the real runbook, step by step, against this incident: it diagnoses which decision-tree Branch applies, mitigates, verifies the alarm returns to OK, and closes with a second blameless postmortem — shorter, but with the same complete discipline.

Resources

This same repository, Module 4, lessons 3, 4, and 5 — the three engines this lesson runs, unmodified, against a new scenario.
This same repository, Module 5, lesson 5 (05-hands-on-andes-cargos-severity-matrix.md) — the severity matrix and its two criteria for SEV1, applied here for the first time with the primary criterion.
This same repository, Module 5, lesson 7 and Module 5, lesson 8 — oncall/schedule.py and INCIDENT-RESPONSE-PLAN.md, the source of Week 3 and the two-person rule.
This same repository, Module 6, lesson 4 (04-hands-on-declaring-the-incident.md) — the exact declaration message pattern this lesson's Step 6 reuses.
Google SRE Workbook — Alerting on SLOs — the source of Table 5-8, governing every threshold in this lesson.

← Previous3. Hands-on: introducing a deterministic synthetic incident Next →5. End-to-end: the runbook mitigates, the postmortem closes