Module 4: Alerting On Error Budget Burn Rate

8. Project: Andes Cargo's alerting policy

Description

Seven lessons built three engines — Python, Prometheus/Alertmanager, CloudWatch — that implement, at different levels of sophistication, the same question: did the error budget's consumption cross the threshold SLO.md allows? This module's final project gathers them into ALERTING-POLICY.md, a portfolio document that doesn't describe the theory — lesson 2 already did that — but proves that all three engines, run side by side over the same two fixed scenarios, reach the same conclusion: a forced drill, with literal results, not an unsupported claim.

Connection to the module

This document directly cites the results already produced in lessons 3, 4, and 5 — it repeats no calculation, it only gathers them and tests them together, with one new data point (the CloudWatch alarm's representative state over both scenarios' complete aggregates, never explicitly calculated until this lesson). Module 5 inherits this document when designing on-call severities; Module 8 cites it when running the final synthetic incident against the same complete machine.

Step 1 — The missing data point: the CloudWatch alarm over the complete aggregates

Lessons 3 and 4 already evaluated the "bad week" and the "normal" scenario with two engines. Lesson 5 built the CloudWatch alarm, but never explicitly evaluated it against these two complete scenarios — only against Module 3's 20-invocation batch. This step closes that gap, applying the alarm's same formula (errors / invocations, threshold 0.001) to the aggregate totals Module 2 already calculated:

Scenario	Valid events	Bad events	`error_ratio`	vs. `0.001` threshold	Alarm state
Bad week (`BAD_WEEK`, M2.7)	2,718	118	118 / 2,718 = 0.04341 (4.341%)	Far above	`ALARM`
Normal (`TRAFFIC_30_DAYS`, M2.4)	12,195	11	11 / 12,195 = 0.00090 (0.090%)	Below	`OK`

The second row is this whole module's most revealing one: 0.090% is below 0.1% — the same tight margin SLO.md already documented (99.9098% SLI, just above the 99.9% SLO). The CloudWatch alarm, evaluated over the "normal" month's complete aggregate, stays OK, but by a margin so narrow it confirms, once more, Module 2's central reading: this system meets its SLO, but with no comfortable margin to spare.

Step 2 — The complete document

At the root of andes-cargo-infra/, create ALERTING-POLICY.md:

# ALERTING-POLICY.md — process-shipment-manifest Burn Rate Alerting

**Status:** Accepted · **Governs:** Modules 5 through 8 of `sre-and-incident-response-guide`
**Source:** Module 4, lessons 2-7 (the Google SRE multi-window pattern, three engines built and run)
**Related:** `SLO.md` (Module 2) defines the SLO this policy alerts against — 99.9% monthly,
0.1% allowed error rate, the exact threshold every engine below evaluates against.

## Why burn rate, not a static threshold

A static threshold on the accumulated monthly SLI would have missed the days 17-18 incident
of Module 2 entirely — the month's SLI (99.9098%) stayed above a 99.9% threshold even with
that incident inside it. Burn rate answers a different question: not "is the SLI below
target right now", but "how fast is the budget being consumed, right now, compared to the
rate the SLO allows" (Module 4, lesson 1-2, citing `sre.google/workbook/alerting-on-slos/`).

## The three engines, and what each one is for

| Engine | Built in | What it is for | Multi-window? |
|---|---|---|---|
| `scripts/burn_rate_evaluator.py` | Lesson 3 | Prototype of the alert decision, zero infrastructure | Yes, all 3 tiers (Table 5-8) |
| Prometheus + Alertmanager | Lesson 4 | Real, portable, multi-provider alerting engine | Yes, via `and ignoring(window)` |
| CloudWatch Alarm (`observability.tf`) | Lesson 5 | Native AWS alarm, no exporter to maintain | No — single window only |

## The two fixed scenarios every engine is tested against

- **`bad_week`** (Module 2, lesson 7): a broken deployment, 7 days, burn rate short window
  17.54x, long window 43.41x.
- **`normal`** (Module 2, lesson 4): the healthy 30-day reference month, burn rate short
  window 0.00x, long window 0.90x.

## Forced drill: results, engine by engine

| Engine | `bad_week` result | `normal` result |
|---|---|---|
| `burn_rate_evaluator.py` (3 tiers: Page fast, Page slow, Ticket) | **All 3 fire** | **None fire** |
| Alertmanager (`ManifestErrorBudgetBurnRate{PageFast,PageSlow,Ticket}`) | **All 3 firing**, confirmed via `/api/v1/rules` and `/api/v2/alerts`, delivered to the webhook receiver | **None appear** in any firing/active list |
| CloudWatch Alarm (`errors / invocations >= 0.001`) | `ALARM` (4.341% >> 0.1%) | `OK` (0.090% < 0.1%, narrow margin) |

Three engines, two scenarios, one conclusion every time: `bad_week` crosses every threshold
this policy defines; `normal` crosses none. No engine disagrees with any other.

## Routing

`aws_cloudwatch_metric_alarm.manifest_error_budget_burn_rate` publishes to
`aws_sns_topic.reliability_alerts` (Lesson 5), subscribed by `email` (real, $0) and, as the
documented pattern for a real team, an HTTPS endpoint matching PagerDuty's Amazon CloudWatch
integration format (Lesson 7) — named, not built, SaaS with no $0 tier.

## What this policy does not do

It does not replace human judgment on severity response (Module 5 builds that framework);
it does not implement the two-window confirmation natively in CloudWatch (Lesson 5's honest
limitation, closed only by the managed alternative named in Lesson 6, CloudWatch Application
Signals, native since November 2024); it does not cover any SLI other than
`process-shipment-manifest`'s error rate — no latency-based burn rate exists in this project.

## Consequences

Module 5 designs the on-call rotation and severity matrix against the exact three tiers this
policy defines (Page fast, Page slow, Ticket). Module 6 measures the real Claude Code incident
against this same policy, retroactively. Module 8's capstone introduces a new synthetic
incident and expects this exact machine — unchanged — to fire correctly on it.

Step 3 — Verifying the document

wc -l ALERTING-POLICY.md
grep -c '^## ' ALERTING-POLICY.md
grep -c 'fire\|ALARM\|OK' ALERTING-POLICY.md

What to expect (literal — you wrote the content, the shape is deterministic):

62
7
3

Sixty-two lines, seven sections (Why burn rate, The three engines, The two fixed scenarios, Forced drill, Routing, What this policy does not do, Consequences), and three lines explicitly mentioning a firing state (fire, ALARM, or OK) — condensed evidence this document isn't a promise in prose, but the record of a drill that really ran, three times, with the same result.

Reading the complete drill: why the agreement among the three engines matters

The three engines reaching the same conclusion isn't a coincidence — it's proof that burn rate math is the same math, no matter what runs it. burn_rate_evaluator.py calculates it with lists of tuples in memory; Alertmanager calculates it with PromQL over scraped series; CloudWatch calculates it with metric math over native Lambda metrics. None of the three "knows" the other two's result — each one independently reaches FIRES/firing/ALARM for bad_week and does not fire/(nothing)/OK for normal, because all three implement, with different syntax, the same formula: observed error rate divided by allowed error rate, compared against a threshold.

This is exactly the kind of evidence a postmortem or an architecture review would ask for: not "we believe the alert works," but "we tested it with a scenario we knew should fire, and one we knew shouldn't, across the three systems that implement it, and all three agreed." This guide's Module 8 is going to repeat exactly this exercise — a new synthetic incident, never seen before — to confirm this agreement wasn't an accident of the chosen data, but a real property of the complete machine.

Common mistakes

Treating the normal row of the CloudWatch table as "a comfortable margin" just because the state is OK (reading the state without reading the margin). What happens: someone sees OK next to normal and concludes the system is far from firing a real alarm. How to spot it: if your reading of "0.090% < 0.1%, OK" doesn't mention how close that margin is. How to fix it: 0.090% against a 0.1% threshold is a margin of barely 0.01 percentage points — the same tight reading SLO.md already made about this same month (4.23 of 43.2 remaining minutes, 90.2% of budget already consumed). OK is the correct state, but "OK with a wide margin" and "OK one step from ALARM" are very different readings, and this document exists, in part, to keep them from being confused.

Assuming the agreement among the three engines guarantees they'll never diverge in the future (over-generalizing from two fixed scenarios). What happens: someone concludes, from this drill, that Python/Alertmanager/CloudWatch are always going to agree, no matter what data they receive. How to spot it: if your conclusion from this lesson is "these three systems are interchangeable in any situation." How to fix it: all three engines implement the same formula, but with real coverage differences — CloudWatch, in this guide's form, has no two-window confirmation; Alertmanager does — that this very document already declares in its "What this policy does not do" section. Agreeing on these two specific scenarios confirms the math is well implemented in all three; it doesn't guarantee a different scenario, with a different failure pattern, wouldn't reveal a real coverage difference between them.

Writing ALERTING-POLICY.md without having really run lessons 3, 4, and 5, trusting the numbers "are probably correct" (repeating the same mistake already named in SLO.md, Module 2). What happens: someone copies this document without having run burn_rate_evaluator.py or brought up Prometheus/Alertmanager on their own. How to spot it: if you can't reproduce, on your own machine, all three Alertmanager alerts firing on bad_week. How to fix it: every figure in this document comes from an engine you yourself ran in the previous lessons — before accepting ALERTING-POLICY.md as final, confirm your own copy of each engine produces, with no differences, the same results cited here.

Exercises

Exercise 1 — Verify by hand Step 1's table CloudWatch row for the bad_week scenario. BAD_WEEK has 2,718 valid events and 2,600 good events (Module 2, lesson 7). Calculate error_ratio and confirm it exceeds 0.001.

See solution

Bad events = 2,718 − 2,600 = 118. error_ratio = 118 / 2,718 ≈ 0.043414 (4.3414%), far above the 0.001 threshold (0.1%) — confirming the table's ALARM row. This figure, moreover, matches the same order of magnitude as the long-window burn rate already calculated in lesson 3 of this module (43.41x): a burn rate of 43.41x means, by definition, the observed error rate is 43.41 times the allowed rate (0.1%) — 0.1% × 43.41 ≈ 4.341%, exactly the error_ratio this exercise just calculated by a different path. Two formulas, the same number, one more confirmation the math is consistent across this module's three engines.

Exercise 2 — A colleague proposes removing the CloudWatch engine from this policy, arguing Alertmanager already covers the complete two-window pattern and is "strictly better." Using this document's "What this policy does not do" section, what argument would you use to keep all three engines?

See solution

CloudWatch has a real limitation (a single window) compared to Alertmanager, but it has an advantage Alertmanager doesn't: it's native to AWS, with no Python exporter to maintain, no Docker container to operate, and it directly evaluates the metrics the Lambda itself already publishes automatically. "Strictly better" ignores that both engines have different operational costs: Alertmanager depends on someone maintaining docker-compose.yml, the exporter, and that whole stack's availability; CloudWatch depends only on the Lambda existing, with no additional piece that could fail on its own. A complete argument would also cite that both engines already demonstrated, in this drill, reaching the same conclusion over the same data — having both isn't useless redundancy, it's the same "don't depend on a single engine" logic lesson 1 of this module already established, now with evidence that both, in fact, work.

Exercise 3 — Explain why this document cites SLO.md in its header (Related) instead of repeating the SLO's definition inside ALERTING-POLICY.md. What document-design principle, already used in this ecosystem, is this?

See solution

It's the same single-source-of-truth principle SLO.md already established for itself in Module 2: every portfolio document in this guide defines one thing with authority, and any later document that needs it cites it, never copies or redefines it on its own. ALERTING-POLICY.md needs the SLO (99.9% monthly, 0.1% allowed error rate) to justify each of its thresholds, but repeating that definition here would create two sources of truth — if SLO.md ever changed, ALERTING-POLICY.md would go stale with nobody noticing. Citing the source, instead, guarantees a future change to the SLO automatically reflects in this document's reasoning, with no duplication and no risk of inconsistency between the two.

Summary and next step

This module's final project wrote ALERTING-POLICY.md: the document that gathers the three burn rate engines built in lessons 3, 4, and 5, tested with a forced drill over this module's two fixed scenarios. The result, matching across all three: bad_week fires all three severities in Python, all three alerts in Alertmanager (confirmed end to end, all the way to the receiver), and turns ALARM in CloudWatch (4.341% error rate, far above the 0.1% allowed); normal fires nothing in any of the three, with CloudWatch staying OK by a margin of barely 0.01 percentage points — the same tight reading SLO.md already documented since Module 2.

Before closing this module you should be able to: explain why the agreement among the three engines is real evidence, not coincidence; calculate CloudWatch's error_ratio by hand for either scenario; and name the real limitation each engine has against the other two.

With this, Module 4 of sre-and-incident-response-guide is complete: Module 2's burn rate math, turned into three real alert engines, tested with evidence, not promises. Module 5 builds the generic incident-response framework — roles, severities, on-call — that's going to operate exactly the alerts this module left working, before applying it to Module 6's real incident.

Resources

This module, lessons 2 through 7 — the direct source for every figure and every decision in this document.
This same repository, Module 2, lesson 8 (08-project-andes-cargos-slo-md.md) — SLO.md, the document ALERTING-POLICY.md cites instead of repeating.
This same repository, Module 3, lesson 8 (08-project-andes-cargos-observability-to-sli-pipeline.md) — OBSERVABILITY-RUNBOOK.md, the same portfolio document format ALERTING-POLICY.md follows.
Google SRE Workbook — Alerting on SLOs — the source for the math this policy's three engines implement.

← Previous7. Hands-on: routing the alert Next →1. Introduction: the framework before the case