Module 4: Alerting On Error Budget Burn Rate

1. Introduction: why a static threshold isn't enough

Description

Module 2 left SLO.md with an uncomfortable figure: 4.23 of 43.2 minutes of budget remaining, a month that was one more bad day away from exhausting its entire error budget. Module 3 connected that formula to real data — CloudWatch, Prometheus, logs, traces — but neither module yet built the piece that turns "the budget is at risk" into "someone finds out, now, that it's at risk." That's this module's job: alerting that fires for the right reason — accelerated error-budget consumption, not an arbitrary threshold — with the logic really executed and an honest contrast against the managed product AWS already sells to solve this exact same problem.

Connection to the module

This module invents no new data. It reuses, unchanged, Module 2 lesson 7's "bad week" dataset (BAD_WEEK, the week of a broken deployment that consumed 43.4 times the monthly budget) and Module 2 lesson 4's healthy scenario (TRAFFIC_30_DAYS, the reference month with 90.2% of budget consumed but no catastrophic day). This module's eight lessons take those two already-familiar scenarios and ask them the same question, with three different engines: should this fire an alert, or not?

This module's analogy: the fuel tank, two ways to watch it

Picture two ways of watching an Andes Cargo truck's fuel tank. The first is a light that turns on when the tank reaches, say, a quarter full — a static threshold: a fixed number, with no memory of how fast it got there. The second is a gauge that doesn't just look at how much is left, but how fast it's draining right now — if the tank drops from half to a quarter in five minutes, something's wrong (a leak), even if the absolute level still hasn't crossed any fixed threshold; if it drops at the same slow pace as always, a quarter tank is simply "time to refuel soon," not an emergency.

The first light — the static threshold — has two flaws any real driver would recognize immediately. It can turn on too late: if the tank has a fast leak, by the time the "quarter full" light turns on, there are minutes left, not hours, before getting stranded. And it can turn on too often with no real need: a truck running long routes crosses the "quarter full" threshold on every trip, every day, without that ever being an emergency — the light becomes noise the driver learns to ignore. Burn rate — this module's central topic — is the second way of watching: not how much is left, but how fast it's being spent, compared against the rate the trip plan allows.

The real problem with a static threshold, with numbers from this same project

A static alerting threshold, applied to process-shipment-manifest's error budget, would look roughly like this: "alert if the SLI drops below 99.9% at any point." Sounds reasonable. It isn't, for two reasons this project's own dataset already demonstrates.

Too late. Day 17 of Module 2's dataset (TRAFFIC_30_DAYS) had a daily burn rate of 13.27x — one step away from Google SRE's most urgent threshold (14.4x). A static threshold evaluated over the full month's cumulative SLI wouldn't even have noticed: the whole month's SLI, including that day, ended at 99.9098% — above the 99.9% the threshold would be watching. A single, nearly catastrophic day, invisible to a threshold that only looks at the cumulative average.

Too often, or never often enough, depending on the window. If the threshold were evaluated over a short window — say, every hour — a brief, unimportant spike (a customer uploading a malformed manifest by mistake, fixed the next minute) would fire an alert nobody needs to handle at 3 AM. If it were evaluated over a long window — the full month, like the example above — a real, sharp incident, like days 17-18, gets diluted into the average and never crosses the threshold. There's no single window that solves both problems at once — and that's exactly why the pattern lesson 2 of this module formalizes exists.

   A STATIC THRESHOLD OVER THE CUMULATIVE SLI — THE TWO FAILURES

   LONG window (e.g. the full month)
   ────────────────────────────────────────────────────
   Day 17: burn rate 13.27x   ──┐
   Day 18: burn rate  6.85x   ──┼── diluted into the
   Rest of the month: near 0x ──┘   monthly average -> SLI = 99.91%
                                     -> the static threshold NEVER crosses

   SHORT window (e.g. every hour)
   ────────────────────────────────────────────────────
   A brief spike, no real importance
   -> crosses the threshold -> alert -> nobody needs it at 3 AM
   -> the team learns to ignore it ("alert fatigue")

This module's map: the 8 lessons

   MODULE 4 — ALERTING ON ERROR BUDGET BURN RATE
   from "a threshold isn't enough" to three real engines that fire for the right reason

   M4.1  Introduction (this lesson)                  the static-threshold problem
   M4.2  Multi-window, multi-burn-rate                 the real Google SRE pattern, cited
   M4.3  The burn rate evaluator                        EXECUTED: burn_rate_evaluator.py
   M4.4  The real rule in Alertmanager                    EXECUTED: Prometheus + Alertmanager
   M4.5  A real CloudWatch alarm                            EXECUTED (validate) + REPRESENTATIVE
   M4.6  What AWS already automates                          REPRESENTATIVE, with exact date
   M4.7  Routing the alert                                     EXECUTED (validate) + REPRESENTATIVE
   M4.8  Project: Andes Cargo's alerting policy                  EXECUTED: forced drill

#	Lesson	What it builds
1	Introduction (this one)	Why a static threshold fires too late or too often
2	Multi-window, multi-rate burn rate	The exact pattern from `sre.google/workbook/alerting-on-slos/`, cited
3	Hands-on: the burn rate evaluator	Executed: `scripts/burn_rate_evaluator.py`, literal output
4	Hands-on: the real rule in Alertmanager	Executed: Prometheus + Alertmanager, really running
5	Hands-on: a real CloudWatch alarm	Executed (`validate`/`plan`) + representative (`apply`/`awslocal`)
6	What AWS already automates	Representative, with the exact date verified
7	Hands-on: routing the alert	Executed (`validate`) + representative (PagerDuty/Opsgenie, named)
8	Project: Andes Cargo's alerting policy	Executed: forced drill, bad week fires, normal doesn't

This module's thread: two fixed scenarios, three engines

Unlike Module 2 — where each lesson ran over a slightly different dataset — and Module 3 — where a single batch was read with three instruments — this module has an even more precise structure: two already-familiar scenarios, evaluated by three different engines, each built on the previous one.

"Normal" scenario — Module 2 lesson 4's 30-day month (TRAFFIC_30_DAYS): healthy overall, with a tight margin but no incident serious enough to fire a real alert.
"Bad week" scenario — Module 2 lesson 7's 7-day week (BAD_WEEK): a broken deployment that consumed 43.4 times the entire monthly budget in just seven days.

This module's three engines, in the order they're built:

Pure Python (lesson 3) — scripts/burn_rate_evaluator.py, the alert decision's prototype, with no infrastructure behind it.
Prometheus + Alertmanager (lesson 4) — the same decision, now really evaluated against real scraped metrics, with an alert that really gets routed to a receiver.
CloudWatch Alarm (lesson 5) — the same decision, this time expressed as infrastructure declared in Terraform, over Andes Cargo's real Lambda.

All three engines, over the same two scenarios, should reach the same conclusion: "bad week" fires, "normal" doesn't. Lesson 8 of this module demonstrates it with a forced drill, running all three engines side by side.

This module's honesty: what really runs, what stays representative

Tool	This module's status	Exact technical reason
`scripts/burn_rate_evaluator.py` (lesson 3)	Real	Pure Python, no external dependencies — ran with `python3` to write this lesson, literal output.
Prometheus `v3.13.2` + Alertmanager `0.33.1` (lesson 4)	Real	`docker compose up` really brought up, with a real alert rule evaluated against a real Python exporter, routed to a real webhook receiver.
`terraform validate` / `terraform plan` (lessons 5, 7)	Real	Really run against `observability.tf`'s HCL — literal output.
`terraform apply` / `awslocal cloudwatch` (lesson 5)	Representative	With no `LOCALSTACK_AUTH_TOKEN` exported in this writing environment, the LocalStack container doesn't start — the same circumstantial limit across this whole ecosystem since `finops-and-cost-guardrails-guide`. CloudWatch is confirmed on the Hobby plan; the representative output is reconstructed field by field from the real HCL, never invented.
CloudWatch Application Signals SLO (lesson 6)	Representative, named, not run	Depends on APM-type infrastructure with no confirmed coverage on any LocalStack tier — named by contrast, with its verified launch date.
PagerDuty / Opsgenie (lesson 7)	Representative, named, not run	Paid SaaS, with no $0 tier equivalent to the rest of this lab.

The rule governing this whole guide holds with no exception: if a command appears in a lesson, it ran to write it. What didn't really run gets labeled at the exact moment it appears, with the precise technical reason.

Common mistakes

Thinking "burn rate" is just a fancier name for "percentage of budget consumed" (confusing two different questions, already resolved in Module 2 but easy to forget here). What happens: someone, starting this module, assumes burn rate is simply the consumed_pct Module 2's calculator already prints. How to spot it: if you can't explain why a month with 90.2% of cumulative budget consumed (Module 2, lesson 4) had, on a single day, a burn rate of 13.27x. How to fix it: consumed_pct is a cumulative figure over the whole window; burn rate is an instantaneous rate, measured over a window much shorter than the SLO's window — the exact distinction Module 2, lesson 6 already established, and that this module turns into a real alert.

Expecting this module to build a single, "definitive" alert (underestimating why there are three engines). What happens: someone reaches lesson 4 expecting that, once the Alertmanager rule is built, lessons 5 and 7 are redundant. How to spot it: if your question after finishing lesson 4 is "why do I need CloudWatch if I already have Prometheus?" How to fix it: a real production system almost never depends on a single alerting engine — lesson 5 exists precisely because process-shipment-manifest runs on AWS, and a native CloudWatch alarm is a second line of defense, not a useless redundancy, with its own strengths and limits (lesson 6 names exactly which) against Prometheus/Alertmanager.

Assuming "representative" in lesson 5 means the Terraform resource isn't real (confusing the infrastructure layer with the layer that runs against LocalStack). What happens: someone reads "representative" next to awslocal cloudwatch describe-alarms and concludes the whole HCL block in that lesson is hypothetical. How to spot it: if you treat observability.tf's HCL as pseudocode instead of the real file terraform validate already validated against the AWS provider's real schema. How to fix it: terraform validate and terraform plan really ran, against the real HCL, with the real provider version (hashicorp/aws ~> 6.0) — what's representative is only the next step, apply against a LocalStack that doesn't start in this specific environment for lack of a token, exactly the same limit already documented since finops-and-cost-guardrails-guide.

Exercises

Exercise 1 — Explain, without using the word "burn rate," why a static threshold evaluated over the cumulative monthly SLI wouldn't have caught Module 2's days 17-18 incident. Use the exact numbers: full-month SLI 99.9098%, hypothetical threshold 99.9%.

See solution

The full month's SLI (99.9098%) lands, by a small margin, above the hypothetical 99.9% threshold — a static threshold evaluated over the month's cumulative figure never gets crossed, so it never fires any alert, no matter how bad a specific day within that month was. The monthly average absorbs and dilutes a sharp two-day event (days 17-18) among 28 mostly healthy days — the same problem this lesson's "too late, or never" section describes: a window long enough to calculate a stable average is, almost by definition, too long to detect a short event within it.

Exercise 2 — Predict, without running anything yet, which of this module's engines (Python, Prometheus/Alertmanager, or CloudWatch) would be the simplest to run on an engineer's laptop with no AWS account and no Docker container running. Justify your answer with what you already know about each tool.

See solution

scripts/burn_rate_evaluator.py (lesson 3) — pure Python, with no external dependency, no Docker, no AWS account, no service running in the background. Prometheus/Alertmanager (lesson 4) already needs Docker running two containers; CloudWatch (lesson 5) needs, at minimum, LocalStack running (or a real AWS account). This progression — from the simplest possible logic toward real infrastructure — is intentional: lesson 3 exists so you understand the alert decision with no operational complexity around it, before seeing that same decision implemented inside two real production systems.

Exercise 3 — A colleague proposes solving the static-threshold problem by simply evaluating it more often (every minute instead of every hour). Explain, using this lesson's fuel-tank analogy, why that doesn't solve the underlying problem.

See solution

Evaluating a static threshold more often changes when it's evaluated, but not what it evaluates — it's still an absolute level (is the SLI below X right now?), with no notion of speed at all. In the tank analogy: checking the fuel level every minute instead of every hour doesn't tell the driver whether the tank is draining fast (a leak) or slow (normal use) — it only tells them the current level, more often. The underlying problem — that a level threshold doesn't distinguish a leak from normal use at the same level — stays exactly the same, no matter how often you look. What does solve the problem is measuring the rate of change (how much the level dropped in the last five minutes, compared against how much it should drop under normal use) — exactly the definition of burn rate lesson 2 of this module formalizes.

Summary and next step

This lesson installed the central problem that governs the seven lessons that follow: a static threshold over the error budget fires either too late (diluted in a long window) or too often with no real need (noisy in a short window), and there's no single window that solves both problems at once. You saw the map of the 8 lessons, the thread connecting them — two already-familiar fixed scenarios (TRAFFIC_30_DAYS, BAD_WEEK), evaluated by three engines built in progression (Python, Prometheus/Alertmanager, CloudWatch) — and this module's complete honesty table.

Before moving on you should be able to: explain, with Module 2's day 17's exact numbers, why a static threshold over the cumulative SLI wouldn't have caught it; name this module's three engines in the order they're built; and explain the difference between evaluating a threshold more often and measuring a consumption rate.

Lesson 2 formalizes the solution: Google SRE's exact multi-window, multi-burn-rate pattern, cited directly from the source, with Module 2 lesson 6's already-familiar result as the first worked example.

Resources

This same repository, Module 2, lessons 4, 6, and 7 — the two fixed datasets (TRAFFIC_30_DAYS, BAD_WEEK) and the first burn rate implementation this module extends.
Google SRE Workbook — Alerting on SLOs — the exact source for the multi-window, multi-burn-rate pattern lesson 2 of this module cites in full.
finops-and-cost-guardrails-guide, Module 5 — the direct precedent for a real CloudWatch alarm in this ecosystem, and the same honesty pattern (real validate/plan, representative apply) lesson 5 of this module follows.

← Previous8. Project: Andes Cargo's observability-to-SLI pipeline Next →2. Multi-window, multi-burn-rate: the real Google SRE pattern