Module 4: Alerting On Error Budget Burn Rate
2. Multi-window, multi-burn-rate: the real Google SRE pattern
Description
Lesson 1 left the problem unresolved: a static threshold fires too late over a long window, or too noisily over a short one. This lesson presents the real solution, exactly as published at sre.google/workbook/alerting-on-slos/ — not a simplification or an approximation from this guide: the multi-window, multi-burn-rate pattern, which combines a long window (confirms the consumption is real, not a five-minute coincidence) with a short window (confirms it's still active now, not already resolved), evaluated together, never separately.
Connection to the module
This lesson introduces no new data — you already know the exact result it produces, because Module 2, lesson 6 already ran it, with a single row of the complete table (Ticket, adapted to daily granularity). What this lesson adds is the complete pattern, with all three severities, cited directly from the source, and the exact reason — not just the result — why two windows avoid both false positives and false negatives. Lesson 3 implements this complete pattern in Python, with all three severities at once.
The analogy: confirming a leak with two gauges, one fast and one slow
Pick lesson 1's fuel tank back up, with one more detail: imagine the truck has two level gauges, not one. The slow gauge (the long window) averages consumption over the last several hours — it's stable, it doesn't react to a pothole or a sharp turn, but for that same reason it takes a while to notice something changed. The fast gauge (the short window) reacts in seconds — it notices a sudden drop almost immediately, but that same sensitivity also makes it react to unimportant noise (a steep turn that tilts the tank for a moment).
Neither gauge, alone, is reliable. The slow gauge, alone, takes too long to warn of a real leak. The fast gauge, alone, generates false alarms for every pothole on the road. But the two together, requiring both to confirm the same problem at the same time, are reliable: if the fast gauge says "it's dropping hard, right now" and the slow gauge says "yes, and it's been dropping like this for a while, it's not an isolated pothole," there's a real leak. If only one of the two confirms it, there isn't. That's exactly the logic Google SRE formalizes for burn rate alerting.
The source, quoted directly
"A simple error-rate threshold produces two critical problems. Too Slow: with a 10-minute window at 99.9% SLO, a 0.1% error rate for 10 minutes would alert, while consuming only 0.02% of the monthly error budget. Too Much Noise: the approach exhibits low precision because the alert fires on many events that do not threaten the SLO, potentially generating 144 alerts daily while still meeting the SLO."
This quote is the formal version, with exact figures, of the problem lesson 1 already introduced with the day 17 example. "Too slow" and "too noisy" aren't intuitions — they're two measurable failures of the same simple threshold, quantified: an alert that fires while consuming barely 0.02% of the budget (irrelevant) or that could fire 144 times a day (pure noise) while the system is still meeting its SLO.
What burn rate is, again, with the full reference figure
"Burn rate is how fast, relative to the SLO, the service consumes the error budget. [...] A 99.9% SLO with constant 0.1% error rate has a burn rate of 1, exhausting the 30-day budget exactly. A 1,000x burn rate (100% errors) depletes budget in 43 minutes."
This quote's second data point — a burn rate of 1,000x exhausts the 30-day budget in 43 minutes — is a useful intuition anchor: at burn rate 1x, the budget lasts exactly the SLO window's full 30 days (by definition); at burn rate 1,000x (100% errors, the worst possible case), that same budget runs out in under an hour. The 43.41x burn rate Module 2's "bad week" reached is, on this scale, much closer to the severe extreme than to the healthy one.
The two-window mechanism, quoted
"[The multi-window strategy reduces false positives by requiring both conditions simultaneously:] send a page-level alert when you exceed the 14.4x burn rate over both the previous one hour and the previous five minutes. [This dual requirement ensures] the alert fires only once you've consumed 2% of the budget, but exhibits a better reset time by ceasing to fire five minutes later."
This is the central piece: both conditions, at once, not one or the other. The long window (1 hour) confirms the elevated consumption isn't an isolated instant — it needs to hold for a full hour to count. The short window (5 minutes) confirms that consumption is still active now — as soon as the problem is resolved, the short window drops below the threshold within minutes, and the alert stops firing, without waiting for the full long window to "clear" (which would take up to a full hour to average out the already-resolved event).
The complete severity table
Table 5-8, Google SRE Workbook — Alerting on SLOs:
| Severity | Long window | Short window | Burn rate | % of budget it consumes |
|---|---|---|---|---|
| Page (urgent) | 1 hour | 5 minutes | 14.4x | 2% |
| Page (urgent) | 6 hours | 30 minutes | 6x | 5% |
| Ticket (not urgent) | 3 days | 6 hours | 1x | 10% |
Three rows, not one — and the order matters. The first two (Page) are urgent alerts: someone gets a notification demanding near-immediate attention, because at that consumption pace, ignoring it for hours has a real budget cost. The third (Ticket) is deliberately less urgent: a burn rate of barely 1x sustained for 3 full days isn't a 3 AM emergency — it's exactly the pace the SLO allows spending the budget at, sustained longer than a sharp event normally lasts, long enough for someone to review it during business hours, without waking anyone up.
Why three rows, not one: each one detects a different pattern
THE THREE SEVERITIES DETECT THREE WAYS OF CONSUMING THE BUDGET
Page (1h/5min, 14.4x) -> a SHARP incident: something broke suddenly,
consumes the budget very fast, very soon
(Module 2's day 17, at 13.27x, nearly crosses this threshold)
Page (6h/30min, 6x) -> a SUSTAINED BUT LESS SHARP incident: slower
than the one above, but still serious if ignored
Ticket (3d/6h, 1x) -> a GRADUAL DEGRADATION: never triggers urgency,
but sustained long enough to deserve
review -- exactly the pattern
Module 2's "bad week" showed day by dayModule 2 lesson 7's "bad week" dataset is already a perfect example of why more than one row is needed: that week's daily burn rate (57.47x on day 2, 75.61x on day 3, dropping gradually to 17.54x on day 7) would have crossed all three rows of this table at some point during the week — a real incident, sharp at the start, that turns into sustained degradation as the mitigation progresses. An alerting system with a single severity would have had to choose between over-alerting at the start or under-alerting at the end; with three rows evaluated in parallel, each one fires and stops firing at the moment that belongs to it.
Applying the pattern to Module 2's already-familiar example
Module 2, lesson 6 already ran a version of this pattern — a single row, the Ticket one, adapted to that lesson's fixed dataset's daily granularity (3-day long window, short window adapted to 1 day instead of 6 hours). The result, already verified back then:
--- Ticket tier (long window 3d / short window 1d, threshold 1.0x) ---
Day 17: long window=4.52x short window=13.27x -> FIRES
Day 18: long window=6.74x short window=6.85x -> FIRES
Day 19: long window=6.67x short window=0.00x -> does not fireDay 19 is that result's most important example, and it's worth repeating here because it's the concrete proof of why the short window exists: the long window (which still includes days 17-18) was still at 6.67x — well above the threshold — but the short window (day 19 alone) was already at 0.00x. Without the short window, the alert would have kept firing on day 19, over a problem that had already resolved — the exact false positive this lesson's quote describes ("a better reset time by ceasing to fire [...] later").
Lesson 3 of this module does what Module 2's lesson 6 couldn't do yet: it implements the table's all three complete rows — not just Ticket — and applies them not to a daily dataset, but to already-calculated burn rates, exactly as a real Prometheus or CloudWatch query would produce them, with the literal hour and minute windows Table 5-8 specifies.
Common mistakes
Thinking "multi-window" means "evaluate the same threshold several times at different frequencies" (confusing evaluation frequency with window length, the same mistake exercise 3 of lesson 1 already dismantled). What happens: someone assumes it's enough to run the same burn rate query every minute instead of every hour. How to spot it: if your explanation of the pattern doesn't mention two windows of different length evaluated simultaneously. How to fix it: the pattern demands two windows of different duration (1 hour and 5 minutes, for example), both evaluated at the same instant, both required to cross the same threshold — not a single window evaluated more often.
Using only the long window, and treating the short window as optional (the mistake that produces day 19's false positive). What happens: someone implements the alert with only the long window's condition, reasoning that "if the long window is already high, that's enough." How to spot it: if your implementation would keep firing on this lesson's example's day 19. How to fix it: the short window isn't an optional refinement — it's half of the AND condition that makes the alert stop firing as soon as the real problem ends. Without it, any burn rate alert inherits the same "too slow to reset" problem a single-window threshold already had.
Assuming the Ticket row (1x, not urgent) is less important than the Page rows, and can be skipped to simplify things (underestimating gradual degradation). What happens: someone, implementing the alert, only builds the two Page rows, reasoning "a burn rate of just 1x isn't serious." How to spot it: if your alerting system would never notify about consumption sustained at exactly the rate the SLO allows, held for several days. How to fix it: a burn rate of 1x sustained for 3 full days consumes 10% of the entire monthly budget — without the Ticket row, that consumption would go completely unnoticed until it was too late to act calmly, exactly the scenario a real on-call team prefers to review during business hours, not discover all at once.
Exercises
Exercise 1 — Without looking at the table, explain from memory what the Page (1h/5min, 14.4x) row requires to fire. Use the words "long window," "short window," and "simultaneously" in your explanation.
See solution
The row fires only when the burn rate calculated over the long window (the last full hour) and, simultaneously, the burn rate calculated over the short window (the last five minutes) are both at 14.4x or higher. If only one of the two windows crosses the threshold — for example, a brief five-minute spike that doesn't raise the full hour's average enough, or the reverse, an hour with an elevated average from an event that ended more than five minutes ago — the alert doesn't fire. Both conditions, at the same time, are mandatory.
Exercise 2 — Using this lesson's quote ("a 1,000x burn rate [...] depletes budget in 43 minutes"), calculate in how many minutes a sustained 100x burn rate would exhaust Andes Cargo's full monthly budget (43.2 minutes at 99.9% monthly, from Module 2).
See solution
The full budget (43.2 minutes) is exhausted in a real time inversely proportional to the burn rate: at burn rate 1x, it's exhausted over the SLO window's full duration (30 days = 43,200 minutes). At burn rate 100x, it would be exhausted 100 times faster: 43,200 / 100 = 432 minutes, that is, 7.2 hours. Verification with the cited figure: at burn rate 1,000x, it would be 43,200 / 1,000 = 43.2 minutes — matches exactly the "43 minutes" the Google SRE quote gives as a reference for that extreme case. This exercise confirms the burn rate formula (time to exhaust the budget = SLO window / burn rate) is consistent between the source's cited figure and Andes Cargo's specific budget.
Exercise 3 — Explain why Table 5-8 uses different burn rate thresholds for each row (14.4x, 6x, 1x) instead of the same threshold with only different windows. What relationship exists between each row's threshold and its percentage of budget consumed?
See solution
Each row is designed so that, if sustained for exactly its long window, it consumes the same target percentage of the budget declared in the last column — it's no coincidence the figures relate: at burn rate 14.4x sustained for 1 hour, consumption is 14.4x × (1 hour / 720 hours in the month) ≈ 2% of the full monthly budget; at 6x sustained for 6 hours, 6 × (6/720) = 5%; at 1x sustained for 3 days, 1 × (72/720) = 10%. Each row answers a different urgency question: "at this pace, if it keeps up for an hour, has it already eaten 2% of the month?" is a question that deserves an immediate page; "at this pace, sustained for three days, has it eaten 10%?" is a question that can wait for someone to review it without getting out of bed. The threshold and the window, together, calibrate how much real urgency each consumption pattern represents.
Summary and next step
This lesson presented Google SRE's complete multi-window, multi-burn-rate pattern, cited directly from sre.google/workbook/alerting-on-slos/: two windows — a long one, confirming consumption is real; a short one, confirming it's still active now — required simultaneously, never separately, across three severities (Page 14.4x/1h+5min, Page 6x/6h+30min, Ticket 1x/3d+6h). You verified, with Module 2 lesson 6's already-familiar example, exactly why the short window avoids day 19's false positive — the system had already recovered, and only the short window reflected that in time.
Before moving on you should be able to: cite from memory Table 5-8's three rows, with their windows and thresholds; explain why both windows must cross the threshold at once, not just one; and calculate, given a burn rate and an SLO window, how long it would take to exhaust the full budget.
Lesson 3 implements this complete pattern — all three severities, not just Ticket — in scripts/burn_rate_evaluator.py, really run over Module 2's "bad week" scenario and "normal" scenario.
Resources
- Google SRE Workbook — Alerting on SLOs — the exact source for every quote in this lesson: the simple threshold problem, burn rate's definition, the two-window mechanism, and the complete Table 5-8.
- This same repository, Module 2, lesson 6 (
06-hands-on-burn-rate-not-just-the-balance.md) — this pattern's first partial implementation (Tickettier, adapted to daily granularity), which this lesson extends to all three severities. - This same repository, Module 2, lesson 7 (
07-hands-on-applying-the-calculator-to-a-real-andes-cargo-scenario.md) — the "bad week" dataset, whose daily burn rate crosses, at some point during the week, all three rows of Table 5-8.