Module 4: Alerting On Error Budget Burn Rate

3. Hands-on: the burn rate evaluator

Description

This lesson builds scripts/burn_rate_evaluator.py: the complete implementation of Table 5-8's three severities — not just Ticket, as far as Module 2 managed to get with daily data — really run over this module's two fixed scenarios: the "bad week" (Module 2, lesson 7) and the "normal" month (Module 2, lesson 4). It's the first of the three engines this module builds, and the simplest: pure Python, no Docker, no Terraform, no infrastructure behind it at all — the exact prototype of the decision Alertmanager (lesson 4) and CloudWatch (lesson 5) are going to make afterward, with real infrastructure around it.

Connection to the module

This script reuses burn_rate_of() from scripts/error_budget_calculator.py (Module 2, lesson 6) unmodified — no new SLI math, only the alert decision layered on top of a number the calculator already knows how to produce. Lesson 4 takes the same decision — the same three severities, the same two scenarios — and reimplements it as a real Prometheus/Alertmanager rule, evaluated against scraped metrics instead of a dataset imported in Python.

Step 1 — Where the numbers come from: two windows per scenario, already familiar

Before the code, the data. This evaluator doesn't recalculate the SLI from scratch — it takes two already-calculable burn rates with burn_rate_of(), one for each window (short and long), for each of this module's two scenarios.

"Bad week" (Module 2, lesson 7, BAD_WEEK dataset). The long window is the entire week (the 7 days aggregated as a single window); the short window is the last day (day 7), already improving relative to the week's peak, but still not back to a healthy pace — exactly as Module 2's lesson 7 left it read.

"Normal" (Module 2, lesson 4, TRAFFIC_30_DAYS dataset). The long window is the full 30-day month; the short window is any healthy day within that month (day 1, with zero errors).

Scenario	Short window	Short burn rate	Long window	Long burn rate
Bad week	Day 7 (`BAD_WEEK[-1]`)	17.54x	Full week (`BAD_WEEK`)	43.41x
Normal	Day 1 (`TRAFFIC_30_DAYS[0]`)	0.00x	Full month (`TRAFFIC_30_DAYS`)	0.90x

Both numbers in each row already appeared, separately, in Module 2: the 17.54x and the 43.41x are literal from lesson 7; the 0.90x is the same 90.2% of budget consumed from lesson 4, expressed as burn rate instead of a percentage (90.2% of budget consumed over the SLO's full window is, by definition, an average burn rate of 0.90x over that same window). This lesson invents no number at all — it just reorganizes them into pairs (short, long) and applies Table 5-8's decision to them.

Step 2 — The complete script

Create scripts/burn_rate_evaluator.py at the root of andes-cargo-infra/:

# burn_rate_evaluator.py
# Multi-window, multi-burn-rate ALERT DECISION. Given the short-window and long-window burn
# rate already observed for a scenario, decides fire/no-fire, tier by tier -- the exact
# decision Alertmanager (Module 4, lesson 4) and CloudWatch (Module 4, lesson 5) apply against
# live metrics, prototyped here first in plain Python before either is built.
#
# Reuses burn_rate_of() from scripts/error_budget_calculator.py (Module 2) -- no new SLI math,
# only the alerting decision layered on top of numbers that calculator already knows how to
# produce. Deterministic: BAD_WEEK (Module 2, lesson 7) and TRAFFIC_30_DAYS (Module 2, lesson 4)
# are the exact fixed datasets already committed to this project -- no random, no datetime.now().

from error_budget_calculator import TRAFFIC_30_DAYS, burn_rate_of

# The "bad week" dataset, unchanged from Module 2, lesson 7.
BAD_WEEK = [
    (1, 420, 418),
    (2, 435, 410),
    (3, 410, 379),
    (4, 428, 400),
    (5, 440, 421),
    (6, 300, 292),
    (7, 285, 280),
]

# Google SRE Workbook, Table 5-8 (sre.google/workbook/alerting-on-slos/).
# "long_window"/"short_window" are the real production windows -- this evaluator's INPUT
# already carries the burn rate for each window (computed the same way M4.4's PromQL
# expressions and M4.5's CloudWatch alarm period will compute it live).
TIERS = [
    {"name": "Page (fast)", "threshold": 14.4, "long_window": "1h", "short_window": "5m"},
    {"name": "Page (slow)", "threshold": 6.0, "long_window": "6h", "short_window": "30m"},
    {"name": "Ticket", "threshold": 1.0, "long_window": "3d", "short_window": "6h"},
]


def evaluate(tier, short_burn_rate, long_burn_rate):
    """Fires only when BOTH windows are at or above the tier's threshold -- the same
    two-window confirmation from Module 2, lesson 6 (ticket_tier_fires), generalized to
    all three severities of the real Google SRE table."""
    return short_burn_rate >= tier["threshold"] and long_burn_rate >= tier["threshold"]


def evaluate_scenario(label, short_burn_rate, long_burn_rate):
    print(f"--- {label} (short={short_burn_rate:.2f}x, long={long_burn_rate:.2f}x) ---")
    for tier in TIERS:
        fires = evaluate(tier, short_burn_rate, long_burn_rate)
        marker = "FIRES" if fires else "does not fire"
        print(
            f"  {tier['name']:<12} >= {tier['threshold']:>4}x "
            f"({tier['long_window']}/{tier['short_window']}): {marker}"
        )


if __name__ == "__main__":
    # "Bad week" (Module 2, lesson 7): long window = the full week (7 days
    # aggregated as a single window); short window = the last day (day 7), already
    # improving but still elevated -- exactly as lesson 7 left it read.
    bad_week_long = burn_rate_of(BAD_WEEK)
    bad_week_short = burn_rate_of([BAD_WEEK[-1]])
    evaluate_scenario("bad week (M2.7)", bad_week_short, bad_week_long)

    print()

    # Normal scenario (Module 2, lesson 4): long window = the full 30 days;
    # short window = any healthy day (day 1, zero errors).
    normal_long = burn_rate_of(TRAFFIC_30_DAYS)
    normal_short = burn_rate_of([TRAFFIC_30_DAYS[0]])
    evaluate_scenario("normal (M2.4)", normal_short, normal_long)

TIERS is, literally, the previous lesson's Table 5-8, turned into data: each severity is a dictionary with its threshold and its two nominal windows (the names "1h", "5m" are just labels for the report — this script doesn't measure real time, it receives the burn rates already calculated). evaluate() implements exactly lesson 2's AND condition: both windows must cross the threshold, never just one. evaluate_scenario() runs all three severities over the same pair of numbers and prints all of them, so it's clear at a glance which ones fire and which don't for a complete scenario.

Step 3 — Running the evaluator

python3 scripts/burn_rate_evaluator.py

What to expect (literal — run it twice and it produces, line for line, the same result):

--- bad week (M2.7) (short=17.54x, long=43.41x) ---
  Page (fast)  >= 14.4x (1h/5m): FIRES
  Page (slow)  >=  6.0x (6h/30m): FIRES
  Ticket       >=  1.0x (3d/6h): FIRES

--- normal (M2.4) (short=0.00x, long=0.90x) ---
  Page (fast)  >= 14.4x (1h/5m): does not fire
  Page (slow)  >=  6.0x (6h/30m): does not fire
  Ticket       >=  1.0x (3d/6h): does not fire

A clean contrast: "bad week" fires all three severities — not just Ticket, like in Module 2. The "normal" scenario fires none. This is exactly the result lesson 8 (this module's project) is going to reproduce with the other two engines, Alertmanager and CloudWatch.

Reading the result: why "bad week" crosses even the most urgent threshold

"Bad week"'s short burn rate (17.54x, day 7) is surprising at first glance: Module 2, lesson 7 already described day 7 as "the week's best day," with a clear improving trend. And yet, 17.54x is still above the most urgent threshold in all of Table 5-8 (14.4x, the Page (fast) row). This isn't a script error — it's the same honest reading Module 2, lesson 7 already previewed in its exercise 2: "this week [...] never drops back below Google's most urgent threshold on any complete day the dataset covers." A real improvement (from 75.61x on day 3 to 17.54x on day 7) can still be, in absolute terms, well above any threshold in the table — "improving" and "already healthy" are two different claims, and only the second one turns an alert off.

The "normal" scenario, by contrast, doesn't even come close to the least demanding threshold: 0.90x over the long window — remembering that a burn rate of 1x is, by definition, exactly the pace the SLO allows sustaining across its whole window — is below what the SLO itself tolerates as sustained consumption. It's the numeric confirmation of something SLO.md already declared in its "Consequences" section: a month with a tight budget (90.2% consumed) can, at the same time, represent no alarming burn rate at all — both readings are true at once, with no contradiction, because they measure different questions.

Common mistakes

Confusing "day 7 improved" with "day 7 shouldn't fire any alert anymore" (repeated from Module 2, lesson 7, now with direct consequences on a real alert decision). What happens: someone, seeing short=17.54x for "bad week," assumes there's an error in the script because "the week was already improving." How to spot it: if your expectation is that the "bad week" scenario should stop firing before the seventh day. How to fix it: 17.54x is still well above any threshold in Table 5-8 — an improving trend isn't the same as an already-healthy value (near or below 1x). The script is doing exactly what it should: keep alerting as long as the real consumption, measured in the short window, stays above the threshold, regardless of which way the trend is going.

Modifying TIERS to "tune" the Ticket threshold to a different number than 1.0x with no source (breaking traceability with Table 5-8). What happens: someone, seeing "bad week" fires all three severities anyway, decides the Ticket threshold is redundant and raises it to, say, 2x, "to reduce noise." How to spot it: if any value in TIERS in your copy of the script doesn't match, digit for digit, lesson 2's Table 5-8. How to fix it: the three thresholds (14.4x, 6x, 1x) aren't arbitrary — each one is calibrated so that, sustained for exactly its long window, it consumes a specific percentage of the budget (2%, 5%, 10%, per lesson 2's exercise 3). Changing a threshold without recalculating that relationship breaks the guarantee that gives the complete pattern its meaning.

Treating evaluate_scenario() as if it needed error_budget_calculator.py and burn_rate_evaluator.py running in separate processes (misunderstanding the import). What happens: someone tries to run burn_rate_evaluator.py without error_budget_calculator.py in the same directory, and the script fails with ModuleNotFoundError. How to spot it: the exact Python error when running the script from a different directory. How to fix it: the line from error_budget_calculator import TRAFFIC_30_DAYS, burn_rate_of requires both files to live in scripts/, at the root of andes-cargo-infra/ — they aren't two independent programs, they're a single script reusing functions from another, exactly as Module 2, lesson 7 (bad_week_scenario.py) already did with the same import pattern.

Exercises

Exercise 1 — Calculate by hand whether the "normal" scenario would fire the Ticket row (threshold 1.0x) if, hypothetically, its long-window burn rate were 1.05x instead of 0.90x, keeping the short-window burn rate at 0.00x. Use evaluate()'s definition.

See solution

It wouldn't fire. evaluate() demands both conditions be met (short_burn_rate >= threshold and long_burn_rate >= threshold) — even though the hypothetical long window (1.05x) does cross the 1.0x threshold, the short window (0.00x) doesn't, so the complete and condition is false. This exercise confirms, with a different number than the lesson's, the same lesson from Module 2's day 19: a single window crossing the threshold is never enough — both have to cross it at once.

Exercise 2 — Describe, in prose (with no code run), which line of the script you'd change to add a hypothetical fourth severity, "Critical," with threshold 50x, long window "15m," and short window "2m." Would you need to change evaluate() or evaluate_scenario()?

See solution

It would be enough to add one more dictionary to the TIERS list: {"name": "Critical", "threshold": 50.0, "long_window": "15m", "short_window": "2m"}. Neither evaluate() nor evaluate_scenario() would need any change — both functions already iterate over TIERS generically, with no severity value hardcoded into their logic. This confirms the same design principle Module 2, lesson 4 already established with budget_report()'s default parameters: separating the data (the severity table) from the logic (how each row is evaluated) lets you extend behavior without touching any already-written function.

Exercise 3 — Explain why this script receives each window's burn rates already calculated, instead of receiving the raw datasets (BAD_WEEK, TRAFFIC_30_DAYS) and calculating the windows internally. What advantage does this separation give, looking ahead to lesson 4 of this module?

See solution

Separating "calculate a window's burn rate" (which burn_rate_of(), from Module 2, already does) from "decide whether an alert fires given a pair of burn rates" (what evaluate() does, new in this lesson) is exactly the same separation Prometheus and Alertmanager use in real production: Prometheus calculates rates and aggregate values with PromQL (equivalent to burn_rate_of()), and a separate alert rule decides fire/no-fire over those already-calculated values (equivalent to evaluate()). Designing evaluate() to receive already-calculated numbers, instead of raw datasets, is what makes it possible for lesson 4 to reimplement exactly the same decision logic as a PromQL expression — which also operates on already-calculated values, never on raw data directly inside the alert rule.

Summary and next step

In this lesson you built and ran scripts/burn_rate_evaluator.py: the complete implementation of Google SRE's Table 5-8 three severities, applied to this module's two fixed scenarios. The result: "bad week" fires all three severities (Page fast, Page slow, Ticket), with a short-window burn rate (17.54x) still above even the most urgent threshold despite the improvement already underway; "normal" fires none, with a long-window burn rate (0.90x) below what the SLO itself tolerates as sustained consumption. You confirmed the script is deterministic and separates the burn rate calculation (already built in Module 2) from the alert decision (new in this lesson).

Before moving on you should be able to: run the script and get this lesson's exact same numbers; explain why "bad week" fires even the most urgent severity despite improving; and explain the advantage of receiving already-calculated burn rates instead of raw datasets.

Lesson 4 takes this exact same decision — the same three severities, the same two scenarios — and reimplements it as a real Prometheus/Alertmanager rule, evaluated against metrics scraped from a real exporter, routed to a real receiver.

Resources

This same repository, Module 4, lesson 2 (02-multiwindow-multiburn-rate-the-real-google-sre-pattern.md) — Table 5-8, which TIERS implements in full.
This same repository, Module 2, lessons 4, 6, and 7 — burn_rate_of(), TRAFFIC_30_DAYS, and BAD_WEEK, the three pieces this script reuses unmodified.
Google SRE Workbook — Alerting on SLOs — the complete source for the pattern this script implements.

← Previous2. Multi-window, multi-burn-rate: the real Google SRE pattern Next →4. Hands-on: the real rule in Alertmanager