Module 5: The Incident Lifecycle

5. Hands-on: Andes Cargo's severity matrix

Description

Lesson 3 gave the four severity levels, with narrative examples. This lesson turns them into something that no longer depends on "feels serious" — a matrix where each SEV ties, with real, verifiable arithmetic, to an exact burn rate threshold ALERTING-POLICY.md (Module 4) already built and tested. It's INCIDENT-RESPONSE-PLAN.md's first real piece: not a table of opinions, a table where every row derives, with a single formula, from the same three thresholds in Google SRE's Table 5-8 this whole project has already been using since Module 4.

Connection to the module

This lesson doesn't introduce any new burn rate data — it reuses, unchanged, the three thresholds scripts/burn_rate_evaluator.py (Module 4, lesson 3) already implemented: Page (fast) at 14.4x, Page (slow) at 6x, Ticket at 1x. The only genuinely new thing here is the formula that connects a burn rate to an observable error rate, and the resulting table — the exact piece lesson 8 (this module's project) is going to copy, unchanged, into INCIDENT-RESPONSE-PLAN.md's "Severity matrix" section.

Step 1 — The missing formula: from burn rate threshold to real error rate

ALERTING-POLICY.md already defines Andes Cargo's SLO: 99.9% monthly, which means an allowed error rate of 0.1% (1 − 0.999). Burn rate, as Module 4, lesson 2 already quoted from Google SRE, is the observed error rate, divided by the allowed error rate:

burn_rate = observed_error_rate / allowed_error_rate
allowed_error_rate = 0.001  (0.1%, fixed, from SLO.md)

Solving for the observed error rate that corresponds to each already-known threshold:

observed_error_rate = burn_rate x allowed_error_rate

`ALERTING-POLICY.md` threshold	Calculation	Observed error rate
`Ticket` — 1.0x	1.0 × 0.001	0.10%
`Page (slow)` — 6.0x	6.0 × 0.001	0.60%
`Page (fast)` — 14.4x	14.4 × 0.001	1.44%

This table is the missing piece: there's no need to invent any new limit for "when is it SEV2 and when is it SEV1" — the three burn rate thresholds already built and tested in Module 4 are, literally, the three limits that separate SEV4 from SEV3, SEV3 from SEV2, and SEV2 from SEV1.

Step 2 — The extreme case, verified with the same Module 4 quote

Lesson 3's most concrete SEV1 example — Shipments not responding at all — has an observed error rate of 100% (every valid invocation fails, none completes). Applying the same formula:

burn_rate = 1.0 / 0.001 = 1000

A burn rate of 1,000x. Module 4, lesson 2 already quoted, verbatim, what that exact number means:

"A 1,000x burn rate (100% errors) depletes budget in 43 minutes."

— Google SRE Workbook — Alerting on SLOs

Verifying with the formula already used in that same lesson (time to exhaustion = SLO window ÷ burn rate):

time_to_exhaustion = 43200 minutes / 1000 = 43.2 minutes

Forty-three point two minutes — the exact same number as Andes Cargo's complete monthly error budget (43.2 minutes, from SLO.md). This isn't a design coincidence of this lesson: at a 1,000x burn rate, by definition, the complete 30-day budget gets consumed 1,000 times faster, so the time it would take to exhaust is the full window (43,200 minutes) divided by 1,000 — which arithmetically gives the same number as the total budget itself in minutes. It's the most direct possible numeric confirmation of why Shipments not responding is, unambiguously, SEV1: at that rate, an entire month's full margin disappears before a one-hour coordination call is even over.

Step 3 — The complete matrix, with the arithmetic in plain sight

Severity	Burn rate threshold (`ALERTING-POLICY.md`)	Observed error rate	% of budget if sustained	Andes Cargo example (Module 5, lesson 3)
SEV1 — Critical	`Page (fast)`, ≥ 14.4x — OR irreversible data loss, regardless of measured burn rate	≥ 1.44% (up to 100%)	2% in 1 hour (100% in 43.2 min at 1,000x)	`Shipments` isn't responding at all
SEV2 — Major	`Page (slow)`, ≥ 6.0x and < 14.4x	0.60%–1.44%	5% in 6 hours	Retries exhausted on a real portion of manifests, silently dropped
SEV3 — Minor	`Ticket`, ≥ 1.0x and < 6.0x	0.10%–0.60%	10% in 3 days	Brief throttling, absorbed by the automatic retry
SEV4 — Low/cosmetic	Below `Ticket`, < 1.0x — or doesn't even count as a bad event	< 0.10%	Negligible	Two extra seconds within the `Timeout`, counts as a good event

Four rows, three numbers that already existed since Module 4 (14.4x, 6x, 1x), and a single new formula (error_rate = burn_rate × 0.001) connecting them to something measurable. This is, literally, the table lesson 8 is going to copy as INCIDENT-RESPONSE-PLAN.md's "Severity matrix" section.

Common mistakes

Forgetting SEV1's explicit exception (irreversible data loss, regardless of measured burn rate) and treating the matrix as if it worked purely by burn rate arithmetic. What happens: someone assumes the only way to classify SEV1 is by calculating a burn rate of 14.4x or higher, and wouldn't know what to do facing a scenario where data was lost but the system, technically, is still responding normally to other requests. How to spot it: if your severity classification depends, without exception, on having a calculable burn rate number in the moment. How to fix it: RELIABILITY-CHARTER.md (Module 1, Decision, point 4) already established that the error budget is "a measurement mechanism, never a preventive control" — and that same honesty applies here: burn rate assumes the system is still receiving measurable traffic. A scenario like the Claude Code incident, where the entire infrastructure was destroyed, might generate no measurable burn rate at all (there are no failing invocations if there's no infrastructure to invoke) — that's why this matrix includes irreversible data loss as an independent SEV1 criterion, one that doesn't wait for any number to cross any threshold.

Calculating burn rate with the wrong allowed error rate (using 0.001 when the scenario isn't about Andes Cargo's main SLO). What happens: someone reuses this lesson's formula for a different system, or a different SLO, without adjusting the allowed error rate (0.001 is specific to SLO.md's 99.9% monthly). How to spot it: if your burn rate calculation produces a number that doesn't match any of the three known thresholds for a scenario that should, by design, cross one of them exactly. How to fix it: the formula burn_rate = observed_rate / allowed_rate is generic, but this lesson's 0.001 is specific to the 99.9% SLO SLO.md already fixed — a system with a different SLO (say, 99% instead of 99.9%) would have a different allowed rate (0.01), and the same burn rate thresholds (14.4x, 6x, 1x) would correspond to completely different observed error rates.

Assuming the "% of budget if sustained" column is the same as "budget already consumed" (confusing a rate with an accumulated total, the same error Module 4, lesson 1 already named). What happens: someone reads "2% in 1 hour" in the SEV1 row and concludes that incident, barely underway, already consumed 2% of the monthly budget. How to spot it: if your reading of the table doesn't distinguish between "at this rate, sustained for an hour this would be consumed" and "this has already been consumed." How to fix it: the column describes what would happen if the rate is sustained over the indicated window — not what already happened. A SEV1 incident mitigated in five minutes consumes far less than the actual 2%; the column is an urgency projection, not an already-spent total, exactly the same rate-versus-balance distinction all of Module 4 exists to teach.

Exercises

Exercise 1 — Calculate, by hand, the burn rate and corresponding severity for a scenario where process-shipment-manifest's observed error rate is 0.72% sustained. Use this lesson's formula.

See solution

burn_rate = 0.0072 / 0.001 = 7.2. A burn rate of 7.2x falls in the Page (slow) range (≥ 6.0x and < 14.4x) — SEV2. Verification against the table: 7.2x is above the Ticket threshold (1.0x) and Page (slow) (6.0x), but below Page (fast) (14.4x), so the correct severity is the SEV2 row, not SEV1 or SEV3.

Exercise 2 — Explain, using this lesson's Step 2, why a burn rate of 1,000x isn't simply "well above" the SEV1 threshold (14.4x), but represents a qualitatively different category of urgency.

See solution

At 14.4x, the complete budget would be exhausted in 43,200 / 14.4 ≈ 3,000 minutes (50 hours) if the rate were sustained exactly at that threshold — still within the order of "days," real time to react before an entire month's margin disappears. At 1,000x, the same budget is exhausted in 43.2 minutes — under an hour. The difference isn't just one of magnitude (1,000 is much more than 14.4): it's the difference between an incident that gives real reaction time measured in hours, and one that exhausts an entire month's full margin before a single coordination call is even over. Both cross the SEV1 threshold, but the second case justifies, through the arithmetic itself, why the response must be immediate, not just "urgent."

Exercise 3 — A classmate proposes simplifying the matrix by removing the "% of budget if sustained" column, arguing that the burn rate column already contains all the necessary information. What gets lost by removing it?

See solution

The burn rate number (14.4x, for example) is correct but abstract — it doesn't, by itself, communicate how fast something concrete and measurable in real minutes or hours gets exhausted. The percentage column translates that abstract multiplier into a question anyone, technical or not, can understand immediately: "if this keeps up for a full hour, how much of the month is already spent?" It's the same reason Module 2 translated the error budget from an abstract percentage into concrete minutes (43.2 minutes, not just "0.1%") — a number without intuitive units is correct, but far less useful for quickly deciding how urgent it is to act.

Summary and next step

This lesson built Andes Cargo's complete severity matrix: four rows, each tied, with a single formula (burn_rate = observed_rate / 0.001), to the same three thresholds ALERTING-POLICY.md already tested in Module 4 — without inventing any new limit. You verified the extreme case (Shipments not responding, 100% error rate, 1,000x burn rate) with the same Google SRE quote from Module 4, confirming that the full monthly budget, 43.2 minutes, would be exhausted in exactly those same 43.2 minutes if the rate were sustained.

Before moving on you should be able to: calculate, given any observed error rate, the corresponding burn rate and the severity it maps to; explain why SEV1 has a criterion independent of burn rate (irreversible data loss); and distinguish "percentage if sustained" from "percentage already consumed."

Lesson 6 shifts topics, toward the human piece that responds to these severities: on-call, with complete honesty about its real cost — the full Hacker News quote on why to treat on-call as a portable layer, not as a promise of total availability.

Resources

This same repository, Module 4, lesson 2 (02-multiwindow-multiburn-rate-the-real-google-sre-pattern.md) — the complete Table 5-8, the source of this lesson's three thresholds.
This same repository, Module 4, lesson 8 (08-project-andes-cargos-alerting-policy.md) — ALERTING-POLICY.md, quoted in this lesson's table header.
This same repository, Module 2, lesson 8 (08-project-andes-cargos-slo-md.md) — SLO.md, the source of the allowed error rate (0.1%) and the 43.2-minute budget.
Google SRE Workbook — Alerting on SLOs — the 1,000x/43-minute case quote, verified again in this lesson's Step 2.

← Previous4. Roles during an incident Next →6. On-call, with honesty about its cost