Module 5: The Incident Lifecycle

1. Introduction: the framework before the case

Description

Module 4 closed with ALERTING-POLICY.md tested with a forced drill: the "bad week" fires all three severities of Google SRE's Table 5-8, the normal scenario fires none. The measurement machine is complete — SLI (Module 2) → real data (Module 3) → burn rate alert (Module 4) — and it works. But when that alert really goes off, at 3 AM or in the middle of a meeting, ALERTING-POLICY.md doesn't answer a single one of the questions that actually matter at that moment: who gets notified? How serious is it, in terms someone can actually decide with? Who decides what to do, and who executes it, and are they the same person? This module builds those answers — lifecycle, severities, roles, on-call rotation — before Module 6 operates the real Claude Code incident, so that module applies a framework already decided, instead of improvising it while the clock runs.

Connection to the module

This is exactly the row that RELIABILITY-CHARTER.md (Module 1, lesson 8) left open for Module 5 in its map: "Who does what, with what severity, with what on-call?". By the close of this module, that row moves from Open to Resolved, with INCIDENT-RESPONSE-PLAN.md as evidence. Nothing that follows is new in the sense of "new math" — this module's severities tie, number for number, to the same three burn rate levels that ALERTING-POLICY.md already built and tested (14.4x, 6x, 1x); what's new is the human layer on top of that math: who responds, and with exactly what urgency.

Why this framework gets built before the fire, not during

Think of an evacuation drill. Nobody designs the exit routes, assigns who counts people at the meeting point, or decides who has the authority to call the fire department, while the building is already filling with smoke. Those decisions get made on a calm afternoon, without pressure, precisely because the quality of a decision made under real pressure is much worse than the same decision made with time to spare — and because, once the smoke appears, there's no time left to design anything: all that's left is to execute what was already decided.

This module is that drill, for Andes Cargo. The Claude Code incident that Module 6 is going to operate already happened — the facts, verified by three sibling guides, aren't going to change — so the real temptation is to jump straight to narrating it with the vocabulary already learned. This guide resists that temptation on purpose, for the same reason a real SRE team would never design its incident roles the same day a system goes down: improvising the framework during real pressure is, almost always, the way a manageable incident turns into a chaotic one — not for lack of technical skill, but for lack of a structure already agreed on beforehand about who decides and who executes.

   TWO POSSIBLE ORDERS, ONE DESIRABLE OUTCOME

   ORDER A (the one this guide follows)      ORDER B (the one this guide avoids)
   ─────────────────────────────────         ─────────────────────────────────
   Module 5: the complete framework          Module 6: the incident is
   is built -- lifecycle, severity,          already underway -- someone asks
   roles, on-call -- WITHOUT real pressure    "who decides here?" for the first
        │                                     time, live, under pressure
        ▼                                          │
   Module 6: the Claude Code incident               ▼
   is operated by APPLYING an already            Every person improvises their
   decided framework -- each role, each          own role, nobody knows whether
   severity, already had an answer before        an incident has been formally
   the first incident data arrived               declared or not

This module's map: the 8 lessons

   MODULE 5 -- THE INCIDENT LIFECYCLE
   the human framework, built before the case that Module 6 is going to operate

   M5.1  Introduction (this lesson)                why the framework comes before the case
   M5.2  The lifecycle of an incident               cited from Google SRE, ASCII diagram
   M5.3  Severity levels                            SEV1-4, narrative Andes Cargo examples
   M5.4  Roles during an incident                   IC / Ops Lead / Comms Lead, cited
   M5.5  Hands-on: the severity matrix               EXECUTED: SEV tied to real burn rate
   M5.6  On-call, with honesty about its cost        full HN quote, what it does and doesn't solve
   M5.7  Hands-on: a deterministic on-call rotation  EXECUTED: oncall/schedule.py, real output
   M5.8  Project: INCIDENT-RESPONSE-PLAN.md          EXECUTED: the complete document

#	Lesson	What it builds
1	Introduction (this one)	Why the framework gets built before the case, not after
2	The lifecycle of an incident	Detection → declaration → response → mitigation → resolution, cited from Google SRE
3	Severity levels, with real Andes Cargo examples	SEV1-4, from "two more seconds" to "`Shipments` isn't responding," narrative
4	Roles during an incident	Incident commander, communications, operations — cited from Google SRE
5	Hands-on: Andes Cargo's severity matrix	Executed: each SEV tied, number for number, to a burn rate level from `ALERTING-POLICY.md`
6	On-call, with honesty about its cost	The full HN quote about the real cost of switching tools
7	Hands-on: a deterministic on-call rotation	Executed: `oncall/schedule.py`, literal output, two identical runs
8	Project: Andes Cargo's `INCIDENT-RESPONSE-PLAN.md`	Executed: the complete document — lifecycle, severity, roles, rotation

This module's thread: no new math, all the human layer

Unlike Module 4 — where each lesson built a different alert engine on the same math — this module doesn't calculate a single new burn rate number. It reuses, unmodified, the three thresholds ALERTING-POLICY.md already left tested: Page (fast) at 14.4x, Page (slow) at 6x, Ticket at 1x. What this module builds is the layer that gives human meaning to those three numbers: which severity each one corresponds to (lesson 3, formalized in lesson 5), who responds when they cross the threshold (lesson 4), and who is on call that specific week to respond (lessons 6 and 7).

This module's honesty: what actually runs, what is a reasoned document

Piece	Status in this module	Exact technical reason
Lifecycle (lesson 2)	Cited, with its own diagram	Official Google SRE source, no infrastructure to run — it's a conceptual framework, not a calculation.
Narrative severities (lesson 3)	Cited + narrative	Andes Cargo examples in prose, still without the formal arithmetic — that arrives in lesson 5.
Roles (lesson 4)	Cited, with its own diagram	Same nature as lesson 2: conceptual framework from the official source.
Severity matrix (lesson 5)	Executed (manual arithmetic, no script)	Same pattern as Module 1, lesson 6: real hand calculation, verifiable line by line, with no infrastructure needed.
On-call quote (lesson 6)	Cited, verified against the source	Hacker News, real comment from `jamiemallers`, verified for this lesson.
`oncall/schedule.py` (lesson 7)	Executed, real	Pure Python, no external dependencies, run with `python3` to write this lesson.
`INCIDENT-RESPONSE-PLAN.md` (lesson 8)	Executed, real	The complete document, verified with `wc -l`/`grep -c`, just like `SLO.md` and `ALERTING-POLICY.md`.

The rule governing this entire guide holds: if a number appears in a lesson, it's traceable to a real calculation or a cited source — never a figure invented to make the example look complete.

Common mistakes

Treating this module as "just paperwork," without the same demand for precision as Module 2 or Module 4 (underestimating a module with no central script). What happens: someone, seeing that this module doesn't build any new central scripts/*.py until lesson 7, assumes the content is opinion with no possible verification. How to spot it: if you treat lesson 5's severity matrix as a list of preferences, instead of a table where every row derives from a burn rate threshold already calculated and tested in Module 4. How to fix it: every severity in this module has an exact number behind it — 14.4x, 6x, 1x, the same three from ALERTING-POLICY.md — and lesson 7's rotation is indeed a real script, actually run. "Little code" doesn't mean "no rigor": this module is framework and documents, with the same deterministic discipline as the rest of the guide.

Assuming that "building the framework beforehand" means it's never going to change once written (reading INCIDENT-RESPONSE-PLAN.md as a frozen document). What happens: someone concludes that, once this module closes, Andes Cargo's severity and roles stay fixed forever, with no possibility of adjustment. How to spot it: if your reading of this module is "this never gets touched again." How to fix it: Module 8 explicitly runs a new synthetic incident against this same framework to confirm it still works — a real incident response framework gets reviewed after every real incident (Module 7 builds exactly that mechanism, with the postmortem's action items), it isn't written once and abandoned.

Expecting this module to already narrate the Claude Code incident (anticipating Module 6 too early). What happens: someone, seeing "Claude Code" mentioned in this module's examples, expects lesson 3 or lesson 5 to already classify that specific incident with severity and roles assigned. How to spot it: if you're searching, in this module's lessons, for a date or a timeline of the real incident. How to fix it: this module builds the generic framework, with hypothetical Andes Cargo examples (sample shipments, latency scenarios) — never the real case. Module 6 is, specifically, the one that takes this now-finished framework and applies it, for the first time, to the Claude Code incident with its real dates and figures.

Exercises

Exercise 1 — Explain, in your own words and without using the word "paperwork," why building this framework in Module 5 (before Module 6) produces a better response to the real incident than building it inside Module 6 itself.

See solution

Building the framework beforehand separates two kinds of decision that, mixed under pressure, interfere with each other: deciding what severity to use, who is the Incident Commander, and how an on-call rotation gets built are design decisions, which benefit from time, comparison of alternatives, and the absence of real-time pressure. Applying those already-made decisions to a specific incident — reading the facts, classifying them against the already-existing matrix, assigning the already-defined roles to actual people — is a completely different kind of work, much faster and with less margin for error, because it doesn't require inventing the structure at the same time as responding to the emergency. Module 6 is going to be faster and more reliable, not in spite of this module, but because of it.

Exercise 2 — This module's map says lesson 5's severities tie "number for number" to ALERTING-POLICY.md's thresholds. Without having read Module 4's lesson 2 yet, what three numbers would you expect to find reused in this module?

See solution

14.4x (Page (fast)), 6x (Page (slow)), and 1x (Ticket) — the three thresholds from Google SRE's Table 5-8 that Module 4, lesson 2 quoted in full, and that Module 4, lesson 3 (burn_rate_evaluator.py) implemented and actually ran over Module 2's two fixed scenarios. This module doesn't invent a fourth threshold or change any of the three — it inherits them exactly as ALERTING-POLICY.md left them, and only adds the question "so who responds, and how fast?" on top of each one.

Exercise 3 — A classmate argues that, since Andes Cargo is a learning project with no real traffic or real customers, building a complete INCIDENT-RESPONSE-PLAN.md is a pointless exercise. How would you respond, using this module's reason for existing?

See solution

This module's value doesn't depend on Andes Cargo having real production traffic — it depends on the discipline of building the framework before the case, with roles and severities traceable to real numbers, being exactly the same discipline a technical interviewer, or a real team, would expect to see applied to any system, large or small. Besides, Module 6 is indeed going to apply this framework to an incident with completely real, verified data — the DataTalks.Club one — and Module 8 is going to test it against a deterministic synthetic incident. INCIDENT-RESPONSE-PLAN.md isn't an isolated exercise in imagination: it's the portfolio piece that proves whoever wrote it knows how to design an incident response framework from scratch, with evidence, not just quote one from memory.

Summary and next step

This lesson installed the reason this entire module exists: the incident response framework — lifecycle, severity, roles, on-call — gets built before operating the real Claude Code incident, exactly like an evacuation drill gets designed before the smoke appears, never during. You saw the complete map of the 8 lessons, the thread connecting them — no new math, all the human layer on top of the three burn rate thresholds ALERTING-POLICY.md already left tested — and this module's honesty table.

Before moving on you should be able to: explain why building this framework beforehand produces a better response than improvising it during the incident; name the three burn rate thresholds this module is going to reuse without changing them; and defend, against a real objection, why this framework is worth it even though Andes Cargo has no real production traffic.

Lesson 2 opens the framework with its first piece: the complete lifecycle of an incident, cited directly from Google SRE's incident management guide.

Resources

This same repository, Module 4, lesson 8 (08-project-andes-cargos-alerting-policy.md) — ALERTING-POLICY.md, the three burn rate thresholds this module reuses without changing them.
This same repository, Module 1, lesson 8 (08-project-andes-cargos-reliability-charter.md) — RELIABILITY-CHARTER.md, the map row this module resolves.
Google SRE — Incident Management Guide — the source this module's lesson 2 quotes in full.
Google SRE Book — Managing Incidents — additional depth on roles and separation of responsibilities, cited in lesson 4.

← Previous8. Project: Andes Cargo's alerting policy Next →2. The lifecycle of an incident