Module 5: The Incident Lifecycle
8. Project: Andes Cargo's `INCIDENT-RESPONSE-PLAN.md`
Description
Seven lessons built each piece separately: the lifecycle (lesson 2), narrative severity (lesson 3) formalized with real arithmetic (lesson 5), the roles (lesson 4), and a deterministic on-call rotation, actually run (lesson 7). This module's final project doesn't calculate anything new — it brings the four pieces together, exactly as each lesson left them, into INCIDENT-RESPONSE-PLAN.md: the document RELIABILITY-CHARTER.md left as Module 5's Open row, and the complete framework Module 6 is going to operate against the real Claude Code incident.
Connection to the module
This document cites SLO.md (Module 2) and ALERTING-POLICY.md (Module 4) instead of repeating their definitions — the same single-source-of-truth principle ALERTING-POLICY.md already applied by citing SLO.md. Module 6 classifies the Claude Code incident against this document's exact matrix, assigns roles from this document's exact list, and operates on the exact lifecycle this document defines — nothing gets re-discussed or redesigned midway through Module 6.
Step 1 — Why this document invents no new number
The temptation, when writing a module's final document, is to add something that feels "more complete" — a fourth severity threshold, a fifth person in the rotation, a special exception for some hypothetical case. This document resists that temptation for the same reason SLO.md and ALERTING-POLICY.md already resisted it: every number that appears here was already calculated, cited, and verified in an earlier lesson of this module. INCIDENT-RESPONSE-PLAN.md is an exercise in disciplined assembly, not invention — proof that the previous seven lessons, together, already contain everything this document needs.
Step 2 — The complete document
At the root of andes-cargo-infra/, create INCIDENT-RESPONSE-PLAN.md:
# INCIDENT-RESPONSE-PLAN.md — process-shipment-manifest Incident Response Framework
**Status:** Accepted · **Governs:** Modules 6 through 8 of `sre-and-incident-response-guide`
**Source:** Module 5, lessons 2-7 (the incident lifecycle, severity levels, roles, the on-call
rotation)
**Related:** `SLO.md` (Module 2) defines the SLO and error budget this plan's severity matrix
measures against. `ALERTING-POLICY.md` (Module 4) defines the exact burn-rate tiers (Page fast
14.4x, Page slow 6.0x, Ticket 1.0x) this plan's severity matrix maps onto — cited here, never
redefined.
## Lifecycle
Every incident this plan governs moves through five stages (Module 5, lesson 2, citing Google
SRE's incident management guide and the SRE book's incident-management chapter): **Detection ->
Declaration -> Response -> Mitigation -> Resolution**. Module 6 operates the real Claude Code
`destroy` incident through all five stages, in order.
## Severity matrix
Maps directly onto the burn-rate tiers `ALERTING-POLICY.md` already built and drilled (Module 4,
lesson 8) — this plan adds no new alerting logic, only the human response each tier earns.
| Severity | Burn-rate tier (`ALERTING-POLICY.md`) | Budget if sustained | Response | Andes Cargo example |
|---|---|---|---|---|
| **SEV1 — Critical** | Page (fast), >= 14.4x — OR unrecoverable data loss, regardless of measured burn rate | 2% per hour (100% in 43.2 min at 1,000x) | Page on-call immediately. Incident Commander assigned. All hands as needed. | `Shipments` stops responding entirely — every `process-shipment-manifest` invocation fails (100% error rate = 1,000x burn rate; Module 4, lesson 2's own citation: a 1,000x burn rate "depletes budget in 43 minutes") |
| **SEV2 — Major** | Page (slow), >= 6.0x and < 14.4x | 5% per 6 hours | Page on-call. Response expected within the same shift. | A bad deploy exhausts both automatic retries on a meaningful share of manifest uploads, silently dropped — the exact pattern of `BAD_WEEK` (Module 2, lesson 7) |
| **SEV3 — Minor** | Ticket, >= 1.0x and < 6.0x | 10% per 3 days | Ticket filed. Triaged next business day. No page. | A traffic spike briefly throttles a handful of invocations against `ReservedConcurrentExecutions: 5`; all recover on automatic retry within seconds |
| **SEV4 — Low/cosmetic** | Below Ticket, < 1.0x — or not a bad event at all | Negligible | Logged only if the pattern repeats. No ticket, no page. | Shipment 4471's manifest takes 2 seconds longer to process during a brief S3 latency blip, completing in 4 seconds — well inside the 10-second `Timeout`, counted as a **good** event under `SLO.md`'s own SLI definition |
**When unsure which severity applies, default to the higher one** (Module 5, lesson 3, citing
PagerDuty's public incident-response documentation) — correcting a severity down after more
information arrives costs far less than correcting it up after minutes of silence.
## Roles
Three roles, never fewer than three people's worth of *responsibility* even when Andes Cargo's
small team means the same person sometimes wears two hats — but never both at once on the same
call (Module 5, lesson 4, citing `sre.google/resources/practices-and-processes/incident-
management-guide/`):
- **Incident Commander (IC)** — coordinates the overall response. Owns the incident document,
the timeline, and every open question about who is doing what. Does not personally run
mitigation commands.
- **Operations Lead (OL)** — the only person modifying `andes-cargo-infra/` during the incident
(citing `sre.google/sre-book/managing-incidents/`: the Ops Lead is "the only group modifying
the system during an incident"). Applies the runbook (Module 7), reports status back to the IC.
- **Communications Lead (CL)** — the point of contact for anyone not actively fixing the
problem. Issues updates on a fixed cadence, translates technical state into plain language.
For a SEV3/SEV4, one person can reasonably hold all three roles. For a SEV1/SEV2, at minimum the
IC and OL must be different people — coordinating and executing at the same time is exactly the
failure mode this separation exists to prevent.
## On-call rotation
Generated by `oncall/schedule.py` (Module 5, lesson 7) — deterministic, no SaaS, no randomness.
First 8 weeks, starting the Monday this plan takes effect:
```text
Week Starts Primary Secondary
1 2026-03-02 Ana Bruno
2 2026-03-09 Bruno Carla
3 2026-03-16 Carla Diego
4 2026-03-23 Diego Ana
5 2026-03-30 Ana Bruno
6 2026-04-06 Bruno Carla
7 2026-04-13 Carla Diego
8 2026-04-20 Diego Ana
```
The Secondary is always the Primary's successor in `ROSTER` — never the same person, always
predictable without consulting a calendar. Google SRE's own limit on on-call time (Module 1,
lesson 7: "no more than 25% can be spent on-call") governs this rotation's minimum size: four
people, each on primary one week in four, is 25% — the ceiling, not a comfortable margin under it.
## What this plan does not do
It does not replace the preventive controls `cloud-security-and-guardrails-guide` already built
(`conftest`, `prevent_destroy`) — this plan governs the response *after* something breaks, the
same boundary `RELIABILITY-CHARTER.md` (Module 1, Decision point 4) already drew for the error
budget itself. It does not staff Andes Cargo with real engineers — `ROSTER` is four example
names, not a real team. It does not select or configure any SaaS on-call tool (PagerDuty,
Opsgenie) — Module 5, lesson 6 names the real switching-cost risk of building directly inside
one, and this plan's own rotation logic is designed portable on purpose, so that risk never
materializes here.
## Alternatives considered
**Assign severity by gut feeling, case by case, instead of a fixed matrix.** Rejected: Module 5,
lesson 3 already showed the two symmetric failure modes of ungoverned severity — crying wolf on
every SEV1, or minimizing a real one — and `ALERTING-POLICY.md` already produces the exact
burn-rate number a fixed matrix needs. A matrix costs nothing to build twice over what already
exists.
**A single on-call person, no rotation, no secondary.** Rejected: no backup means a missed page
has no second line of defense, and one person absorbing 100% of on-call time triples Google SRE's
own 25% ceiling (Module 1, lesson 7) — an unsustainable design this plan explicitly avoids.
**Adopt a SaaS on-call platform now, instead of `schedule.py`.** Rejected, for now: Module 5,
lesson 6's citation on switching costs applies directly — adopting a platform before the team is
large enough to need its escalation/integration features risks exactly the lock-in `jamiemallers`
warned about on Hacker News. `schedule.py`'s plain, portable output can migrate into any real tool
later without rewriting the rotation logic itself.
## Consequences
Module 6 operates the real Claude Code `destroy` incident through the five-stage lifecycle this
plan defines, classifies it against this exact severity matrix, and assigns roles from this exact
list. Module 7's postmortem cites this plan directly. Module 8's capstone runs a new synthetic
incident through the same framework, unchanged, to confirm it works on a case nobody saw coming.Step 3 — Verifying the document
wc -l INCIDENT-RESPONSE-PLAN.md
grep -c '^## ' INCIDENT-RESPONSE-PLAN.md
grep -c 'Rejected' INCIDENT-RESPONSE-PLAN.mdWhat to expect (literal — you wrote the content, the shape is deterministic):
110
7
3
One hundred ten lines, seven sections (Lifecycle, Severity matrix, Roles, On-call rotation, What this plan does not do, Alternatives considered, Consequences), and three explicitly rejected alternatives — severity by gut feeling, a single on-call person, adopting a SaaS now — each with its own reason, none simply omitted without explanation. The same deterministic verification pattern SLO.md and RELIABILITY-CHARTER.md already established.
How to read this document, six months later
The same test SLO.md and ALERTING-POLICY.md already passed: facing this document, without asking anyone, a new reader should be able to answer five questions. What happens when something breaks, in what order? (Lifecycle section, five stages, each cited). How serious is this, with a number, not an opinion? (Severity matrix, each row tied to an exact burn rate threshold). Who decides, and who executes, and are they the same person? (Roles, with the explicit rule that IC and OL can never be the same person in a SEV1/SEV2). Who's on call this week, without having to ask anyone? (On-call rotation, verifiable with schedule.py itself). And what does this plan deliberately not solve? (What this plan does not do, three limits precisely named). If any of those five questions requires rereading a whole lesson of this module, the document failed its purpose.
Module 5's close
With INCIDENT-RESPONSE-PLAN.md written, this module delivers exactly what it promised in lesson 1: the complete incident response framework, built before the case, not during it — a lifecycle cited from Google SRE, severity tied to the same burn rate math ALERTING-POLICY.md already tested, roles that separate deciding from executing, and a real, deterministic on-call rotation, actually run. RELIABILITY-CHARTER.md can update its Module 5 row from Open to Resolved, citing this document as evidence. You enter Module 6 with everything you need already decided: when the Claude Code incident gets operated, every "so what do we do now?" question already has a written answer in this document, waiting to be applied.
Common mistakes
Treating INCIDENT-RESPONSE-PLAN.md as a document that gets completed once and never reviewed (a repeat of the same error already named with RELIABILITY-CHARTER.md, now with direct consequences for an operational document). What happens: someone assumes that, once this document is written, the severity matrix and the on-call rotation stay fixed forever, with no review mechanism. How to spot it: if your maintenance plan for this document is "never touch it again." How to fix it: this guide's Module 8 deliberately runs a new synthetic incident against this same framework, precisely to confirm it still works — a real incident response plan gets reviewed after every real incident, with the action items Module 7's postmortem produces, it's never written once and filed away.
Assuming the "Roles" section of this document assigns specific Andes Cargo people, instead of defining the role structure (reading the document as a contact list, not a framework). What happens: someone looks, in this document, for the real name of who would be the Incident Commander for a specific future incident. How to spot it: if you expect to find a proper name permanently assigned to the IC role somewhere in this document. How to fix it: the "What this plan does not do" section is explicit — this document defines the role structure (what each one does, and why they must be separated), not a fixed assignment of people to roles forever. Module 6 is, specifically, the one that assigns these roles to specific people for the real incident it's going to operate.
Copying this lesson's document without having actually run oncall/schedule.py, trusting that the rotation table "is probably correct" (a repeat of the same error already named with SLO.md and ALERTING-POLICY.md, Modules 2 and 4). What happens: someone pastes this document's "On-call rotation" section without having run their own copy of schedule.py in lesson 7. How to spot it: if you can't reproduce, by running your own script, exactly the same eight-week table cited here. How to fix it: every row in this document's rotation table comes from a script you yourself wrote and ran in lesson 7 — before accepting this document as final, confirm your own copy of oncall/schedule.py produces, line by line, the same output.
Exercises
Exercise 1 — Verify this document's "Severity matrix" section against your own calculation from lesson 5. Confirm that the SEV2 row (Page (slow), ≥ 6.0x and < 14.4x) corresponds to an observed error rate between 0.60% and 1.44%, using the formula error_rate = burn_rate × 0.001.
See solution
6.0 × 0.001 = 0.006 (0.60%) at the lower end, and 14.4 × 0.001 = 0.0144 (1.44%) at the upper end — matching exactly the range cited in this document's SEV2 row, and lesson 5's full table. This exercise confirms that INCIDENT-RESPONSE-PLAN.md's "Severity matrix" section is traceable to a formula executable by hand, not a table written from memory with no verification.
Exercise 2 — Defend the "Roles" section against a real objection. A technical interviewer asks: "with only four people on Andes Cargo's team, isn't formally defining three separate roles for every incident a waste of time?" How do you respond, using the document itself?
See solution
A complete answer: "This document's Roles section already answers that directly — for a SEV3 or SEV4, a single person can hold all three roles without a problem, so there's no waste at all in minor incidents. The hard rule only applies to SEV1/SEV2, where IC and OL must be different people, precisely because coordinating and executing at the same time, under real pressure, is exactly the pattern that produced the Claude Code incident's full blast radius: a single entity with permission to decide and execute at once, with no second person coordinating the decision. It's not that a small team needs more people — it's that, in the incidents that actually matter, it needs the people it does have to not mix two functions that, mixed together, already proved dangerous."
Exercise 3 — Explain why this document's "Alternatives considered" section includes "adopting a SaaS now" as an explicitly rejected option, instead of simply not building schedule.py and using a commercial tool from the start.
See solution
Explicitly naming and rejecting "adopting a SaaS now" leaves a record that the decision to build schedule.py was deliberate, with a concrete cited reason (lesson 6's switching-cost risk), not a budget limitation disguised as a technical decision. Without that entry, someone might assume Andes Cargo simply couldn't afford a commercial tool, and that the decision would change as soon as budget allowed. By naming it and rejecting it with a reason — adopting the platform before actually needing its escalation features risks exactly the lock-in the Hacker News quote describes — the document makes clear that portability was the criterion, not cost, and that the decision could legitimately be revisited the day the team does need those features — but always keeping schedule.py's portable layer underneath whatever tool gets adopted.
Summary and next step
In this module's final project you wrote INCIDENT-RESPONSE-PLAN.md: the five-stage lifecycle (cited from Google SRE), the complete severity matrix (each SEV tied to an exact burn rate threshold from ALERTING-POLICY.md), the three roles that separate deciding from executing (cited from Google SRE, with the hard rule of never merging IC and OL in a SEV1/SEV2), and the first eight weeks of a real, deterministic on-call rotation, run with oncall/schedule.py. You verified the document with the same discipline as this entire ecosystem: 110 lines, 7 sections, 3 rejected alternatives with evidence, no decision without justification.
Before closing this module you should be able to: recite the five lifecycle stages without looking at the document; classify any new Andes Cargo scenario into one of the four severities, with the arithmetic in plain sight; and explain, with the "Roles" section itself, why IC and OL can never be the same person in a SEV1/SEV2.
With this, sre-and-incident-response-guide's Module 5 is complete: the human incident response framework, built with the same deterministic discipline as the SRE math of the previous modules. Module 6 takes this document, unmodified, and applies it for the first time to a real incident: Claude Code's terraform destroy that erased 2.5 years of DataTalks.Club's data, operated start to finish through the five stages, the severity matrix, and the roles this module just finished defining.
Resources
- This module, lessons 2 through 7 — the direct source of every section of this document.
- This same repository, Module 1, lesson 8 (
08-project-andes-cargos-reliability-charter.md) —RELIABILITY-CHARTER.md, the row this document resolves. - This same repository, Module 2, lesson 8 and Module 4, lesson 8 —
SLO.mdandALERTING-POLICY.md, the two documents this document cites instead of repeating. - Google SRE — Incident Management Guide and Google SRE Book — Managing Incidents — the sources of this document's lifecycle and roles.