Module 5: The Incident Lifecycle
4. Roles during an incident
Description
Lesson 2 gave the lifecycle; lesson 3 gave the severity. This lesson answers the third question: once an incident is declared and classified, who does what? The answer, quoted directly from Google SRE's incident management guide and deepened with the SRE Book's chapter on managing incidents, separates three functions that, under real pressure, almost always blur together if nobody distinguishes them on purpose beforehand: who decides, who executes, and who communicates.
Connection to the module
These three roles are the piece lesson 5 is going to assign to specific people inside INCIDENT-RESPONSE-PLAN.md, and the same piece Module 6 is going to put into action, with real people assigned, on the Claude Code incident. Without clear roles, every person in an incident ends up deciding, executing, and communicating at the same time — exactly the pattern that produced the Claude Code incident's full blast radius: a single agent with permission to decide and execute at the same time, with no second person coordinating the decision.
The analogy: a film director, not the camera operator
On a movie set, the director doesn't hold the camera. It's not because they don't know how a camera works — many directors do — it's because holding it would force them to look through a single lens, at a single angle, at the exact moment their real job is to see the entire scene: what's happening with the cast, with the lighting, with the next shot, with the whole day's schedule. The camera operator, in contrast, has exactly the opposite responsibility: focus completely on framing that specific shot well, without having to think, at the same time, about whether the lead actor is running late for the next scene.
An Incident Commander is that director. They coordinate, decide priorities, hold the complete view of the incident — and, precisely because of that, they are not the one running the mitigation commands with their own hands. That distinction isn't a hierarchical whim: it's the same practical reason a director doesn't hold the camera. Someone applying a real technical change, under pressure, with full attention on not breaking anything else, cannot simultaneously hold the complete view of the whole incident without losing something important in one of the two jobs.
The three roles, quoted
"Incident Commander (IC): coordinates the overall incident response." "Communications Lead (CL): provides regular updates to stakeholders and acts as a point of contact for incoming communications." "Operations Lead (OL): focus on mitigating the issue, minimize user impact, and resolving the problem."
Three roles, three responsibilities that don't overlap by design. The Incident Commander doesn't mitigate with their own hands; the Operations Lead doesn't unilaterally decide to change the whole incident's priority; the Communications Lead doesn't apply any technical change. The exact reason this separation matters, quoted in more depth from the SRE Book:
"A clear separation of responsibilities allows individuals more autonomy than they might otherwise have, since they need not second-guess their colleagues."
Each person, with a clear role, can act confidently within their own space without needing to constantly verify that nobody else is doing something contradictory. The same chapter describes the Operations role with a phrase worth quoting in full, because it resolves a real coordination problem on its own:
"[The Ops Lead] works with the incident commander to respond to the incident by applying operational tools [...] [and is] the only group modifying the system during an incident."
"The only group modifying the system" — not "the main group," the only one. This is the concrete rule that prevents the most chaotic possible scenario during a real incident: several people, each with good intentions, applying different changes to the same broken system at the same time, with none of them knowing what the other is doing. That exclusive coordination is, literally, what separates a well-handled incident from one that gets worse because of the response to it.
A fourth role, for longer incidents
The SRE Book names a fourth role Google uses in extended incidents, worth knowing even though INCIDENT-RESPONSE-PLAN.md (lesson 8) doesn't formalize it for Andes Cargo, given the team's small size:
"[The Planning Lead] supports Ops by dealing with longer-term issues, such as filing bugs, ordering dinner, arranging handoffs."
The detail about "ordering dinner" isn't a trivial example — it's literal confirmation that Google formally recognizes that an incident stretching over hours has real human logistical needs, not just technical ones. For a small team like Andes Cargo's, this responsibility doesn't disappear — it simply gets absorbed by the Incident Commander or the Communications Lead, whichever has more slack at that specific moment, instead of having a fourth dedicated person.
The three roles, in a diagram
THE THREE ROLES -- WHO DECIDES, WHO EXECUTES, WHO COMMUNICATES
INCIDENT COMMANDER (IC)
────────────────────────
Coordinates the full response.
Keeps the incident's state.
Does NOT execute technical changes.
│
┌────────────┴────────────┐
▼ ▼
OPERATIONS LEAD (OL) COMMUNICATIONS LEAD (CL)
─────────────────── ─────────────────────────
The ONLY one modifying Point of contact for
the system during the anyone not fixing the
incident. Applies the problem. Updates
runbook. Reports to the IC. stakeholders, translates
technical state into
clear language.No role has the authority to bypass another: if the Operations Lead needs to change the whole incident's priority, that's a decision for the Incident Commander, not something decided unilaterally while working the technical problem. If someone outside the incident asks "what's happening?", that question goes to the Communications Lead, it doesn't directly interrupt the Operations Lead mid-command.
Why the same small team doesn't mean the same problem
Andes Cargo, as RELIABILITY-CHARTER.md already acknowledged, is a small team — the real temptation is to think these three roles are "for big companies with lots of engineers available," and that a small team simply can't afford this separation. The flaw in that reasoning isn't about team size, it's about confusing number of people with number of roles. A SEV3/SEV4 incident, with only one person available, can have that person wearing all three roles — but even then, it's still useful for that person to know, at every moment, which of the three "hats" they're wearing: coordinating (what priority does this have?), executing (what command do I run now?), or communicating (who else needs to know this?). The separation stops being optional exactly at the moment an incident crosses into SEV1/SEV2 and more than one person is available: there, mixing deciding and executing in the same person is exactly the pattern that produced the Claude Code incident's blast radius.
Common mistakes
Assuming the Incident Commander must always be the person most technically expert in the broken system (confusing two different skills). What happens: a team reflexively picks, as IC, whoever knows the most about process-shipment-manifest's architecture, without considering whether that person coordinates well under pressure. How to spot it: if your criterion for choosing an IC is "who knows the most about this system," instead of "who coordinates best a response with incomplete information." How to fix it: this lesson's quote describes the IC coordinating the full response, not solving the technical problem with their own hands — that's precisely the Operations Lead's function. Often, the most technically expert person is more valuable as the Operations Lead, applying the fix with deep system knowledge, than distracted by the whole incident's general coordination.
Letting the Operations Lead also decide, without consulting the IC, to change the incident's priority or scope (breaking the separation under pressure). What happens: while applying a mitigation, the Operations Lead decides, on their own, that the incident isn't as serious anymore and lowers the urgency of the response, without running that decision by the Incident Commander. How to spot it: if a decision about the incident's severity or priority changed without it getting recorded in the document the IC maintains. How to fix it: this lesson's quote is explicit — the Ops Lead is "the only group modifying the system," but coordinating that modification with the IC is still mandatory. Role separation doesn't mean each role acts completely independently of the others; it means each has a clear responsibility, coordinated through the IC.
Not assigning any Communications Lead in a SEV1/SEV2 incident, letting updates come from whoever has free time at that moment (underestimating communication as secondary). What happens: during a serious incident, nobody is formally assigned to communicate status, so updates arrive late, inconsistent, or don't arrive at all, while the Operations Lead is busy applying the real mitigation. How to spot it: if, in your simulation of an incident, whoever is solving the technical problem is also the one answering questions from people outside the incident. How to fix it: the Communications Lead role exists exactly to prevent that interruption — without it, every question from an external stakeholder directly interrupts the Operations Lead, exactly the kind of under-pressure distraction this whole lesson exists to prevent.
Exercises
Exercise 1 — Classify each action under the correct role (IC, OL, or CL): (a) deciding an incident moves from SEV3 to SEV2 after new information; (b) rolling back process-shipment-manifest's broken deployment; (c) sending stakeholders an update saying "still investigating, next update in 30 minutes."
See solution
(a) Incident Commander — changing severity is a general incident-coordination decision, not a technical action or an external communication. (b) Operations Lead — it's literally "the only group modifying the system during an incident," per this lesson's quote. (c) Communications Lead — keeping stakeholders updated on a fixed cadence is exactly their quoted responsibility: "provides regular updates to stakeholders."
Exercise 2 — A two-person team responds to a SEV2. Explain how you would distribute the three roles across just two people, without losing the central separation this lesson defends (never deciding at the same time as executing).
See solution
A reasonable distribution: one person takes IC + CL (coordinating the response and communicating status — both are "full-view" functions, not direct technical execution, so combining them doesn't break the central separation), and the other person takes OL alone (applies the technical changes, without getting distracted coordinating general priorities or answering external questions). The combination this lesson explicitly avoids is merging IC with OL into the same person — that's the combination that mixes "deciding" with "executing," exactly the pattern the director-and-camera analogy warns doesn't work well, no matter how many more people are available.
Exercise 3 — Explain, using the SRE Book quote about "the only group modifying the system," what concrete problem gets avoided if an engineer who is NOT the Operations Lead decides, on their own initiative, to apply an additional change during an incident already in progress.
See solution
If more than one person modifies the system at the same time, neither one has the complete view of what changed, in what order, or whether their changes interact unexpectedly with each other — the same kind of chaos a film director avoids by keeping a single clear flow of decisions. An additional change, applied by someone other than the Operations Lead, can accidentally undo progress the Operations Lead had already made, apply a redundant mitigation, or — worse — introduce a second technical problem on top of the first, all while the Incident Commander believes there's only one active flow of changes because nobody told them about the second one. Restricting system modification to a single coordinated group is what makes it possible for the IC to maintain a reliable state of the whole incident, instead of blindly chasing uncoordinated changes.
Summary and next step
This lesson defined the three central roles of an incident response — Incident Commander (coordinates, doesn't execute), Operations Lead (the only one who modifies the system), Communications Lead (the external point of contact) — quoted from Google SRE's incident management guide and deepened with the SRE Book, including the explicit rule that only one coordinated group modifies the system during the incident. You saw why a small team like Andes Cargo's doesn't need more people to respect this separation — it needs each person to know which role they're thinking in at every moment.
Before moving on you should be able to: name the exact responsibility of each of the three roles; explain, with the director-and-camera analogy, why the IC doesn't execute technical changes with their own hands; and distribute the three roles across a team of two or three people without merging "deciding" with "executing."
Lesson 5 turns lessons 3 and 4's concepts into this module's first real deliverable: Andes Cargo's severity matrix, with each SEV tied, number for number, to an exact burn rate threshold from ALERTING-POLICY.md.
Resources
- Google SRE — Incident Management Guide — the definition of the three roles (IC, CL, OL) quoted at the start of this lesson.
- Google SRE Book — Managing Incidents — the separation of responsibilities, the Operations Lead's exclusive role, and the fourth role (Planning Lead) for extended incidents.
- This same repository,
terraform-and-iac-guide(Module 8) andcloud-security-and-guardrails-guide(Module 1) — the sibling guides that already narrated the Claude Code incident without this role vocabulary; this guide's Module 6 applies it for the first time.