Module 5: The Incident Lifecycle
2. The lifecycle of an incident
Description
This lesson answers the whole module's first question: when something breaks, what happens, in what order, and who decides what stage the system is in at each moment? The answer isn't invented here — it's quoted, directly, from Google SRE's incident management guide and from the SRE Book's chapter on managing incidents, the two official sources this lesson contrasts and synthesizes into five concrete stages: detection → declaration → response → mitigation → resolution.
Connection to the module
Each of these five stages reappears, under its own name, at a specific moment in Module 6: detection is what ALERTING-POLICY.md's alert already knows how to do (Module 4); declaration is the formal message Module 6, lesson 4 is going to write; response needs the roles this module's lesson 4 defines; mitigation is the decision tree Module 6, lesson 6 is going to reconstruct; resolution closes with the timeline Module 6, lesson 8 delivers. This lesson is the complete map; Module 6 walks through it, step by step, with the real incident.
The five stages, quoted and defined
1. Detection
Not everything that looks wrong is, automatically, an incident. The cited source gives an explicit criterion, not an intuition:
"[...] if any of the following is true, the event is an incident: Do you need to involve a second team in fixing the problem? Is the outage visible to customers? Is the issue unsolved even after an hour's concentrated analysis?"
Three questions, not a single technical signal: does it need a second team to fix it? Is it visible to customers? Is it still unresolved after an hour of focused analysis? If the answer to any of the three is "yes," it's already an incident — no matter how small it feels at first. Detecting, in this guide's vocabulary, is the moment when an ALERTING-POLICY.md alert fires (the technical data already exists), but detecting is not the same as declaring — that distinction is exactly the next stage.
2. Declaration
Declaring an incident is the formal act of saying, out loud and in writing, "this is an incident, and from now on it's handled as one." It's a step that often gets skipped by accident: someone sees the alert, starts investigating on their own, and never formally communicates that the system is in incident mode. Google SRE's incident management guide frames preparing for this moment as a complete, separate phase:
"Prepare for incidents [...]: Establishing alerting mechanisms and on-call processes before outages occur"
Preparation — already having the alerting mechanisms and on-call processes in place, exactly what Module 4 and this module's lessons 6-7 build — is what makes it possible to declare an incident in seconds, instead of minutes lost deciding whether "this counts" or not. Declaring is the exact point where ambiguity ends and the formal roles begin.
3. Response
With the incident declared, active coordination kicks in: someone takes on the Incident Commander role, someone else focuses on mitigating, someone else communicates. The source summarizes it as the phase of:
"Respond and manage incidents [...]: Active coordination and mitigation during the event"
This guide treats "response" as the moment when this module's lesson 4 roles come into play — it's not the technical fix itself yet, it's the coordination that makes it possible for the technical fix to happen without two people applying contradictory changes at the same time.
4. Mitigation
Mitigating is not the same as fully resolving — it's limiting the damage while the real cause is understood or corrected:
"[...] limiting the disruption caused by an incident and restoring normal business operations as quickly as possible."
A concrete example, on process-shipment-manifest: if a deployed version starts failing half its invocations, mitigating could mean rolling back to the previous deployment — the system works again, but the root cause of why the new version failed hasn't been thoroughly investigated yet. That arrives in Module 7's postmortem, not in this stage.
5. Resolution
An incident closes when the condition that triggered it is no longer active — not when "nobody is watching the alert anymore":
"[...] incidents are closed once mitigated."
Formally closing the incident is also the point where this guide's next complete discipline starts, still outside this module's scope: systemic learning. The same source calls it the "Remediate and learn from incidents" phase — resolution isn't the end of the story, it's the transition toward the blameless postmortem Module 7 builds.
The five stages, in a diagram
THE LIFECYCLE OF AN INCIDENT -- FIVE STAGES
DETECTION DECLARATION RESPONSE
────────── ─────────── ─────────
The burn rate Someone says, Lesson 4's
alert fires formally: roles come
(ALERTING-POLICY.md "this is an into play:
already built) incident" IC, Ops, Comms
│ │ │
└──────────┬──────────┴──────────┬──────────┘
▼ ▼
MITIGATION RESOLUTION
────────── ──────────
Damage is limited, The condition that
the service is triggered the incident
restored -- the root is no longer active --
cause is not yet the incident closes
fully resolved formally
│ │
└──────────┬──────────┘
▼
LEARNING (Module 7)
────────────────────
Outside this module's
scope -- the blameless
postmortem, built afterThe five stages aren't optional, and skipping one has a real cost: skipping declaration leaves a team responding informally, with nobody certain whether the incident is "already official" or not; confusing mitigation with resolution closes an incident before the system is truly healthy, leaving the real condition unresolved until the same alert fires again.
Common mistakes
Confusing "detection" with "declaration" (assuming the alert already is the incident). What happens: someone sees ALERTING-POLICY.md fire an alert and assumes that, by itself, is already a formally ongoing incident. How to spot it: if nobody in your scenario said, out loud or in writing, "this is an incident" before starting to work the problem. How to fix it: an alert firing is the signal that something could be an incident — this lesson's three quoted questions (does it need a second team? is it visible to customers? is it still unresolved after an hour?) are the real criterion for deciding whether it crosses over into formal declaration. Not every alert becomes a declared incident; every alert does deserve those three questions.
Treating "mitigation" as a synonym for "resolution" (closing too soon). What happens: someone rolls back a broken deployment, the system starts responding normally again, and calls the incident closed without investigating the root cause. How to spot it: if your incident closes with no plan at all for why what happened, happened. How to fix it: this lesson's cited definition is explicit — mitigating is "restoring normal operations as quickly as possible," not finding the root cause. Resolving means the condition that triggered the incident is truly no longer active, not just that the immediate symptom disappeared; this guide's Module 7 is precisely where the work of finding the full root cause lives, after resolution has already closed the active incident.
Skipping the "preparation" stage entirely, treating the lifecycle as if it started at detection (ignoring the first half of the cited source). What happens: someone reads this lesson's five stages and concludes that incident discipline begins when something breaks. How to spot it: if your incident response plan doesn't mention anything that exists before the first incident occurs. How to fix it: this lesson's cited source is explicit that "preparing" — already having the alerting mechanisms and on-call processes in place — is its own phase, prior to detection. This entire module, in fact, is that preparation phase for Andes Cargo: ALERTING-POLICY.md (Module 4) already built the alerting mechanisms; this module's lessons 6-7 build the on-call process. The five-stage lifecycle describes what happens once something breaks — but it only works well if the preparation already existed.
Exercises
Exercise 1 — Using the three quoted detection questions, decide whether the following scenario crosses into "incident": an Andes Cargo engineer notices, on their own, that a single test shipment took three seconds longer than normal to process, with no errors, and fixes it by checking their own laptop's connection in five minutes.
See solution
It doesn't cross into "incident." The three questions: did it need to involve a second team? No. Was it visible to customers? No — it was a test shipment, not even a valid event under SLO.md's SLI definition. Was it still unresolved after an hour of focused analysis? No, it resolved in five minutes, and the cause was local to the engineer's own laptop, not the system. None of the three conditions hold, so this scenario never needs to go through formal declaration, response, mitigation, or resolution — it's exactly the kind of low-level noise a well-defined incident lifecycle exists to not unnecessarily escalate.
Exercise 2 — Explain why this lesson's source places "declaration" as a step separate from "detection," instead of merging them into one. What is lost if a team treats both as the same moment?
See solution
Merging detection and declaration would assume that any technical signal that something is wrong automatically deserves the full weight of a formal incident — assigned roles, active communication, a tracking document — with no human judgment filter in between. Separating them lets detection happen far more often and with more sensitivity (any alert, any anomaly) than declaration, which requires crossing a higher bar (this lesson's three questions). Without that separation, a real team would end up declaring formal incidents over every minor blip, exhausting exactly the same kind of "alert fatigue" this guide's Module 4, lesson 1 already identified as the central problem with a badly designed threshold — just applied now to people, not to an alerting system.
Exercise 3 — A classmate argues that the "resolution" stage is unnecessary, because once mitigation restores the service, "the real work is already done." How would you respond, using this lesson's cited definition?
See solution
The cited definition of "resolution" — closing the incident only "once mitigated," not before — implicitly assumes that mitigation and resolution are distinct events that can happen at different times: the service can start responding again (mitigated) while the underlying condition that broke it is still present, just hidden behind the temporary mitigation (for example, a rolled-back deployment, but the bug that caused the original failure is still in the code, waiting for the next deployment). Treating mitigation as the end of the real work leaves that underlying condition with no formal closure, with the real risk that the same incident repeats as soon as the temporary mitigation is no longer in place. The resolution stage exists precisely to force the question "is the real cause no longer active, or did we just hide the symptom?" before calling the incident closed.
Summary and next step
This lesson defined the incident lifecycle's five stages — detection, declaration, response, mitigation, resolution — each quoted directly from Google SRE's incident management guide and the SRE Book, with the precise distinction between the ones that tend to get confused: detection isn't declaration, mitigation isn't resolution. This lesson's diagram connects the five stages in sequence, with systemic learning (Module 7) as the stage that comes next, outside this module's scope.
Before moving on you should be able to: name the five stages in order, without looking at the diagram; apply the three detection questions to a new scenario; and explain, with the cited source, why mitigation and resolution aren't the same thing.
Lesson 3 takes on this module's second central question: when an incident does get declared, how serious is it? Severity levels — SEV1 through SEV4 — with concrete Andes Cargo examples.
Resources
- Google SRE — Incident Management Guide — the three phases (prepare, respond, learn) and this lesson's main source.
- Google SRE Book — Managing Incidents — the three-question criterion for detection, and the definitions of mitigation and resolution cited.
- This same repository, Module 4, lesson 8 (
08-project-andes-cargos-alerting-policy.md) —ALERTING-POLICY.md, the detection mechanism this lesson treats as already built. - This same repository, Module 6 — the module that walks through these five stages, in order, with the real Claude Code incident.