Mitigate first, diagnose later

Description

Lesson 4's runbook had a detail that was left not fully explained: the mitigate step (shutting off the kill switch) comes before any investigation into why latency broke. This lesson names that order as a general principle of incident response, not just a particular decision in the recommendations runbook: mitigate first, diagnose later — never the other way around, no matter how curious or urgent it feels in the moment to understand the root cause.

Connection to the module. Lessons 2 and 3 gave you the tool (the kill switch) and the criterion (rollbackDecision()) to mitigate fast. Lesson 4 showed the order within a concrete runbook. This lesson explains why that order matters, with numbers that show the real cost of inverting it — before lesson 6 talks about who to notify, and lesson 7 measures how fast the response was overall.

An analogy: putting out the fire before investigating why it started

A kitchen fire doesn't wait for someone to determine whether it started from a frayed wire, a pan with too much oil, or a poorly calibrated stove. The first reaction, the one any safety training teaches, is to put it out — with an extinguisher, cutting the gas, whatever fits the type of fire — not to investigate its origin while the flames keep growing. The investigation into the cause — checking the electrical wiring, examining the pan, calibrating the stove — comes afterward, with the fire already out, with no one breathing smoke in the meantime.

Diagnosing the root cause of a broken guardrail, with the guardrail still broken and real users still exposed, is investigating the fire's origin while it's still burning. It can be done — but every minute spent investigating instead of putting it out is one more minute of real damage, accumulating on real people, while legitimate curiosity about "what exactly happened" competes with the urgency of making it stop.

Worked example: the real cost of inverting the order

// simulateResponse: compares the ORDER of actions -- diagnose first vs mitigate
// first -- on the same recommendations incident. The mitigation itself is
// identical in both orders (shutting off the kill switch); what changes is WHEN
// it happens within the sequence.
function timeToMitigateFor(steps) {
  let elapsed = 0;
  for (const step of steps) {
    elapsed += step.minutes;
    if (step.isMitigation) return elapsed;
  }
  return null;
}

function totalElapsed(steps) {
  return steps.reduce((sum, s) => sum + s.minutes, 0);
}

const orders = {
  diagnoseFirst: [
    { action: 'read logs and dashboards looking for the root cause', minutes: 35 },
    { action: 'confirm hypothesis: query without an index on the recommendations join', minutes: 40 },
    { action: 'only now, shut off the kill switch', minutes: 2, isMitigation: true },
  ],
  mitigateFirst: [
    { action: 'confirm the guardrail is broken (step 2 of the runbook)', minutes: 3 },
    { action: 'shut off the kill switch immediately', minutes: 2, isMitigation: true },
    { action: 'with traffic already protected, now look for the root cause', minutes: 75 },
  ],
};

console.log('=== the same actions, two different orders ===\n');
Object.keys(orders).forEach((order) => {
  console.log(order + ':');
  let elapsed = 0;
  orders[order].forEach((s) => {
    elapsed += s.minutes;
    console.log('  ' + s.action + '  (t=' + elapsed + 'min)' + (s.isMitigation ? '  <- MITIGATED HERE' : ''));
  });
  console.log('');
});

console.log('=== the real cost of the order ===\n');
Object.keys(orders).forEach((order) => {
  const ttm = timeToMitigateFor(orders[order]);
  const total = totalElapsed(orders[order]);
  console.log(order.padEnd(14) + 'timeToMitigate=' + String(ttm).padStart(3) + 'min   total time to root cause=' + total + 'min');
});

const diff = timeToMitigateFor(orders.diagnoseFirst) - timeToMitigateFor(orders.mitigateFirst);
console.log('\nDifference in timeToMitigate: ' + diff + 'min more of users exposed to the broken latency, just from the order of the actions.');

What to expect. Running the file with Node, the output is exactly this:

=== the same actions, two different orders ===

diagnoseFirst:
  read logs and dashboards looking for the root cause  (t=35min)
  confirm hypothesis: query without an index on the recommendations join  (t=75min)
  only now, shut off the kill switch  (t=77min)  <- MITIGATED HERE

mitigateFirst:
  confirm the guardrail is broken (step 2 of the runbook)  (t=3min)
  shut off the kill switch immediately  (t=5min)  <- MITIGATED HERE
  with traffic already protected, now look for the root cause  (t=80min)

=== the real cost of the order ===

diagnoseFirst timeToMitigate= 77min   total time to root cause=77min
mitigateFirst timeToMitigate=  5min   total time to root cause=80min

Difference in timeToMitigate: 72min more of users exposed to the broken latency, just from the order of the actions.

This result is, perhaps, the most important one in the entire module, and it's worth reading carefully. The same three actions happen in both orders — confirm, mitigate, investigate; the only thing that changes is which position in the sequence the mitigation falls into. With diagnoseFirst, the kill switch doesn't activate until minute 77 — 25,000 users stayed exposed to the broken latency for over an hour, while the team investigated. With mitigateFirst, the kill switch activates at minute 5 — the damage stops almost 15 times faster.

And notice the second number, the one almost no one expects: the total time to finding the root cause is practically the same in both orders — 77 minutes versus 80. Mitigating first didn't make investigating the root cause take much longer (80 versus 77, a difference of just 3 minutes, the time of the confirmation step). What did change, and drastically, was how long users stayed exposed while that investigation happened. That's exactly the argument for mitigating first: it costs almost nothing in diagnosis speed, and it saves an enormous amount of real exposure.

Why "mitigate first" isn't the same as "don't investigate"

It's easy to misread this principle as "the root cause doesn't matter" — and that would be as serious a mistake as the one this lesson corrects. Look again at the mitigateFirst order: the third step, "look for the root cause," is still there, and is still necessary — no one settles for "we already shut off the flag, no need to understand what happened." The difference isn't whether you investigate, but when: with the damage already stopped, or while the damage is still happening. Investigating calmly, after mitigating, also has an advantage the example doesn't show explicitly but is worth naming: the investigation itself tends to be better when there's no pressure of "every minute that passes affects more people" pushing you to rush conclusions — the same rush that can lead to a hasty, wrong diagnosis.

This principle, moreover, is exactly what justified the order of lesson 4's runbook: the mitigate step comes in position 3, before any investigation step — not because investigating doesn't matter, but because the correct order always resolves the ongoing damage first, and only afterward devotes time to understanding why it happened.

Common mistakes

Diagnosing the root cause with the site down, instead of mitigating first. What happens: facing a broken guardrail, someone on the team starts directly reviewing logs, running diagnostic queries, or testing hypotheses about the cause — while exposure stays active and affects real users — postponing mitigation until they have a clear explanation. Why it happens: understanding the problem feels like an engineer's "real" work, and shutting something off without knowing exactly why it broke can feel, incorrectly, like an incomplete or unrigorous solution. How to spot it: if at some point during an active incident someone says "wait, let me understand this first" before having mitigated, that's the mistake happening in real time. How to fix it: as this lesson's contrast shows — 77 minutes versus 5 — mitigating first costs almost nothing in diagnosis speed and saves an enormous amount of exposure; the correct order is mitigate, and then investigate with the same or greater depth.

Confusing "mitigate first" with "no need to investigate the root cause." What happens: the team shuts off recommendations with the kill switch, breathes a sigh of relief, and no one revisits why latency broke in the first place — as if mitigating were equivalent to resolving. Why it happens: the immediate relief of seeing the guardrail return to normal can feel like the end of the work, when in reality it's only the end of the first half. How to spot it: if after mitigating no one is assigned to investigate the root cause, the underlying problem is exactly where it was — just without users exposed for now — ready to repeat the next time someone ramps the rollout up again. How to fix it: the runbook's mitigate step was never the last step — the investigation is still needed, it just happens afterward, without the pressure of an active incident.

Measuring the response's success only by how fast the root cause was found, without measuring how long exposure stayed active. What happens: when reviewing how a past incident's response went, the team congratulates itself because "we found the problem in under an hour" — the total time to diagnosis — without noting or reporting how much of that time users were exposed to the problem without mitigation. Why it happens: time to diagnosis is easier to count as a technical success story than exposure time, which sounds more uncomfortable to report. How to spot it: if an incident report mentions "we found the cause in X minutes" but doesn't separately mention "we mitigated in Y minutes," it's missing the metric that actually matters to the affected users. How to fix it: as in this example's output, always report the two figures separately — timeToMitigate and the total time to full diagnosis; they're different questions, and the first one almost always matters more to the people who were exposed.

Exercises

Exercise 1 — Change the cost of confirming. In the mitigateFirst order, the first step ("confirm the guardrail is broken") takes 3 minutes. If that confirmation step took 20 minutes instead of 3 — imagine the dashboard is slow to update — what would the new timeToMitigate be for mitigateFirst? Would it still be much better than diagnoseFirst?

See solution

The new timeToMitigate would be 20 + 2 = 22 minutes (the confirmation step plus the mitigation step, the only one marked isMitigation: true). Compared to diagnoseFirst's 77 minutes, 22 minutes is still dramatically better — less than a third of the time — although no longer as extreme as the original 5 versus 77 contrast. This shows something important: the "mitigate first" principle still holds even if the confirmation step becomes slower — what would change, in that case, isn't whether mitigating first is still better, but exactly how much of an advantage it gives. It's also worth separately investigating why the dashboard is so slow to confirm a broken guardrail — that's a real improvement too, though different from the decision of how to order the steps.

Exercise 2 — Find the case where diagnosing first would actually make sense. Think (no code needed) of a hypothetical situation where mitigating before understanding the problem could, by itself, cause more damage than waiting a bit to diagnose. Describe it in 2-3 sentences.

See solution

One possible situation: if the available "mitigation" weren't a low-risk kill switch like recommendations's, but an action with its own high or risky cost — for example, restarting an entire database without knowing whether that could cause data loss in some transaction stuck mid-processing. In that case, "mitigate first" with no diagnosis at all could trade a latency problem for a much worse one. The key lesson doesn't change: it's still true that the full diagnosis should be postponed as much as reasonably possible — but "mitigate first" assumes the available mitigation is low-risk, like a flag's kill switch. When the mitigation itself is risky, at least a minimal diagnosis is needed — not the full root-cause diagnosis, but enough to know the mitigation won't make things worse.

Exercise 3 — Explain the principle using the fire analogy. In two or three sentences, explain to someone who never worked in software why an engineering team shuts off a broken feature first and only afterward investigates why it broke, using the kitchen fire analogy.

See solution

One example answer: "When there's a fire in the kitchen, you put out the fire first — with an extinguisher, cutting the gas — and only afterward do you investigate whether it was a frayed wire or a pan with too much oil. Nobody stays investigating the cause while the flames keep growing. We do the same thing: when something breaks for our users, we shut it off first — so it stops affecting them — and only afterward, calmly and without anyone else getting hurt in the meantime, do we investigate exactly what caused it." The central idea: the order protects the affected people first, at almost no cost to the quality or speed of the investigation that follows.

Summary and next step

In this lesson you named and measured the central principle of every incident response: mitigate first, diagnose later. With timeToMitigateFor() you compared two orderings of the same actions on the same recommendations incident — diagnoseFirst mitigated only at minute 77, mitigateFirst mitigated at minute 5 — and you saw that this 72-minute difference in avoided exposure cost almost nothing in diagnosis speed (80 minutes versus 77).

Before moving on you should be able to: explain why mitigating first doesn't mean "don't investigate the root cause"; distinguish timeToMitigate from the total time to full diagnosis, as two separate metrics; and name a situation where the available mitigation is, itself, risky enough to justify a minimal diagnosis before acting.

You already know what to do (rollbackDecision()), how to execute it without improvising (the runbook), and in what order (mitigate before diagnosing). Lesson 6 adds the organizational piece that's missing: not every incident is the same — some warrant waking up the entire on-call team, and others can wait until the next business day. How is that decided, and who finds out about each type?

Resources

• Google SRE Book, Chapter 13, "Emergency Response" — sre.google/sre-book/emergency-response. Documents real Google cases where acting fast to contain damage, before understanding the full cause, made the difference between a controlled incident and a much bigger one. In English.
• PagerDuty Incident Response Documentation, "What is an Incident?" — response.pagerduty.com/before/what_is_an_incident. Establishes the distinction between containing an incident's impact and resolving its root cause as two separate phases of the process — the formal basis for this lesson's principle. In English.
• Charity Majors, "Deploys Are the WRONG Way to Change User Experience" — honeycomb.io/blog/deploys-wrong-way-change-user-experience. Reinforces, from the feature-flag perspective, why containing impact (shutting off the flag) must be separated from investigating the code that caused the problem. In English.