The runbook: the steps you don't improvise

Description

rollbackDecision() already knows how to decide what to do: revert, fix forward, or continue. But knowing what to do and executing it well under pressure are two different things. When the recommendations guardrail breaks at three in the morning, the on-call person doesn't have time — and shouldn't need it — to think from scratch "who do I notify first? do I confirm the guardrail before or after shutting off the flag? what do I tell the rest of the team?". This lesson builds the runbook: the sequence of steps, written and agreed on before the incident happens, that the team follows without improvising.

Connection to the module. This lesson's runbook doesn't invent any new step — it organizes into a fixed sequence what lessons 2 and 3 already built (the kill switch, rollbackDecision()) and what lessons 5, 6, and 7 still need to formalize (mitigating before diagnosing, severity, communication). This is the document that brings all those pieces together in the exact order they run during a real incident.

An analogy: the pilot's emergency checklist

A commercial pilot doesn't react to an emergency — an engine failure, a loss of cabin pressure — by thinking from scratch what to do. They react by pulling out a physical, printed checklist, with numbered steps followed in an exact order, without skipping any even if the pilot's memory tells them they already know what needs to be done. That checklist wasn't written in the moment of the emergency — it was written calmly, on the ground, by people who carefully thought through each step without the pressure of a cabin in alarm. The reason isn't that pilots don't trust their own judgment: it's that anyone's judgment, under real stress, at 10,000 meters up, is worse than that same judgment applied calmly, weeks earlier.

A software incident's runbook serves exactly that function. It isn't written while the guardrail is broken and 25,000 users are exposed — it's written beforehand, with the full team, thinking through each step without the pressure of the moment. When the incident hits, no one improvises the order: someone opens the runbook and follows it.

Worked example: runRunbook() on the recommendations incident

// runRunbook: executes the steps of a PRE-WRITTEN runbook, in order, against the
// incident's real state -- nobody improvises the order or skips steps mid-crisis.
function runRunbook(steps, state) {
  let current = { ...state };
  const log = [];
  for (const step of steps) {
    const result = step.action(current);
    current = { ...current, ...result.nextState };
    log.push({ id: step.id, name: step.name, outcome: result.outcome });
  }
  return { log, finalState: current };
}

const recommendationsRunbook = [
  { id: 1, name: 'acknowledge', action: (s) => ({
    outcome: 'on-call acknowledges the guardrail dashboard alert (M4)',
    nextState: { acknowledged: true },
  }) },
  { id: 2, name: 'confirm guardrail', action: (s) => ({
    outcome: s.p95Latency > s.ceiling
      ? 'guardrail CONFIRMED broken (' + s.p95Latency + 'ms > ' + s.ceiling + 'ms)'
      : 'guardrail within range -- not a real alert',
    nextState: { guardrailConfirmed: s.p95Latency > s.ceiling },
  }) },
  { id: 3, name: 'mitigate', action: (s) => ({
    outcome: s.guardrailConfirmed ? 'recommendationsFlag.enabled = false (kill switch, M2 L5)' : 'no action -- guardrail not confirmed',
    nextState: { mitigated: s.guardrailConfirmed },
  }) },
  { id: 4, name: 'verify', action: (s) => ({
    outcome: s.mitigated ? 'p95Latency returns to baseline (~650ms) for all users' : 'nothing to verify',
    nextState: { verified: s.mitigated },
  }) },
  { id: 5, name: 'communicate', action: (s) => ({
    outcome: s.verified ? 'status update sent: mitigated, SEV2, investigating root cause' : 'no update',
    nextState: { communicated: s.verified },
  }) },
];

const initialState = { p95Latency: 910, ceiling: 800 };
const { log } = runRunbook(recommendationsRunbook, initialState);

console.log('=== recommendations runbook on the rollout 10% incident ===\n');
log.forEach((l) => console.log('step ' + l.id + ' (' + l.name + '): ' + l.outcome));

What to expect. Running the file with Node, the output is exactly this:

=== recommendations runbook on the rollout 10% incident ===

step 1 (acknowledge): on-call acknowledges the guardrail dashboard alert (M4)
step 2 (confirm guardrail): guardrail CONFIRMED broken (910ms > 800ms)
step 3 (mitigate): recommendationsFlag.enabled = false (kill switch, M2 L5)
step 4 (verify): p95Latency returns to baseline (~650ms) for all users
step 5 (communicate): status update sent: mitigated, SEV2, investigating root cause

Notice something the code makes clear: each step depends on the state left by the previous one, not on a fixed list of disconnected actions. Step 3 (mitigate) only flips off the kill switch if step 2 confirmed the guardrail is actually broken — if step 2 had determined the alert was noise (p95Latency within range), step 3 would have done nothing, and the runbook would have stopped there, without needing a rollback that wasn't necessary. That chaining — each step builds on the previous one's confirmation, never acts blindly — is exactly what separates a real runbook from a simple list of suggestions.

Why the runbook has exactly these five steps, in this order

The order isn't a coincidence, and each step has a reason to exist that the next lessons dig into further:

1. ACKNOWLEDGE     a specific person takes responsibility -- nobody assumes "someone else already saw it"
2. CONFIRM          verify the guardrail is truly broken, not dashboard noise
3. MITIGATE         stop the damage -- kill switch, the fastest option (lesson 5: why it goes BEFORE diagnosing)
4. VERIFY           confirm the mitigation worked -- it's measured, not assumed
5. COMMUNICATE      report the status -- severity, what was done, what's next (lessons 6 and 7)

Notice that diagnosing the root cause of the broken latency doesn't appear in any step of this runbook. That isn't an oversight — it's the whole of lesson 5, previewed here in one sentence: an incident-response runbook stops at "mitigated and communicated," not "fixed at the root." Investigating why the query became slow is real, important work, but it doesn't block or delay any of these five steps — it happens afterward, with exposure already under control, without the pressure of affected users in that moment.

Common mistakes

Not having any written runbook, and trusting that "the team already knows what to do." What happens: an incident occurs, and each person involved makes different decisions about what to do first — someone tries to diagnose, another person tries to notify leadership, a third looks for the kill switch button without knowing whether the guardrail has actually been confirmed broken. Why it happens: a competent technical team reasonably trusts its own individual judgment — but a live incident isn't the best moment for five individual judgments to spontaneously converge on the same order of actions. How to spot it: if you ask two people on the team to describe, from memory, the exact steps they'd follow when the recommendations guardrail breaks, and they describe different orders, no real runbook exists — an implicit expectation exists, which is the same as having nothing. How to fix it: write the runbook with the full team, calmly, before it's needed — exactly like the pilot's checklist is written on the ground.

Writing a runbook without verification steps — assuming "mitigate" and "verify the mitigation worked" are the same thing. What happens: the team has a runbook with a "shut off the flag" step, but no step afterward that confirms, with data, that the guardrail is back within range — it's assumed that shutting off exposure automatically solves the problem, without checking. Why it happens: the act of mitigating (flipping off the kill switch) feels so final that verifying its effect afterward seems redundant. How to spot it: if the runbook ends at the mitigation step, with no later step that reviews the guardrail dashboard, step 4 from this lesson's example is missing. How to fix it: as runRunbook() shows, the verify step isn't optional — it confirms with real data that the mitigation had the expected effect, instead of assuming it because "it should have worked."

Writing a runbook so detailed and rigid that no one can follow it under real pressure. What happens: someone drafts a twenty-step runbook, with extremely specific instructions for every possible variant of the incident, and in practice no one consults it during a real emergency because it's too long to read while the system is affected. Why it happens: it's tempting to anticipate every possible case in writing, thinking more detail is always better. How to spot it: if the runbook has so many steps or branches that no one can describe its general structure from memory, it's likely to end up ignored in a real incident. How to fix it: like this lesson's five-step runbook, a useful runbook fits in the team's head — few steps, each with a clear purpose, and room for human judgment within each step, not a decision tree impossible to memorize.

Exercises

Exercise 1 — Simulate the case where the alert was noise. Mentally run runRunbook() with initialState = { p95Latency: 720, ceiling: 800 } (a latency that's actually below the ceiling). What changes in the output from step 2 onward, and why does the runbook "stop" safely without anyone having to decide that manually?

Exercise 2 — Add a sixth step. Mercado's team wants to add an escalate step between mitigate (step 3) and verify (step 4), which only activates if mitigation took more than 10 minutes to confirm (imagine the state carries a mitigationMinutes field). Write, in JavaScript, the object for that new step, following the same format as the others.

{ id: 3.5, name: 'escalate', action: (s) => ({
  outcome: s.mitigated && s.mitigationMinutes > 10
    ? 'escalated to tech lead -- mitigation took longer than expected'
    : 'not escalated -- mitigation within expected time',
  nextState: { escalated: s.mitigated && s.mitigationMinutes > 10 },
}) }

Exercise 3 — Explain the runbook without using the word "runbook." In two or three sentences, explain to someone new on the team why Mercado has a sequence of steps written in advance to respond to recommendations's broken guardrail, instead of trusting each on-call person to decide on the fly.

Summary and next step

In this lesson you built runRunbook(), the five-step sequence — acknowledge, confirm guardrail, mitigate, verify, communicate — that executes, in order, with each step building on the previous one's state, the full response to the recommendations incident. You saw that each step depends on the previous one's confirmation — the runbook doesn't act blindly — and that none of the five steps includes diagnosing the root cause: that comes later, with exposure already under control.

Before moving on you should be able to: explain why a runbook is written before the incident and not during it; describe the difference between "mitigating" and "verifying the mitigation worked" as two separate steps; and modify runRunbook() to add a new step that depends on the state left by the previous ones.

This lesson's runbook mentions, in passing, that diagnosing the root cause doesn't block any of the five steps. Lesson 5 turns that observation into the central principle of every incident response: mitigate first, diagnose later — and shows, with numbers, how much it costs to invert that order.

Resources

• PagerDuty Incident Response Documentation, "During an Incident" — response.pagerduty.com/during/during_an_incident. The industry's practical reference on the structure of steps during an active incident — very close in spirit to this lesson's five-step sequence. In English.
• Google SRE Book, Chapter 14, "Managing Incidents" — sre.google/sre-book/managing-incidents. Explains why structured, predefined processes beat individual improvisation, even among highly skilled engineers, during a real incident. In English.
• Atlassian, "How we respond to an incident" — atlassian.com/incident-management/handbook/incident-response. Atlassian's public handbook on its own response process, with steps written in advance similar in structure to this lesson's runbook. In English.