Module 7: Blameless Postmortems And Runbooks
5. What a runbook is (and is not)
Description
POSTMORTEM.md looks backward: what happened, why, what went right, and what went wrong. A runbook looks forward: what to do, step by step, the next time a specific condition repeats — without needing to understand, from scratch, all the context of the incident that motivated it. This lesson establishes that distinction precisely, with a directly cited market source, before lesson 6 builds Andes Cargo's first real runbook.
Connection to the module
Lesson 4's action item #3 — writing runbooks/manifest-processor-error-rate.md — is exactly the deliverable this lesson conceptually prepares. Lesson 6 can't build a good runbook without this lesson first establishing what distinguishes a good runbook from a numbered imitation of a postmortem.
The analogy: the emergency checklist taped to the cockpit
A commercial pilot doesn't memorize, from pure memory, what to do for every possible airplane failure — there are too many combinations, and under real pressure memory fails in unpredictable ways. Instead, every cockpit has an emergency checklist, physical or digital, with numbered steps, clear conditionals ("if cabin pressure drops below X, then..."), and no paragraph of historical context about why that list exists. Someone half-asleep, in the middle of the night, with an alarm sounding, can follow it without thinking — literally without needing to think, because thinking under real pressure is exactly what the list exists to replace.
An SRE runbook serves the same function, for the same kind of moment: Module 4's alarm fires at 3 AM, the on-call person (Module 5) doesn't need to reconstruct, from scratch, what that alarm means or why it exists — they follow the exact steps someone, with a clear head, already thought through beforehand and verified work.
POSTMORTEM vs RUNBOOK -- TWO DOCUMENTS, TWO DIFFERENT MOMENTS
POSTMORTEM (Lesson 3) RUNBOOK (Lesson 6)
─────────────────────── ────────────────────
Written AFTER an incident, Written BEFORE the next
with time to think carefully incident, to be used IN A HURRY
│ │
▼ ▼
Looks backward: "what happened, Looks forward: "what do I
and why?" do, step by step, right now?"
│ │
▼ ▼
Dense with context, root cause, Dense with actionable steps,
analysis -- read once, calmly, sparse on context -- followed
to learn under pressure, without thinking
│ │
▼ ▼
Success = understanding the Success = resolving the
system better than before specific condition fasterStep 1 — The definition, cited from the market
PagerDuty — already cited in this guide's Module 4 as a production alert's real destination — publishes a public runbook definition worth quoting in full, because it precisely distinguishes two terms often confused:
"A runbook is a detailed 'how-to' guide for completing a commonly repeated task or procedure within a company's IT operations process."
And, on the specific difference from a playbook — a broader-scope document that can include several runbooks:
"A playbook deals with the overarching responses to larger issues and events, and can include multiple runbooks and team members as part of the complete workflow."
The same source offers an analogy almost identical to this lesson's, with a useful twist: if a runbook is a recipe, the playbook is the complete guide for organizing the event — the recipe resolves one specific component of a larger, orchestrated response. Lesson 6's runbook — what to do when the manifest-processor-error-rate alarm fires — is, precisely, a recipe: it resolves one specific condition, not "how to respond to any Andes Cargo incident in general" (INCIDENT-RESPONSE-PLAN.md already covers that, this guide's closest equivalent to a playbook).
Step 2 — What a good runbook contains, per the market
The same source cites Tom Limoncelli's work — a recognized reference in systems operations — on the sections a complete runbook should have:
| Section (Limoncelli, quoted by PagerDuty) | What it answers |
|---|---|
| Service overview | What is this service, in one sentence? |
| Service build information | How is this service built/deployed? |
| Instructions for deploying the software | Concrete deployment steps |
| Instructions for common tasks | The most frequent routine operations |
| "Pager playbook" (monitoring alerts, step by step) | What to do when a specific alert fires |
| Disaster recovery plans | What to do in the event of total loss |
| Service level agreement | What service level is promised |
Lesson 6's runbook — manifest-processor-error-rate.md — corresponds precisely to this list's fifth row: a "pager playbook" for a specific alert, not a document trying to cover all seven rows at once. This lesson adopts that same narrow-scope discipline: a complete, useful runbook doesn't need to be a complete service manual — it needs to resolve one specific condition well.
Step 3 — The three properties the market source demands, and why each matters under pressure
The same source is explicit about what makes a runbook actually work, not just exist:
"Runbooks should be clear and simple [...] use easy to understand language [...] be specific and unique to your processes [...] remain flexible and adaptable to changes."
Three of these four properties deserve an explanation of why they matter specifically under pressure, not just in the abstract:
- Clear and simple, with easy-to-understand language: at 3 AM, with an alarm sounding, anyone's cognitive processing capacity is reduced — a runbook written with the same density as an architecture document forces rereading every step twice, exactly the time a runbook exists to save.
- Specific and unique to your own processes: a generic runbook, copied from a blog or a template, almost never matches the real system exactly — this module's lesson 6 builds the runbook against the real alarm, the real SNS, Andes Cargo's real commands, not a generic version of "how to respond to a Lambda with errors."
- Flexible and adaptable to changes: a runbook nobody updates when the system changes becomes, over time, worse than having no runbook at all — because it generates false confidence in steps that no longer apply. This property is why lesson 6 precisely labels which parts of the runbook are literal (steps that actually ran) and which are representative (
awslocalwithoutLOCALSTACK_AUTH_TOKEN) — a runbook honest about its own limits is easier to keep updated than one that fakes a certainty it doesn't have.
Step 4 — Why a runbook and a postmortem can't merge into a single document
It's worth precisely resolving why this guide — and the SRE discipline in general — insists on keeping these two documents separate, instead of combining them into one "more complete" document:
| Property | Postmortem | Runbook |
|---|---|---|
| When it's read | Once, after the incident, to learn | Repeatedly, every time the condition repeats |
| Under what pressure it's read | No rush, time to reflect | In a hurry, under an active alarm |
| What context density it needs | High — root cause, what went right/wrong | Low — actionable steps, minimal context |
| What happens if it's outdated | Still a valid historical record | Becomes actively dangerous — steps that no longer apply, followed under pressure |
| Who writes it | The Incident Commander, after the incident (Module 5, lesson 4) | Anyone with the operational knowledge, before it's needed |
The last row is, perhaps, the most important difference in practice: an outdated postmortem is still useful as history; an outdated runbook is dangerous, because someone under real pressure can follow a step that no longer matches the current system, with real consequences. Merging both documents into one would mix content that needs to be kept precisely updated (the operational steps) with content that is, by nature, a fixed historical record (the analysis of an incident already past) — the separation isn't bureaucracy, it's the same single-responsibility discipline that already governs every document in this guide.
Common mistakes
Writing a runbook that opens with two pages of historical context about the incident that motivated it (confusing runbook with postmortem, structurally). What happens: someone, writing lesson 6's runbook, opens the document narrating the entire Claude Code incident before reaching the first actionable step. How to spot it: if someone has to read more than a couple of lines before finding the first step they can execute. How to fix it: historical context — why this runbook exists, what incident motivated it — belongs, at most, to a one-line reference to the corresponding postmortem, not a complete narration. Step 1's source is explicit: a runbook is a "how-to" guide, not a "why we got here" document.
Assuming a runbook, once written, never needs to be reviewed again (losing Step 3's "flexible and adaptable" property). What happens: someone treats lesson 6's runbook as a document finished forever, with no process for updating it when the alarm, command, or infrastructure it describes changes. How to spot it: if the runbook mentions an alarm name, a command, or a threshold that no longer matches current observability.tf. How to fix it: every time andes-cargo-infra/ changes something a runbook describes — an alarm's name, a threshold, a diagnostic command — that runbook needs a corresponding review, the same maintenance discipline any piece of infrastructure as code already demands.
Writing a generic runbook, copied from a public template, without adapting it to Andes Cargo's real commands and names (violating Step 3's "specific and unique" property). What happens: someone writes steps like "check your service's logs" or "verify the relevant metric," without naming the exact command, the exact alarm name, or the exact table involved. How to spot it: if your runbook could apply, with no changes, to any project other than Andes Cargo. How to fix it: this module's lesson 6 builds the runbook against andes-cargo-manifest-error-budget-burn-rate (Module 4, lesson 5's exact alarm name), process-shipment-manifest (the exact Lambda), and awslocal commands with real resource names — never a generic version that forces someone, under pressure, to "translate" the step to the real system.
Exercises
Exercise 1 — Classify each of the following documents from this guide as a runbook, a postmortem, or neither, and justify each answer with this lesson's Step 4: TIMELINE.md, INCIDENT-RESPONSE-PLAN.md, runbooks/manifest-processor-error-rate.md.
See solution
TIMELINE.md isn't exactly either one — it's a chronological record of facts, the postmortem's input, but without the root-cause section or the action items that define a complete postmortem (Module 6, lesson 8 already established it: "does not analyze root cause — that is Module 7"). INCIDENT-RESPONSE-PLAN.md looks more like what this lesson's source calls a playbook — a broader-scope framework (lifecycle, severity, roles) that can include multiple specific runbooks, not a step-by-step procedure for a single condition. runbooks/manifest-processor-error-rate.md is, precisely, a runbook: scoped to a single condition (Module 4's alarm fires), with actionable steps, minimal historical context, written before the next time the alarm actually fires.
Exercise 2 — Explain, using Step 1's PagerDuty quote, why INCIDENT-RESPONSE-PLAN.md (Module 5) looks more like a playbook than a runbook, even though this guide never uses that word to describe it.
See solution
Step 1's quote defines a playbook as something that "deals with the overarching responses to larger issues and events, and can include multiple runbooks and team members as part of the complete workflow" — exactly INCIDENT-RESPONSE-PLAN.md's description: it covers any incident's complete lifecycle (not a specific condition), coordinates multiple roles (IC, OL, CL), and is designed so that, within it, specific runbooks (like lesson 6's) exist and get invoked depending on the incident type. This guide never uses the word "playbook" to describe it because it isn't a term the rest of this ecosystem has adopted, but the structure — broad scope, coordination of several roles, capable of including multiple specific procedures — matches the market definition cited here.
Exercise 3 — A classmate argues that, since a runbook "doesn't need to explain the why," any runbook that includes a single sentence of context is badly written. Do you agree, using this lesson's Step 4?
See solution
Disagree, with a nuance. Step 4 doesn't say a runbook can never include context — it says the context density must be low, not zero, and that extensive context (the full "why") belongs to the postmortem, not the runbook. A single reference line — for example, "this alarm exists because it measures the same error ratio SLO.md defines as acceptable; see POSTMORTEM.md for the full analysis of the incident that motivated this document" — doesn't violate the "clear and simple" property: it helps whoever follows it understand, in one sentence, why the step matters, without forcing them to read a complete document before being able to act. What does violate the discipline is replacing the actionable steps with that context, or extending it beyond one or two reference lines.
Summary and next step
This lesson established the central distinction between a postmortem (looks backward, dense with context, read once calmly) and a runbook (looks forward, dense with actionable steps, followed under pressure), citing PagerDuty's market definition and Tom Limoncelli's seven sections of a complete runbook. You confirmed why both documents must be kept separate — an outdated postmortem is still valid history; an outdated runbook is actively dangerous — and the three properties a real runbook needs: clear and simple, specific and unique to the real system, flexible in the face of change.
Before moving on you should be able to: quote PagerDuty's runbook definition; explain the difference between a runbook and a playbook; and justify why merging a postmortem with a runbook into one document would be a design error, not just a style preference.
Lesson 6 builds, with this complete discipline, Andes Cargo's first real runbook: what to do when Module 4's andes-cargo-manifest-error-budget-burn-rate alarm actually fires.
Resources
- PagerDuty — What Is a Runbook? — the source of every verbatim quote in this lesson, including the runbook/playbook distinction and Limoncelli's seven sections.
- Google SRE Workbook — Postmortem Culture — the document this lesson contrasts with a runbook, already built in lesson 3.
- This same repository, Module 4, lesson 5 (
05-hands-on-a-real-cloudwatch-alarm-on-the-lambda.md) — the real alarm lesson 6's runbook is going to operate. - This same repository, Module 5, lesson 8 (
08-project-andes-cargos-incident-response-plan.md) —INCIDENT-RESPONSE-PLAN.md, the broader-scope document this lesson compares to a playbook.