Module 7: Blameless Postmortems And Runbooks

1. Introduction: from the timeline to systemic learning

Description

TIMELINE.md was finished at Module 6's close: fourteen facts, SEV1 severity verified, roles assigned, the complete mitigation decision tree, and the error budget recalculated with the real tool — 33.3x the monthly budget. That document precisely answers what happened and in what order. This module answers a different question, one Module 6 deliberately left untouched: why it happened, at the system level, and what changes to make it less likely to happen again. That question has a name in the SRE discipline, and a very specific way of being answered badly: a postmortem that ends up pointing at a person instead of a system.

Connection to the module

Everything this module needs already exists: TIMELINE.md (Module 6, lesson 8) is the direct input for lesson 3, this guide's central postmortem. INCIDENT-RESPONSE-PLAN.md (Module 5) already established the roles that draft that postmortem. cloud-security-and-guardrails-guide, Module 1, lesson 6 already precisely named the declared gap that this module's lessons 4 and 6 turn into a real action item and a real runbook, respectively. This module doesn't recalculate anything from Module 6 — it doesn't touch the error budget, the severity, or the mitigation tree again — it builds, on top of that already-finished work, the three artifacts Andes Cargo had been missing since Module 1: a blameless postmortem, action items that actually get done, and this whole ecosystem's first operational runbook.

The analogy: an airplane's black box, not a pilot's file

When an airplane has a serious incident, there's a body — in the United States, the National Transportation Safety Board; in most countries, an equivalent authority — whose only job is to investigate what happened. That body has access to the black box: the flight data recorder, the cockpit voice recorder, every instrument, every word spoken in the preceding minutes. And yet, the investigation doesn't exist to determine whether the pilot "is guilty." It exists to reconstruct, with forensic precision, the complete chain of conditions — mechanical, procedural, human, regulatory — that made the outcome possible, and to produce recommendations that make it less likely for the same chain to repeat on any other airplane, with any other pilot, at any other airline.

That investigation does document that the pilot made a specific decision at a specific moment — that's a fact, not an opinion, and hiding it would produce a useless investigation. What that investigation does not do is stop there. The question that truly drives the investigation isn't "did the pilot decide badly?" — a question that, in isolation, leads to no systemic change — but "what information did they have available at that moment, what operational pressure were they facing, and what safeguard, had it existed, would have made that decision impossible to make, no matter who was in the cockpit?" A blameless postmortem asks that exact same question about a software incident.

   TWO INVESTIGATIONS OF THE SAME FACT -- THE DIFFERENCE THAT MATTERS

   AN INVESTIGATION SEEKING A CULPRIT         A BLAMELESS INVESTIGATION (this guide)
   ─────────────────────────────────          ─────────────────────────────────
   Central question:                          Central question:
   "who made the bad decision?"                "what made that decision seem
                                                 reasonable at the time?"
        │                                            │
        ▼                                            ▼
   Ends in a name and a                        Ends in a list of system
   personal consequence                        controls (Lesson 4: prevent_destroy,
                                                 extended policy, new runbook)
        │                                            │
        ▼                                            ▼
   The next person, under the                  The next person, under the same
   same pressure, can make the                  pressure, encounters a system
   same decision -- nothing about               that makes the same decision
   the system changed                           harder or impossible

What you're going to build in this module

Eight lessons, with the second-heaviest EXECUTED weight of the whole guide, alongside Module 3:

#	Lesson	What it produces
1	This introduction	The module's complete map, and the central distinction between a blameless postmortem and an investigation seeking someone to blame
2	What makes a postmortem "blameless"	The definition quoted from `sre.google/sre-book/postmortem-culture/`, applied to the exact point where the Claude Code incident gets tested: the human approval of the `destroy`
3	Hands-on: writing the postmortem	`incidents/2026-02-26-claude-code-destroy/POSTMORTEM.md` — this guide's central document, built on top of `TIMELINE.md`
4	Action items that actually get done	SMART criteria applied to real action items, with an owner and priority — including the `prevent_destroy` gap on `Shipments` that `cloud-security-and-guardrails-guide` left declared
5	What a runbook is (and isn't)	The distinction from a postmortem, with cited market examples
6	Hands-on: the first real runbook	`runbooks/manifest-processor-error-rate.md` — what to do when Module 4's alarm fires
7	Hands-on: the honest backup/restore attempt	`awslocal dynamodb create-backup`/`restore-table-from-backup` on a disposable copy of `Shipments`, with the real result documented exactly as it comes out
8	Project: the postmortem and runbooks package	The four artifacts integrated — this module's final portfolio piece

Why this module comes after Module 6, not before it or alongside it

It might seem more efficient to write the postmortem at the same time as reconstructing the timeline — after all, you're already looking at the same facts. This ecosystem, deliberately, doesn't do it that way, for a discipline reason worth understanding before writing lesson 3's postmortem's first line: reconstructing the facts and analyzing their cause are two different activities, with two different risks if mixed. If you wrote the root cause while still confirming the exact order of events, retrospective bias (hindsight bias, already named in Module 6, lesson 6) would contaminate the reconstruction itself: you'd know from the start "how the story ends," and without noticing you'd reorder or reinterpret early facts to better fit the conclusion you'd already reached. Separating the two modules — Module 6 reconstructs without analyzing cause, Module 7 analyzes on top of an already-closed reconstruction — is the same discipline an aviation incident investigator applies: first the complete black box, with no interpretation; afterward, and only afterward, the cause analysis.

An honesty holding up this entire module

The Claude Code incident, again, happened to DataTalks.Club, not to Andes Cargo — the same boundary Module 6, lesson 1 already precisely established, and that this module inherits without repeating the full explanation. Lesson 3's postmortem analyzes root cause of the real incident, with the same verified sources as always; lesson 4's action items, in contrast, are indeed real for Andes Cargo — prevent_destroy on Shipments, lesson 6's runbook — because this guide has the authority to build new controls over its own project, even though it doesn't have the authority to change what already happened to DataTalks.Club. Every lesson in this module, wherever it matters, keeps that distinction visible.

Common mistakes

Starting lesson 3 thinking "blameless" means "without analyzing the human decision" (over-correcting toward vagueness). What happens: someone, afraid of sounding accusatory, writes a postmortem that entirely avoids mentioning that a human approved the destroy, leaving the root cause vague and useless ("there was a process problem"). How to spot it: if your postmortem draft can't precisely point to the exact step in the chain where a specific decision changed the outcome. How to fix it: this module's lesson 2 is going to be explicit about this — a blameless postmortem does document that the human approved the destroy, as a verified fact; what it doesn't do is treat that approval as the end of the analysis. "The human approved the destroy" is necessary information; "the human is guilty" is a conclusion a blameless postmortem never writes, because it leads to no system change.

Treating this module as a repeat of Module 6 under a different name (losing the boundary between reconstructing and analyzing). What happens: someone, reading lesson 3's postmortem, expects to find the complete fourteen-fact timestamped table again, already present in TIMELINE.md. How to spot it: if your version of the postmortem repeats the entire timeline instead of citing it. How to fix it: the same disciplined-assembly principle INCIDENT-RESPONSE-PLAN.md and TIMELINE.md already applied — cite, don't repeat — also governs the postmortem: the summary and impact sections lean on TIMELINE.md, without reconstructing its fourteen facts from scratch.

Assuming this module's final goal is "finding someone responsible for Andes Cargo not adding prevent_destroy" (reintroducing blame through the back door, at the project level instead of the person level). What happens: someone drafts lesson 4's action items with a tone of "this should have been done earlier," instead of "this is what the system needs now." How to spot it: if your action item's wording includes any assessment of why it wasn't done earlier, instead of limiting itself to what's going to be done, by whom, and when. How to fix it: a SMART action item (lesson 4) is, by design, a forward-looking statement — what changes, not why it didn't change before; the same blameless discipline governing the whole postmortem extends, without exception, to how each of its action items is written.

Exercises

Exercise 1 — In your own words, explain the difference between "the human approved the destroy" and "the human is guilty," using this lesson's black-box analogy.

See solution

"The human approved the destroy" is a verifiable fact, the same kind as "the pilot activated autopilot at a specific moment" — necessary information for understanding the complete sequence of events, without which the investigation would be incomplete or outright false. "The human is guilty" is a conclusion that assigns moral or professional responsibility to a person, and that — just like in the aviation analogy — leads to no verifiable system change: punishing or singling out that person doesn't make the next person, under the same pressure and with the same absence of automatic safeguards, make a different decision. A blameless postmortem retains the first kind of statement (necessary, verifiable) and explicitly discards the second (unverifiable, not actionable) — the same distinction that separates a flight safety body's investigation from an airline's internal disciplinary investigation.

Exercise 2 — Explain why separating fact reconstruction (Module 6) from cause analysis (Module 7) into two different modules protects against retrospective bias, instead of just being an arbitrary organizational decision.

See solution

Retrospective bias (hindsight bias) makes it almost automatic, once an event's outcome is known, to reinterpret early facts as if they'd been "obvious signals" of that outcome — Module 6, lesson 6 already named this risk when warning against treating the path to recovery as "obvious from the start." If fact reconstruction and cause analysis happened in the same step, whoever writes the document would already know, from the timeline's first line, "how this ends" and "why it happened" — and that certainty would contaminate, without the person noticing, how they describe the early facts: the order would subtly become narrative instead of forensic. Finishing the complete timeline first, with no cause conclusion yet written (the exact discipline Module 6, lesson 8 closed with "does not analyze root cause — that is Module 7"), forces the sequence of facts to stand on its own, independently verifiable from any later conclusion.

Exercise 3 — A classmate argues that, since Andes Cargo is a fictional project, this module's lesson 4 action items "don't count as real" the same way the incident's postmortem does analyze a real case. Do you agree?

See solution

Disagree, with the exact distinction this lesson already drew. The postmortem (lesson 3) analyzes root cause for an incident that happened to a real company (DataTalks.Club) — there, "real" refers to the facts investigated. The action items (lesson 4), in contrast, are real in a different sense: they're genuine changes this module declares over Andes Cargo's real infrastructure within this ecosystem — prevent_destroy on Shipments, lesson 6's runbook — with the same level of execution (valid HCL, verified document) as any other artifact in this guide. Andes Cargo being a fictional project doesn't make the artifacts this guide builds on top of it fictional, the same way a systems exercise in a LocalStack lab isn't "less real" just because the AWS account behind it (000000000000) isn't anyone's production account.

Summary and next step

This lesson established the whole module's central distinction: a blameless postmortem documents facts — including a specific human decision, when that decision is part of the chain — without turning those facts into an assignment of personal blame, because personal blame produces no verifiable system change. You saw the map of the eight lessons, each building an artifact Andes Cargo had been missing since this guide's Module 1, and the design reason this module comes after, not alongside, Module 6's fact reconstruction.

Before moving on you should be able to: explain, in your own words, the difference between a verified fact and an assignment of blame; name the four artifacts this module is going to produce; and explain why separating reconstruction from analysis protects against retrospective bias.

Lesson 2 deepens the very definition of "blameless," quoted directly from the source that coined it as formal engineering practice: the Google SRE book.

Resources

Google SRE Book — Postmortem Culture — the source of the definition lesson 2 quotes in full.
This same repository, Module 6, lesson 8 (08-project-andes-cargos-final-timeline-for-this-case.md) — TIMELINE.md, lesson 3's direct input.
cloud-security-and-guardrails-guide, Module 1, lesson 6 (06-blast-radius-revisited-what-would-have-stopped-it.md) — the source of the declared prevent_destroy gap lesson 4 turns into an action item.
Alexey Grigorev — How I Dropped Our Production Database — the primary source supporting every fact in lesson 3's postmortem.

← Previous8. Project: Andes Cargo's final `TIMELINE.md` for this case Next →2. What makes a postmortem "blameless"