Module 7: Blameless Postmortems And Runbooks
3. Hands-on: writing the Claude Code incident postmortem
Description
This is this entire guide's central document. TIMELINE.md (Module 6) already answered what happened, with what severity, with what roles, and at what cost measured in error budget. This lesson answers the question that document deliberately left open: why it happened, at the system level, following Google SRE's real postmortem template structure — summary, impact, root cause distinguished from trigger, what went well, what went wrong, action items — applied to the blameless criterion lesson 2 just established.
Connection to the module
POSTMORTEM.md cites TIMELINE.md, never repeats it — the same disciplined-assembly principle that has governed every document in this guide since SLO.md. This document's root-cause section leans, point by point, on the seven-failure chain cloud-security-and-guardrails-guide, Module 1, lesson 6 already built for this exact incident. This module's lesson 4 takes the brief action-items table this document leaves at the end and expands it with full SMART criteria: owner, priority, verification criteria.
Step 1 — Why "root cause" and "trigger" are two sections, not one
Google SRE's postmortem template precisely distinguishes two questions that sound similar but aren't: the trigger is the specific event that set the incident in motion; the root cause is the systemic condition that made it possible for that trigger to cause the damage it caused. Collapsing the two into one section produces a postmortem that "fixes" the trigger — in this case, it would be tempting to write "the problem was the stale state" — and leaves the system exactly as exposed as before, because the next trigger will never be exactly that same stale state. It will be something else — an exposed credential, a misread plan, a wrong configuration field — but, if the systemic root cause remains unresolved, the result will be the same kind of incident again.
TRIGGER vs ROOT CAUSE -- WHY "FIXING THE TRIGGER" ISN'T ENOUGH
TRIGGER (this specific incident) ROOT CAUSE (the system, in general)
───────────────────────────────── ─────────────────────────────────
A stale .tfstate file gets loaded No automated control, not
as if it were the infrastructure's dependent on human attention,
real state stands between "a destructive
action is proposed" and
"it executes"
│ │
▼ ▼
"Fixing" this: validate the "Fixing" this: a policy gate
specific file before loading (conftest) + a resource
it -- resolves THIS EXACT protection (prevent_destroy)
trigger -- resolves ANY trigger
attempting the same class
│ of damage
▼ │
The next, different trigger ▼
(another type of error) is The next, different trigger
NOT covered by this fix DOES get blocked, because the
control doesn't depend on
knowing in advance what it
will beStep 2 — The complete document
In incidents/2026-02-26-claude-code-destroy/, alongside TIMELINE.md, create POSTMORTEM.md:
# POSTMORTEM.md — 2026-02-26 Claude Code `destroy` Incident
**Status:** Final · **Severity:** SEV1 · **Postmortem owner:** Ana (Incident Commander)
**Real incident, external:** happened to DataTalks.Club, not to Andes Cargo. Analyzed here as a
drill against Andes Cargo's own reliability framework — see Module 6, lesson 1 and Module 7,
lesson 1 for the full honesty note on this distinction.
**Built on:** `incidents/2026-02-26-claude-code-destroy/TIMELINE.md` (Module 6, lesson 8). This
document does not re-derive any fact already established there — it cites, never repeats.
**Sources:** Alexey Grigorev, "How I Dropped Our Production Database" (alexeyondata.substack.com);
Hacker News #47278720; incidentdatabase.ai #1424; Tom's Hardware; `cloud-security-and-guardrails-
guide`, Module 1, lesson 6 (the seven-step failure chain this document's root cause section maps
onto).
## Summary
On 2026-02-26, an AI coding agent (Claude Code) executed `terraform destroy -auto-approve`
against DataTalks.Club's production infrastructure, using a stale state file as its only source
of truth. The agent had explicitly flagged the destructive action as risky before running it; the
human operator approved it anyway. VPC, ECS cluster, load balancers, bastion host, and the RDS
database — including all automated snapshots — were destroyed in a single command. The
`courses_answer` table (1,943,200 rows, 2.5 years of data) was gone. Recovery took exactly 24
hours and depended on an AWS-side snapshot that was not visible in the customer's own console.
## Impact
- **Duration:** 24 hours 0 minutes, `T+0` to `T+24h00m` (`TIMELINE.md`).
- **Scope:** total infrastructure loss — every service DataTalks.Club ran was unreachable from
`T+0` until partial restoration began.
- **Data:** `courses_answer`, 1,943,200 rows (2.5 years of student submissions, projects, and
course leaderboards), unrecoverable through any means under the team's own control.
- **Cost:** upgrade to AWS Business Support tier (~+10% of monthly cloud spend) to escalate the
recovery request.
- **Error budget (measured against Andes Cargo's own `SLO.md`, Module 6, lesson 7):** 1,440
minutes consumed against a 43.2-minute monthly budget — **33.3x** the full monthly budget in a
single event, a -1,396.8-minute deficit. Burn rate itself is not calculable (Module 6, lesson 3
and lesson 7): no infrastructure remained to generate measurable traffic once `T+0` executed.
- **Users:** the primary source does not report a precise affected-user count; the destroyed data
represents 2.5 years of accumulated course activity on a platform serving DataTalks.Club's full
learner base. This document does not fabricate a number the source does not provide.
## Detection
Immediate and total — every service became unreachable the instant `terraform destroy
-auto-approve` completed (`TIMELINE.md`, `T+0` to `T+~5min`). No monitoring system "detected" this
incident in the sense of an alert firing on a threshold: the infrastructure that would have hosted
any such alerting system was destroyed in the same command. Detection here was direct observation
of total outage, not instrumentation — a limit already named and analyzed in Module 6, lesson 3
(burn rate is mathematically undefined, `0/0`, when no traffic-generating infrastructure remains).
## Resolution
Summarized here; full branch-by-branch detail lives in `TIMELINE.md`'s "Mitigation decision tree"
section (Module 6, lesson 6). Two self-service recovery paths were checked and exhausted within
minutes: reverting via Terraform state (not applicable — the state itself was the cause, and the
infrastructure it described no longer existed) and locating an automated RDS snapshot in the
account's own console (none — destroyed alongside the database). The team escalated to AWS
Business Support (`T+~1h`); AWS confirmed an internal snapshot not listed in the console
(`T+~1h30min` — the single point of genuine uncertainty in the entire incident); phone escalation
followed (`T+~2h` to `T+~3h`); AWS restored the snapshot at `T+24h00m`, with all 1,943,200 rows
confirmed intact.
## Root cause(s) and trigger
Google SRE's postmortem template distinguishes a **trigger** (the specific event that set the
incident in motion) from **root cause(s)** (the systemic conditions that made the trigger possible,
or that let it cause the damage it caused). Collapsing the two into one produces a postmortem that
"fixes" the trigger and leaves the system exactly as exposed as before.
**Trigger:** during a routine task (migrating a static site into the existing Terraform project to
avoid $5-10/month of duplicate infrastructure), the AI agent loaded an outdated
`terraform.tfstate`, extracted from an archived `.zip`, as the active state file. Against that
stale state, `terraform plan` showed a long list of resources marked for *creation* — the agent's
own first anomaly signal — because the state no longer matched what actually existed in the
account.
**Root cause(s), at the system level, per the seven-step chain `cloud-security-and-guardrails-
guide` (Module 1, lesson 6) already mapped for this exact incident:**
1. **No automated validation existed between loading a state file and trusting it as ground
truth.** A stale state was accepted without any check against the account's real resources —
the first point where a systemic control, not a person's attention, could have stopped the
chain before it started.
2. **No automated, non-bypassable barrier existed between "a destructive plan is proposed" and
"it executes."** The agent *did* flag the risk (chain step 4) — an honest, correctly-functioning
signal. The chain's real failure is step 5: that signal could be approved past with a single
human decision, under normal-feeling work pressure, with no independent mechanical check behind
it. A policy-as-code gate evaluating the `plan` for delete actions on critical resources — the
exact mechanism `cloud-security-and-guardrails-guide` built for Andes Cargo as
`no-destroy-shipments.rego` (Module 4, lesson 6) — would have failed the pipeline job before
`apply` was even an option, independent of who or what proposed the change.
3. **No deletion protection existed on the database as a last line of defense.** Even if every
upstream control had failed, `deletion protection` (RDS) or `prevent_destroy` (Terraform
`lifecycle` block) would have blocked the actual delete operation at the resource level — the
last possible point of the chain, and the one `cloud-security-and-guardrails-guide` explicitly
left as a declared, unbuilt gap for Andes Cargo (Module 1, lesson 6): *"No guide in this
ecosystem builds it yet for Andes Cargo — it is, honestly, a declared gap, not a hidden one."*
4. **No backup mechanism existed under the team's own control.** Recovery depended entirely on an
AWS-side snapshot the team did not know existed and had not configured — not a self-service
backup the team owned and could restore on their own timeline (Module 6, lesson 6, Step 4).
None of these four root causes describes a person acting in bad faith. Per Module 7, lesson 2's
citation of Google SRE: *"You can't 'fix' people, but you can fix systems and processes."* Each of
these four is a system gap, not a character judgment — and each has a corresponding, concrete
action item (below, detailed in Module 7, lesson 4).
## What went well
- **The agent's warning was real and specific**, not a generic disclaimer — it explicitly flagged
`terraform destroy` as the proposed action before running it. The signal existed; the gap was
that a signal alone, without an independent enforcement mechanism, is information, not a
control.
- **Escalation to AWS support was fast** — the ticket opened within roughly an hour of the data
loss, with an immediate decision to upgrade to Business Support rather than wait on the standard
tier.
- **AWS's internal recovery mechanism worked, and worked completely** — all 1,943,200 rows were
confirmed intact, with no partial or silently corrupted recovery.
- **The recovery timeline was bounded and predictable once escalated** — exactly 24 hours, a
number precise enough that Grigorev could state it verbatim, which itself suggests AWS's internal
process, once engaged, followed a defined and repeatable procedure.
## What went wrong
- **No automated, non-bypassable control existed between a destructive plan being proposed and it
executing.** The single human approval at step 5 of the chain was, in practice, the entire
system's defense against this class of incident.
- **A stale state file was trusted without any validation against real infrastructure**, letting a
corrupted premise ("this infrastructure doesn't exist") propagate into a destructive action
without ever being checked against reality.
- **No deletion protection existed on the database**, so even a successful `destroy` execution had
no resource-level last line of defense.
- **Recovery depended entirely on a mechanism outside the team's knowledge and control.** The
favorable outcome was not the result of a safeguard the team had built — Module 6, lesson 6
already named this precisely: *"DataTalks.Club's recovery depended on a snapshot retention
mechanism on AWS's side that the affected person themselves didn't know existed."* A team
without that particular undocumented AWS behavior working in their favor would have faced
permanent data loss.
## Action items
Full SMART treatment — owner, priority, and verification criteria for each — is Module 7, lesson
4's subject; this section names them, without expanding them, as the direct output of the four
root causes above.
| # | Action item | Root cause it addresses |
|--:|---|---|
| 1 | Add a `prevent_destroy` lifecycle block to the `Shipments` table in `andes-cargo-infra/` | Root cause 3 — no deletion protection |
| 2 | Verify `no-destroy-shipments.rego`'s coverage extends to `terraform destroy` plans, not only `apply` plans containing delete actions | Root cause 2 — automated gate coverage |
| 3 | Write `runbooks/manifest-processor-error-rate.md`, Andes Cargo's first operational runbook | Root cause 1 and 2 — no documented, mechanical response path independent of who is on call |
| 4 | Evaluate and document DynamoDB backup/point-in-time-recovery options for `Shipments`, owned by Andes Cargo rather than depending on an undisclosed provider-side mechanism | Root cause 4 — no self-service backup |
## What this document does not do
It does not re-derive the incident's chronology, severity classification, assigned roles, or
error-budget arithmetic — all of that is `TIMELINE.md` (Module 6, lesson 8), cited by section
above, never repeated. It does not assign personal blame to the human operator or to the agent —
Module 7, lesson 2 already established, with Google SRE's own definition, why this document
instead asks what system-level gap let a good-faith decision, made with incomplete signals, cause
the damage it caused. It does not claim the four action items above are exhaustive — they are the
ones this postmortem's root-cause section directly supports; a real team's postmortem review might
surface others. It does not evaluate whether DataTalks.Club, in real life, should have had these
exact controls in place before 2026-02-26 — that is a judgment about a real company this guide has
no standing to make; the action items in this document are Andes Cargo's own response, built for
its own infrastructure.
## Consequences
Module 7, lesson 4 expands each action item into a SMART commitment with an owner and a priority.
Module 7, lesson 6 delivers action item #3 in full. Module 7, lesson 7 begins the honest work
behind action item #4. Module 7, lesson 8 packages all four alongside this document as Andes
Cargo's postmortem and runbooks deliverable.Step 3 — Verifying the document
wc -l POSTMORTEM.md
grep -c '^## ' POSTMORTEM.md
grep -c '^| [0-9]' POSTMORTEM.mdWhat to expect (literal — you assembled the content, the shape is deterministic):
171
10
4
One hundred seventy-one lines, ten sections (Summary, Impact, Detection, Resolution, Root cause(s) and trigger, What went well, What went wrong, Action items, What this document does not do, Consequences), and four action-item rows — the same deterministic verification pattern TIMELINE.md already established in Module 6.
Step 4 — Reading the document's most important section: why it has four root causes, not one
Notice something deliberate about the "Root cause(s) and trigger" section: it doesn't collapse the incident into a single explanation. Four root causes, each at a different point in the seven-failure chain — state validation, an automated gate before apply, resource-level protection, backup under the team's own control — because an incident of this magnitude almost never has a single cause that, if fixed, would have prevented it entirely. This isn't a stylistic choice — it's the same defense-in-depth discipline cloud-security-and-guardrails-guide already named in its Module 1, lesson 6: no single control is "the complete solution." A postmortem that reduces an incident of this scale to a single root cause is almost always oversimplifying, and that simplification has a real cost: every unnamed root cause is a root cause no action item is going to fix.
Common mistakes
Writing the "Root cause(s) and trigger" section with the trigger and the root cause mixed into a single paragraph (losing Step 1's distinction). What happens: someone writes "the root cause was the stale .tfstate file," without distinguishing that statement (correct as a trigger) from the systemic question of why no control stopped the damage once the trigger occurred. How to spot it: if your postmortem has a single cause section that doesn't distinguish "what specific event started this" from "what was the system missing in general." How to fix it: Step 2's document explicitly separates Trigger (one paragraph) from Root cause(s) (a numbered list of four points) — every root cause on the list stays valid even if the specific trigger (this particular .tfstate) never repeats in exactly the same form again.
Treating "What went well" as a decorative or optional section, written only to soften the tone (underestimating its real function). What happens: someone writes a minimal version of "What went well" — a single generic line — assuming the important section is only "What went wrong." How to spot it: if your "What went well" section can't point to any specific mechanism that actually worked. How to fix it: Step 2's document identifies four things that worked — the agent's real warning, the fast escalation, AWS's mechanism working completely, a predictable recovery timeline once escalated — each with real value: the section doesn't exist to make anyone feel better, it exists to identify which behaviors and mechanisms are worth preserving or reinforcing, not just what needs fixing.
Writing action items directly in this lesson, instead of leaving them as a brief table lesson 4 expands (jumping ahead of work that breaks the module's sequence). What happens: someone, seeing Step 2's short action-items table, immediately rewrites it with owner, priority, and date, without waiting for lesson 4. How to spot it: if your version of POSTMORTEM.md already has "Owner" and "Priority" columns in the action items section. How to fix it: Step 2's document, on purpose, leaves that table with only two columns — the action item and the root cause it addresses — and explicitly says "full SMART treatment [...] is Module 7, lesson 4's subject." Writing the full SMART criteria here would duplicate the next lesson's real work with nothing gained — the sequence (complete postmortem first, SMART action items after) exists so every document has one clear responsibility.
Exercises
Exercise 1 — Verify, by running Step 3's commands on your own copy of POSTMORTEM.md, that you get exactly 171, 10, and 4.
See solution
Copying Step 2's document exactly as it appears, wc -l POSTMORTEM.md counts 171 total lines (including blank lines between paragraphs and table rows), grep -c '^## ' finds ten level-2 section headers (Summary through Consequences), and grep -c '^| [0-9]' finds the four action-items table rows that start with a number between vertical bars. If your count differs, the most common cause is a whitespace character added or removed while copying the code block — the same kind of deterministic verification TIMELINE.md (Module 6, lesson 8) already established as this guide's standard.
Exercise 2 — A classmate argues that root cause #2 (no automated gate before apply) and root cause #3 (no deletion protection on the resource) are, at bottom, "the same root cause repeated twice." Do you agree?
See solution
Disagree. Both share the same general principle (an automated control, not dependent on human attention, would have stopped the damage), but they act at different points in the chain, with different consequences if only one of the two existed. Root cause #2 (a policy gate like conftest) acts before apply is even possible — if it works, the destroy never gets to execute. Root cause #3 (prevent_destroy/deletion protection) acts during execution itself, as a last line of defense if the earlier gate, for any reason (a bug in the policy, a misconfigured exception), didn't stop the action. A system with only root cause #2 resolved would still be exposed if that specific policy had a gap; a system with only #3 resolved would let the plan with the destructive action get generated and reviewed unnecessarily before failing. Defense in depth — the same principle cloud-security-and-guardrails-guide already named — demands both layers, not just one, exactly why Step 2's document lists them as separate action items.
Exercise 3 — Explain why the "What this document does not do" section explicitly includes "It does not evaluate whether DataTalks.Club, in real life, should have had these exact controls in place before 2026-02-26."
See solution
Without that explicit statement, a reader might interpret this document's four root causes and four action items as a direct criticism of the real decisions DataTalks.Club made before the incident — a real company, with a real founder, who already went through a genuinely difficult incident. That kind of retrospective judgment about a real external company entirely exceeds this exercise's authority and purpose: this guide has the authority to decide what controls Andes Cargo, its own fictional project, builds, but no authority or sufficient information basis to judge a real external company's actual operational decisions. Explicitly declaring this boundary — the same honesty discipline governing every document in this guide — keeps the root-cause analysis, which must be rigorous, from reading as an unsolicited judgment of real third parties.
Summary and next step
In this lesson you wrote this entire guide's central document: POSTMORTEM.md, with Google SRE's real template structure — summary, impact, detection, resolution, root cause distinguished from trigger (four root causes, not one), what went well, what went wrong, and a brief action-items table — built on top of TIMELINE.md without repeating any of its facts. You verified the document with this entire ecosystem's same deterministic pattern: 171 lines, 10 sections, 4 action items. You applied, point by point, lesson 2's blameless criterion: every root cause is a system gap, none is a judgment about a person.
Before moving on you should be able to: explain the difference between trigger and root cause with this incident's concrete example; recite the four root causes from memory; and defend why "What went well" isn't a decorative section.
Lesson 4 takes this document's brief action-items table and expands it with full SMART criteria — owner, priority, and how you'd know, with evidence, that each one actually got done.
Resources
- Google SRE Workbook — Postmortem Culture — the template structure (summary, impact, detection, resolution, root cause, lessons learned, action items) this document follows.
- Google SRE Book — Postmortem Culture — the blameless definition applied in every section, quoted in full in lesson 2.
cloud-security-and-guardrails-guide, Module 1, lesson 6 (06-blast-radius-revisited-what-would-have-stopped-it.md) — the source of the seven-failure chain the root-cause section maps.- This same repository, Module 6, lesson 8 (
08-project-andes-cargos-final-timeline-for-this-case.md) —TIMELINE.md, the cited source for every fact in this document. - Alexey Grigorev — How I Dropped Our Production Database — the primary source for every verified fact in this document.