Module 7: Blameless Postmortems And Runbooks

8. Project: Andes Cargo's postmortem and runbooks package

Description

Four lessons built four artifacts separately: POSTMORTEM.md with root cause distinguished from trigger (lesson 3), four SMART action items with an owner and priority (lesson 4), Andes Cargo's first real runbook (lesson 6), and the honest result of the backup/restore attempt (lesson 7). This final project doesn't write any new document — it brings the four together into a single package, with an index confirming each piece serves its function, and with the same deterministic verification discipline every project in this guide has already demanded of itself.

Connection to the module

This document does, for Module 7's complete package, the same thing TIMELINE.md (Module 6, lesson 8) did for that module's six documents: assemble without inventing, cite without repeating. Module 8 — this guide's capstone — is going to cite this complete package as evidence that Andes Cargo's reliability machine (SLI → SLO → observability → alert → incident → postmortem → runbook) works end to end, not just in each module's theory separately.

Step 1 — The complete inventory, verified file by file

Before the portfolio document itself, confirm the four artifacts exist exactly where DISENO.md predicted:

find andes-cargo-infra/incidents/2026-02-26-claude-code-destroy -type f
find andes-cargo-infra/runbooks -type f

What to expect (literal — you created this structure in this module's lessons 3 and 6, plus Module 6's lessons 2 and 8):

andes-cargo-infra/incidents/2026-02-26-claude-code-destroy/TIMELINE.md
andes-cargo-infra/incidents/2026-02-26-claude-code-destroy/POSTMORTEM.md
andes-cargo-infra/runbooks/manifest-processor-error-rate.md

Three files, two directories — TIMELINE.md (Module 6) and POSTMORTEM.md (this module) share a folder because they document the same incident from two different angles; runbooks/ is a new directory, a sibling of incidents/, because a runbook doesn't belong to any specific incident — it exists for any future time the condition it describes repeats. Lesson 7's result (the backup/restore attempt) doesn't produce a new fourth file: it stays documented within the lesson itself, the same decision DISENO.md already made by not listing it among this guide's new artifacts.

Step 2 — The portfolio document

At the root of andes-cargo-infra/, create RELIABILITY-POSTMORTEM-PACKAGE.md:

# RELIABILITY-POSTMORTEM-PACKAGE.md — Module 7 Deliverable Index

**Status:** Final · **Governs:** the four artifacts this module produced, indexed together
**Built on:** `TIMELINE.md` (Module 6, lesson 8) — this package does not re-derive any fact
already established there or in `POSTMORTEM.md` itself.

## What this package contains

| Artifact | Path | Lesson | What it answers |
|---|---|---|---|
| Postmortem | `incidents/2026-02-26-claude-code-destroy/POSTMORTEM.md` | Module 7, lesson 3 | Why did this happen, at the system level — blameless, four root causes, distinguished from the trigger |
| Action items | (table inside `POSTMORTEM.md`, expanded in lesson 4) | Module 7, lesson 4 | What changes now, with an owner, a measurable criterion, and a deadline for each |
| Runbook | `runbooks/manifest-processor-error-rate.md` | Module 7, lesson 6 | What to do, step by step, the next time the Module 4 alarm fires |
| Backup/restore attempt result | Documented in lesson 7 itself, not a separate file | Module 7, lesson 7 | Whether Andes Cargo can recover `Shipments` on its own, without depending on an undisclosed provider-side mechanism |

## The four action items, current status

| # | Action item | Owner | Priority | Status as of this package |
|--:|---|---|---|---|
| 1 | `prevent_destroy` on `Shipments` | Bruno | P1 | Declared, not yet applied to `andes-cargo-infra/` — this guide's scope ends at the postmortem and runbook layer, not a new HCL commit to the shared infrastructure module |
| 2 | `no-destroy-shipments.rego` coverage for `terraform destroy` plans | Carla | P2 | Declared, verification test not yet written |
| 3 | `runbooks/manifest-processor-error-rate.md` | Ana | P1 | **Done** — delivered in full, Module 7, lesson 6 |
| 4 | DynamoDB backup/restore evaluation for `Shipments` | Diego | P2 | **Done, with an honest result** — Module 7, lesson 7 confirmed the network-level root cause directly, and documented the exact response shape a successful attempt would have, without confirming LocalStack's specific API coverage |

Two of four action items are fully delivered within this guide's own scope (#3, the runbook itself;
#4, the honest investigation it asked for). The other two (#1, #2) are infrastructure changes to
`andes-cargo-infra/` outside a documentation-and-runbook module's scope — declared with full SMART
detail, ready for whoever picks them up next, the same honest boundary `POSTMORTEM.md` already
drew between "root cause identified" and "root cause resolved."

## What a reader should be able to answer from this package alone

The same test `TIMELINE.md`, `SLO.md`, and `INCIDENT-RESPONSE-PLAN.md` already passed: **What
happened, and why, at the system level, without blaming a person?** (`POSTMORTEM.md`'s Root
cause(s) section, four causes, none a character judgment.) **What changes now, and who is
accountable for it?** (the action items table above, each with a single owner.) **What does
someone on call do the next time this specific alarm fires?** (`runbooks/manifest-processor-
error-rate.md`, seven steps, three branches.) **Can Andes Cargo recover this table on its own,
without relying on an AWS-side mechanism nobody knew existed?** (lesson 7's honest answer: partly
confirmed, partly still open, documented precisely which part is which.)

## What this package does not do

It does not re-run or re-verify any of the four artifacts' own content — this index trusts each
lesson's own verification (`POSTMORTEM.md`'s 171 lines / 10 sections, the runbook's 220 lines / 9
sections) rather than repeating it. It does not close action items #1 and #2 — those remain real,
open, SMART commitments against `andes-cargo-infra/`, not resolved by this guide's own scope. It
does not claim DataTalks.Club, the real company behind this incident, ever built or needed any of
these four artifacts — every one of them is Andes Cargo's own response, built on a real incident's
verified facts.

## Consequences

Module 8's capstone cites this package as evidence that Andes Cargo's reliability machine — SLI
(Module 2) measured with real data (Module 3), alerted on burn rate (Module 4), operated through
a real incident (Module 6), and closed with a blameless postmortem and a working runbook (this
module) — produces artifacts a real interview or a real on-call rotation could use unmodified.

Step 3 — Verifying the package

wc -l RELIABILITY-POSTMORTEM-PACKAGE.md
grep -c '^## ' RELIABILITY-POSTMORTEM-PACKAGE.md
grep -c '^| [0-9]' RELIABILITY-POSTMORTEM-PACKAGE.md

What to expect (literal — you assembled the content):

57
5
4

Fifty-seven lines, five sections (What this package contains, The four action items, current status, What a reader should be able to answer from this package alone, What this package does not do, Consequences), and four rows in the action items status table — the same count of four POSTMORTEM.md already established in lesson 3, now with a status column added.

Step 4 — Reading the package's most important honesty: two of four, not four of four

Notice something this document doesn't soften: only two of POSTMORTEM.md's four action items are fully resolved within this guide's scope — the runbook (#3) and the honest backup investigation (#4). The other two — prevent_destroy (#1) and conftest's extended coverage (#2) — are infrastructure changes to andes-cargo-infra/ that precisely exceed a postmortems-and-runbooks module's declared scope: they're real HCL work, the same kind cloud-security-and-guardrails-guide built for its own controls, not the work of a module that documents and operates. Declaring this with the same honesty every document in this guide already demands — "two of four, not four of four" — is more valuable than forcing an "everything resolved" claim that wouldn't be true. An action item precisely declared with SMART criteria, ready for someone to pick up, is a real outcome of this module, even if the HCL itself isn't written here.

Common mistakes

Writing this package in a way that implies all four action items are already resolved, so the project "looks more complete" (inflating the outcome, the same error TIMELINE.md already warned against with "Resolved" with no nuance). What happens: someone, filling in Step 2's table's "Status" column, marks all four as "Done," including prevent_destroy and conftest's coverage, which weren't actually applied within this module. How to spot it: if your status table doesn't distinguish between "the HCL already ran" and "the action item is declared with SMART criteria, ready to be applied." How to fix it: the same discipline cloud-security-and-guardrails-guide already applied when writing RISK-MAP.md's row for CloudTrail — "it doesn't just say 'Resolved'" — governs this table: two action items fully delivered, two declared and ready, none inflated.

Treating the absence of a fourth new file (for the backup/restore result) as a gap in this package (expecting symmetry DISENO.md never asked for). What happens: someone, seeing Step 1 finds only three files instead of four, assumes an additional document needs to be created for lesson 7's result. How to spot it: if your file inventory includes an invented file, not listed in DISENO.md, just to "complete" the one-file-per-lesson pattern. How to fix it: DISENO.md is explicit about which new artifacts this guide produces, and the backup/restore attempt's result isn't among them — it stays documented within lesson 7 itself, a deliberate design decision, not an oversight this final project needs to fix by adding a file nobody asked for.

Reconstructing POSTMORTEM.md's or the runbook's full content inside this portfolio document, instead of citing them by path (repeating, for the fifth time in this guide, the same disciplined-assembly error). What happens: someone copies POSTMORTEM.md's complete sections into RELIABILITY-POSTMORTEM-PACKAGE.md, so the package "has it all in one place." How to spot it: if your version of this document goes well beyond Step 3's 57 lines. How to fix it: the same principle TIMELINE.md, INCIDENT-RESPONSE-PLAN.md, and SLO.md already applied governs this document — an index cites, with a table and a file path, it never repeats the full content of what it indexes.

Exercises

Exercise 1 — Verify Step 3 against your own copy of RELIABILITY-POSTMORTEM-PACKAGE.md and confirm you get exactly 57, 5, and 4.

See solution

Copying Step 2's document exactly as it appears, wc -l counts 57 total lines, grep -c '^## ' finds five level-2 section headers, and grep -c '^| [0-9]' finds the four rows in the action items status table starting with a number. This document's deterministic verification is shorter than POSTMORTEM.md's or the runbook's because, by design, it's an index — it cites paths and statuses, it doesn't reconstruct content.

Exercise 2 — A technical interviewer, looking at this package, asks: "why are only two of four action items resolved? Didn't this postmortem fully work?" How do you respond, using this lesson's Step 4?

See solution

A complete answer: "The postmortem worked exactly as it should — it identified four real root causes and produced four SMART action items, each with an owner, a measurable criterion, and a deadline. That two of the four require a real infrastructure change against andes-cargo-infra/ — a Terraform lifecycle block, a new conftest test — and don't get resolved within a module that documents postmortems and runbooks isn't a process failure, it's a precisely declared scope boundary: this module delivers the two action items that were fully its own to deliver (the runbook, the honest backup investigation) and leaves the other two ready, with all the necessary criteria, for whoever builds Andes Cargo's next piece of infrastructure to pick up unambiguously. A postmortem that produces real SMART action items, even if not all get resolved within the same module that wrote them, is more honest and more useful than one that pretends to resolve them all immediately."

Exercise 3 — Explain why Step 2's "What a reader should be able to answer from this package alone" section phrases its four questions in the same order this module's lessons 3, 4, 6, and 7 built their artifacts, instead of a different order.

See solution

The order reproduces this module's real logical sequence: first establish why something happened, at the system level, without blaming a person (lesson 3, the root-cause question); then decide what changes as a result, with an owner and a deadline (lesson 4, the action-items question); then build the operational tool for the next time a similar — though not identical — condition repeats (lesson 6, the runbook question); and finally close the most specific gap the incident itself revealed — dependence on a recovery mechanism outside the team's control — with an honest investigation (lesson 7). Each question logically depends on the previous one: it wouldn't make sense to ask "what changes" before knowing "why it happened," nor to ask "what does someone on call do" before knowing "what changed." The section's order isn't arbitrary — it's the same order a real, well-built postmortem has to reason through.

Summary and next step

This final project integrated Module 7's four artifacts into RELIABILITY-POSTMORTEM-PACKAGE.md: POSTMORTEM.md (blameless root cause), the four SMART action items with their real status (two fully delivered, two declared and ready), Andes Cargo's complete runbook, and the honest result of the backup/restore attempt. You verified the package with the same deterministic pattern as always — 57 lines, 5 sections, 4 status rows — and confirmed, without inflating the outcome, that two of four action items exceed this documentation module's scope and remain declared, not resolved.

Before closing this module you should be able to: name this module's four artifacts and their exact location within andes-cargo-infra/; explain why only two of four action items are fully resolved, without that taking away from the postmortem's value; and defend, with this document's four questions, that the complete package stands on its own without needing to reread any individual lesson.

With this, sre-and-incident-response-guide's Module 7 is complete: the Claude Code incident, already operated end to end in Module 6, now has its blameless systemic analysis, its action items with real owners, and this entire ecosystem's first operational runbook. Module 8, this guide's capstone, runs a new synthetic incident — deterministic, never random — through the complete machine the previous seven modules built: SLI, SLO, observability, alerting, lifecycle, and now, postmortem and runbook.

Resources

This module, lessons 2 through 7 — the direct source of every artifact this package indexes.
This same repository, Module 6, lesson 8 (08-project-andes-cargos-final-timeline-for-this-case.md) — TIMELINE.md, the document POSTMORTEM.md was built on.
cloud-security-and-guardrails-guide, Module 1, lesson 6 (06-blast-radius-revisited-what-would-have-stopped-it.md) — the source of this package's action item #1.
Google SRE Book — Postmortem Culture and Google SRE Workbook — Postmortem Culture — the complete discipline governing every artifact in this module.

← Previous7. Hands-on: the honest backup/restore attempt Next →1. Introduction to the capstone