Module 7: Blameless Postmortems And Runbooks

4. Action items that actually get done

Next →

Description

POSTMORTEM.md ended with a table of four action items, each tied to a specific root cause — but, as lesson 3 left it, that table has a real problem: nobody owns anything, there's no date, and there's no way to confirm, with evidence, that any of the four actually got done. This lesson closes that gap with a concrete criterion — SMART — and with the same Google SRE quote that precisely explains why an action item with no owner almost never gets done.

Connection to the module

This lesson expands, without modifying any root cause already established, the brief table POSTMORTEM.md (lesson 3) left at the end. This lesson's action item #1 — prevent_destroy on Shipments — is the exact piece cloud-security-and-guardrails-guide, Module 1, lesson 6 left declared as a pending gap across the whole ecosystem: "No guide in this ecosystem builds it yet for Andes Cargo — it is, honestly, a declared gap, not a hidden one." Action item #3 — the runbook — is, literally, the deliverable this same module's lesson 6 builds in full.

Step 1 — Why most postmortem action items never get done

Google SRE doesn't leave this problem as an anecdotal observation — it treats it as a design requirement of the process itself:

"All action items have both an owner and a tracking number" [...] "All action items are assigned a priority level."

— Google SRE Workbook — Postmortem Culture

The reason behind this rule, cited in the same source: an action item with no single clear owner tends to get spread across "the team" in general, and what belongs to the whole team, in practice, belongs to nobody in particular — nobody feels responsible for moving it from "pending" to "done." The same source recommends a single owner, with additional collaborators if needed, instead of responsibility distributed evenly across several people.

And there's a second quote, from SRE discipline's own founder at Google, that directly connects this lesson to a postmortem's entire purpose:

"To our users, a postmortem without subsequent action is indistinguishable from no postmortem."

— Ben Treynor Sloss, quoted in Google SRE Workbook — Postmortem Culture

This quote is, precisely, the reason this lesson exists as a separate step — not as a footnote to POSTMORTEM.md: an impeccably written postmortem, with root cause correctly distinguished from trigger, with the full blameless discipline, that ends in a list of good intentions with no owner or date, has, from the perspective of whoever depends on the system becoming more reliable, the same value as never having written a postmortem at all.

Step 2 — The SMART criteria, applied with no filler

SMART is an acronym for five conditions that, together, distinguish a real commitment from a vague intention:

LetterConditionThe question it answers
SpecificSpecificWhat, exactly, is going to change? No ambiguity about scope.
MeasurableMeasurableHow would you know, with verifiable evidence, that it's already done?
AssignableAssignableWho, one single person, is responsible for making this move forward?
RealisticRealisticCan it be completed with the available resources and time?
Time-boundTime-boundBy when? Without a date, there's no way to know if it's overdue.

The most common mistake when applying this criterion isn't forgetting a letter — it's writing an action item that sounds specific but, read carefully, doesn't answer any of the five questions with verifiable precision. Before building this lesson's four real action items, it's worth seeing the exact contrast:

   VAGUE ACTION ITEM                          SMART ACTION ITEM
   (fails the criteria)                       (passes all five letters)
   ─────────────────────                      ───────────────────────
   "Investigate why the human                 "Add prevent_destroy to the
   didn't stop the agent"                      aws_dynamodb_table.shipments
                                                resource in andes-cargo-infra/"
        │                                            │
        ▼                                            ▼
   Specific?   NO -- "investigate" doesn't    Specific?   YES -- one resource,
               say what's going to change                 one exact HCL block
   Measurable? NO -- what evidence             Measurable? YES -- terraform plan
               confirms it's done?                         shows the lifecycle
   Assignable? NO -- no owner                  Assignable? YES -- Bruno (Operations
                                                             Lead, INCIDENT-RESPONSE-
                                                             PLAN.md)
   Time-bound? NO -- no date                   Time-bound? YES -- before the
                                                             next sprint cycle
                                                             (2 weeks)
        │                                            │
        ▼                                            ▼
   Also: violates Lesson 2's                   Also: it's a system action,
   blameless discipline --                     not a judgment about a
   "investigating a person" isn't              person -- exactly the kind
   a system action item                        of action item Lesson 2
                                                demands

Step 3 — Andes Cargo's four real action items, with full SMART criteria

Each of POSTMORTEM.md's (lesson 3) four action items, expanded with the five conditions, an owner from INCIDENT-RESPONSE-PLAN.md (Module 5), and a priority:

Action item #1 — prevent_destroy on Shipments

ConditionHow it's met
SpecificAdd a lifecycle { prevent_destroy = true } block to the aws_dynamodb_table.shipments resource in andes-cargo-infra/.
Measurableterraform plan against any change attempting to delete or replace the table fails with Terraform's native error: "Instance cannot be destroyed [...] Resource has lifecycle.prevent_destroy set" — verifiable by running the plan a single time.
AssignableBruno — Operations Lead (INCIDENT-RESPONSE-PLAN.md, Module 5), the only person with permission to modify andes-cargo-infra/ outside an active incident.
RealisticA three-line lifecycle block, with no external dependencies — the same kind of low-risk change as any other .tf file in this ecosystem.
Time-boundBefore the next sprint cycle — two weeks from this postmortem's date, the same window INCIDENT-RESPONSE-PLAN.md already uses as a reference for P1-priority action items.
PriorityP1 — closes root cause #3 (POSTMORTEM.md), the last line of defense the real incident never had, and the gap cloud-security-and-guardrails-guide left declared but unbuilt.

Action item #2 — no-destroy-shipments.rego's coverage of terraform destroy

ConditionHow it's met
SpecificVerify, with a dedicated conftest test, that the Rego policy built in cloud-security-and-guardrails-guide (Module 4, lesson 6) correctly evaluates a plan generated by terraform plan -destroy, not only an apply plan that contains an incidental delete action.
MeasurableA new test file, destroy_plan_test.rego, with a synthetic plan.json simulating terraform plan -destroy against Shipments; conftest test must fail that plan with the same message that already fails an equivalent apply plan.
AssignableCarla — the rotation's third person (oncall/schedule.py, Module 5, lesson 7), assigned here as the policy verification owner, distinct from Bruno so the four action items aren't concentrated on a single person.
RealisticDoesn't require rewriting the policy — only adding a new test case and confirming the result; if the coverage already exists, the action item's result is "confirmed," not "rewritten."
Time-boundA full sprint (four weeks) — lower priority than #1 because no-destroy-shipments.rego already exists and works against the most common case (apply); this verifies an edge case, it doesn't close an open gap.
PriorityP2 — reinforces root cause #2, but doesn't leave it open if it slips: the apply gate already exists.

Action item #3 — Write runbooks/manifest-processor-error-rate.md

ConditionHow it's met
SpecificA new document in andes-cargo-infra/runbooks/, with the exact steps for responding when observability.tf's alarm (Module 4, lesson 5) fires.
MeasurableThe document exists, every command it contains was verified (or precisely labeled as representative, with the reason), and it passes the "someone new to the team can follow it without asking anything" test — the same criterion this module's lesson 6 applies when writing it.
AssignableAna — Incident Commander (INCIDENT-RESPONSE-PLAN.md), owner of the team's operational documentation.
RealisticCompletely realistic — this same module delivers it in full in lesson 6, with no pending external dependency.
Time-boundOne week — the shortest window of the four, because it closes a total absence (Andes Cargo had no runbook at all before this module).
PriorityP1 — closes root causes #1 and #2 from the angle of "what does a human do, not just what does a system block": even with prevent_destroy and conftest in place, a runbook reduces response time for any other class of incident those two controls don't cover.

Action item #4 — Evaluate own backup/PITR for Shipments

ConditionHow it's met
SpecificAttempt, against LocalStack, awslocal dynamodb create-backup and restore-table-from-backup on a disposable copy of Shipments, and document the real outcome — whether it works, fails, or ends up partially confirmed.
MeasurableA document with the attempt's exact outcome, without inventing a result that didn't happen — the same standard cloud-security-and-guardrails-guide already applied with CloudTrail in its Module 7.
AssignableDiego — the rotation's fourth person, so this postmortem's four action items have four distinct owners, none overloaded.
RealisticPartially uncertain by design — LocalStack's own documentation confirms DynamoDB on the Hobby plan, but doesn't explicitly confirm the backup API's coverage (see this module's lesson 7); the action item is "document the real outcome," not "guarantee it works," so it stays realistic even if the outcome is a documented technical limit.
Time-boundTwo weeks — this module's lesson 7, immediately after, already completes this attempt within the guide's own ecosystem.
PriorityP2 — depends on an uncertain technical outcome; it blocks nothing else while under investigation.

Step 4 — Why no action item in this lesson names anyone from the real incident

Notice a detail that connects directly to lesson 2: none of this lesson's four action items say "train the human who approved the destroy" or "review the agent's judgment before proposing a destructive action." Both ideas sound reasonable at first glance, and both fail the same criterion: they're not verifiably measurable (how would you confirm someone "has better judgment" now?), and they depend, again, on a specific person's attention or judgment being the safeguard — the same single point of failure that produced the incident in the first place. This lesson's four real action items, in contrast, are verifiable with a command (terraform plan, conftest test) or a finished document (the runbook, the backup attempt's result) — none of them depends on anyone "learning the lesson" in an unverifiable sense.

Common mistakes

Writing an action item that does have all five SMART letters but is still, at bottom, a task about a person (passing the criterion superficially). What happens: someone writes "Bruno manually reviews every plan before any production apply, every week, starting today" — it has an owner, a deadline, and looks specific. How to spot it: if your action item, even with all five conditions, still depends on a specific person paying attention every single future time, instead of an automated mechanism doing the work. How to fix it: SMART is a necessary condition, not a sufficient one — an action item also has to pass lesson 2's question: "is this a system change, or does it again depend on human attention at the critical moment?" This lesson's four action items pass both tests: SMART, and "doesn't depend on someone remembering to do something every time."

Assigning all four action items to the same person, "because they know the most about infrastructure" (repeating, at the action-item level, the same responsibility-concentration pattern that already caused part of the incident). What happens: someone, assigning owners, puts the most technically capable person on all four, without considering the resulting load. How to spot it: if your action items table has a single name in the "Assignable" column. How to fix it: this lesson's Step 3 distributes the four across Ana, Bruno, Carla, and Diego — the four names from INCIDENT-RESPONSE-PLAN.md's real rotation — precisely so no action item waits on a single person having available time; Google SRE recommends a single owner per item, not a single owner for all of one postmortem's action items.

Marking an action item "complete" without the measurable evidence its own definition demands (declaring without verifying, the same error already named with TIMELINE.md and POSTMORTEM.md). What happens: someone closes action item #1 saying "already resolved" without having run terraform plan to confirm the lifecycle block really blocks the deletion. How to spot it: if you can't point to a command's exact output, or the exact document, that demonstrates that action item's "Measurable" criterion was met. How to fix it: every "Measurable" row in Step 3's table describes a runnable check, not a promise — before marking any action item in this guide as closed, that row's described verification must actually be run, exactly the same discipline SLO.md, INCIDENT-RESPONSE-PLAN.md, and TIMELINE.md already demanded of themselves.

Exercises

Exercise 1 — Rewrite, with full SMART criteria, this vague action item: "Improve the Terraform documentation so it's clearer."

See solution

One possible SMART version: "Add, to andes-cargo-infra/README.md, a 'State file handling' section that explicitly documents the active state's expected origin and the procedure for validating it against real infrastructure before any plan — Specific: a new section, with defined content; Measurable: the section exists and contains, at minimum, the three validation steps described; Assignable: a single named owner (say, Ana); Realistic: it's a documentation change, with no technical dependencies; Time-bound: within one week." The original version ("improve the documentation so it's clearer") fails all five conditions at once: it doesn't say which documentation, it doesn't say how "clearer" would be measured, it has no owner, there's no way to know if it's realistic without knowing the scope, and it has no deadline.

Exercise 2 — Explain why this lesson's action item #4 (evaluate backup/PITR) has the phrase "document the real outcome" in its "Specific" condition, instead of "confirm it works."

See solution

Because the real technical outcome isn't yet known at the moment this action item is written — lesson 3 itself and this table's "Realistic" section already acknowledge that LocalStack's backup API coverage isn't confirmed with the same certainty as CloudWatch. If the action item promised "confirm it works," an honest but negative result (the API isn't supported, or fails for a specific technical reason) would make the action item look "failed" or "unmet," when in fact exactly what it asked for was done: investigate and document the truth, whatever it is. This is the same discipline cloud-security-and-guardrails-guide already applied with CloudTrail in its Module 7 — an honest investigation action item is completed by documenting the real outcome, not by guaranteeing in advance what that outcome will be.

Exercise 3 — A classmate proposes merging action items #1 and #2 into one ("add automated controls against destroy on Shipments"), arguing both point to the same goal. Do you agree, using this module's lesson 3, Exercise 2?

See solution

Disagree. This module's lesson 3, Exercise 2 already established that root causes #2 and #3 — the automated gate before apply, and resource-level protection — act at different points in the chain, with different consequences if only one works: the gate stops the plan before the option to apply it even exists; prevent_destroy blocks the resource's own execution as a last line of defense, even if the gate fails for any reason. Merging action items #1 and #2 into one would lose exactly that distinction — a single "Specific" action item that actually covers two independent mechanisms is harder to verify precisely ("Measurable"), because completing only one of the two could incorrectly get marked as the entire action item resolved. Keeping them separate, each with its own owner and its own verification, stays truer to the defense-in-depth principle underlying both root causes.

Summary and next step

This lesson expanded POSTMORTEM.md's brief table into four complete SMART action items: prevent_destroy on Shipments (Bruno, P1), no-destroy-shipments.rego's coverage of terraform destroy (Carla, P2), Andes Cargo's complete runbook (Ana, P1), and the honest backup/restore attempt (Diego, P2) — each with a single owner from INCIDENT-RESPONSE-PLAN.md's real rotation, a measurable condition with verifiable evidence, and a date. You confirmed, with Ben Treynor Sloss's quote, why this lesson isn't an optional step: a postmortem whose action items don't get done has, for whoever depends on the system, the same value as no postmortem at all.

Before moving on you should be able to: apply the five SMART conditions to any new action item; explain why an action item can pass SMART and still, at bottom, be a task about a person; and name this lesson's four owners and priorities from memory.

Lesson 5 shifts documents entirely: what a runbook is (and isn't), the piece this lesson's action item #3 still has to deliver.

Resources

  1. Google SRE Workbook — Postmortem Culture — the source of the quotes on a single owner, tracking number, and priority for every action item.
  2. cloud-security-and-guardrails-guide, Module 1, lesson 6 (06-blast-radius-revisited-what-would-have-stopped-it.md) — the source of the declared prevent_destroy gap on Shipments, closed by this lesson's action item #1.
  3. This same repository, Module 5, lesson 8 (08-project-andes-cargos-incident-response-plan.md) — INCIDENT-RESPONSE-PLAN.md, the source of the four names assigned as owners.
  4. This same repository, Module 7, lesson 3 (03-hands-on-writing-the-claude-code-incident-postmortem.md) — POSTMORTEM.md, the source of the brief table this lesson expands.
← Previous3. Hands-on: writing the Claude Code incident postmortemNext →5. What a runbook is (and is not)