Module 7: Blameless Postmortems And Runbooks
6. Hands-on: Andes Cargo's first real runbook
Description
Lesson 5 established what a runbook is and isn't. This lesson builds the first one Andes Cargo has had in its entire history: runbooks/manifest-processor-error-rate.md, the exact procedure for when andes-cargo-manifest-error-budget-burn-rate — the real alarm built in Module 4, lesson 5 — fires. Every awslocal command is labeled with its exact reason; the decision tree and mitigation steps are the real document, verified with the same discipline as every deliverable in this guide.
Connection to the module
This is POSTMORTEM.md's (lesson 3) action item #3, with an assigned owner (Ana) from lesson 4: "Write runbooks/manifest-processor-error-rate.md, Andes Cargo's first operational runbook [...] Root cause 1 and 2 — no documented, mechanical response path independent of who is on call." This runbook operates on Module 4, lesson 5's exact alarm (threshold = 0.001, metric math errors / invocations), reuses the exact log format Module 3, lesson 4 already confirmed for this same Lambda, and its escalation branch (Branch C, Step 5) invokes INCIDENT-RESPONSE-PLAN.md and Module 6, lesson 4's declaration pattern without rewriting either.
Step 1 — Why this runbook reuses already-confirmed data, instead of inventing a new scenario
Before the complete document, a design decision worth explaining: this runbook's example scenario — three failed invocations out of twenty, with three different validation reasons — is exactly the same batch Module 3, lesson 3 built, and that Module 4, lesson 5, Exercise 1 already confirmed fires this exact alarm (3/20 = 15%, well above 0.1%). This lesson doesn't invent a new synthetic incident to illustrate the runbook — it reuses a datum already verified, already cited, already traceable to an earlier lesson. It's the same "never random" discipline governing this entire guide, applied here in a specific way: if an example already exists and is already verified, reusing it is more honest than fabricating a new one just so the runbook "looks different."
Step 2 — The complete document
At the root of andes-cargo-infra/, create the runbooks/ directory and, inside it, manifest-processor-error-rate.md:
# manifest-processor-error-rate.md — Andes Cargo Runbook
**Alarm:** `andes-cargo-manifest-error-budget-burn-rate` (`observability.tf`, Module 4, lesson 5)
**Fires when:** `process-shipment-manifest`'s error ratio (`errors / invocations`) crosses `0.001`
(0.1%, `SLO.md`'s allowed error rate) over a 1-hour window.
**Severity if unclassified:** this alarm alone maps to **SEV3** in `INCIDENT-RESPONSE-PLAN.md`'s
severity matrix (Ticket tier, burn rate >= 1.0x and < 6.0x). Step 2 below may reveal a worse
condition — re-classify per the matrix, do not assume the alarm's own tier caps the real severity.
**Who runs this:** whoever is Operations Lead on call (`oncall/schedule.py`, Module 5).
**Related documents:** `SLO.md` (Module 2), `observability.tf` (Module 4),
`INCIDENT-RESPONSE-PLAN.md` (Module 5), `POSTMORTEM.md` (Module 7, lesson 3, action item #3 — this
runbook is that action item, delivered).
## Before you start
- This alarm measures a **ratio**, not a raw count. One error out of five invocations breaches
`threshold = 0.001` exactly as surely as three hundred errors out of a hundred thousand — always
read `Invocations` alongside `Errors` (Step 2) before deciding how urgent this actually is.
- Every `awslocal` command below is labeled **representative** or **literal**. Representative means
reconstructed field by field from real HCL and LocalStack's confirmed API coverage, not executed
against a live container in this repository's authoring environment — no `LOCALSTACK_AUTH_TOKEN`
is exported here, the same limit named since this guide's Module 1. `jq` commands over a saved
log file are literal: the binary ran, in this environment, to produce that output.
## Step 1 — Confirm the alarm actually fired
```bash
awslocal cloudwatch describe-alarms \
--alarm-names andes-cargo-manifest-error-budget-burn-rate \
--query 'MetricAlarms[0].{State:StateValue,Reason:StateReason,Updated:StateUpdatedTimestamp}'
```
**What to expect (representative — reconstructed from the real alarm definition, Module 4, lesson
5; no `LOCALSTACK_AUTH_TOKEN` in this environment):**
```json
{
"State": "ALARM",
"Reason": "Threshold Crossed: 1 datapoint [0.15 (14/08/26 15:00:00)] was greater than or equal to the threshold (0.001).",
"Updated": "2026-08-14T15:05:00.000Z"
}
```
If `State` is not `ALARM` (for example, it already recovered to `OK`, or `INSUFFICIENT_DATA`
because the container just came up), stop here — Step 6 already applies, there is nothing further
to mitigate.
## Step 2 — Pull the raw numbers behind the ratio
```bash
awslocal cloudwatch get-metric-statistics \
--namespace AWS/Lambda --metric-name Invocations \
--dimensions Name=FunctionName,Value=process-shipment-manifest \
--start-time 2026-08-14T14:00:00Z --end-time 2026-08-14T15:00:00Z \
--period 3600 --statistics Sum
awslocal cloudwatch get-metric-statistics \
--namespace AWS/Lambda --metric-name Errors \
--dimensions Name=FunctionName,Value=process-shipment-manifest \
--start-time 2026-08-14T14:00:00Z --end-time 2026-08-14T15:00:00Z \
--period 3600 --statistics Sum
```
**What to expect (representative — same reason as Step 1; the shape matches Module 3, lesson 3's
already-confirmed format for this exact metric pair):**
```
Invocations (Sum): 20.0
Errors (Sum): 3.0
```
`3 / 20 = 0.15` (15%) — well above the `0.001` threshold, consistent with the `ALARM` state from
Step 1. This is the same batch shape Module 4, lesson 5's Exercise 1 already confirmed would
breach this alarm: a small sample with three errors reads as a severe ratio, even though the
absolute count (three) sounds minor on its own — exactly why Step 2 never skips reading
`Invocations`.
## Step 3 — Pull the failing invocations themselves
```bash
awslocal logs filter-log-events \
--log-group-name /aws/lambda/process-shipment-manifest \
--filter-pattern '?"Invalid manifest" ?"Status: error"' \
--start-time 2026-08-14T14:00:00Z --end-time 2026-08-14T15:00:00Z
```
**What to expect (representative — same reason as Steps 1-2; identical in shape to the log events
Module 3, lesson 4 already confirmed for this Lambda). Save this output as
`observability/manifest-log-events.json` before continuing:**
```json
{
"events": [
{ "message": "Invalid manifest [...]/05-shipment-4471-manifest.txt: ['missing required field: weightKg']", "...": "..." },
{ "message": "REPORT RequestId: a47f3e21-8b6a-4c9d-9f12-3d8e7b1a2c44\t[...]Status: error[...]", "...": "..." },
{ "message": "Invalid manifest [...]/12-shipment-4472-manifest.txt: ['missing required field: carrier']", "...": "..." },
{ "message": "REPORT RequestId: f3c91a08-2e4d-4b7f-8a3c-5e9d1f6b8a72\t[...]Status: error[...]", "...": "..." },
{ "message": "Invalid manifest [...]/17-shipment-4473-manifest.txt: ['weightKg must be numeric']", "...": "..." },
{ "message": "REPORT RequestId: c8e42d15-9a3b-4f8e-b6c1-7d2a4e9f3b58\t[...]Status: error[...]", "...": "..." }
]
}
```
Now group the failure reasons — this is the step that decides which branch of Step 4 applies, and
`jq` runs for real from here on, against the file you just saved:
```bash
jq -r '.events[] | select(.message | contains("Invalid manifest")) | .message
| capture(": .(?<reason>[^]]+).") | .reason' observability/manifest-log-events.json \
| sort | uniq -c | sort -rn
```
**What to expect (literal — `jq` ran in this environment against the reconstructed file above):**
```
1 'weightKg must be numeric'
1 'missing required field: weightKg'
1 'missing required field: carrier'
```
## Step 4 — The decision tree
```text
MANIFEST-PROCESSOR-ERROR-RATE -- DECISION TREE
Step 3's grouped reasons
│
┌─────────┼──────────────────────────────┐
│ │
▼ ▼
ONE reason dominates Reasons are scattered,
(3+ failures, same no single reason has more
validation rule, e.g. than 1-2 occurrences
all "missing required (the case in this run:
field: X") three DIFFERENT reasons)
│ │
▼ ▼
BRANCH A Check Step 2's Invocations
Likely a recent deploy against the normal baseline
changed the manifest for this hour of day
schema, or a data │
producer upstream ┌─────┴─────┐
started sending a │ │
malformed field ▼ ▼
│ Invocations Invocations
▼ near normal far below
Go to Step 5, Branch A BRANCH B normal
Background BRANCH C
noise from Possible
malformed infra-level
uploads -- failure --
normal, no Go to Step 5,
code change Branch C
needed
```
The run in Step 3 lands in **Branch B**: three failures, three different validation reasons, no
single rule repeating — the signature of ordinary malformed-manifest noise (the same kind of fixed,
deliberate malformation Module 3, lesson 3 used to build its own batch), not a systemic regression
in the validation code itself. A dominant, repeated reason across most or all failures is the
signal that points to Branch A instead.
## Step 5 — Mitigation by branch
**Branch A — deploy-caused regression.** Confirm the timing: does the failure window in Step 1's
`Reason` line start shortly after the last deploy of `process-shipment-manifest` or of whatever
system produces manifests upstream? If yes, revert that deploy through the normal pipeline path
(`cicd-and-gitops-on-aws-guide`'s CI/CD flow, not a manual `awslocal` edit against the running
Lambda) and re-run Step 1-2 once the revert completes to confirm the ratio drops back under
`0.001`.
**Branch B — background noise (this run's case).** No code change. Log this occurrence in the
team's weekly reliability review as a normal data-quality event, not an incident. If this pattern
repeats with increasing frequency week over week, that trend — not any single occurrence — is
itself worth raising as a new `POSTMORTEM.md`-style investigation, but a single Ticket-tier alarm
firing on isolated malformed uploads does not, by itself, warrant escalation.
**Branch C — possible infrastructure failure.** This is no longer a runbook-scale problem — declare
an incident following `INCIDENT-RESPONSE-PLAN.md`: assign Incident Commander, Operations Lead, and
Communications Lead per the on-call rotation; write the declaration message following the pattern
Module 6, lesson 4 already built (severity, roles, first known state, relative timestamps from a
new `T+0`); start a new `TIMELINE.md` for this incident. This runbook's job ends at the point where
"follow these steps" stops being sufficient and "coordinate a response" begins.
## Step 6 — Verify resolution
```bash
awslocal cloudwatch describe-alarms \
--alarm-names andes-cargo-manifest-error-budget-burn-rate \
--query 'MetricAlarms[0].StateValue'
```
**What to expect (representative, same reason as Step 1):**
```
"OK"
```
If the state is still `ALARM` after the mitigation in Step 5, do not repeat the same branch a
second time on the assumption it "just needs more time" — re-run Step 2-3 to confirm the branch
classification was correct in the first place. A wrong branch, repeated, wastes exactly the kind of
time this runbook exists to save.
## Step 7 — After the alarm clears
If Branch A or B applied, no formal postmortem is required — log the resolution and move on. If
Branch C required declaring an incident, that incident's own postmortem (following the same
structure `POSTMORTEM.md`, Module 7, lesson 3 already used) is the next document, not this runbook.
## Known limitations
| Step | Status | Why |
|---|---|---|
| 1, 2, 3, 6 (`awslocal cloudwatch`/`logs`) | Representative | No `LOCALSTACK_AUTH_TOKEN` exported in this repository's authoring environment — the same limit named since Module 1. Both `cloudwatch` and `logs` are confirmed on LocalStack's Hobby plan (Module 3, Source #8); output is reconstructed field by field from that confirmed behavior, never invented. |
| 3 (`jq` commands) | Literal | `jq` does not depend on LocalStack — it ran, in this environment, against the saved log file. |
| Decision tree (Step 4) | Structural, not statistical | The three branches are the reasonable classification for the failure signatures this Lambda can produce (a fixed set of `validate_manifest()` rules, Module 3, lesson 3) — not a machine-learned or dynamically-tuned classifier. A new failure mode not covered by these three branches would need this runbook itself updated, per the "flexible and adaptable" property Module 7, lesson 5 already named.
This runbook is Andes Cargo's first — the earlier absence of any runbook at all is exactly what
`POSTMORTEM.md`'s root cause #1 and #2 (Module 7, lesson 3) named as part of why the Claude Code
incident had no documented, mechanical response path to fall back on.Step 3 — Verifying the document
wc -l manifest-processor-error-rate.md
grep -c '^## ' manifest-processor-error-rate.mdWhat to expect (literal):
220
9
Two hundred twenty lines, nine sections (Before you start, Step 1 through Step 7, Known limitations) — a runbook-sized document, correctly proportioned: complete enough to leave no decision uncovered, tight enough to be read start to finish in the time it takes an alarm to fire.
Step 4 — Actually running jq, with a different question than Module 3's
Notice something deliberate in Step 3 (within the document): this runbook's jq command doesn't exactly repeat the one Module 3, lesson 4 already ran on the same file — that one extracted the three requestIds and the three reasons separately. This runbook asks a different operational question, the one that actually matters for deciding the tree's branch: do the reasons repeat, or are they scattered? sort | uniq -c | sort -rn groups and counts, and the result — three reasons, each with exactly one occurrence — is direct evidence that none dominates, the exact condition Step 4's Branch B needs to justify itself. Reusing the same data file with a new question, instead of repeating the previous question, is why this runbook, even though built on Module 3's same data, isn't a copy of that lesson — it draws a different operational conclusion that lesson never needed to draw.
Common mistakes
Writing the runbook with the "obvious" severity already in mind, without checking against INCIDENT-RESPONSE-PLAN.md's real matrix (assuming instead of citing). What happens: someone, seeing an error alarm fired, automatically assumes "this is serious" without confirming which level of the matrix it falls under. How to spot it: if your version of the runbook doesn't explicitly mention which severity corresponds to this specific alarm. How to fix it: Step 2's document header states it explicitly — this alarm, on its own, maps to SEV3 (Ticket), the matrix's lowest tier with an automatic page; only if Step 2 reveals a worse pattern (for example, Invocations near zero, indicating almost nothing is processing) could the real severity be higher, and the runbook flags that precisely instead of assuming.
Following Branch A (reverting a deploy) without first confirming, with Step 4, that the reasons really cluster into a single dominant pattern (jumping straight to the most "active" mitigation). What happens: someone, under pressure, sees a fired alarm and assumes the correct response is always "revert the last change," without first running Step 3's diagnosis. How to spot it: if your response to a fired alarm starts with a mitigation action, not a diagnosis. How to fix it: Step 4's decision tree exists exactly to prevent this — this runbook classifies before acting, and this lesson's worked example ends up in Branch B (normal noise, no code action), not Branch A, precisely because the diagnosis revealed scattered reasons, not a dominant pattern. Acting without diagnosing first can revert a completely innocent deployment, without solving the real problem.
Treating the awslocal commands' "representative" label as a reason not to trust the whole runbook (a repeat, in the specific context of an operational document, of the error already named in Module 3). What happens: someone, seeing Steps 1, 2, 3, and 6 marked representative, concludes the entire runbook is less trustworthy than a "fully real" one. How to spot it: if your evaluation of the runbook treats the "Known limitations" section as a weakness of the document instead of honest operational information. How to fix it: Step 2's "Known limitations" table precisely distinguishes what's representative (the awslocal commands, reconstructed field by field from already-confirmed behavior) from what's literal (the jq commands, which did run) — the decision tree's structure, the three branches' logic, and the mitigation steps are the real document, verified with the same discipline as any other artifact in this guide; what's representative is exclusively the specific output of commands that depend on a container this authoring environment can't spin up.
Exercises
Exercise 1 — Run Step 3's jq command yourself, on your own copy of observability/manifest-log-events.json, and confirm you get exactly the same grouped result.
See solution
Copying the JSON exactly as it appears in Step 2 and running jq -r '.events[] | select(.message | contains("Invalid manifest")) | .message | capture(": .(?<reason>[^]]+).") | .reason' observability/manifest-log-events.json | sort | uniq -c | sort -rn, the result is the same: three lines, each with count 1, for 'weightKg must be numeric', 'missing required field: weightKg', and 'missing required field: carrier' (the exact order among the three may vary depending on how sort breaks ties on equal counts, but the three counts of 1 don't change). This verification confirms that Branch B's diagnosis — scattered reasons, none dominant — is reproducible by anyone following the runbook, not an unsupported claim.
Exercise 2 — Modify, in prose (without running anything), Step 2's observability/manifest-log-events.json file so that Step 3's same jq command produces a result that classifies into Branch A of the decision tree, instead of Branch B.
See solution
You'd need to change the content of the three "Invalid manifest" messages so all three share exactly the same validation reason — for example, all three lines ending in : ['missing required field: weightKg'], instead of the three current distinct reasons. With that change, Step 3's jq command would group the three occurrences into a single row: 3 'missing required field: weightKg', a clearly dominant pattern Step 4's decision tree would classify into Branch A — suggesting a recent deploy probably removed that field somewhere earlier in the manifest-generation chain, instead of it being scattered noise from individually malformed manifests.
Exercise 3 — Explain why Step 2's runbook includes an explicit row for "what if State is no longer ALARM?" at the end of Step 1, instead of assuming whoever follows the runbook always does so while the alarm is active.
See solution
Because an alarm can recover on its own — a transient traffic spike, or a batch of malformed manifests that already finished processing — between the moment someone receives the notification and the moment they actually open the runbook to follow it, especially if the notification arrived minutes or hours before the on-call person could get to it. Without that explicit check at the end of Step 1, someone could run all of Steps 2 through 5 — including, in the worst case, reverting a deploy in Branch A — against a condition that no longer exists, generating unnecessary work and risk. The same discipline this module's lesson 5 already named as a good runbook's property — "clear and simple" — includes anticipating the most common false-positive case, not just the happy path where the alarm still matches exactly what the notification described.
Summary and next step
This lesson built runbooks/manifest-processor-error-rate.md: Andes Cargo's first operational runbook, with seven numbered steps, a three-branch decision tree, and a known-limitations table that precisely distinguishes what's representative (the awslocal commands, with no LOCALSTACK_AUTH_TOKEN) from what's literal (the jq commands, which really ran against data already confirmed in Module 3). You verified the document with the same deterministic pattern as always — 220 lines, 9 sections — and saw, with executed evidence, how Step 3's diagnosis classifies the example scenario into Branch B (normal noise), not Branch A (deploy regression).
Before moving on you should be able to: recite the runbook's seven sections in order; explain how jq's result decides between the tree's three branches; and defend why the runbook's representative parts don't make it less trustworthy as an operational document.
Lesson 7 closes the loop on POSTMORTEM.md's action item #4: the honest attempt at backing up and restoring Shipments against LocalStack, with the same truth standard this runbook already applied.
Resources
- This same repository, Module 4, lesson 5 (
05-hands-on-a-real-cloudwatch-alarm-on-the-lambda.md) — the real alarm (andes-cargo-manifest-error-budget-burn-rate) this runbook operates. - This same repository, Module 3, lessons 3 and 4 — the origin of the dataset and log format reused in this document's Step 3.
- This same repository, Module 5, lesson 8 (
08-project-andes-cargos-incident-response-plan.md) —INCIDENT-RESPONSE-PLAN.md, the severity matrix and roles this runbook's Branch C invokes. - This same repository, Module 7, lesson 5 (
05-what-a-runbook-is-and-is-not.md) — the properties of a good runbook, applied here. - jqlang/jq — Manual — reference for
capture,select, and the expressions used in Step 3.