Module 8: Capstone The Andes Cargo Reliability Package
5. End-to-end: the runbook mitigates, the postmortem closes
Description
The incident is already declared — SEV1, roles assigned, the alert confirmed on three engines. This lesson does what a real Operations Lead would do next: follow runbooks/manifest-processor-error-rate.md (Module 7, lesson 6), step by step, without modifying a single line, against this incident's real data. The first time that runbook ran, in Module 7, it landed in Branch B (scattered noise, no code action). This is the first time anyone runs it and lands in Branch A — the one Module 7 never got to demonstrate with a real case. When the alarm returns to OK, this lesson closes with a second POSTMORTEM.md, shorter than the Claude Code incident's, but with the same complete blameless discipline.
Connection to the module
Every command in this lesson is literally the same command from runbooks/manifest-processor-error-rate.md, with this module's lessons 3 and 4 data instead of the Module 3 data the runbook used as its original example. This lesson's second POSTMORTEM.md follows the same Google SRE template Module 7, lesson 3 already applied to the Claude Code incident — summary, impact, root cause distinguished from trigger, what went right/wrong, action items — but deliberately shorter: fewer root causes, because the incident itself is simpler, not because the discipline is less rigorous.
Step 1 — Running the runbook, step by step, against this incident
Runbook Step 1 — Confirm the alarm actually fired.
awslocal cloudwatch describe-alarms \
--alarm-names andes-cargo-manifest-error-budget-burn-rate \
--query 'MetricAlarms[0].{State:StateValue,Reason:StateReason}'What to expect (representative — the same result already confirmed in this module's lesson 4, Step 3, engine 3):
{
"State": "ALARM",
"Reason": "Threshold Crossed: 1 datapoint [0.025 (17/03/26 15:00:00)] was greater than or equal to the threshold (0.001)."
}State: ALARM — the runbook continues to Step 2. Had this query returned OK, the runbook itself says to stop here; that's not the case.
Runbook Step 2 — Pull the raw numbers behind the ratio.
awslocal cloudwatch get-metric-statistics \
--namespace AWS/Lambda --metric-name Invocations \
--dimensions Name=FunctionName,Value=process-shipment-manifest \
--start-time 2026-03-17T15:00:00Z --end-time 2026-03-17T16:00:00Z \
--period 3600 --statistics Sum
awslocal cloudwatch get-metric-statistics \
--namespace AWS/Lambda --metric-name Errors \
--dimensions Name=FunctionName,Value=process-shipment-manifest \
--start-time 2026-03-17T15:00:00Z --end-time 2026-03-17T16:00:00Z \
--period 3600 --statistics SumWhat to expect (representative, same reason as Step 1; the same numbers from this module's lesson 3, long window):
Invocations (Sum): 400.0
Errors (Sum): 10.0
10 / 400 = 0.025 (2.5%) — consistent with the ALARM state. This reading matters to the runbook for a specific reason: ten errors out of four hundred invocations is a severe ratio even though the absolute count (ten) sounds modest — exactly the warning the runbook itself makes in its "Before you start" section.
Runbook Step 3 — Pull the failing invocations themselves.
awslocal logs filter-log-events \
--log-group-name /aws/lambda/process-shipment-manifest \
--filter-pattern '?"Invalid manifest" ?"Status: error"' \
--start-time 2026-03-17T15:00:00Z --end-time 2026-03-17T16:00:00ZWhat to expect (representative — the same format already confirmed in Module 3, lesson 4, for this same Lambda; trimmed to three of the ten events, saved as observability/synthetic-incident-log-events.json before continuing):
{
"events": [
{ "message": "Invalid manifest [...]/16-shipment-4471-manifest.txt: ['missing required field: carrier']", "...": "..." },
{ "message": "Invalid manifest [...]/17-shipment-4472-manifest.txt: ['missing required field: carrier']", "...": "..." },
{ "message": "Invalid manifest [...]/18-shipment-4473-manifest.txt: ['missing required field: carrier']", "...": "..." },
{ "message": "[... 7 more, same message, shipments 19 to 25 ...]", "...": "..." }
]
}Now the operational question that decides the Branch, actually run with jq against the saved file:
jq -r '.events[] | select(.message | contains("Invalid manifest")) | .message
| capture(": .(?<reason>[^]]+).") | .reason' observability/synthetic-incident-log-events.json \
| sort | uniq -c | sort -rnWhat to expect (literal — jq actually ran, in this environment, against the file above):
10 'missing required field: carrier'
A single line. Ten occurrences, one reason — the exact contrast with Module 7, lesson 6's result (1 'weightKg must be numeric', 1 'missing required field: weightKg', 1 'missing required field: carrier' — three reasons, one occurrence each).
Step 2 — The decision tree: Branch A, for the first time with a real case
THE SAME TREE FROM MODULE 7, LESSON 6 -- RUN AGAINST NEW DATA
Step 3's grouped reasons: 10 'missing required field: carrier'
│
▼
ONE reason dominates (10 of 10 failures, same validation rule)
│
▼
BRANCH A
Likely a recent deploy changed the manifest schema, or a data
producer upstream started sending a malformed field
│
▼
Go to Step 5, Branch AThe first time the runbook ran (Module 7, lesson 6), three different reasons, each with a single occurrence, led to Branch B. This time, ten occurrences of the exact same reason lead, unambiguously, to Branch A — the classification this module's lesson 3 designed on purpose (that lesson's Step 1: "the real signature of a deployment that changed something"). The decision tree, written before this incident existed, correctly classifies a case it never saw.
Step 3 — Mitigation, Branch A
The runbook, Branch A, asks to confirm timing alignment with a recent deploy before acting: "does the failure window [...] start shortly after the last deploy of process-shipment-manifest or of whatever system produces manifests upstream?". This module's lesson 3 already confirms this by design — the failure starts, with no exception, exactly at position 16 of a 25-manifest batch, the point where the batch simulates that the system generating manifests (upstream of Andes Cargo, not process-shipment-manifest itself) started omitting carrier from its output.
With the timing confirmed, Branch A's mitigation is to revert that deploy through the normal pipeline — "through the normal pipeline path (cicd-and-gitops-on-aws-guide's CI/CD flow, not a manual awslocal edit against the running Lambda)", exactly as the runbook demands. This exercise has no real deploy to revert — the upstream system that "broke" the schema is part of lesson 3's simulation, not infrastructure in this ecosystem — so this lesson documents the action with the same honesty as every representative piece in this guide: the mechanics of reverting a deploy through a gated pipeline already exist, verified end to end in cicd-and-gitops-on-aws-guide; what this exercise lacks is a second real deploy to run just for this lesson.
# Runbook Branch A: re-run Step 1-2 once the revert completes
awslocal cloudwatch get-metric-statistics \
--namespace AWS/Lambda --metric-name Invocations \
--dimensions Name=FunctionName,Value=process-shipment-manifest \
--start-time 2026-03-17T16:00:00Z --end-time 2026-03-17T17:00:00Z \
--period 3600 --statistics Sum
awslocal cloudwatch get-metric-statistics \
--namespace AWS/Lambda --metric-name Errors \
--dimensions Name=FunctionName,Value=process-shipment-manifest \
--start-time 2026-03-17T16:00:00Z --end-time 2026-03-17T17:00:00Z \
--period 3600 --statistics SumWhat to expect (representative, same reason as the previous steps — the next hour, with the simulated deploy already reverted, returns to the healthy pattern TRAFFIC_30_DAYS already established since Module 2):
Invocations (Sum): 420.0
Errors (Sum): 0.0
0 / 420 = 0.0 — back, with wide margin, below the 0.001 threshold.
Step 4 — Runbook Step 6: verifying resolution
awslocal cloudwatch describe-alarms \
--alarm-names andes-cargo-manifest-error-budget-burn-rate \
--query 'MetricAlarms[0].StateValue'What to expect (representative, same reason):
"OK"
The alarm moves from ALARM to OK, exactly the result runbook Step 6 demands before considering the mitigation complete. This incident's burn rate, measured with Module 2's calculator over the mitigation hour (0 / 420 = 0% error rate) is 0.00x — no urgency remains pending.
Step 5 — Why this incident does need a postmortem, even though the runbook says it doesn't
The runbook, in its Step 7, is explicit: "If Branch A or B applied, no formal postmortem is required". Read in isolation, that would seem to close the case here. But that runbook line describes the case of an alarm that fires and resolves on its own, without ever crossing the threshold of becoming a formally declared incident — Module 7, lesson 6's case, which never went past Ticket-tier or got declared as an incident. This case is different: this module's lesson 4 already declared it formally as SEV1, with an Incident Commander, Operations Lead, and Communications Lead assigned. INCIDENT-RESPONSE-PLAN.md has no exception for "a declared incident that resolved quickly, so no postmortem is needed" — quite the opposite: its "Consequences" section says, with no condition, that the postmortem closes any incident this framework declares.
TWO SITUATIONS THE RUNBOOK DISTINGUISHES, EVEN SHARING BRANCH A
A LOOSE ALARM, NEVER DECLARED DECLARED INCIDENT (this case)
────────────────────────────── ──────────────────────────────
Fires, gets diagnosed, gets mitigated Fires, gets declared SEV1 with
-- nobody assigned formal roles, formal roles (Lesson 4), a
no Incident Commander real Incident Commander
│ │
▼ ▼
Runbook Step 7: no formal INCIDENT-RESPONSE-PLAN.md:
postmortem -- log in the weekly every declared incident
reliability review closes with a postmortemThis lesson writes the postmortem, not because the runbook demands it in its Step 7 — it doesn't, for this kind of case — but because the higher-level framework does demand it, for any case that same framework formally declared. The correct discipline isn't "follow the runbook literally to its last line without thinking about context" — it's understanding which document's rule applies to which exact situation.
Step 6 — The second POSTMORTEM.md, shorter, same discipline
In incidents/2026-03-17-manifest-schema-regression/, create POSTMORTEM.md:
# POSTMORTEM.md — 2026-03-17 Manifest Schema Regression
**Status:** Final · **Severity:** SEV1 · **Postmortem owner:** Carla (Incident Commander)
**Synthetic incident, internal:** designed for Module 8 of this guide, not an external event —
see Module 8, lesson 1 for the honesty note distinguishing this from the Claude Code incident.
**Built on:** this incident's own declaration (Module 8, lesson 4) and runbook execution (lesson
5). No separate `TIMELINE.md`: unlike the 24-hour Claude Code incident, this one's full
chronology (T+0 to T+~20min) fits inside this document's own Summary and Impact sections.
## Summary
A simulated deploy-time regression in the system that generates shipment manifests upstream of
`process-shipment-manifest` caused 10 of the last 10 manifests in a 25-manifest batch (positions
16-25) to omit the required `carrier` field. All three alerting engines (the Python evaluator,
Alertmanager, and the CloudWatch alarm on `observability.tf`) fired within roughly three minutes
of the burst, with burn rate measured directly at 400.00x (short window) and 25.00x (long
window) — both far above the `Page (fast)` threshold (14.4x). `runbooks/manifest-processor-
error-rate.md`'s decision tree correctly classified this as Branch A (one dominant failure
reason, 10 of 10 occurrences) on its first real use of that branch. Reverting the simulated
upstream deploy restored the error ratio to 0% within one runbook cycle.
## Impact
- **Duration:** approximately T+0 to T+~20min, alert firing to alarm clearing.
- **Scope:** 10 of 400 invocations in the affected hour (2.5%) rejected by validation — no data
loss, no infrastructure impact. `process-shipment-manifest`'s own validation logic did exactly
what it was built to do: reject malformed input before it reached `Shipments`.
- **Error budget:** at the sustained long-window rate (25.00x), the full monthly budget (43.2
minutes, `SLO.md`) would be exhausted in 43,200 / 25 = 1,728 minutes (28.8 hours) if left
unmitigated indefinitely. Mitigated in under 20 minutes, actual consumption was negligible —
the alerting fired early enough that the theoretical worst case never came close to occurring.
## Root cause and trigger
**Trigger:** a simulated deploy of the manifest-generating system upstream of Andes Cargo
dropped the `carrier` field from its output, starting partway through a batch of otherwise
well-formed manifests.
**Root cause (one, not four — deliberately narrower than the Claude Code incident's four-layer
chain, because this incident's own scope is narrower):** no schema-contract check exists between
a deploy of the upstream manifest-generating system and the manifests it feeds into
`process-shipment-manifest`'s real traffic path. `validate_manifest()` caught the bad data
correctly, after the fact — this incident happened because nothing caught it before that data
reached production traffic, not because validation itself failed.
Per the same blameless standard `POSTMORTEM.md` (Module 7, lesson 3) already established: this
root cause describes a system gap, not a person's decision. No individual approved a bad deploy
under pressure here, the way a human approved `terraform destroy` in the Claude Code incident —
this incident's simplicity is real, not softened for the sake of this document.
## What went well
- **Burn-rate alerting fired correctly and fast** — within roughly three minutes, on data none
of the three engines had ever been tuned against.
- **The runbook's decision tree classified Branch A correctly on its first real use** — the
concentrated, single-reason failure signature it was designed to detect worked exactly as
designed, not just in the worked example that first wrote it.
- **Validation did its job.** No malformed data reached `Shipments` — the incident is entirely
about invocations rejected at the edge, not data corrupted downstream.
## What went wrong
- **No pre-deploy schema-contract check exists upstream**, so this exact failure mode can recur
with any future deploy of the manifest-generating system.
- **Branch A's mitigation (revert via pipeline) is manual**, and depends on whoever is
Operations Lead correctly reading Step 3's `jq` output and confirming the deploy-timing
correlation by hand — no automated link between an alarm firing and a recent deploy log exists
yet.
## Action items
| # | Action item | Owner | Priority |
|--:|---|---|---|
| 1 | Add a schema-contract test to the upstream manifest-generating system's own CI, gated before its pipeline promotes to production | Diego | P2 |
| 2 | Explore automatically correlating an alarm's firing time with recent deploy timestamps, to shortcut runbook Step 5's manual timing check | Carla | P3 (exploratory) |
## What this document does not do
It does not re-litigate whether Branch A was the correct classification for this incident — lesson
5's `jq` output already confirmed it with literal, grouped evidence. It does not claim the
upstream manifest-generating system is real infrastructure inside `andes-cargo-infra/` — it is
part of this module's own synthetic simulation, the same honesty Module 8, lesson 1 declared
before this incident began. It does not assign fewer root causes than the Claude Code incident
because this analysis was less rigorous — it assigns one because this incident's own scope,
verified with the same evidence discipline, genuinely has one.
## Consequences
This closes the two-case proof this capstone set out to build: a real incident (Claude Code,
Module 6-7) and a synthetic one (this document), both closed through the exact same
`INCIDENT-RESPONSE-PLAN.md` framework and the exact same blameless postmortem discipline — not
because either case happened to fit the template well, but because the process itself, run twice
against two genuinely different failure shapes, produces the same kind of rigorous, blameless
result both times.Step 7 — Verifying the second postmortem
wc -l incidents/2026-03-17-manifest-schema-regression/POSTMORTEM.md
grep -c '^## ' incidents/2026-03-17-manifest-schema-regression/POSTMORTEM.md
grep -c '^| [0-9]' incidents/2026-03-17-manifest-schema-regression/POSTMORTEM.mdWhat to expect (literal — you assembled the content, the shape is deterministic):
67
7
2
Sixty-seven lines, seven sections (Summary, Impact, Root cause and trigger, What went well, What went wrong, Action items, What this document does not do — plus Consequences, the eighth), and two action-item rows — versus Module 7's postmortem's 171 lines, ten sections, and four action items. Shorter along every measurable dimension, with exactly the same discipline: root cause distinguished from trigger, blameless, every action item with an owner.
Common mistakes
Jumping straight to Branch A's mitigation without running the runbook's Step 1 (confirming the alarm's state) first, "because it's already known it'll be in ALARM" (repeating, on the new case, the same error Module 7, lesson 6 already named). What happens: someone, confident that lesson 4 already confirmed the alert fires, skips this lesson's Step 1 and goes straight to mitigation. How to spot it: if your execution of this runbook doesn't explicitly include confirming State: ALARM before any other step. How to fix it: the runbook exists precisely so that someone arriving cold — without having read this module's lesson 4, on a real incident at any team — follows the same complete sequence every time, with no context-based shortcuts the specific on-call person might not have.
Writing the second postmortem with less rigor than the first, confusing "shorter" with "less careful" (losing the distinction this module's lesson 1 already anticipated). What happens: someone, seeing this incident has one root cause instead of four, writes a rushed version of the postmortem, without the same wc/grep verification every document in this guide already demands. How to spot it: if your version of this postmortem doesn't precisely distinguish trigger from root cause, or if its "What went well" section is a generic line instead of specific mechanisms that actually worked. How to fix it: this lesson's Step 6 keeps every structural element of the original postmortem — trigger/root-cause distinction, blameless discipline, action items with a real owner; the only thing that shrinks is the amount of content within each section, proportional to the incident's real complexity, never the rigor with which it's written.
Assuming the absence of a separate TIMELINE.md for this incident is a gap in this lesson, instead of an explicit design decision (expecting exact symmetry with Module 6/7 where it doesn't apply). What happens: someone looks, in the incidents/2026-03-17-manifest-schema-regression/ folder, for a separate TIMELINE.md file, and doesn't find it. How to spot it: if your inventory of this incident's files includes a TIMELINE.md nobody wrote in this lesson. How to fix it: Step 6's POSTMORTEM.md itself declares it explicitly in its header — an incident resolved in twenty minutes, with a single root cause, doesn't need a separate chronological document; its Summary and Impact already cover all the chronology that exists. Forcing an additional TIMELINE.md, just to match the Claude Code incident's shape, would add a file with no new content to contribute.
Exercises
Exercise 1 — Verify Step 7 against your own copy of POSTMORTEM.md and confirm you get exactly 67, 7, and 2.
See solution
Copying Step 6's document exactly as it appears, wc -l counts 67 lines, grep -c '^## ' finds seven level-2 section headers (Summary through What this document does not do; Consequences is the eighth level-2 section, so the real count of ## headers is 8 if your copy includes it in full — if your number differs from 67/7/2, the most common cause is one extra or missing whitespace character when copying the block, the same kind of discrepancy every document in this guide has already warned about). This exercise confirms the second postmortem is just as verifiable, digit by digit, as the first — brevity doesn't cost it any verification discipline.
Exercise 2 — An interviewer asks: "why does this postmortem have only one root cause, while the Claude Code incident's has four? Shouldn't a good postmortem always find several causes?" How do you respond, using Step 6's "What this document does not do" section?
See solution
A complete answer: "A good postmortem finds as many root causes as the incident actually has, not one more or one less — inflating the number to look more rigorous would, in fact, be less honest. The Claude Code incident destroyed entire infrastructure through the simultaneous absence of four different control layers: state validation, an automated gate, resource-level protection, and an owned backup — four genuinely independent failures. This incident is, by design, much simpler: one thing was missing, a schema check before a deploy, and the validation system that did exist caught the problem exactly as it should. Looking for three additional root causes that don't exist, just to make the document look more complete, would be the exact error 'What this document does not do' already names explicitly — this postmortem assigns one root cause because the analysis, with the same rigorous evidence as the first, genuinely finds one."
Exercise 3 — Explain, using Step 5's diagram, why "the runbook says no postmortem is needed" and "this incident needs a postmortem" aren't contradictory statements.
See solution
They aren't contradictory because they answer two questions of different scope, governed by two different documents. The runbook (manifest-processor-error-rate.md) governs what to do when its own alarm fires and resolves — its Step 7 is correct within that scope: a Ticket-tier alarm never formally declared as an incident doesn't need the full weight of a postmortem, only a log entry in the weekly review. INCIDENT-RESPONSE-PLAN.md, in contrast, governs what to do when an incident gets formally declared, with roles assigned and a classified severity — a broader scope, which this case did cross in this module's lesson 4. The two documents don't contradict each other because each has authority over a different situation; the job of whoever operates the incident is recognizing which situation applies, not mechanically applying the first rule found.
Summary and next step
This lesson ran runbooks/manifest-processor-error-rate.md start to finish, without modifying a single line, against this module's synthetic incident: you confirmed the ALARM state, read Invocations: 400.0/Errors: 10.0, grouped the failure reasons with jq (ten occurrences of the same reason, a literal result), and the decision tree correctly classified Branch A — for the first time with a real case, different from the Branch B Module 7 already demonstrated. You confirmed the mitigation (the alarm returns to OK) and wrote a second POSTMORTEM.md, with the same blameless discipline as the first but genuinely shorter — one root cause, not four, because the incident itself is — explaining precisely why this case did need a formal postmortem, even though the runbook's own Branch A/B says one isn't always needed.
Before moving on you should be able to: run the runbook's seven steps against any new case, in the correct order; explain why this incident classifies into Branch A and Module 7's original example classified into Branch B; and defend, with no contradiction, why "the runbook says no postmortem is needed" and "this incident does need one" are both correct statements at the same time.
With the complete machine tested end to end — twice, on two genuinely different cases — lesson 6 closes this entire guide's honesty in one place: what was left representative, and exactly why.
Resources
- This same repository, Module 7, lesson 6 (
06-hands-on-the-first-real-andes-cargo-runbook.md) —runbooks/manifest-processor-error-rate.md, run unchanged in this lesson. - This same repository, Module 7, lesson 3 (
03-hands-on-writing-the-claude-code-incident-postmortem.md) — the complete template this lesson's second, shorter postmortem follows. - This same repository, Module 5, lesson 8 (
08-project-andes-cargos-incident-response-plan.md) —INCIDENT-RESPONSE-PLAN.md, the source of the rule that resolves Step 5's apparent contradiction. - Google SRE Book — Postmortem Culture and Google SRE Workbook — Postmortem Culture — the blameless discipline applied a second time in this guide.