Module 8: Capstone The Andes Cargo Reliability Package

1. Introduction to the capstone

Description

Seven modules built, one at a time, every piece of a complete reliability machine: a real SLI definition (Module 2), measured with real data from the legacy Lambda (Module 3), watched by an alert that fires for the right reason — accelerated error budget consumption, not a loose threshold — across three different engines (Module 4), operated with a lifecycle, severity, roles, and on-call framework (Module 5), applied end to end against the real Claude Code incident (Module 6), and closed with a blameless postmortem and Andes Cargo's first operational runbook (Module 7). Every module, until now, tested its own piece separately.

This module doesn't add a new piece. It runs the whole machine, one more time, against a case no earlier lesson built on purpose to work well. A new synthetic incident — deterministic, never random, as verifiable as any dataset in this guide — enters through the same point any real traffic would: a sequence of manifests uploaded to the real bucket. From there, with no code or configuration change to any of the seven already-built pieces, the complete machine has to do, on its own, everything it promised: measure, alert, declare, mitigate, and close with a blameless document. If this guide's capstone has one single question, it's this: does the machine you built work on a case you didn't see coming?

Connection to the module

Every piece this module uses already exists, built and verified in the previous seven modules: SLO.md (Module 2), scripts/error_budget_calculator.py and scripts/burn_rate_evaluator.py (Modules 2 and 4), the complete observability/ — Prometheus, Grafana, Alertmanager, Jaeger, the burn rate exporter — (Modules 3 and 4), observability.tf with the real CloudWatch alarm (Module 4), INCIDENT-RESPONSE-PLAN.md with the lifecycle, severity matrix, roles, and on-call rotation (Module 5), the complete precedent of operating a real incident start to finish (Module 6), and Andes Cargo's first operational runbook alongside the blameless postmortem template (Module 7). This module declares no new business HCL resource, introduces no tool the previous seven guides haven't already used — its only genuinely new work is a second data scenario, as fixed and deterministic as any other in this guide, and the final portfolio document that brings everything together.

The analogy: the end-to-end emergency drill

Every previous module in this guide trained an isolated piece of the same protocol: how to measure whether something's wrong (Module 2), how to get the real data that measurement needs (Module 3), how to sound the right alarm at the right moment (Module 4), who responds and with what rules (Module 5). Practicing each piece separately is necessary, but it isn't the same as a real end-to-end drill: the alarm sounds for real, with nobody knowing beforehand exactly when or with exactly what texture it will sound; the whole team — not a single role practicing in isolation — has to coordinate under the same relative time pressure a real incident would impose; the fire gets put out with the same tools that already existed, not one improvised for the occasion; and at the end, someone has to write the report, with the same blameless discipline as always, so the protocol itself improves next time.

This module is that complete drill. The difference from a generic drill — the same one Module 6 already established when operating the Claude Code incident against real facts, not invented ones — is that here it isn't even necessary to invent the final test scenario: Module 8.3 builds a fixed, deterministic sequence of malformed invocations, new, never run before in this guide, and everything else — the alert, the declaration, the runbook, the postmortem — has to react without any earlier lesson giving it a head start.

   SEVEN MODULES, ONE PIECE EACH                THIS MODULE -- THE COMPLETE MACHINE
   ──────────────────────────────────          ─────────────────────────────────
   M2: how to measure (SLI/SLO/budget)           A NEW synthetic incident enters
   M3: where the real data comes from            through the same point real
   M4: how the right alarm sounds                traffic would
   M5: who responds, with what rules                          |
   M6: the protocol run once,                                 v
       against an already-known case             M2-M4 measure and alert on it
   M7: how it closes blamelessly                  with NO changes -- they never
        │                                         saw these specific numbers before
        ▼                                                     |
   Each piece tested separately                                v
                                                  M5-M7 operate and close it
                                                  with the SAME framework, no
                                                  adjustments custom-fit to this case

What you're going to deliver in this module

Eight lessons, each a step of the same closing journey:

#	Lesson	What it produces
1	This introduction	The capstone's complete map, and why "running the complete machine" is different from "reviewing each piece"
2	Architecture review	The complete diagram of the reliability machine, overlaid on `andes-cargo-infra/` as it exists today, with the seven inherited pieces named by their exact path
3	Introducing a deterministic synthetic incident	`observability/upload_synthetic_incident_batch.py` — a fixed sequence of 25 manifests, 10 malformed for the same reason, never seen before in this guide
4	The alert fires, the incident gets declared	Module 4's three engines, with no code change, firing on the new incident; the formal declaration with roles from a different week of the rotation
5	The runbook mitigates, the postmortem closes	Module 7's runbook executed step by step against this case; a second `POSTMORTEM.md`, shorter, that proves the process — not just the original example — works
6	What this guide left representative	One single place, with the exact technical reason for each unexecuted piece: X-Ray, Application Signals, Incident Manager, PagerDuty/Opsgenie, and the real result of Module 7's backup attempt
7	What Andes Cargo still needs	The map toward the rest of the ecosystem: EKS and its own SRE discipline, SRE for AI systems, deep observability as an independent discipline
8	Final project: the complete reliability package	`RELIABILITY-PACKAGE.md` — the index of everything this guide built, as a portfolio piece

An honesty that needs to be clear from this first lesson

This module's synthetic incident (lessons 3 through 5) is different, in one important way, from the Claude Code incident Module 6 operated: it didn't happen to anyone, real or fictional, outside this guide. It has no external primary source to verify, because none is needed — its only job is to be deterministic and new: a fixed sequence, written into a script, that none of this guide's seven earlier pieces ever saw during their own construction. That's, in fact, the proof this module needs: if Module 4's alert, Module 5's framework, and Module 7's runbook only worked on the exact data they were built and tested with — BAD_WEEK, Module 3's 20-manifest batch, the Claude Code incident — they would prove nothing about whether the machine generalizes. A genuinely new case, even one invented on purpose for this module, is the only honest way to test that generalization.

The same hard rule as always still applies with no exception: never random, never datetime.now(). Lesson 3's synthetic incident is as fixed, as committed, as reproducible digit by digit as TRAFFIC_30_DAYS or BAD_WEEK from Module 2 — the only difference is that nobody, in any previous module, adjusted any piece of the machine so this specific case would work out well.

Common mistakes

Expecting this module to introduce a new tool, a different alert engine, or an alternative incident framework (anticipating content this module doesn't have). What happens: someone arrives at this capstone expecting something genuinely different from what the previous seven modules already built — maybe a new integration, an additional dashboard, a managed service not yet named. How to spot it: if your expectation for this module is "what new piece of infrastructure am I going to learn here?" How to fix it: this module, deliberately, adds no new piece of infrastructure — it runs the seven already-built pieces against a new case. The value isn't in learning something different, it's in confirming, with executed evidence, that what you already learned generalizes beyond the exact case you built it with.

Treating this module's synthetic incident as if it carried the same narrative weight as the Claude Code incident, with a primary source to cite (confusing "deterministic" with "externally verified"). What happens: someone looks, for lesson 3's incident, for the same kind of independent source — Hacker News, incidentdatabase.ai, technical coverage — Module 6 cited for the Claude Code incident. How to spot it: if your summary of this module's incident tries to attribute it to a real external company or case. How to fix it: this module's synthetic incident needs no external source, and has none — it's test data designed on purpose, with the same fixed, deterministic discipline as any dataset in this guide since Module 2, but with no pretense of having happened to anyone real. Its function is to test the process, not to document a fact of the world.

Assuming "running the complete machine" means rewriting every script from scratch, instead of reusing exactly what already exists (underestimating how much of this module is reuse, not construction). What happens: someone, reaching lesson 4, starts rewriting burn_rate_exporter.py or alert_rules.yml from scratch, instead of extending them with the minimum necessary change. How to spot it: if your copy of any observability/ file in this module differs, by more than the minimum necessary for the new scenario, from the version Module 4 already finished. How to fix it: this module's lesson 4 is explicit about this — alert_rules.yml's PromQL rules don't change a single line; the only thing added is a third scenario value in the exporter. That the machine works without touching its own logic is precisely the proof this module is looking for.

Exercises

Exercise 1 — In your own words, explain why a synthetic incident, designed on purpose for this module, is a more honest test of the complete machine than repeating the Claude Code incident a second time.

See solution

Repeating the Claude Code incident would necessarily produce the same result Module 6 already got — the same numbers, the same severity classification, the same mitigation — because every piece of the machine was already tested exactly against those facts. That would prove nothing new: it would confirm the machine works on the case it's already known to work on. A new synthetic incident, in contrast, forces every piece — the burn rate exporter, the Alertmanager rules, the severity matrix, the runbook — to react to numbers nobody adjusted on purpose to come out well. If the complete machine produces the correct result on data it never saw, that's real proof the design generalizes, not just that it was calibrated for a specific case.

Exercise 2 — Lesson 5 is going to write a second POSTMORTEM.md, shorter than Module 7's. Before reading it, predict: which parts of a blameless postmortem's structure (Module 7, lesson 2) would you expect to stay the same, and which would you expect to shrink?

See solution

What should stay the same: the complete blameless discipline (no person's name as a cause), the trigger-vs-root-cause distinction, and the general section structure (summary, impact, root cause, what went right/wrong, action items) — that discipline doesn't depend on the incident's scale. What would reasonably shrink: the number of root causes (the Claude Code incident had four, across different layers of a completely destroyed system; a single error-alarm incident probably has a more concentrated cause), the impact section (an incident resolved in minutes or hours, not 24 hours, has less to narrate), and potentially the number of action items (fewer root causes, fewer direct actions to derive from them). The correct proportion is "as rigorous as needed, not as long as the previous example" — a short, blameless postmortem isn't an incomplete postmortem, if the incident itself was simpler.

Exercise 3 — INCIDENT-RESPONSE-PLAN.md (Module 5) ends with the phrase: "Module 8's capstone runs a new synthetic incident through the same framework, unchanged, to confirm it works on a case nobody saw coming." Explain why the word "unchanged" in that sentence is this entire module's central promise.

See solution

"Unchanged" is the condition that makes this module's exercise mean something: if any piece of the machine — the lifecycle, the severity matrix, the roles, the rotation, the alert, the runbook — had to be modified for the synthetic incident to work correctly, that would prove the original framework had a gap only discovered by forcing it with a new case, not that the framework generalizes. This word's promise is that the seven pieces built in Modules 2 through 7 are, truly, a reusable framework — not a solution custom-fit to the one case it was tested with — and this module exists precisely to put that promise to the test with evidence, not with an unverified claim.

Summary and next step

This lesson established the capstone's complete purpose: run, with no changes, the complete reliability machine the previous seven modules built — SLI, real data, burn rate alert, incident lifecycle, roles, mitigation, blameless postmortem — against a new, deterministic synthetic incident that no piece in this guide saw during its own construction. You saw the map of the eight lessons, and the honesty holding up the whole module: this incident didn't happen to anyone external, real or fictional — it's test data designed to confirm the framework generalizes, with this entire guide's same "never random, never datetime.now()" discipline.

Before moving on you should be able to: explain the difference between "repeating the Claude Code incident" and "testing the machine with a new case"; name the seven inherited pieces this module is going to run unmodified; and anticipate, in general terms, which parts of a blameless postmortem reasonably shrink when the incident itself is simpler.

Lesson 2 reviews, with a complete diagram, how the seven inherited pieces fit together before introducing the new incident — the complete map of andes-cargo-infra/ as it exists today, with every file in its exact place.

Resources

This same repository, Module 5, lesson 8 (08-project-andes-cargos-incident-response-plan.md) — INCIDENT-RESPONSE-PLAN.md, the source of this module's promise, quoted in full in its "Consequences" section.
This same repository, Module 7, lesson 8 (08-project-andes-cargos-postmortem-and-runbooks-package.md) — RELIABILITY-POSTMORTEM-PACKAGE.md, the same portfolio-document pattern this module is going to extend in its lesson 8.
Google SRE — Incident Management Guide — the complete framework this module puts to the test a second time.
Google SRE Workbook — Alerting on SLOs — the burn rate math this module's lesson 4 runs, unchanged, on new data.

← Previous8. Project: Andes Cargo's postmortem and runbooks package Next →2. Architecture review: the full reliability machine