Module 8: Capstone The Andes Cargo Reliability Package

2. Architecture review: the full reliability machine

Next →

Description

Before introducing any new data, this lesson does something no earlier lesson in this guide had reason to do: look at the complete andes-cargo-infra/, all at once, the way someone who never saw the previous seven modules would, having to understand, in minutes, what each file builds and how it connects to the others. It's a review, not a new lesson — every piece appearing here is already built, already verified, and already has its own origin lesson cited.

Connection to the module

This lesson verifies the complete inventory with a real command, not a list written from memory: find andes-cargo-infra -type f | sort on the repository you built across Modules 1 through 7 should return exactly the files this lesson names. Lesson 3 introduces this whole module's only genuinely new file on top of this same structure, without modifying any of the ones that already exist.

Step 1 — The complete inventory, verified

find andes-cargo-infra -type f \
  \( -name "*.md" -o -name "*.py" -o -name "*.tf" -o -name "*.yml" -o -name "*.json" \) \
  | sort

What to expect (literal — you built this structure, file by file, across Modules 1 through 7; this command only confirms it):

andes-cargo-infra/ALERTING-POLICY.md
andes-cargo-infra/INCIDENT-RESPONSE-PLAN.md
andes-cargo-infra/RELIABILITY-CHARTER.md
andes-cargo-infra/RELIABILITY-POSTMORTEM-PACKAGE.md
andes-cargo-infra/SLO.md
andes-cargo-infra/incidents/2026-02-26-claude-code-destroy/POSTMORTEM.md
andes-cargo-infra/incidents/2026-02-26-claude-code-destroy/TIMELINE.md
andes-cargo-infra/observability.tf
andes-cargo-infra/observability/alert_rules.yml
andes-cargo-infra/observability/alert_webhook_receiver.py
andes-cargo-infra/observability/alertmanager.yml
andes-cargo-infra/observability/burn_rate_exporter.py
andes-cargo-infra/observability/docker-compose.yml
andes-cargo-infra/observability/instrument_manifest_flow.py
andes-cargo-infra/observability/manifest-log-events.json
andes-cargo-infra/observability/prometheus.yml
andes-cargo-infra/observability/upload_manifest_batch.py
andes-cargo-infra/oncall/schedule.py
andes-cargo-infra/runbooks/manifest-processor-error-rate.md
andes-cargo-infra/scripts/burn_rate_evaluator.py
andes-cargo-infra/scripts/error_budget_calculator.py

Twenty-one files, none of them business logic — no main.tf, nothing from process-shipment-manifest itself, nothing from the bucket or from Shipments. Every one of these twenty-one files is, exclusively, the reliability layer this guide added on top of what the six sibling guides already built: five portfolio documents, two SRE-math scripts, an observability directory with nine pieces, one Terraform file, an on-call generator, a runbook, and two documents from the already-operated incident.

Step 2 — The complete chain, from measurement to operation

   ANDES CARGO'S RELIABILITY MACHINE -- SEVEN MODULES, ONE CHAIN

   ┌─────────────────────────────────────────────────────────────────────┐
   │  M2 -- WHAT TO MEASURE                                               │
   │  SLO.md: SLI = good events / valid events for                        │
   │  process-shipment-manifest. SLO = 99.9% monthly.                     │
   │  scripts/error_budget_calculator.py: the formula, executable.        │
   └─────────────────────────────┬─────────────────────────────────────┘
                                  │ the SLI needs real data
                                  ▼
   ┌─────────────────────────────────────────────────────────────────────┐
   │  M3 -- WHERE THE REAL DATA COMES FROM                                │
   │  observability/upload_manifest_batch.py uploads manifests via S3     │
   │  (the real trigger, never lambda invoke) -> CloudWatch Invocations/  │
   │  Errors -> real logs (jq) -> real traces (Jaeger, OTel)               │
   └─────────────────────────────┬─────────────────────────────────────┘
                                  │ the real data feeds the alert
                                  ▼
   ┌─────────────────────────────────────────────────────────────────────┐
   │  M4 -- WHEN THE ALARM SOUNDS                                         │
   │  ALERTING-POLICY.md: burn rate, not a static threshold. Three         │
   │  engines: scripts/burn_rate_evaluator.py (Python prototype) +        │
   │  Prometheus/Alertmanager (observability/alert_rules.yml,             │
   │  burn_rate_exporter.py) + CloudWatch (observability.tf, real          │
   │  metric math)                                                        │
   └─────────────────────────────┬─────────────────────────────────────┘
                                  │ the alert fires -> someone responds
                                  ▼
   ┌─────────────────────────────────────────────────────────────────────┐
   │  M5 -- WHO RESPONDS, WITH WHAT RULES                                 │
   │  INCIDENT-RESPONSE-PLAN.md: 5-stage lifecycle, severity matrix        │
   │  tied to M4's burn rate thresholds, three roles (IC/OL/CL),           │
   │  oncall/schedule.py -- a real deterministic rotation                  │
   └─────────────────────────────┬─────────────────────────────────────┘
                                  │ the framework already exists -> it gets operated against a case
                                  ▼
   ┌─────────────────────────────────────────────────────────────────────┐
   │  M6 -- THE OPERATED INCIDENT (case 1: Claude Code destroy)           │
   │  incidents/2026-02-26-claude-code-destroy/TIMELINE.md: the 5 stages  │
   │  run against real, verified facts, SEV1 by the independent            │
   │  data-loss criterion (no measurable burn rate)                        │
   └─────────────────────────────┬─────────────────────────────────────┘
                                  │ the incident ends -> learning happens, blamelessly
                                  ▼
   ┌─────────────────────────────────────────────────────────────────────┐
   │  M7 -- BLAMELESS CLOSE + THE TOOL FOR NEXT TIME                      │
   │  incidents/.../POSTMORTEM.md: 4 root causes, none a person.           │
   │  runbooks/manifest-processor-error-rate.md: the first real runbook    │
   │  -- what whoever's on call follows the NEXT time this fires           │
   └─────────────────────────────┬─────────────────────────────────────┘
                                  │
                                  ▼
   ┌─────────────────────────────────────────────────────────────────────┐
   │  M8 -- THIS MODULE: THE SAME CHAIN, UNCHANGED, AGAINST A NEW          │
   │  CASE (case 2: deterministic synthetic incident, lesson 3)            │
   └─────────────────────────────────────────────────────────────────────┘

Notice this diagram's central property: every arrow points forward exactly once. SLO.md doesn't depend on INCIDENT-RESPONSE-PLAN.md; INCIDENT-RESPONSE-PLAN.md does depend on ALERTING-POLICY.md (it cites its burn rate thresholds without repeating them); TIMELINE.md depends on INCIDENT-RESPONSE-PLAN.md (it applies its lifecycle without redefining it); POSTMORTEM.md depends on TIMELINE.md (it cites its facts without repeating them). It's the same single-source-of-truth discipline every project in this guide has already applied separately — seen here, for the first time, as one complete seven-link chain.

Step 3 — The three alert engines, side by side, again

Module 4's lesson 8 already tested this once, with a forced drill over two fixed scenarios (bad_week, normal). It's worth recalling before this module's lesson 3 introduces a third scenario:

EngineWhere it livesWindow coverageOperational cost
scripts/burn_rate_evaluator.pyModule 4, lesson 3Multi-window, all 3 severities in fullNone — pure Python
Prometheus + Alertmanagerobservability/alert_rules.ymlReal multi-window, via and ignoring(window)Docker, an exporter to maintain
CloudWatch Alarmobservability.tfSingle window only (a real, declared limitation)None — native to AWS, no exporter

All three implement the same formula (burn_rate = observed_error_rate / allowed_error_rate, with allowed_error_rate = 0.001 from SLO.md), with different syntax and coverage. This module's lesson 4 is going to run all three again, against a third scenario none of them has seen before.

Step 4 — What's business logic and what's reliability, in the same repository

A detail worth stating explicitly before continuing: andes-cargo-infra/ isn't exclusively this guide's. It contains the andes-cargo-shipment-docs bucket, the Shipments table, the process-shipment-manifest Lambda, the IAM roles, the .github/workflows/ pipeline, the conftest/Trivy/cosign security gate, and the Infracost cost gate — all inherited whole from the six sibling guides, with not a single resource rewritten. This guide never touched that code. Step 1's twenty-one files are, exclusively, the layer added on top: it measures how well what already exists is working, and what to do when it isn't. No file on this list changes process-shipment-manifest's behavior — all of them observe it, alert on it, or document the response when something goes wrong with it.

   TWO LAYERS, ONE REPOSITORY

   BUSINESS LAYER (6 sibling guides)          RELIABILITY LAYER (this guide)
   ────────────────────────────────           ─────────────────────────────────
   main.tf, S3, Shipments, Lambda,             SLO.md, ALERTING-POLICY.md,
   IAM, .github/workflows/,                    INCIDENT-RESPONSE-PLAN.md,
   conftest, cosign, Infracost                 observability/, oncall/,
                                                incidents/, runbooks/
        │                                              │
        ▼                                              ▼
   "Does the system do what                    "Do we know if the system
    it's supposed to do?"                       is still doing it well, and
                                                 what do we do if it stops?"

Common mistakes

Treating this lesson as if it needed to reconstruct the content of the seven inherited pieces, instead of just citing them (repeating the same disciplined-assembly error every project in this guide has already avoided). What happens: someone, writing their own summary of this lesson, copies whole paragraphs from SLO.md or INCIDENT-RESPONSE-PLAN.md, instead of referencing them by path. How to spot it: if your version of this lesson duplicates textual content from a document already written in an earlier module. How to fix it: this lesson is, deliberately, a map — every box in Step 2's diagram points to a file and an origin lesson; this lesson's job is to show how they connect, not to repeat what each one already says on its own.

Confusing "the complete machine" with "every file in andes-cargo-infra/," including the business ones inherited from the six sibling guides (losing Step 4's boundary). What happens: someone, describing this guide in an interview, includes the Lambda, the bucket, or the CI/CD pipeline as if this guide had built them. How to spot it: if your inventory of "what this guide built" includes any file not on Step 1's list of twenty-one. How to fix it: Step 4's boundary is exact — this guide adds the reliability layer, it never touches or rewrites the business layer. An interviewer who asks "did you build the Lambda?" deserves a "no, I inherited it from aws-serverless-and-containers-guide — what I built is how it's measured and operated when something goes wrong with it," not an incorrect attribution.

Assuming Step 2's diagram's arrow order is just a pretty way of presenting it, with no real consequence if read in a different order (underestimating why the direction matters). What happens: someone tries to explain INCIDENT-RESPONSE-PLAN.md without first understanding ALERTING-POLICY.md, and ends up unable to justify where their severity matrix's exact thresholds come from. How to spot it: if, explaining any piece of the chain, you have to "guess" a number instead of pointing to which earlier piece it comes from. How to fix it: the diagram's order is a real dependency, not a presentation preference — every link cites the one before it because it literally needs it to justify itself (the severity matrix would have no threshold to cite without ALERTING-POLICY.md; ALERTING-POLICY.md would have no threshold to evaluate without SLO.md). Understanding the chain in order means understanding why every document exists, not just what it says.

Exercises

Exercise 1 — Without looking at Step 1, write from memory the five portfolio documents (.md, at the root of andes-cargo-infra/, not counting the ones living inside incidents/ or runbooks/) this guide built through Module 7. Check your answer against Step 1.

See solution

RELIABILITY-CHARTER.md (Module 1), SLO.md (Module 2), ALERTING-POLICY.md (Module 4), INCIDENT-RESPONSE-PLAN.md (Module 5), RELIABILITY-POSTMORTEM-PACKAGE.md (Module 7) — five documents, each its own module's final project, each citing the previous one instead of repeating it. If your list had one document too many or too few, check against Step 1: the incident's two documents (TIMELINE.md, POSTMORTEM.md) live inside incidents/2026-02-26-claude-code-destroy/, not at the root, and the runbook lives inside runbooks/ — none of the three counts as one of the five "root portfolio documents."

Exercise 2 — Explain, using Step 2's diagram, why TIMELINE.md (Module 6) couldn't have been written before INCIDENT-RESPONSE-PLAN.md (Module 5) existed.

See solution

To make sense, TIMELINE.md needs a framework to classify the incident against: without INCIDENT-RESPONSE-PLAN.md, there would be no severity matrix to decide the Claude Code incident is SEV1 against, no set of roles (IC/OL/CL) to assign to specific people, and no five-stage lifecycle to structure the document against. Module 5 exists, deliberately, before Module 6 — Module 5's introduction already said it explicitly: "the framework before the case" — precisely so operating the real incident means applying an already-finished framework, not improvising one as the incident gets analyzed.

Exercise 3 — A classmate argues Step 4's "two layers" diagram is unnecessary, because "in the end everything lives in the same Terraform repository, so the distinction doesn't matter in practice." Do you agree?

See solution

Disagree. Both layers living in the same repository doesn't erase a real difference in purpose and ownership: the business layer answers "does the system do what it's supposed to do?" and the six sibling guides modify it; the reliability layer answers "do we know if it's still doing it well, and what to do if it isn't?" and this guide modifies it. This distinction matters in practice in a very concrete way: if someone needs to change Shipments's schema, that change belongs to aws-serverless-and-containers-guide, not this guide; if someone needs to adjust an alarm's threshold, that change belongs here, not to that guide. Without the explicit distinction, it would be easy for someone to try "fixing" a reliability problem by rewriting the Lambda, when the correct change lives in observability.tf or in the runbook — exactly the kind of layer confusion this lesson exists to prevent.

Summary and next step

This lesson confirmed, with a real command (find), the complete inventory of the twenty-one files this guide built across Modules 1 through 7, and traced the complete seven-link chain connecting them: what to measure (M2) → where the data comes from (M3) → when the alarm sounds (M4) → who responds (M5) → the operated incident (M6) → the blameless close and the tool for next time (M7). You confirmed, with the two-layer diagram, the exact boundary between what this guide built (the reliability layer) and what it inherited untouched (the six sibling guides' business layer).

Before moving on you should be able to: recite the chain's seven links in order, with the exact file representing each; explain why that chain's order reflects real dependencies, not just a way of presenting it; and defend the boundary between business layer and reliability layer against someone who considers it irrelevant.

Lesson 3 introduces this module's first genuinely new data: a fixed, deterministic synthetic incident, which is going to enter through the same point as always — the real bucket — and is going to test, for the first time, whether this complete chain reacts without anyone having prepared it for this specific case.

Resources

  1. This same repository, all of Modules 2, 4, 5, 6, and 7 — the source of every piece named in Step 2's diagram.
  2. This same repository, Module 4, lesson 8 (08-project-andes-cargos-alerting-policy.md) — the direct precedent for the three-engine drill this lesson's Step 3 summarizes.
  3. Google SRE Book — Table of Contents — the original source of the complete discipline this seven-link chain implements.
← Previous1. Introduction to the capstoneNext →3. Hands-on: introducing a deterministic synthetic incident