Module 8: Capstone The Andes Cargo Reliability Package

8. Final project: Andes Cargo's reliability package as a deliverable

Description

Eight modules built, one at a time, a complete reliability machine, and this module just ran it twice — against a real incident and against a synthetic one — with the same correct result both times. This final project writes no new calculation, no new alert, no new incident document — it brings together everything that exists in andes-cargo-infra/ into RELIABILITY-PACKAGE.md, this entire guide's complete portfolio index, with the same assemble-without-inventing discipline every project in every module has already applied separately.

Connection to the module

This document does, for the complete guide, what RELIABILITY-POSTMORTEM-PACKAGE.md (Module 7) did for its four artifacts: cite paths and results, never repeat content. It's the last document this guide produces — the one an interviewer, a technical reviewer, or you yourself, six months later, would open first to understand what you built and how it all connects.

Step 1 — The final inventory, verified

find andes-cargo-infra -type f \
  \( -name "*.md" -o -name "*.py" -o -name "*.tf" -o -name "*.yml" -o -name "*.json" \) \
  | sort

What to expect (literal — twenty-three files: the twenty-one Module 8, lesson 2 already confirmed, plus the two this module added):

andes-cargo-infra/ALERTING-POLICY.md
andes-cargo-infra/INCIDENT-RESPONSE-PLAN.md
andes-cargo-infra/RELIABILITY-CHARTER.md
andes-cargo-infra/RELIABILITY-POSTMORTEM-PACKAGE.md
andes-cargo-infra/SLO.md
andes-cargo-infra/incidents/2026-02-26-claude-code-destroy/POSTMORTEM.md
andes-cargo-infra/incidents/2026-02-26-claude-code-destroy/TIMELINE.md
andes-cargo-infra/incidents/2026-03-17-manifest-schema-regression/POSTMORTEM.md
andes-cargo-infra/observability.tf
andes-cargo-infra/observability/alert_rules.yml
andes-cargo-infra/observability/alert_webhook_receiver.py
andes-cargo-infra/observability/alertmanager.yml
andes-cargo-infra/observability/burn_rate_exporter.py
andes-cargo-infra/observability/docker-compose.yml
andes-cargo-infra/observability/instrument_manifest_flow.py
andes-cargo-infra/observability/manifest-log-events.json
andes-cargo-infra/observability/prometheus.yml
andes-cargo-infra/observability/synthetic-incident-log-events.json
andes-cargo-infra/observability/upload_manifest_batch.py
andes-cargo-infra/observability/upload_synthetic_incident_batch.py
andes-cargo-infra/oncall/schedule.py
andes-cargo-infra/runbooks/manifest-processor-error-rate.md
andes-cargo-infra/scripts/burn_rate_evaluator.py
andes-cargo-infra/scripts/error_budget_calculator.py

Twenty-four files, with this lesson's RELIABILITY-PACKAGE.md as the twenty-fifth. incidents/ now has two subdirectories — physical proof the machine ran twice, not once; observability/ has two invocation batches (Module 3's upload_manifest_batch.py, this module's upload_synthetic_incident_batch.py) and two saved log files, one for each case a runbook actually diagnosed.

Step 2 — The final portfolio document

At the root of andes-cargo-infra/, create RELIABILITY-PACKAGE.md:

# RELIABILITY-PACKAGE.md — Andes Cargo's Complete Reliability Machine

**Status:** Final · **Closes:** `sre-and-incident-response-guide`, all 8 modules
**Built on:** every document and script this guide produced — this index cites each one by
path, it does not repeat any of their content.

## What "reliable" means for this system, and how it's measured

`RELIABILITY-CHARTER.md` (Module 1) names what reliability means for `process-shipment-manifest`
and maps the 8-module plan. `SLO.md` (Module 2) defines the SLI (good events / valid events),
the SLO (99.9% monthly), and the error budget (43.2 minutes) in an executable calculator,
`scripts/error_budget_calculator.py` — not just a formula in prose.

## Where the data comes from

`observability/` — Prometheus `v3.13.2`, Grafana `13.1.3`, Alertmanager `v0.33.1`, Jaeger v2
(`jaegertracing/jaeger`, never `all-in-one`), all verified running in this environment (Module 3).
`observability/upload_manifest_batch.py` generates real, deterministic traffic through the real
S3 trigger path — never a manual `lambda invoke` — feeding CloudWatch's native `Invocations`/
`Errors` metrics for `process-shipment-manifest`.

## When it alerts, and why

`ALERTING-POLICY.md` (Module 4) is the burn-rate policy — never a static threshold — implemented
three times over: `scripts/burn_rate_evaluator.py` (the Python prototype), `observability/
alert_rules.yml` (real Prometheus/Alertmanager rules, `and ignoring(window)` for the two-window
confirmation), and `observability.tf` (a real CloudWatch alarm with metric math). All three agree
on every scenario tested against them — `bad_week`, `normal`, and `synthetic_incident` (Module 8).

## Who responds, and how

`INCIDENT-RESPONSE-PLAN.md` (Module 5) is the response framework — five-stage lifecycle, a
severity matrix mapped directly onto `ALERTING-POLICY.md`'s burn-rate tiers, three roles (IC/OL/CL)
that never merge decision and execution in a SEV1/SEV2, and `oncall/schedule.py`, a deterministic
rotation with no SaaS and no randomness.

## Two incidents, the same framework, unchanged

| Incident | Path | Severity criterion | Postmortem |
|---|---|---|---|
| Claude Code `destroy` (real, external, DataTalks.Club) | `incidents/2026-02-26-claude-code-destroy/` | SEV1 via the independent data-loss criterion — no burn rate was measurable | `POSTMORTEM.md`, 171 lines, 4 root causes (Module 7) |
| Manifest schema regression (synthetic, Module 8) | `incidents/2026-03-17-manifest-schema-regression/` | SEV1 via the primary burn-rate criterion (400.00x short window, 25.00x long window) | `POSTMORTEM.md`, 67 lines, 1 root cause (Module 8) |

One incident this framework did not design itself to handle; one it did. Same lifecycle, same
severity matrix, same roles, same runbook — `runbooks/manifest-processor-error-rate.md` resolved
the second one on its first real use of Branch A, the branch its original worked example never
demonstrated.

## What this package does not do

It does not re-verify any individual artifact's own content — `SLO.md`, `ALERTING-POLICY.md`,
`INCIDENT-RESPONSE-PLAN.md`, and `RELIABILITY-POSTMORTEM-PACKAGE.md` each already passed their
own module's verification (line counts, section counts, forced drills), cited here, not repeated.
It does not claim this machine is complete for every kind of system — Module 8, lesson 7 named,
with precision, the three directions (EKS, AI systems, deep observability) where the same
vocabulary needs a different guide and a different set of signals. It does not replace the
preventive controls `cloud-security-and-guardrails-guide` already built — this package measures
and responds; it does not gate a deploy or protect a resource from deletion.

## What a reader should be able to answer from this package alone

**What does "reliable" mean here, with a number, not a feeling?** (`SLO.md`, cited above.)
**What data feeds that number, and is it real?** ("Where the data comes from", Module 3's stack,
verified running.) **When does this system alert, and why is that the right moment?**
(`ALERTING-POLICY.md`, three engines, proven to agree.) **Who responds, in what order, with what
authority?** (`INCIDENT-RESPONSE-PLAN.md`, cited above.) **Does the framework actually work, or
only on the one case it was built to explain?** (the two-incident table — one case built the
framework, one case tested it against something new.)

## Consequences

This is the last document `sre-and-incident-response-guide` produces. Module 8, lesson 7's map
(`kubernetes-and-eks-in-production-guide`, `genai-on-aws-production-guide`,
`monitoring-observability-guide`) names where this same vocabulary goes next, with different
systems behind it — not built here, by design.

Step 3 — Verifying the final package

wc -l RELIABILITY-PACKAGE.md
grep -c '^## ' RELIABILITY-PACKAGE.md
grep -c '^|' RELIABILITY-PACKAGE.md

What to expect (literal — you assembled the content, the shape is deterministic):

55
7
5

Fifty-five lines, seven sections, five table lines (two data rows plus three header/separator lines in the two-incidents table) — the shortest document in this entire guide's portfolio, exactly as fits an index of indexes: every statement in this document points to an artifact already verified, line by line, in its own module.

Step 4 — Defending the complete project in an interview

With RELIABILITY-PACKAGE.md finished, you have an hour-long technical conversation's summary reduced to a fifty-five-line document, with every statement backed by an actually executed artifact. A reasonable walkthrough, if an interviewer asked "walk me through your SRE project":

  1. "What is reliability here, in numbers?" — open SLO.md, show python3 scripts/error_budget_calculator.py, and the literal output: 99.9098% measured SLI, 4.23 minutes of budget remaining out of 43.2.
  2. "How do you know that number is real, not invented?" — observability/upload_manifest_batch.py, the real S3 trigger, awslocal cloudwatch get-metric-statistics reading real Invocations/Errors.
  3. "When does the alarm sound, and why at that moment and not sooner?" — ALERTING-POLICY.md, Google SRE's multi-window, multi-burn-rate pattern, the three engines agreeing on the same verdict over the same data.
  4. "What happens when it actually sounds?" — INCIDENT-RESPONSE-PLAN.md, and the Claude Code incident operated end to end with that structure.
  5. "How do you know that framework isn't custom-fit to a single case?" — Module 8's synthetic incident: same framework, no change at all, against a case nobody adjusted for it to work out well.

Five questions, five artifacts, no improvised answer — each one is, literally, the result of a command you actually ran.

Common mistakes

Presenting RELIABILITY-PACKAGE.md as the document that "contains" the complete project, instead of as the index pointing toward it (losing, in the guide's final document, the assembly discipline that governed every earlier one). What happens: someone, preparing this project for an interview or a portfolio, shares only RELIABILITY-PACKAGE.md, without the twenty-four files it indexes. How to spot it: if your delivery of this project is a single fifty-five-line file, with no link or access to the documents it cites. How to fix it: this document is, deliberately, a map — its complete value depends on every path it cites existing and being consultable; sharing it in isolation, without the full repository behind it, is like handing over only a book's index and calling it the book.

Treating the "Two incidents" table as if both carried the same narrative weight, without distinguishing which is real and which is synthetic (repeating, in the final document, the same care Module 8, lesson 1 and lesson 5 already demanded separately). What happens: someone, summarizing this project, describes the table's two incidents as if both had happened to a real company. How to spot it: if your description of "the two incidents this project handles" doesn't distinguish which one has a verifiable external primary source and which is an internal construction of this module. How to fix it: Step 2's table already labels it precisely — "real, external, DataTalks.Club" versus "synthetic, Module 8" — that distinction, already established in every lesson where it appeared, keeps applying in the final document just as much as the first time it was declared.

Considering the project "finished" without having run, at least once, Step 3's verification commands on your own copy (repeating, for the last time in this guide, the same error every earlier project already warned about). What happens: someone copies Step 2's RELIABILITY-PACKAGE.md and calls the project complete without confirming their own Step 1 find returns the same twenty-four files. How to spot it: if you can't reproduce, on your own copy of the repository, Step 1's exact inventory. How to fix it: this guide's deterministic verification discipline — wc -l, grep -c, a command whose output admits no ambiguity — doesn't relax in the final document; if your inventory differs from Step 1, there's a lesson in this module, or an earlier one, left incomplete.

Exercises

Exercise 1 — Verify Step 3 against your own copy of RELIABILITY-PACKAGE.md and confirm you get exactly 55, 7, and 5.

See solution

Copying Step 2's document exactly as it appears, wc -l counts 55 lines, grep -c '^## ' finds seven level-2 section headers, and grep -c '^|' finds five lines starting with a vertical bar — the two-incidents table's three lines (header, separator, and the two data rows count as four; if your count gives a different number, check that Step 2's table has no extra or missing line). This final document's deterministic verification follows the exact same pattern as every earlier document in this guide.

Exercise 2 — An interviewer, after hearing Step 4's five-question walkthrough, asks: "of all this, what was hardest to build?" Using what you know from the whole guide, how would you answer with a specific, not generic, response?

See solution

A specific, defensible answer would cite the multi-window burn rate pattern (Module 4) as the technically most demanding piece — not for the math itself (a division and a comparison), but for understanding why two windows are needed at once, and correctly expressing that condition in three different syntaxes (pure Python, PromQL with and ignoring(window), and the honest limitation that CloudWatch, in its simplest form, can't do it with a single alarm). A generic answer ("everything was hard" or "learning Terraform") doesn't demonstrate the same level of understanding as precisely pointing to the exact point where the real difficulty was — the same specificity discipline every "Common mistakes" section in this guide already modeled, applied here to an interview question.

Exercise 3 — Explain why RELIABILITY-PACKAGE.md's "Consequences" section cites this module's lesson 7 (the map toward other guides) instead of simply ending the document after "What a reader should be able to answer."

See solution

Ending without that section would leave the impression that Andes Cargo's reliability package is a closed, complete system in itself, with no direction to grow toward — exactly the opposite of what a real portfolio project should communicate. Citing lesson 7's map in the consequences section closes the document with the same honesty that governed the entire guide: this is what was built, verified with evidence, and this is, with the same precision, where the same vocabulary extends next — neither a claim that the work is finished forever, nor a vague list of "possible future improvements," but three concrete directions, each with its guide named.

Summary and next step

This final project integrated andes-cargo-infra/'s twenty-four files — five portfolio documents, two SRE-math scripts, nine observability/ pieces, one Terraform file, an on-call generator, a runbook, and two complete incidents, one real and one synthetic — into RELIABILITY-PACKAGE.md, this whole guide's final index. You verified the document with the same deterministic discipline as every earlier project — 55 lines, 7 sections, 5 table lines — and practiced defending it in an interview with five questions, each backed by a command you actually ran, not by an unsupported claim.

With this, sre-and-incident-response-guide is complete: the CRITICAL SRE gap VALIDACION.md flagged for the entire aws-cloud-ecosystem — SLI/SLO/error budgets, honest on-call, incident response, blameless postmortem — now has a complete machine, actually executed, tested twice against two different cases, and documented with the exact same honesty discipline every guide in this ecosystem has already demanded of itself. Andes Cargo's business layer — bucket, table, Lambda, pipeline, security gate, cost gate — remains exactly what the six sibling guides built; this guide never rewrote it, it only added the discipline of knowing whether it's still working well, and what to do when it isn't.

Resources

  1. This same repository, every module — the source of every artifact this final document indexes.
  2. This same repository, Module 7, lesson 8 (08-project-andes-cargos-postmortem-and-runbooks-package.md) — the direct precedent for the portfolio-document pattern this project extends to the whole guide.
  3. src/paths/aws-cloud-ecosystem/VALIDACION.md — the CRITICAL SRE gap this complete guide closes, cited since this same guide's Module 1.
  4. Google SRE Book — Table of Contents and Google SRE Workbook — the complete source of the discipline this guide implemented, start to finish, with executed evidence in every module.
← Previous7. What Andes Cargo still needs