Module 1: What Is Sre And Reliability As A Feature

1. Guide introduction: from "it works, it's secure, it fits the budget" to "it's reliable"

Description

If you completed aws-core-services-guide, aws-serverless-and-containers-guide, terraform-and-iac-guide, cicd-and-gitops-on-aws-guide, cloud-security-and-guardrails-guide, and finops-and-cost-guardrails-guide, you have something very few learning portfolios can show: the complete infrastructure of Andes Cargo — an S3 bucket, two IAM roles, a Lambda function, a DynamoDB table — declared in Terraform inside andes-cargo-infra/, deployed through a GitHub Actions pipeline you ran with act against LocalStack, hardened with a real security gate (conftest, Trivy, cosign) and, on top of that, with a second gate that never says "this is insecure" but instead "this is expensive" (Infracost, a cost threshold, allocation tags). The plan comes out clean. The pipeline goes green, twice: secure and sustainable. And yet, none of the six previous guides has answered the question that opens this one: can a system that passes all those gates still fail in production?

The answer, which you already know if you paid attention to the five cases in this same ecosystem that mention the Claude Code destroy incident, is yes — and in the most expensive way possible. A clean plan never measured how much downtime Andes Cargo can tolerate before its customers notice. A secure pipeline never measured how fast something broken gets detected. A budget that stayed on track never measured whether anyone knows what to do at 3 AM when Shipments stops responding. This guide — SRE and Incident Response — starts from a central idea you're going to install in this first lesson and use in the seven that follow: "reliable" is not a state a system reaches and keeps. It is a measurement, with a number, a time window, and a budget that gets spent.

Connection to the module

This Module 1 doesn't build any calculator yet — that starts in Module 2, with scripts/error_budget_calculator.py. Its job is the same kind of work cloud-security-and-guardrails-guide and finops-and-cost-guardrails-guide already did in their own Module 1, before touching a single tool: install the vocabulary, take the first inventory, and present the guide's central case from the specific angle that belongs to this layer. Lessons 2 and 3 give the conceptual framework — what SRE is, and why 100% availability is almost never the right target. Lesson 4 applies that framework to Andes Cargo's real inventory, read for the first time through an SRE lens, not a security or cost lens. Lessons 5 and 6 bring this guide's central case — the Claude Code destroy incident — with the first real measurement at this layer: a manual calculation of how much error budget it burned. Lesson 7 formalizes the full vocabulary. Lesson 8 closes with the first real deliverable: RELIABILITY-CHARTER.md.

What this guide assumes (and won't explain again)

This guide is not an entry point into Andes Cargo's AWS ecosystem. It assumes, without re-explaining it:

The six previous guides, complete: andes-cargo-infra/ deployed exactly as it stood at the close of finops-and-cost-guardrails-guide — bucket andes-cargo-shipment-docs, table Shipments (partition key shipmentId, PAY_PER_REQUEST), function process-shipment-manifest (Python 3.13, role LambdaManifestProcessorRole), least-privilege IAM roles — with its .github/workflows/ci.yml pipeline running two gates in parallel: the security one (policy-check/iac-scan/verify-artifact) and the cost one (cost-estimate/cost-check/cost-tags). No lesson in this guide re-explains what a Terraform resource is, a GitHub Actions job, a Rego policy, or how a cost is estimated with Infracost.
docker-essentials-guide (transitive): docker compose up, ports, volumes — used without re-explanation to bring up Prometheus, Grafana, Alertmanager, and Jaeger in Module 3.
Basic terminal and shell, basic Python (transitive): the error budget calculator and the burn rate evaluator are short Python scripts using the standard library — Python isn't taught from scratch.

What this guide does teach, and none of the six previous ones covered: what an SLI is and why Google SRE's formula (good events ÷ valid events) is more rigorous than "is it up or not?"; how to choose an SLO that means something; what an error budget is and how it's calculated in real minutes; observability seen specifically as the data source that feeds an SLI; alerting based on error budget consumption rate (burn rate), not a static threshold; the complete lifecycle of an incident, with roles and severities; on-call, with honesty about its real cost; and, at the center of the guide, operating end to end the Claude Code destroy incident — already familiar from three previous guides — through that complete lifecycle, all the way to the blameless postmortem and the runbooks that come out of it.

The full map: this guide's 8 modules

   SRE AND INCIDENT RESPONSE — THE 8 MODULES

   M1  What SRE is and reliability as a feature       ← you are here: vocabulary + the case, measured for the first time
   M2  SLIs, SLOs, and the error budget                       the real calculator, over fixed data
   M3  Observability as SLI input                              real data from the inherited Lambda, not simulated
   M4  Alerting on error budget burn rate                       fires for the right reason, not a threshold
   M5  The incident lifecycle                                    the framework — roles, severity, on-call — before the case
   M6  Operating the Claude Code incident                          the central module: you operate it, not narrate it once more
   M7  Blameless postmortems and runbooks                          the systemic learning, without hunting for someone to blame
   M8  Capstone: Andes Cargo's reliability package                 a new incident, the full machine, end to end

#	Module	What it builds	New piece in `andes-cargo-infra/`
1	What SRE is and reliability as a feature	The vocabulary and the first real deliverable	`RELIABILITY-CHARTER.md`
2	SLIs, SLOs, and the error budget	The real error budget calculator	`SLO.md`, `scripts/error_budget_calculator.py`
3	Observability as SLI input	Real metrics/logs/traces feeding the SLI	`observability/docker-compose.yml`, `observability/instrument_manifest_flow.py`
4	Alerting on error budget burn rate	An alert that fires for the right reason	`scripts/burn_rate_evaluator.py`, `observability.tf`
5	The incident lifecycle	The roles, severity, and on-call framework	`INCIDENT-RESPONSE-PLAN.md`, `oncall/schedule.py`
6	Operating the Claude Code incident	The real case, operated with the M5 framework	`incidents/2026-02-26-claude-code-destroy/TIMELINE.md`
7	Blameless postmortems and runbooks	The systemic learning and the first real runbook	`.../POSTMORTEM.md`, `runbooks/manifest-processor-error-rate.md`
8	Capstone: Andes Cargo's reliability package	The full machine, against a new synthetic incident	The complete repository, as a portfolio piece

Notice the progression: first you install the vocabulary and measure the case once, by hand (M1), then you build the tool that formalizes that measurement (M2), then you feed that tool real data instead of a sample dataset (M3), then you turn that data into an alert that fires for the right reason (M4), then you build the response framework — roles, severity, on-call — before you need it (M5), then you use that complete framework to operate the guide's central case, not narrate it again (M6), then you turn what you learned into a blameless postmortem and Andes Cargo's first real runbook (M7), and you close by running the complete machine against a new incident nobody saw coming line by line (M8).

This module's map: the 8 lessons

#	Lesson	What you practice
1	Introduction (this one)	The full map, what's inherited from the six previous guides, the guide's thesis
2	What SRE is (and isn't)	Ben Treynor's definition, quoted; SRE vs. DevOps vs. traditional operations
3	Reliability as a feature, not an accident	Why 100% is almost never the right target; the error budget as negotiation
4	Hands-on: reading Andes Cargo like an SRE would	Representative: the `awslocal` inventory reframed as "what could fail here"
5	Two real incidents, two different questions	The Claude Code incident, new angle; the `us-east-1` outage, cited
6	Hands-on: the error budget the Claude Code incident burned	Executed: the guide's first real calculation, by hand
7	The vocabulary you'll use all guide long	SLI/SLO/SLA/error budget/toil/blast radius/on-call/postmortem, cited
8	Project: Andes Cargo's reliability charter	Executed (the document): `RELIABILITY-CHARTER.md`

What's genuinely new in this guide

No module in this guide declares a single new business HCL resource, or modifies a single security or cost policy already written. The bucket is still andes-cargo-shipment-docs, the table is still Shipments, the function is still process-shipment-manifest. What's new, always at the reliability layer, never the business layer, always in English as this ecosystem's convention:

RELIABILITY-CHARTER.md (Module 1, this lesson) and SLO.md (Module 2) — the first two documents, the same structural role THREAT-MODEL.md had in cloud-security-and-guardrails-guide and COST-PROFILE.md had in finops-and-cost-guardrails-guide.
scripts/error_budget_calculator.py and scripts/burn_rate_evaluator.py — the two $0 scripts from Module 2 and Module 4, reused with real data in Module 3 and in the Module 8 capstone.
observability/docker-compose.yml — Prometheus, Grafana, Alertmanager, and Jaeger, built in Module 3.
observability/instrument_manifest_flow.py — the OpenTelemetry instrumentation script from Module 3, which sends traces of the upload → Lambda → DynamoDB flow to Jaeger.
observability.tf — new, a sibling of finops.tf (which already declares its own billing alarm, untouched), with a real CloudWatch alarm on AWS/Lambda/Errors.
INCIDENT-RESPONSE-PLAN.md and oncall/schedule.py — the Module 5 framework: roles, severity, and a deterministic on-call rotation, with no PagerDuty.
incidents/2026-02-26-claude-code-destroy/TIMELINE.md and .../POSTMORTEM.md — the two central deliverables of Modules 6 and 7: the real incident, operated with this guide's vocabulary and tools.
runbooks/manifest-processor-error-rate.md — Module 7's first real runbook, every step verified against LocalStack before being written.
.github/workflows/ci.yml unchanged: this guide doesn't add a CI job. Its artifacts are operational — SLO, on-call, postmortem, runbooks — not build/deploy, and that's why there's no third lane to chain onto the pipeline. It's the honest difference from cloud-security-and-guardrails-guide and finops-and-cost-guardrails-guide, declared here from the start so that when you reach Module 8 you don't expect a pipeline change this guide has no reason to build.

The $0 commitment, and exact honesty about what really runs

SRE is, by nature, a discipline that in the real world runs on production traffic, PagerDuty bills, and months of historical data — none of those three things exist in a $0 lab. This guide handles that with the same discipline as the six before it: nothing is simulated in prose; if a command appears in a lesson, it ran to write it, and everything that didn't really run is labeled at the exact moment it appears, with its technical reason.

What really runs, end to end: the error budget calculator — 100% Python, 100% deterministic, no external tools at all; Prometheus v3.13.2, Grafana OSS 13.1.3, Alertmanager 0.33.1, and Jaeger v2 v2.20.0 (never all-in-one, discontinued since Jaeger v1's end of life on December 31, 2025), all four tested end to end in this environment; and CloudWatch Logs, metrics, and alarms, confirmed on LocalStack's Hobby plan — with the two native Lambda metrics (Invocations, Errors) this guide needs to measure a real SLI.

What stays representative, with its exact reason, always declared the moment it appears: AWS X-Ray (absent from the Hobby plan, Ultimate-only); Amazon CloudWatch Application Signals (native SLO Recommendations, launched March 2026, named by contrast); AWS Systems Manager Incident Manager (closed to new accounts since November 2025); PagerDuty/Opsgenie (SaaS with no $0 tier). One case is left with an open result, documented exactly as it comes out: the DynamoDB backup/restore attempt in Module 7, in the spirit of the real mechanism that saved the Claude Code incident's data.

Also, in this specific writing environment — with no LOCALSTACK_AUTH_TOKEN exported — the LocalStack container doesn't start, the same limit you already saw in the previous guides. Lesson 4 of this module carries that same honesty forward for the inventory awslocal commands: the output you'll read there is what each command would produce against the infrastructure already applied, reconstructed field by field from what the previous guides already confirmed running for real — never invented, always labeled.

A hard rule the sibling guides don't share: random/datetime.now() are forbidden in any "What to expect" block in this guide, with no exceptions declared. Every metric that feeds an SLI comes from a fixed sequence of invocations, written into the script itself.

Common mistakes

Assuming this module already builds the error budget calculator (expectation mismatch). What happens: someone finishes lesson 8, with RELIABILITY-CHARTER.md written, and expects a running Python script or an SLO.md file with formal numbers to already exist. How to spot it: if you look in this module for an executable .py file. How to fix it: this module installs vocabulary and does a manual calculation, by hand, in lesson 6 — deliberately before the tool, not instead of it — so that when Module 2's calculator exists, you understand exactly what it automates. SLO.md and error_budget_calculator.py belong to Module 2, not this one.

Confusing "reliable" with "never fails" (wrong intuition, the most common one in this whole guide). What happens: someone, reading this lesson's title, assumes this guide's goal is teaching how to build a system that never fails. How to spot it: if your mental definition of "reliable system" has no number attached — no percentage, no time window. How to fix it: lesson 3 develops this in depth with the exact Google SRE source, but the preview worth installing now: a perfect system (100% availability, forever) isn't the goal of any serious SRE team — it's, literally, more expensive than it's worth, and this guide will show you why with a number, not an opinion.

Treating the Claude Code destroy incident as an anecdote you've already heard three times, and therefore skippable (case fatigue). What happens: someone, seeing the incident's name in lesson 5's map, assumes this guide narrates it again the same way terraform-and-iac-guide or cloud-security-and-guardrails-guide already did, and decides to skip those two lessons. How to spot it: if your plan is "I already know how this story ends, no need to read it again." How to fix it: none of the three previous guides measured the incident with an error budget, or operated it through a complete response lifecycle with roles, severity, and a postmortem. Lesson 6 of this module is the first time, in the whole ecosystem, that this incident becomes a number: how many times the full monthly budget got spent in a single event.

Exercises

Exercise 1 — Explain, in your own words, this guide's central thesis. Without using yet the technical terms you'll formalize in lesson 7 (SLI, SLO, error budget), describe in two or three sentences why Andes Cargo's infrastructure, already secure and already financially sustainable for two full guides now, might still not be reliable.

See solution

A complete answer sounds, roughly, like this: "A secure pipeline confirms that a change doesn't open the bucket to the public or delete Shipments without control. A sustainable pipeline confirms that change fits the monthly budget. Neither guarantee says anything about how fast someone finds out the system stopped working, how much downtime is acceptable before a customer notices, or what the team does when something fails at 3 AM. 'Secure' and 'sustainable' are two questions already answered; 'reliable' is a third one, measurable with a number, that none of the six previous guides has answered yet." The key piece: each previous guardrail measured a different thing, and none measured the system's operational reliability in production.

Exercise 2 — Locate this guide's eight deliverables, without looking back. From memory, name the eight new artifacts this complete guide is going to add to andes-cargo-infra/, and in which module each one appears.

See solution

RELIABILITY-CHARTER.md (Module 1, this module); SLO.md + scripts/error_budget_calculator.py (Module 2); observability/docker-compose.yml + observability/instrument_manifest_flow.py (Module 3); scripts/burn_rate_evaluator.py + observability.tf (Module 4); INCIDENT-RESPONSE-PLAN.md + oncall/schedule.py (Module 5); incidents/2026-02-26-claude-code-destroy/TIMELINE.md (Module 6); .../POSTMORTEM.md + runbooks/manifest-processor-error-rate.md (Module 7); the complete repository, as a portfolio piece (Module 8). If you remembered at least six of the eight without looking at the table, you have the guide's progression clear.

Exercise 3 — Explain why this guide doesn't add a new job to the CI/CD pipeline. A colleague, familiar with cloud-security-and-guardrails-guide and finops-and-cost-guardrails-guide — each of which added its own lane to ci.yml — asks why this guide, being the ecosystem's seventh, doesn't do the same. Answer with the exact reason.

See solution

Because the two previous gates evaluate proposed changes before they're applied — is this plan secure? does this plan fit the budget? — while this guide's artifacts (SLO.md, INCIDENT-RESPONSE-PLAN.md, a postmortem, a runbook) are operational: they describe how the team measures and responds to what happens after a change is already in production, not a condition a pull request must meet to merge. There's no rule like "the plan fails if the SLO isn't met" that makes sense to evaluate in a PR — the SLO is measured against real traffic, over time, not against a proposed HCL file. That's why this guide doesn't chain a third lane: it builds a different layer, one that lives alongside the pipeline, not inside it.

Summary and next step

In this lesson you saw the full map of this guide's 8 modules, confirmed what's inherited without repeating it from the ecosystem's six previous guides (the complete andes-cargo-infra/, with its security gate and cost gate already chained), and what's genuinely new: eight reliability-layer artifacts, none of them business, security, or cost. You installed the central thesis that governs the rest of the guide: "reliable" isn't a state, it's a measurement with a number, a time window, and a budget that gets spent. And you saw, up front, this guide's honesty commitment: what really runs with Python/Prometheus/Grafana/Alertmanager/Jaeger/CloudWatch, and what stays representative with its exact technical reason.

Before moving on you should be able to: name the 8 modules in order and what each one builds; explain why this Module 1 doesn't build any calculator yet; and explain, in one sentence, why "secure" and "sustainable" aren't the same as "reliable."

Lesson 2 opens with the formal vocabulary: what SRE is, quoted directly from the person who first defined it at Google, and why the distinction from DevOps and from traditional operations matters beyond marketing.

Resources

aws-core-services-guide (NIEVA) — the prerequisite that leaves Andes Cargo's four core services (S3, Lambda, DynamoDB, IAM) deployed for the first time.
aws-serverless-and-containers-guide (NIEVA) — the prerequisite that hardens process-shipment-manifest to production level (concurrency, dead-letter queues).
terraform-and-iac-guide (NIEVA) — the prerequisite that leaves andes-cargo-infra/ complete in HCL, and the original, verified source of the Claude Code destroy incident (its Module 8, lesson 5).
cicd-and-gitops-on-aws-guide (NIEVA) — the prerequisite that leaves the GitHub Actions pipeline running under act.
cloud-security-and-guardrails-guide (NIEVA) — the prerequisite that leaves the complete security gate, and this ecosystem's first lesson to ask "what would have stopped this?" about the same incident (its Module 1, lesson 6).
finops-and-cost-guardrails-guide (NIEVA) — the prerequisite that leaves the complete cost gate, the exact point where this guide begins.
Google SRE Book — Table of Contents — the official complete source this guide cites, lesson by lesson, starting with the next one.
src/paths/aws-cloud-ecosystem/VALIDACION.md — the market audit that identified the CRITICAL SRE gap this guide closes.

Next →2. What SRE is (and isn't)