Module 2: Slis Slos And The Error Budget
8. Project: Andes Cargo's `SLO.md`
Description
Seven lessons built the evidence; this one turns it into a decision. This module's final project writes SLO.md, the document RELIABILITY-CHARTER.md explicitly left open in Module 1: "What SLI and SLO make sense for process-shipment-manifest, with evidence?" The answer is no longer an empty promise or a number picked "because it sounds responsible" — it's the SLI defined in writing in lesson 3, the SLO chosen among three scenarios run with real data in lesson 5, and the burn rate behavior from lessons 6 and 7, gathered into a single document with the same ADR discipline RELIABILITY-CHARTER.md already established.
Connection to the module
This is the deliverable the rest of this guide measures and cites, without reopening the discussion: Module 3 builds the real telemetry pipeline that replaces lesson 4's fixed dataset with live CloudWatch/Prometheus data, feeding exactly the SLI this document defines. Module 4 alerts on burn rate against this document's exact SLO. Module 6 measures the real Claude Code incident against this same SLO. Module 7 cites it directly in the postmortem. Module 8's capstone runs a new synthetic incident against the same 43.2-minute budget this document sets here.
Step 1 — Why this document can pick an SLO, unlike RELIABILITY-CHARTER.md
RELIABILITY-CHARTER.md deliberately refused to pick an SLO in Module 1 — the evidence didn't exist yet. That same document was explicit about what it would take to make that decision with judgment: the four golden signals (lesson 2 of this module) and a real dataset (lesson 4). Both now exist. On top of that, this document has something RELIABILITY-CHARTER.md never had: three real calculator runs, over the same traffic, against three different SLOs (lesson 5), showing with numbers — not intuition — which of the three is defensible. Picking an SLO now, with that evidence behind it, is exactly the opposite of the mistake RELIABILITY-CHARTER.md avoided — it's the decision all that evidence was building toward.
Step 2 — The complete document
At the root of andes-cargo-infra/, create SLO.md:
# SLO.md — process-shipment-manifest Reliability Target
**Status:** Accepted · **Date:** this module's close · **Supersedes:** none
**Governs:** Modules 3 through 8 of `sre-and-incident-response-guide`
**Source:** Module 2, lessons 2-7 (the four golden signals, the SLI definition, three calculator
runs across a real 30-day dataset and a synthetic bad week)
## SLI
**Definition:** the proportion of valid events that were good, applied to `process-shipment-manifest`
(Module 2, lesson 3, citing Google's *Art of SLOs*).
- **Valid event:** any invocation triggered by a real S3 upload of a shipment manifest to
`andes-cargo-shipment-docs`. Manual test invocations (console/CLI, synthetic payloads used only to
verify a deployment) are excluded from the denominator entirely — they do not represent real
customer traffic.
- **Good event:** a valid invocation that completes without an unhandled exception, within the
10-second `Timeout`, and writes a correct record to `Shipments`. Explicitly counted as **not good**
(bad, within valid events, never excluded from the denominator): an invocation throttled by
`ReservedConcurrentExecutions: 5`, an invocation that exceeds the 10-second timeout, and an
invocation that exhausts its two automatic retries and is silently dropped — the three risks
Module 1, lesson 4 identified and left unresolved.
```text
SLI = good_events / valid_events
```
## SLO
**99.9% monthly**, measured over a 30-day rolling window.
**Why not 99%:** Module 2, lesson 5 ran the calculator's real 30-day dataset (SLI 99.9098%) against
a 99% target and found 393.03 minutes of budget remaining — 90% of the monthly budget untouched. A
99% target would never have flagged the two-day degradation of days 17-18 as anything but routine; it
sets a bar this system would clear even during a real incident, which defeats the purpose of having a
target at all.
**Why not 99.99%:** the same real dataset, run against 99.99%, produced a budget deficit of -34.65
minutes — a system with no architectural changes and no unusual failure mode would have "failed" its
SLO nearly every month. `RELIABILITY-CHARTER.md` (Module 1, lesson 8, Decision point 2) already
rejected chasing 99.99%+ for a system that is "not a payments system or a life-support system" — an
asynchronous manifest processor, where a few extra minutes of delayed processing carries no real cost
to the business. Module 2, lesson 3 of `terraform-and-iac-guide`'s sibling reasoning (Embracing Risk,
Google SRE Book, Chapter 3) applies directly: each additional nine can cost 100x more than the last,
and nothing about this system's business need justifies that cost.
**Why 99.9%:** it is the only one of the three scenarios tested where the real 30-day dataset landed
close to the line without crossing it — 90.2% of budget consumed, 4.23 minutes remaining. This is a
target the system can meet under real, measured conditions, but not so loose that a genuine
degradation (like the days 17-18 incident, or the synthetic bad week of Module 2, lesson 7) goes
unnoticed. A number that is barely met is a number that still means something.
## Error budget
- **43.2 minutes per 30-day window** — `(1 - 0.999) × 43200 minutes`.
- **Measured against the real 30-day traffic dataset (Module 2, lesson 4):** 38.97 minutes consumed
(90.2%), 4.23 minutes remaining. This is the reference measurement this document commits to — not a
live number (Module 3 builds the real telemetry pipeline that keeps it current), but the evidence
this SLO was chosen with, not guessed at.
- **Burn rate context (Module 2, lessons 6-7):** the two-day incident inside that same 30-day window
reached a daily burn rate of up to 13.27x, and the synthetic bad week of lesson 7 reached 43.41x
sustained over seven days — both far above the 1x sustainable rate, both examples of exactly the
kind of event this budget exists to quantify, not prevent.
## Alternatives considered
**99% (two nines).** Rejected: Module 2, lesson 5 showed the real dataset clears this target with 90%
of its budget untouched even during a real two-day degradation — a target too loose to mean anything
operationally for this system.
**99.99% (four nines).** Rejected: the same real dataset breaches this target by -34.65 minutes with
no unusual failure mode present — a target this system's current architecture cannot sustain, and one
`RELIABILITY-CHARTER.md` already ruled out on business grounds (Decision, point 2).
**No SLO chosen yet, deferring further (matching Module 1's approach).** Rejected: `RELIABILITY-
CHARTER.md` deferred choosing an SLO specifically until the four golden signals (Module 2, lesson 2)
and a real dataset (Module 2, lesson 4) existed. Both now exist. Deferring further would repeat the
same "100% because it sounds responsible" mistake Module 1, lesson 3 already named, this time as "no
number because it feels safer than committing to one."
## Consequences
Every module from here forward measures against this document, not around it: Module 3 builds the
real telemetry pipeline that replaces the fixed 30-day dataset with live CloudWatch/Prometheus data
feeding the same SLI definition. Module 4 alerts on burn rate against this exact SLO. Module 6
measures the real Claude Code incident against this SLO, retroactively. Module 7's postmortem cites
this document's numbers directly. Module 8's capstone runs a new synthetic incident through the same
machine, against the same 43.2-minute budget defined here.Step 3 — Verifying the document
wc -l SLO.md
grep -c '^## ' SLO.md
grep -c 'Rejected' SLO.mdWhat to expect (literal — you wrote the content, the shape is deterministic):
88
5
3
Eighty-eight lines, five sections (SLI, SLO, Error budget, Alternatives considered, Consequences), and three explicitly rejected alternatives — 99%, 99.99%, and "keep not deciding" — each with its reason, none simply omitted with no explanation.
How to read this document, six months later
The same test RELIABILITY-CHARTER.md already passed in Module 1: facing this document, with nobody to ask, a new reader should be able to answer four questions. What counts as "good" for this system? (the SLI section, with the three explicit exclusions — timeout, saturation, exhausted retries — named, not hidden). Why 99.9% and not another number? (the SLO section, with lesson 5's three scenarios cited with their exact figures, not an opinion). How much margin is there, really, today? (the Error budget section, 4.23 of 43.2 minutes — a tight margin, declared as such, not dressed up). What other options were considered and why were they dropped? (Alternatives considered, three entries, each with evidence). If any of those four questions requires rereading an entire lesson from this module, the document didn't do its job — the same standard RELIABILITY-CHARTER.md already set.
Module 2's close
With SLO.md written, this module delivers exactly what lesson 1 promised: not a one-time calculation like Module 1's, but a real, reusable calculator (scripts/error_budget_calculator.py, built in lesson 4 and extended in lesson 6), run over real data twice (lessons 4-6 and lesson 7), and the formal decision — with evidence, not a hunch — that closes the Open row RELIABILITY-CHARTER.md left pending. You enter Module 3 with a defined SLI, an SLO chosen with data, and an error budget measured in exact minutes — the foundation the rest of this guide builds on, without reopening any of these three things.
Common mistakes
Treating the 4.23 in "Error budget" as if it were a number that updates itself (expecting automation that doesn't exist yet). What happens: someone assumes that, from this document on, the remaining budget recalculates automatically every time real traffic runs. How to spot it: if you expect SLO.md to change without anyone editing it. How to fix it: the document is explicit that that 4.23 is a reference measurement, taken on lesson 4's fixed dataset — not a live value. This guide's Module 3 is, specifically, what builds the real telemetry pipeline that feeds the same formula with data that actually changes over time; until then, this number is the evidence the decision was made with, not an active counter.
Copying the "why 99.9%" justification without verifying the figures match what you yourself got in lesson 5 (copying without running). What happens: someone pastes this lesson's document without having actually run lessons 4-7's calculator, trusting the numbers are "probably right." How to spot it: if you can't reproduce, by running your own copy of scripts/error_budget_calculator.py, the 90.2% of budget consumed cited in this document. How to fix it: every figure in SLO.md comes from a real run of a script you already wrote and executed in this module's previous lessons — before accepting this document as final, confirm your own calculator produces, digit for digit, the same numbers cited here.
Adding a fourth considered alternative "to make it look more complete" with no real evidence behind it (the same mistake from Module 1's lesson 8, reappearing here). What happens: someone, uncomfortable with only three alternatives, adds a fourth (for example, "hourly SLO instead of monthly") without having run it through the calculator. How to spot it: if any "Alternatives considered" entry in your version of SLO.md cites no specific figure from a lesson in this module. How to fix it: RELIABILITY-CHARTER.md's Alternatives considered section already set the standard — every rejected alternative needs its own evidence, not just its own paragraph. If you want to explore a real fourth alternative, the right way is to run the calculator with that scenario first, and only then document the real result, whatever it is.
Exercises
Exercise 1 — Verify the complete document against your own run of the calculator. Run scripts/error_budget_calculator.py with SLO=0.999 over TRAFFIC_30_DAYS and confirm it produces exactly 90.2% of budget consumed and 4.23 remaining minutes, the same numbers cited in this document's "Error budget" section.
See solution
Running budget_report(TRAFFIC_30_DAYS) (default SLO, 0.999) gives: Consumed this period: 38.97 minutes (90.2% of budget) and Remaining budget: 4.23 minutes — matching SLO.md exactly. This exercise isn't a formality: it's the verification that the portfolio document you just wrote is traceable back to an executable script, not an unsupported claim. Anyone auditing this document can run the same script and land in exactly the same place.
Exercise 2 — Defend the SLO section against a real objection. A technical interviewer asks: "if the system is already using 90% of its budget under normal traffic, shouldn't you have just picked a looser SLO, like 99%, to have more breathing room?" How do you respond, using the document itself?
See solution
A complete answer: "This document's 'Why not 99%' section already answers that question with evidence, not preference: at 99%, the same real traffic that consumes 90.2% of the budget at 99.9% would have consumed barely 9% at 99%, leaving 393 minutes untouched even during the real days 17-18 incident. An SLO that loose wouldn't have distinguished that incident from a perfectly healthy month — the number would stop meaning anything, exactly the problem this document deliberately avoids, citing Module 2's lesson 5. Picking an SLO 'with more margin' sounds prudent, but in practice it chooses not to measure anything: leftover margin is margin that never gets used to catch a real problem. We prefer a tight SLO, one the system barely meets, because it's the only one that gives useful information when something goes wrong."
Exercise 3 — Explain why this document's "Alternatives considered" section includes "not choosing an SLO yet" as an explicitly rejected option, instead of simply not mentioning it. What does the document gain by naming and rejecting that option, instead of just omitting it?
See solution
Explicitly naming and rejecting "keep not deciding" closes a door that would otherwise stay open by omission: without that entry, someone could reason the team simply never considered waiting for more evidence before committing to a number. By naming it and rejecting it with a concrete reason — the same evidence RELIABILITY-CHARTER.md asked for (four golden signals, a real dataset) already exists, so waiting longer wouldn't add rigor, only delay a decision that was already ready to make — the document demonstrates the decision to commit to 99.9% now was deliberate, not rushed or avoided by carelessness. It's the same pattern RELIABILITY-CHARTER.md used when explicitly rejecting "pick a placeholder SLO now" in Module 1 — declaring the option not taken, with its reason, instead of leaving it as a silent gap someone could question later.
Summary and next step
In this module's final project you wrote SLO.md: process-shipment-manifest's SLI defined with precision (good vs. valid, with Module 1's three risks explicitly included as failures, never excluded for convenience), the chosen SLO — 99.9% monthly — with lesson 5's three exact scenarios as evidence, and the error budget — 43.2 minutes, with only 4.23 remaining against measured real traffic — with the burn rate context from lessons 6 and 7. You verified the document with this whole ecosystem's same deterministic discipline: 88 lines, 5 sections, 3 rejected alternatives with evidence, no decision left unjustified.
Before closing this module you should be able to: recite this document's complete SLI definition without looking at it; explain, with exact figures, why 99.9% and not 99% or 99.99%; and reproduce, by running your own calculator, every number this document cites.
With this, Module 2 of sre-and-incident-response-guide is complete: real SRE math, actually run over fixed, committed data, with a reusable tool the rest of this guide uses without rebuilding it. Module 3 takes exactly the SLI this document defines and feeds it, for the first time, with data that doesn't come from a fixed dataset — real invocations from the inherited Lambda, read with awslocal cloudwatch get-metric-statistics.
Resources
- This module, lessons 2 through 7 — the direct source for every figure and every decision in this document.
- This same repository, Module 1, lesson 8 (
08-project-andes-cargos-reliability-charter.md) —RELIABILITY-CHARTER.md, the document that left the questionSLO.mdanswers. - Google — The Art of SLOs (Participant Handbook) — the SLI formula this document's SLI section cites.
- Google SRE Workbook — Implementing SLOs — the error budget formula this document's Error budget section applies.
finops-and-cost-guardrails-guide, Module 1, lesson 8 andcloud-security-and-guardrails-guide, Module 1, lesson 8 — the same portfolio ADR formatSLO.mdfollows, already used twice in this ecosystem.