5. Choosing an SLO that means something

Description

Lesson 4 measured process-shipment-manifest's real traffic against a reference SLO, 99.9%, and found an almost-exhausted budget (4.23 of 43.2 minutes). That figure isn't a fixed property of the system — it's the result of comparing the same real performance against a specific number someone chose. This lesson asks the question Module 1 lesson 3 already raised in theory, now with the real calculator: what happens to that same reading if the chosen SLO were different? The answer, run three times over exactly the same data, is the most direct demonstration of why the SLO isn't a technical detail — it's a decision with measurable, immediate consequences on how much margin for error a team has.

Connection to the module

Nothing in this lesson changes lesson 4's dataset or lesson 3's SLI definition — process-shipment-manifest's real performance during those 30 days was what it was, a single fact. The only thing that varies is the SLO, the parameter you already saw budget_report accepts without touching the rest of the code (Module 2, lesson 4, exercise 3). This lesson runs that function three times, with three SLOs, and lets the numbers — not an opinion — show why "pick the highest possible SLO" (the most common mistake Module 1, lesson 3 already named) is, in practice, a way of turning a calm month into a manufactured crisis.

---

The nines table, remembered from Module 1

You already saw this table in Module 1, lesson 3, with the exact Google SRE quote on why the cost of each additional nine grows non-linearly. Worth having it in front of you before running the calculator:

SLO	Allowed downtime / month (30 days)	What it usually costs to get there
99% ("two nines")	~7.3 hours (432 minutes)	Basic redundancy, manual restart acceptable
99.9% ("three nines")	~43.2 minutes	Recovery automation, active monitoring
99.99% ("four nines")	~4.3 minutes	Multi-zone redundancy, automatic failover, serious on-call

This lesson doesn't repeat that table's theory — it runs it against real data, so the jump between rows stops being an abstraction of "costs more" and becomes a concrete number of margin minutes, measured over the same month you already know.

---

Running the calculator, three times, without changing the dataset

Using exactly TRAFFIC_30_DAYS from lesson 4 — the same real traffic, the same measured SLI of 99.9098% — the only line that changes is the slo argument:

from error_budget_calculator import TRAFFIC_30_DAYS, budget_report, print_report

for slo in [0.99, 0.999, 0.9999]:
    report = budget_report(TRAFFIC_30_DAYS, slo=slo)
    print_report(report, f"SLO {slo:.2%}")
    print()

What to expect (literal — the measured SLI, 99.9098%, is identical across all three runs; the only thing that changes is the SLO):

--- SLO 99.00% ---
Valid events:            12195
Good events:             12184
Measured SLI:            99.9098%
Target SLO:              99.0%
Budget (monthly):        432.0 minutes
Consumed this period:    38.97 minutes (9.0% of budget)
Remaining budget:        393.03 minutes

--- SLO 99.90% ---
Valid events:            12195
Good events:             12184
Measured SLI:            99.9098%
Target SLO:              99.9%
Budget (monthly):        43.2 minutes
Consumed this period:    38.97 minutes (90.2% of budget)
Remaining budget:        4.23 minutes

--- SLO 99.99% ---
Valid events:            12195
Good events:             12184
Measured SLI:            99.9098%
Target SLO:              100.0%
Budget (monthly):        4.3 minutes
Consumed this period:    38.97 minutes (902.0% of budget)
Remaining budget:        -34.65 minutes

(The 99.99% SLO prints as 100.0% because print_report rounds to one decimal — the real value used in the calculation, unrounded, is still 99.99%; exercise 1 of this lesson comes back to this detail.)

---

The same month, three completely different stories

This is the lesson's central result, and it's worth reading slowly because it's counterintuitive the first time you see it with real numbers: process-shipment-manifest had exactly the same performance across the 30 days — the same 11 errors, over the same 12,195 valid invocations, the same 99.9098% SLI — and yet:

• At 99%, that month was calm: 9.0% of budget consumed, 393 minutes of margin. A team with this SLO could run risky experiments for the rest of the month with no worries.
• At 99.9%, that same month was on the edge: 90.2% consumed, only 4.23 minutes of margin. A team with this SLO should be on alert, reviewing what caused the errors on days 17-18 before risking anything else.
• At 99.99%, that same month was, unambiguously, a failure: 902% of budget consumed, a deficit of 34.65 minutes. A team with this SLO already exhausted, several times over, any margin at all — and exhausted it before day 17, because a 4.3-minute budget would have been consumed with far fewer than this dataset's 11 real errors.

None of the three readings is "wrong" — the arithmetic in all three is perfect. What changes is whether the number that comes out of that arithmetic means something useful for the business that decided the SLO. Choosing 99.99% for process-shipment-manifest wouldn't have made the system actually more reliable — the real performance was the same across all three scenarios — it would only have turned every normal month into a "crisis" on paper, training the team to ignore alerts because they fire all the time, exactly the "threshold that cries too loud" problem this guide's Module 4 is going to solve with burn rate.

---

So, what SLO makes sense for Andes Cargo?

This lesson doesn't close that question yet — that's, formally, lesson 8's job, with SLO.md — but it already leaves enough evidence to steer the decision: process-shipment-manifest is, per RELIABILITY-CHARTER.md (Module 1, lesson 8, Decision section position 2), "not a payments system or a life-support system" — an asynchronous manifest processor, where the real cost of a few extra minutes of downtime is low. With that business evidence, plus this lesson's result — at 99.9% the real system is genuinely close to the limit, neither trivially comfortable nor catastrophically broken — the most defensible candidate among the three scenarios run is 99.9%: it's the only one of the three where the number has room to mean something — it lands close to zero but not negative, so a slightly worse month would actually matter, and a slightly better month would actually be noticeable. Lesson 8 picks this evidence back up, together with lesson 6's burn rate evidence, to set the final decision in SLO.md.

---

Common mistakes

Picking 99.99% because "the system should aim for the best possible" (the same mistake from Module 1 lesson 3, reappearing with real numbers). What happens: someone, seeing the nines table, assumes aiming higher is always the more responsible decision. How to spot it: if your justification for a high SLO includes no argument about what missing it would cost the business. How to fix it: this lesson already showed it with numbers — the same real performance that "comfortably passes" at 99% "fails by almost 35 minutes" at 99.99%. An SLO isn't an aspiration, it's a target the system must be able to sustainably meet with the design and engineering budget it actually has.

Interpreting the 902% from the 99.99% run as a script error (distrusting an extreme number). What happens: someone sees "902.0% of budget" and "-34.65 minutes" and assumes there's a bug in budget_report. How to spot it: if your first reaction to a percentage above 100% is "this can't be right." How to fix it: a budget can be consumed at more than 100% — it literally means the system spent more downtime minutes than the SLO allowed, and the deficit (the negative minutes) is exactly that amount. There's no mathematical limit preventing minutes_consumed from exceeding budget_minutes; the limit that does exist is a design one: an SLO so strict that the real system misses it by such a large margin, most months, isn't a useful SLO.

Treating this lesson's 99.9% "candidate" as the final, already-closed decision (jumping ahead of lesson 8). What happens: someone, finishing this lesson, writes in SLO.md that Andes Cargo's SLO is 99.9%, closing lesson 8 ahead of time. How to spot it: if you already cite "Andes Cargo's SLO" as a fixed fact before reaching lesson 8 of this module. How to fix it: this lesson gathers evidence — three scenarios run, one of them clearly more defensible — but lesson 6 still has to add the burn rate perspective, which could reinforce or qualify this choice before SLO.md sets it with its full justification.

---

Exercises

Exercise 1 — Explain why print_report showed 100.0% for a 99.99% SLO, and why that doesn't affect the calculation's result. What exact part of the code produces that rounding, and what would happen if you wanted to show more precision?

See solution

The responsible line is print(f"Target SLO: {report['slo']:.1%}") inside print_report — the .1% format specifier rounds to a single decimal when converting to a percentage, and 99.99% rounded to one decimal is, indeed, 100.0%. The real value stored in report['slo'] remains 0.9999 with no rounding at all — the proof is that budget_minutes was correctly calculated as 4.3 minutes, which is (1 - 0.9999) × 43200, not (1 - 1.0) × 43200 (which would give zero). To show more precision, it would be enough to change the format specifier to .2% or .3% on that single line of print_report, with no calculation touched.

Exercise 2 — Calculate, without running anything, how many additional errors (on top of the real 11) would have been enough to also leave the 99.9% scenario with a negative budget. Use the numbers already calculated in this lesson.

See solution

The remaining budget at 99.9% is 4.23 minutes. Each additional bad event, applied over 12,195 valid events, adds an error rate of 1/12,195 ≈ 0.0082%, which over the 43,200-minute window represents 0.0082% × 43,200 ≈ 3.54 additional minutes consumed. With just 2 more errors (13 instead of 11), the additional consumption would be roughly 7.08 minutes — more than enough to exhaust the 4.23 remaining minutes and leave the budget negative. This confirms, with simple arithmetic, how close this real month was to a deficit: it didn't take 11 more errors, not even 5 — with just 2 additional bad invocations across the whole month, the 99.9% SLO would also have been missed.

Exercise 3 — Argue, using this lesson's result, why an SLO shouldn't be chosen by looking only at Module 1's nines table, with no real traffic data. What was that table missing that this lesson does provide?

See solution

Module 1's nines table shows how many minutes of downtime each SLO allows — necessary information, but abstract: it never says whether the real system is capable of meeting that figure. An engineer looking only at that table could reason "43.2 minutes sounds like a comfortable margin" with no evidence at all of whether the real system, with its historical error rate, lands close to that limit or far from it. This lesson provides exactly what was missing: the same real traffic, run against three different SLOs, showing that 43.2 minutes (99.9%) isn't "comfortable" at all for this specific system — it's barely enough, with only 4.23 minutes of margin. Choosing an SLO without this step would be exactly the mistake RELIABILITY-CHARTER.md already rejected in Module 1: a number picked "because it sounds responsible," with no evidence of whether the real system can sustain it.

---

Summary and next step

In this lesson you ran the same calculator, over the same 30 days of real traffic, with three different SLOs: at 99% the month was comfortable (393 minutes of margin); at 99.9% it was on the edge (4.23 minutes); at 99.99% it was a clear failure (a 34.65-minute deficit). The same real performance, read with three different yardsticks, told three completely different stories — the most direct demonstration of why the SLO is a decision with measurable consequences, not a detail chosen by instinct. With that evidence, you identified 99.9% as the most defensible candidate for Andes Cargo, without closing the final decision yet.

Before moving on you should be able to: run the calculator against at least three different SLOs and explain the difference across the three results; calculate by hand how many additional errors would have taken a positive-budget scenario to a negative one; and explain why Module 1's nines table, alone, isn't enough to choose a real SLO.

Lesson 6 adds the piece this lesson still doesn't have: not just how much budget is left at the end of the month, but how fast it was being spent at each moment — burn rate, the formula from sre.google/workbook/alerting-on-slos/, applied to the same dataset you already know.

Resources

1. This same repository, Module 1, lesson 3 (03-reliability-as-a-feature-not-an-accident.md) — the nines table and the Google SRE quote on the non-linear cost of each additional nine.
2. This same repository, Module 1, lesson 8 (08-project-andes-cargos-reliability-charter.md) — RELIABILITY-CHARTER.md, Decision section position 2, the business evidence steering this lesson's SLO candidate.
3. Google SRE Book, Chapter 3 — Embracing Risk — the original source for reliability's non-linear cost, already cited in Module 1.
4. This same repository, Module 2, lesson 4 (04-hands-on-the-error-budget-calculator.md) — the dataset and the budget_report function this lesson reuses unmodified.