Module 1: What Is Sre And Reliability As A Feature
3. Reliability as a feature, not an accident
Description
It's intuitive to think the goal of any system in production is "never fail" — 100% availability, all the time, no exceptions. Google SRE, with two decades of real data behind it, reached the opposite conclusion, and documented it with the same precision it would document any other engineering result: chasing 100% isn't just impossible, it's a bad business decision. This lesson explains why, with the exact figures from the Google SRE book, and introduces the tool that turns that abstract idea into something a team can use every day to decide: the error budget.
Connection to the module
Lesson 2 established that SRE measures reliability instead of promising it. This lesson puts the exact number on that measurement: how much failure is acceptable, why that number is almost never zero, and what mechanism turns "how much failure is acceptable" into a real operational decision, every day, about when to ship something new and when to hold back. Lesson 6 is going to take exactly this vocabulary and apply it, for the first time with a real number, to the Claude Code incident.
Analogy: the prepaid card of "allowed downtime"
An error budget works exactly like the balance on a prepaid card someone tops up once a month, with a fixed limit. It's not a line of credit that grows if you need more — it's a closed balance. You can spend it fast, all at once, on something worthwhile (a big launch, a risky experiment); you can spread it carefully across the whole month; or you can barely touch it. But once the balance hits zero, that's it — it doesn't get replenished early, no matter how urgent the next thing you'd want to spend it on seems. Chasing 100% availability is the equivalent of deciding the card can never be used, not a single cent, ever — which sounds responsible until you realize a card nobody can use isn't useful for anything either: no experiment, no risky launch measures its real cost, because the cost of failure was always treated as if it were infinite.
Why 100% is almost never the right target
The source is explicit, and it contradicts intuition directly:
"You might expect Google to try to build 100% reliable services—ones that never fail. It turns out that past a certain point, however, increasing reliability is worse for a service (and its users) rather than better!"
And, even more directly, in the same chapter's conclusions section:
"100% is probably never the right reliability target: not only is it impossible to achieve, it's typically more reliability than a service's users want or notice."
Two separate ideas in that second quote, worth keeping apart:
- It's impossible to achieve. No real system — not even Google's own infrastructure — controls 100% of the variables between the server and the end user: the user's connection, their internet provider, their device's hardware, all have their own failure rate, outside any engineering team's control. Chasing 100% on the server side doesn't move the user's real experience if the 0.3% of failure they perceive comes, either way, from their own network.
- Even if it were possible, almost nobody needs it or notices it. The difference between 99.99% and 100% availability — 52.6 minutes of downtime a year versus zero — is, for the vast majority of systems, invisible to the end user next to the variability that already exists in their own connection.
The marginal cost of each additional nine
The reason "just aim higher" isn't free advice: the cost of improving reliability doesn't grow linearly. The source states it precisely:
"As we build systems, cost does not increase linearly as reliability increments—an incremental improvement in reliability may cost 100x more than the previous increment."
Translated into real minutes, over a one-year window (365 days = 525,600 minutes), here's the ladder of "nines" you're going to use for the rest of this guide:
| SLO (availability) | Allowed downtime / year | Allowed downtime / month (30 days) | What it usually costs to get there |
|---|---|---|---|
| 99% ("two nines") | ~3.65 days | ~7.3 hours | Basic redundancy, manual restart acceptable |
| 99.9% ("three nines") | ~8.76 hours | ~43.2 minutes | Recovery automation, active monitoring |
| 99.99% ("four nines") | ~52.6 minutes | ~4.3 minutes | Multi-zone redundancy, automatic failover, serious on-call |
| 99.999% ("five nines") | ~5.26 minutes | ~26 seconds | Active/active multi-region, dedicated on-call engineering, a much bigger infrastructure budget |
Each row down doesn't cost "a bit more" than the one above — according to the source, it costs an order of magnitude more. Going from 99% to 99.9% (from 7.3 hours of downtime a month to 43 minutes) can be achieved with reasonable engineering discipline. Going from 99.99% to 99.999% (from 4.3 minutes a month to 26 seconds) typically demands active multi-region architecture, a much larger on-call team, and a budget most products — including Andes Cargo, a shipment-tracking system, not a real-time payments system or life support — don't need and can't justify.
The right question is never "how much more reliable can we be?" — it's "how much additional reliability would someone who actually uses this system really pay for?"
The error budget: the mechanism that turns this idea into a daily decision
If 100% isn't the target, and each additional nine costs more than the last, then someone has to pick a specific number — the SLO — and, even more important, someone has to decide what to do when that number is about to be missed. That "what to do" is exactly the problem the error budget solves, and the source is explicit about why it works:
"The main benefit of an error budget is that it provides a common incentive that allows both product development and SRE to focus on finding the right balance between innovation and reliability."
"An error budget aligns incentives and emphasizes joint ownership between SRE and product development."
The mechanism, in its simplest form: if the SLO is 99.9% monthly availability, the error budget is the remaining 0.1% — the equivalent of the prepaid card balance from the analogy above, this time measured in concrete minutes (43.2 minutes a month, per the table above). As long as the budget has balance, the product team literally has permission to take risk: ship a new feature without exhaustively testing it, run an aggressive experiment, deploy more often. When the budget runs out, the conversation stops being an opinion ("I think we should hold back") and becomes a verifiable fact with a number: no downtime minutes are left this month, so new launches wait until the system stabilizes or until the next window starts. No team has to win an argument — the budget already decided.
This is the "negotiation mechanism between speed and stability" this section's title promises: it's not a negotiation of opinions every week, it's a shared number both teams — the one that wants to ship fast and the one that responds when something breaks — can check and trust equally, exactly the answer to the split-incentives problem lesson 2 already identified in the traditional operations model.
Common mistakes
Picking the highest possible SLO "because it sounds better" (the most common mistake in this whole guide). What happens: someone, when defining an SLO for a new service, picks 99.99% or 99.999% with no business justification at all, just because a higher number feels more responsible. How to spot it: if your justification for an SLO is "we want to be as reliable as possible" instead of "our users would notice/pay for this specific difference." How to fix it: use this lesson's nines table in reverse — before picking an SLO, ask what would really happen if the system were down for the number of minutes that SLO allows. For process-shipment-manifest, an asynchronous manifest-processing system, 43 minutes of downtime a month (99.9%) probably costs nothing real — a manifest that arrives 43 minutes late for processing isn't a crisis; 26 seconds a month (99.999%) would demand a completely unjustified architecture investment for that same system. This guide's Module 2 picks this decision back up with the real calculator.
Treating the error budget as a punishment when it runs out (a culture mistake, the most expensive one). What happens: when the error budget runs out, someone looks for who to blame for "spending it," instead of treating it as neutral operational information. How to spot it: if your team's reaction to an exhausted budget includes the question "who caused this?" before the question "what decision does this make for us now?" How to fix it: the error budget isn't a performance score — it's, literally, the tool the source describes to align incentives, not to assign blame. An exhausted budget means exactly one thing, operationally: pause new risk until margin recovers. This guide's Module 7 builds the same idea, in the same spirit, for the blameless postmortem.
Confusing "zero incidents this month" with this discipline's real goal. What happens: someone celebrates a month with no incidents as if it were SRE's ultimate goal, and treats any future incident as a process failure. How to spot it: if your personal success metric is "zero incidents" instead of "the error budget was respected, and when it wasn't, we learned something systemic from it." How to fix it: a month with no incidents at all, with a 99.9% SLO, can mean the team was too conservative and left budget unspent — budget that, by this lesson's own logic, exists precisely to take reasonable risk. The goal isn't zero incidents; it's that the amount of downtime, whatever it is, stays within the agreed budget, and that every time it's exceeded, the system — not a person — is the focus of the fix.
Exercises
Exercise 1 — Explain, in your own words, why 99.99% isn't automatically "better" than 99.9% for every system. Using this lesson's nines table, argue why picking the highest available SLO isn't, by default, the right decision.
See solution
A complete answer: "99.99% allows only 4.3 minutes of downtime a month, versus 43.2 minutes at 99.9% — an order-of-magnitude difference, not a minor tweak. According to the Google SRE source, that jump doesn't cost 'a bit more'; it can cost a hundred times more in engineering (multi-zone redundancy, recovery automation, much more aggressive on-call). If the system in question has no real business cost tied to those 39 extra minutes of difference — for example, an asynchronous processing system where nobody notices if a piece of data arrives a few minutes late — that extra engineering cost would be paid with no real benefit to the user. The right decision depends on the real cost of failure for that specific system, not on a 'higher is more responsible' instinct."
Exercise 2 — Apply the prepaid-card analogy to a concrete situation. A team has a monthly SLO of 99.9% (43.2 minutes of budget) and already used 40 of those 43.2 minutes halfway through the month, in an incident that's already resolved. Someone proposes shipping an experimental, risky feature that same afternoon. Using this lesson's analogy, what should you answer?
See solution
With the prepaid card nearly empty — only 3.2 minutes of balance left for the rest of the month — shipping something risky that same afternoon is exactly the equivalent of trying a big purchase with a card that's nearly out of funds: any new problem, however small, exhausts the remaining budget and leaves the team with no margin for the rest of the month, including incidents that have nothing to do with the experimental launch. The right answer, following the error budget's logic: hold back the risky launch until the budget recovers in the next window, or until the team decides, with explicit awareness of the risk, to spend the little remaining balance on something worth that bet — never launch it without checking the balance first.
Exercise 3 — Explain the difference between "the error budget ran out" and "someone made a mistake." A teammate, after the team exhausts its error budget in an incident, suggests the person who approved the change that caused it should receive a formal reprimand. What do you tell them, using this lesson's vocabulary?
See solution
The error budget exists, according to the source cited in this lesson, to align incentives between development and reliability — not to assign individual blame. Exhausting the budget is operational information (pause new risk until it recovers), not a verdict on a person. Treating budget consumption as grounds for an individual reprimand reintroduces exactly the problem this discipline was designed to avoid — a culture where people hide or downplay problems out of fear of personal consequences, instead of reporting them precisely so the system gets fixed. The right response is to treat the event as an input to the process, the same philosophy this guide's Module 7 formalizes with the blameless postmortem.
Summary and next step
In this lesson you established, with the exact Google SRE quotes, why 100% availability is almost never the right target — it's impossible to achieve, and it's typically more reliability than users notice or need; why the cost of each additional nine grows non-linearly, with a concrete table of allowed downtime minutes for each SLO level; and what the error budget is — the mechanism, not the promise, that turns that decision into something operational every day, aligning the incentives between whoever wants to ship fast and whoever responds when something breaks.
Before moving on you should be able to: explain why 100% is impossible and, on top of that, undesirable; calculate from memory how many minutes of budget a monthly 99.9% SLO gives; and explain, without using the word "punishment," what it really means for an error budget to run out.
Lesson 4 puts this vocabulary to work on Andes Cargo's real infrastructure, read for the first time with an SRE's specific question: not "is it secure?", not "is it cheap?" — "what could fail here?"
Resources
- Google SRE Book, Chapter 3 — Embracing Risk — the primary source for every quote in this lesson: why 100% is the wrong target, the non-linear cost of each nine, and the error budget as an incentive-alignment mechanism.
- Google SRE Workbook — Implementing SLOs — the complete error budget formula (1 − SLO) with the numeric example over real request volume, picked back up with the Module 2 calculator.
- Google SRE Book, Chapter 4 — Service Level Objectives — the formal SLI/SLO/SLA definitions, picked back up in lesson 7 of this module.