Module 7: Reliability and the Consistency Tradeoff

7. SLA, SLO and the nines, executed

Next →

Description

By the end of this lesson you will know how to translate an availability percentage into a number of downtime hours —by running the table of the nines— and to distinguish the three terms almost everyone confuses: SLI, SLO and SLA. You will see, run in Python, that 99% is 87.6 hours of downtime a year, 99.9% is 8.76 hours, 99.99% is 52.6 minutes and 99.999% is 5.3 minutes —each additional nine divides the downtime by ten—. You will compute the error budget (a monthly SLO of 99.9% gives you 43.2 minutes of allowed downtime a month, a budget that gets spent) and the dependency ceiling (you cannot promise more availability than the product of your providers'). And you will propose a concrete SLO for Enlace —99.9% on resolve— justified with its numbers, understanding why promising "100%" is not ambition but a lie.

This matters because "availability" without a number is an empty slogan, and percentages close to 100 fool intuition systematically: 99% and 99.9% sound almost identical and differ by a factor of ten —78 hours of downtime a year—. Without the table, people promise levels they cannot meet, or spend fortunes chasing one extra nine that nobody needed. With the table, availability becomes what it should be: a budget of hours you decide conscientiously, knowing what each nine costs (roughly 10× more effort per nine) and what your dependencies allow. This lesson gives you that table executed, teaches you to read an SLA without being fooled by the percentage, and closes the module with the yardstick that measures everything before it: it is useless to have removed the SPOFs (lesson 2) and chosen consistency well (lesson 6) if you cannot say, in an honest number, how reliable your system is.

Connection to the module: this lesson is the third block —the yardstick— and it puts a number on the two before it. The availability calculations of lesson 2 (series, parallel, how much downtime each configuration has) flow into here: those percentages are now translated into concrete hours and into a budget. And the error budget connects with the consistency of lesson 6: part of your failure budget is "spent" tolerating lags or failover windows, decisions you made in lessons 3 to 6. The border: here we define and compute SLI/SLO/SLA and the error budget; the organizational practice of using the error budget to decide when to freeze deploys, and the SRE culture around it, is more about operations and the decisions guide —we stop at the number and its meaning—.

The store's "almost always open"

Think of it this way. Two stores hang a sign. The first says "we are open 99% of the year"; the second, "we are open 99.9% of the year". At a glance, they sound almost the same —both promise to be open "almost always", and a 0.9% difference seems insignificant—. But translate the percentages into closed days and the illusion breaks. The first store is closed 3.65 days a year; the second, closed barely 8.76 hours. The "insignificant" difference of 0.9 points is actually a difference of ten times in closed time —from almost four days to less than a workday—. The sign fools you because human intuition is not calibrated for percentages so close to 100: we do not feel the difference between 99% and 99.9%, even though it is a factor of ten.

Now imagine the first store promises something stronger: it signs a contract with its providers that says "if I am closed more than 1% of the year, I pay you a penalty". That changes everything: it is no longer an aspiration, it is a commitment with consequences. And to sign it with a clear conscience, the store needs to know three different things that people mix up: how much it is really open (the measured datum), how much it aims to be open (its internal goal, set a bit stricter than the contract to have margin), and how much it promises to be open by contract (the penalty). Those three —the measured, the goal, the promise— are SLI, SLO and SLA, and confusing them is how people sign contracts they cannot meet.

The lesson I want you to take from the store is this: an availability percentage means nothing until you translate it into time, and it cannot be promised seriously without distinguishing what you measure, what you aim for and what you sign. This lesson does both things: it runs the table that translates percentages into hours, and it separates the three terms precisely.

SLI, SLO, SLA: the three terms everyone confuses

Let us define them from least to most commitment, because they build on one another.

SLI — Service Level Indicator. It is the number you measure. An SLI is a concrete, observed metric of the service's real behavior: for example, "the percentage of resolve requests answered successfully in under 100 ms over the last month". The SLI is neither a goal nor a promise; it is the fact —what really happened, measured—. Everything else rests on having a well-defined SLI: if you do not measure, you can neither aim nor promise.

SLO — Service Level Objective. It is the goal you set internally on an SLI: for example, "the resolve SLI must be ≥ 99.9%". The SLO is an internal engineering objective —you do not sign it with anyone outside—, and that is why it is usually set stricter than the SLA you promise the customer, to have a cushion. If your SLA promises 99.9%, your internal SLO could be 99.95%: that way, if you start missing the SLO, you have time to react before breaking the SLA and paying the penalty.

SLA — Service Level Agreement. It is the contractual promise with consequences: for example, "we guarantee 99.9% monthly availability; if we do not meet it, we refund 10% of the bill". The SLA is outward-facing, it is legal, and it has a cost if broken (credits, penalties, customers who leave). That is why it is the most conservative of the three: you never promise in the SLA the best you could do, but something you can meet comfortably even in a bad month.

   SLI  ──►  what you MEASURE     "99.93% of resolves successful <100ms this month"  (the fact)
   SLO  ──►  what you AIM FOR     "we want SLI >= 99.9%"                              (internal goal)
   SLA  ──►  what you PROMISE     "we guarantee 99.9% or there is a penalty"          (contract)

   golden rule:   SLA  <=  SLO   (you promise less than you aim for, to have a cushion)

The rule that ties the three together: SLA ≤ SLO, and the two rest on the SLI. You promise (SLA) a bit less than you aim for (SLO), and you aim based on what you can measure (SLI). Confusing them leads to expensive mistakes: promising in the SLA the best SLI you have ever seen (and breaking it the first bad month), or setting an SLO without an SLI that measures it (and never knowing whether you meet it).

Worked example: the table of the nines, executed

Here is the module's central calculation. A "nine" is each 9 in the availability percentage: 99% is "two nines", 99.9% "three nines", etc. The table translates each level into allowed downtime over different time windows. We run it instead of quoting it:

SECONDS_PER_YEAR  = 365 * 24 * 3600      # non-leap year
SECONDS_PER_MONTH = 30  * 24 * 3600      # 30-day month (SLA convention)
SECONDS_PER_WEEK  = 7   * 24 * 3600
SECONDS_PER_DAY   = 24 * 3600

def human(seconds):
    """Format seconds as the most natural multiple."""
    if seconds >= 3600:
        return f"{seconds/3600:.2f} h"
    if seconds >= 60:
        return f"{seconds/60:.1f} min"
    return f"{seconds:.1f} s"

levels = [
    ("90%     (one nine)",    0.90),
    ("99%     (two nines)",   0.99),
    ("99.9%   (three nines)", 0.999),
    ("99.99%  (four nines)",  0.9999),
    ("99.999% (five nines)",  0.99999),
]

header = f"{'Availability':<24}{'/year':>12}{'/month':>12}{'/week':>12}{'/day':>12}"
print(header)
print("-" * len(header))
for name, a in levels:
    unavail = 1 - a
    print(f"{name:<24}"
          f"{human(unavail*SECONDS_PER_YEAR):>12}"
          f"{human(unavail*SECONDS_PER_MONTH):>12}"
          f"{human(unavail*SECONDS_PER_WEEK):>12}"
          f"{human(unavail*SECONDS_PER_DAY):>12}")

print()
print("Direct check of 99.9% per year:")
seconds = (1 - 0.999) * SECONDS_PER_YEAR
print(f"  (1 - 0.999) x {SECONDS_PER_YEAR:,} s = {seconds:,.0f} s = {seconds/3600:.2f} h/year")

What to expect. When you run it:

Availability                   /year       /month       /week        /day
------------------------------------------------------------------------
90%     (one nine)          876.00 h     72.00 h     16.80 h      2.40 h
99%     (two nines)          87.60 h      7.20 h      1.68 h    14.4 min
99.9%   (three nines)         8.76 h    43.2 min    10.1 min     1.4 min
99.99%  (four nines)        52.6 min     4.3 min     1.0 min       8.6 s
99.999% (five nines)         5.3 min      25.9 s       6.0 s       0.9 s

Direct check of 99.9% per year:
  (1 - 0.999) x 31,536,000 s = 31,536 s = 8.76 h/year

This table is one worth having burned into memory, because it dismantles the sign's illusion forever. Read it by columns and notice the pattern: each nine you add divides the downtime by ten. From 99% to 99.9%: from 87.6 h/year to 8.76 h/year. From 99.9% to 99.99%: from 8.76 h to 52.6 min. From 99.99% to 99.999%: from 52.6 min to 5.3 min. That factor of ten per nine is the key to everything: 99.9% is not "a little better" than 99%; it is ten times better, and it costs —roughly— ten times more effort to achieve (more redundancy, more failover automation, more people on call).

And now the translation that matters for writing an SLA: when someone promises "three nines" (99.9%), they are promising that the system can be down up to 8.76 hours a year —more than a full workday— and still comply. "Five nines" (99.999%), the standard of traditional telephony, allows only 5.3 minutes a year —so little that no human intervention fits in there; it has to be all automatic failover—. When you read an SLA, do not read the percentage: translate to the column you care about (per year to plan, per month because that is how it is billed) and you will see what it really promises.

The error budget: downtime is a budget that gets spent

Out of the table comes one of the most useful ideas in modern reliability: the error budget. If your SLO is 99.9% monthly, then you allow yourself to fail 0.1% of the month —and that 0.1% is a budget you can spend as you like—. Let us compute it:

SECONDS_PER_MONTH = 30 * 24 * 3600
slo = 0.999
budget_s = (1 - slo) * SECONDS_PER_MONTH
print(f"error budget of a monthly 99.9% SLO:")
print(f"  (1 - {slo}) x {SECONDS_PER_MONTH:,} s = {budget_s:,.0f} s/month = {budget_s/60:.1f} min/month")

What to expect. When you run it:

error budget of a monthly 99.9% SLO:
  (1 - 0.999) x 2,592,000 s = 2,592 s/month = 43.2 min/month

You have 43.2 minutes of allowed downtime a month, and the key word is budget: it is yours to spend. On what is it spent? On risky deploys, on experiments, on the failover windows of lesson 3, on planned maintenance. As long as you do not exhaust the 43.2 minutes, you are within the SLO and you can keep taking risks (deploying new features). If you get close to exhausting it, the signal is clear: stop taking risks, freeze the changes, dedicate the team to stabilizing until the budget recovers the next month. The error budget turns reliability from a discussion of opinions ("do we deploy on Friday?") into a decision with a number ("do we have budget left? yes → go ahead; no → we wait"). It is the idea that makes peace between those who want to move fast and those who want not to go down: the budget is exactly how much risk you can afford.

The dependency ceiling: you cannot promise more than your providers

There is a hard limit many people ignore when setting an SLA: your availability cannot exceed that of the things you depend on. If Enlace runs on a compute provider, a managed database and a DNS/CDN, and all of them are in series on the path of a request (lesson 2), your ceiling is the product of their availabilities:

SECONDS_PER_YEAR = 365 * 24 * 3600
deps = {"cloud compute": 0.9995, "managed db": 0.9995, "DNS/CDN": 0.9999}
ceiling = 1.0
for name, a in deps.items():
    ceiling *= a
    print(f"  x {name}: {a}")
print(f"availability ceiling = {ceiling:.6f} ({ceiling*100:.4f}%)")
downtime_h = (1 - ceiling) * SECONDS_PER_YEAR / 3600
print(f"implicit downtime of the ceiling = {downtime_h:.2f} h/year")

What to expect. When you run it:

  x cloud compute: 0.9995
  x managed db: 0.9995
  x DNS/CDN: 0.9999
availability ceiling = 0.998900 (99.8900%)
implicit downtime of the ceiling = 9.63 h/year

Look at the number: even though each provider promises "four long nines", the product of the three is 99.89% —below 99.9%—, with 9.63 h/year of implicit downtime just from the dependencies. The consequence is hard and concrete: Enlace cannot seriously promise a 99.99% SLA (not even 99.9% comfortably) while it depends on those three providers in series, however perfect its own code is. Each dependency in series lowers your ceiling, exactly like the chain of lesson 2. To promise more, you would have to: make the dependencies redundant (multi-region, multi-provider), or remove dependencies from the critical path. This calculation is the first one anyone about to sign an SLA should do: does my dependency ceiling give me margin for what I want to promise? If not, the SLA is a promise you do not control.

The SLO we propose for Enlace

Let us put everything together into a concrete recommendation for Enlace.

  • The SLI: the percentage of resolve requests answered successfully (correct redirect) in under, say, 100 ms, measured monthly. We choose resolve because it is 99% of the traffic (100:1) and the path that really matters to the user —being redirected fast—. Creation (shorten) can have its own SLO, more lax.
  • The SLO: 99.9% on resolve. It is a serious but achievable objective: it gives 43.2 min/month of error budget, enough to absorb the primary's failover windows (lesson 3, which moreover only degrade writes, not reads) and the occasional risky deploy. Chasing 99.99% (52.6 min/year) would multiply the cost —perfect automatic failover, multi-region redundancy— for a URL shortener, where 8.76 h/year of possible downtime is perfectly tolerable.
  • The SLA: something below the SLO, for example 99.5% toward paying customers, to have a cushion: if a bad month the SLI drops to 99.7%, you miss your internal SLO (alarm signal) but you do not break the SLA (no penalty paid). SLA ≤ SLO, always.
  • The dependency check: before promising 99.9%, the calculation above says the series dependency ceiling is 99.89% —dangerously close—. Honest conclusion: to sustain 99.9% comfortably, Enlace needs to reduce its series dependency on the weakest component (for example, cache in the CDN so as not to depend on the db on every resolve, or make the db redundant as in lesson 2). The SLO sets the goal; the dependency ceiling says how much architecture work is needed to meet it.

And the uncomfortable truth that closes the topic: nobody promises 100%, and whoever does, lies. 100% would mean zero downtime, ever, not for maintenance, nor in the face of a provider's failure, nor in a disaster —physically impossible in a real system—. SLAs are written in nines precisely because 100% does not exist; the question is never "down or not?", but "how many nines can I sustain and what does each one cost me?".

Common mistakes

Confusing SLI, SLO and SLA (of definition). What happens: someone uses the three as synonyms —"our SLA is 99.9%" when they mean the internal objective, or "we measure the SLA" when they measure the SLI—. Why it happens: they are three similar acronyms for three layers of the same topic. How to spot it: ask yourself "is this what I measure (SLI), what I aim for (SLO) or what I promise with a penalty (SLA)?"; if you cannot answer, you are mixing them. How to fix it: remember the chain —the SLI is the measured fact, the SLO the internal goal, the SLA the contractual promise— and the rule SLA ≤ SLO.

Not translating the percentage into time (of calibration). What happens: someone promises or demands "99.99%" without having computed that it is 52.6 min/year, and then is surprised —in either direction— by what that implies in effort or in tolerance. Why it happens: percentages close to 100 say nothing to intuition. How to spot it: if you cannot say from memory (or in ten seconds) the yearly downtime of a level, you do not really understand it. How to fix it: memorize the three anchors —99% ≈ 87.6 h, 99.9% ≈ 8.76 h, 99.99% ≈ 52.6 min a year— and keep in mind that each nine is a factor of ten.

Promising an SLA above the dependency ceiling (of architecture). What happens: a team signs 99.99% while its three series dependencies give a product of 99.89%, guaranteeing that it will break the SLA for reasons outside its code. Why it happens: the availability of one's own service is looked at and it is forgotten that series dependencies multiply downward. How to spot it: compute the product of the availabilities of everything on the critical path; if it is less than your SLA, you already lost. How to fix it: before promising, do the ceiling calculation; if it does not suffice, reduce series dependencies or make them redundant (lesson 2) until the ceiling has margin over the SLA.

Exercises

Exercise 1 — Translate the nines. Without running the code, using that each nine divides the downtime by ten and that 99% ≈ 87.6 h/year: (a) how much yearly downtime does 99.9% allow? (b) And 99.99%? (c) A provider promises "four and a half nines" (99.995%). Approximately how much yearly downtime is that?

See solution
  • (a) 99.9%: one more nine than 99%, so 87.6 / 10 = 8.76 h/year.
  • (b) 99.99%: another nine, 8.76 / 10 = 0.876 h = 52.6 min/year.
  • (c) 99.995%: it is between 99.99% (52.6 min/year) and 99.999% (5.3 min/year). "Four and a half nines" is half the downtime of four nines: 52.6 / 2 ≈ 26 min/year. (Exact calculation: (1 - 0.99995) × 8760 h ≈ 0.438 h ≈ 26.3 min/year.)

Exercise 2 — Spend the error budget. The resolve SLO is 99.9% monthly (43.2 min/month of budget). This month there have already been: a primary failover that degraded the service for 8 minutes, a failed deploy that caused 12 minutes of errors, and a DNS provider outage of 15 minutes. (a) How much budget is left? (b) Should the team deploy a risky feature this week? (c) What decision does the error budget make for you?

See solution
  • (a) Spent: 8 + 12 + 15 = 35 min. Left: 43.2 - 35 = 8.2 min of budget for the rest of the month.
  • (b) It is not advisable. With only 8.2 minutes of margin, a risky deploy (which historically can cost more than 8 minutes if it goes wrong) could break the SLO. The signal is "stabilize, do not take risks".
  • (c) The error budget turns the question of opinion ("do we deploy?") into a rule with a number: as long as there is budget, you can take risks; when it runs out or very little is left, changes are frozen and the team dedicates itself to reliability until the budget renews the next month. It decides by evidence, not by discussion.

Exercise 3 — Enlace's SLA, with ceiling. Enlace wants to promise a 99.9% SLA on resolve. Its series dependencies are: compute (99.95%), database (99.9%) and CDN (99.99%). (a) Compute the dependency ceiling. (b) Can Enlace sustain 99.9% with this ceiling? (c) Propose an architecture change that raises the ceiling, using what you learned in lesson 2.

See solution
  • (a) Ceiling = 0.9995 × 0.999 × 0.9999 = 0.998401, that is 99.84%. Implicit downtime: (1 - 0.998401) × 8760 h ≈ 14 h/year.
  • (b) Not comfortably. The ceiling (99.84%) is below the SLA it wants to promise (99.9%). Enlace would break the SLA because of its dependencies even if its own code were perfect —the weak link is the database at 99.9%, which drags the product down—.
  • (c) The change from lesson 2: make the database redundant (the weakest link). Two instances at 99.9% in parallel give 1 - (1-0.999)^2 = 0.999999, almost 99.9999%. New ceiling: 0.9995 × 0.999999 × 0.9999 = 0.999400, that is 99.94% —now above the 99.9% SLA, with margin—. Complementary alternative: serve many resolves from the CDN to take the db off the critical path of most reads, reducing its weight in the product. The rule: make redundant or take off the path the weakest link until the ceiling has margin over what you promise.

Summary and next step

In this lesson you put a number on reliability. With the store's sign you saw that a percentage close to 100 fools you —99% and 99.9% sound the same and differ by ten times— and that only translating it into time makes it honest. You separated the three terms: SLI (what you measure), SLO (the internal goal you aim for) and SLA (the contractual promise with consequences), with the rule SLA ≤ SLO. You ran the table of the nines —99% = 87.6 h/year, 99.9% = 8.76 h/year, 99.99% = 52.6 min/year, 99.999% = 5.3 min/year— and you saw that each nine divides the downtime by ten and costs ~10× more. You computed the error budget (99.9% monthly = 43.2 min/month of allowed downtime, a budget that gets spent and that decides when to take risks) and the dependency ceiling (the product of the series availabilities, which you cannot exceed however perfect you are). And you proposed for Enlace an SLO of 99.9% on resolve, justified with its numbers, understanding why 100% is a lie.

Before moving on you should be able to: distinguish SLI, SLO and SLA with an example of each; reproduce the nines table from memory (the three anchors and the factor of ten); compute an error budget and explain what it is for; and compute the dependency ceiling of a system and say whether it sustains a given SLA.

What comes next is putting it all together. You have already gone through the three blocks of the module —reliability (SPOF, redundancy, failover), the consistency tradeoff (CAP, PACELC, strong versus eventual) and the yardstick (nines, SLO, error budget)—. In lesson 8, the project, you will produce a written and defensible decision for Enlace: its reliability plan (the SPOFs and how you eliminate them, with a diagram), its CAP/PACELC classification (PA/EL, justified), its consistency model (eventual for resolve, with read-your-writes and the uniqueness guarantee), and its complete SLO (the SLI, the objective, the executed error budget and the dependency ceiling check). It is where the six lessons become a design document.

Resources

← Previous6. Strong versus eventual consistency, and Enlace's choiceNext →8. Project: choose and justify Enlace's consistency model and its SLO