Module 3: Metrics Latency Throughput Errors

7. Average vs p95 on a tailed distribution

Overview

This lesson is the module's climax. In lesson 2 you dealt the first blow to the average trap; in lesson 3 you learned to compute percentiles; here you see, with all the numerical force that can be gathered, how much the average lies about a latency distribution with a tail —and why that isn't a statistical curiosity, but the reason latency SLOs the world over are written in percentiles and never in averages—. By the end, you won't just know that "the average lies": you'll be able to prove it with a number that leaves anyone speechless.

That number is this: over /quote_slow's distribution with a tail, 88% of the requests were faster than the average. Read it again. The average isn't "the midpoint" of anything: it's a value that almost nine in every ten users beat, dragged upward by a slow minority. A number that 88% of people surpass doesn't describe "the typical user"; it describes no one. This lesson breaks down why that happens (the shape of a latency distribution, asymmetric, with a tail to the right), which percentile to choose according to what matters to you, and how all this translates into the practice of writing a performance target. It's the lesson that turns "use percentiles" from advice into conviction.

Connection to the module: here lessons 2 (why percentiles) and 3 (how to compute them) culminate, and it connects with what's coming: in module 5 you'll write thresholds like p(95)<500 —limits on the percentile you learn here to choose and defend—. The distinction between p50, p95, and p99 you close here is exactly the one you'll need to decide which percentile to put your limit on. We reuse the /quote_slow endpoint declared in lesson 1 and the usual generator.

The bar's average salary

There's a classic and perfect analogy for this. You're in a bar with nine friends; you all earn a similar salary, say 30,000 a year. The average salary at the table is 30,000, and it describes everyone well. Suddenly a billionaire who earns 100 million a year walks into the bar and sits with you. Now the table's average salary is over 9 million. Does that average describe anyone at the table? No one: not the ten of you who earn 30,000, not her who earns 100 million. The average jumped to a place where no one is sitting, dragged by a single extreme value. If a journalist wrote "the customers of this bar earn an average of 9 million," they'd say something technically true and deeply false.

Latencies with a tail are that bar. The vast majority of requests are "normal salaries" —fast, clustered—, and a few from the tail are "the billionaire" —extremely slow, a database spike, a lock, a garbage collector—. The average, which sums and divides, is incapable of resisting those few giant values: it weighs them the same as everyone else and spikes. The median (p50), by contrast, is immune: it's the value of whoever is right in the middle, and the billionaire walking into the bar doesn't change who's in the middle of the line. That's why, for latencies, the median describes the typical one and the average describes a ghost. And that's why the p95 —the one in place 95 of 100— captures how bad the tail is without letting itself be dragged to no man's land.

In a distribution with a tail, the average isn't the center: it's a value the majority beats, pushed by an extreme minority. The median (p50) describes the typical user; the p95/p99 describe those who suffer the tail. The average, alone, describes no one.

Worked example: 88% beat the average

Let's prove it with measured numbers. The generator hits /quote_slow with 2000 requests and 50 clients, saves the 2000 latencies, and computes not just the percentiles but something revealing: how many requests fell below the average.

import json, statistics, time, urllib.request
from concurrent.futures import ThreadPoolExecutor

def one(url, payload):
    s = time.perf_counter()
    req = urllib.request.Request(url, data=payload,
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=10) as r:
        r.read()
    return (time.perf_counter() - s) * 1000

url = f"http://127.0.0.1:{PORT}/quote_slow"
payload = json.dumps({"room": "Focus", "tier": "basic", "hours": 3}).encode()
with ThreadPoolExecutor(max_workers=50) as pool:
    latencies = sorted(pool.map(lambda _: one(url, payload), range(2000)))

q = lambda p: statistics.quantiles(latencies, n=100, method="inclusive")[p - 1]
avg = statistics.fmean(latencies)
below_avg = sum(1 for x in latencies if x < avg)

print(f"average  (avg)  : {avg:7.2f} ms")
print(f"median   (p50)  : {q(50):7.2f} ms")
print(f"p95             : {q(95):7.2f} ms")
print(f"p99             : {q(99):7.2f} ms")
print(f"requests BELOW the average: {below_avg} of {len(latencies)} "
      f"({below_avg / len(latencies) * 100:.1f}%)")

What to expect. The average will fall well above the median, and the vast majority of the requests will be below that average —the unmistakable signature of a tail to the right—. This is the real output:

average  (avg)  :   29.34 ms
median   (p50)  :   11.95 ms
p95             :  187.27 ms
p99             :  241.47 ms
requests BELOW the average: 1764 of 2000 (88.2%)

There's the blow. The average says 29.34 ms. The median says 11.95 ms: the typical user saw the latency at less than half the average. And the devastating part: 1764 of the 2000 requests (88.2%) were faster than the average. A number that 88% surpass isn't "the center"; it's a value inflated by the 5-10% of the tail (the p95 at 187 ms, the p99 at 241 ms) that drags the mean upward. If you reported "the API takes 29 ms on average," you'd be giving a number that almost no one experienced as their reality: the fast ones saw ~12 ms, the slow ones saw ~190 ms, and "29" is the no man's land between the two. The p50 (11.95) speaks to the typical; the p95 (187) speaks to the one who suffers; the average speaks to no one.

The shape of a latency distribution

Why do latencies always have this shape —clustered to the left, with a long tail to the right—? Because there's a floor but no ceiling. A request can't take less than its minimum service time (there's a physical limit: the computation, the network), so the latencies pack against that floor on the left. But on the right there's no cap: a request can take double, ten times, a hundred times more if it happens to wait for a lock, a database spike, a retry. That asymmetry —hard floor on the left, open tail on the right— is universal in systems, and it's exactly the shape that makes the average lie.

Visualize it with /quote_slow's latencies grouped into ranges:

  range (ms)     how many requests (approx.)
   0 –  15   ████████████████████████████  the vast majority (the fast bulk)
  15 –  50   ████████                       some
  50 – 120   ██                             few
 120 – 250   ███                            the TAIL (the ~10% slow that spikes the average)

The bulk lives glued to the left (0-15 ms); a minority lives in the tail (120-250 ms). The average is the "center of gravity" of the whole mass, and that right tail, however small, pulls the center of gravity toward it. The median and the percentiles don't let themselves be pulled: they count positions, not mass. That's why they capture the real shape —"most here, a few there"— that the average crushes into a single misleading number.

Which percentile to choose: p50, p95, or p99

If the average is discarded, which percentile do you look at? It depends on the question, and it's worth having the map:

Percentile	Who it describes	When it's your metric
p50 (median)	The typical user.	To know how "the normal experience" is going. Never alone: it says nothing about the tail.
p90 / p95	The slow users (1 in every 10 / 1 in every 20).	The standard of a latency SLO. The p95 is the most common: "almost everyone should see this or better."
p99	The extreme tail (1 in every 100).	When the tail matters a lot: payment systems, healthcare, or where a slow user is a lost user.
p99.9	The very long tail (1 in every 1000).	Huge-scale systems, where 1 in every 1000 is millions of requests a day.

The practical lesson: you choose the percentile according to how much the unlucky user matters to you. A p95 says "I accept that 1 in every 20 has it worse than my limit"; a p99, "only 1 in every 100." The higher the percentile you watch, the more you commit to the tail —and the more expensive it usually is to meet, because taming the tail is the hardest part of a system—. Google, in its SRE book, insists on this point: measure the tail's latency (high percentiles), because the average and the median can look healthy while a fraction of users suffer unacceptable latencies that that summary hides.

From the metric to the SLO (preview of module 5)

All this flows into how a performance target, an SLO (Service Level Objective), is written. A real-world latency SLO is never written over the average; it's written over a percentile:

"95% of the requests to /quote must respond in under 200 ms."
   -> p(95) < 200 ms

Notice that that statement is, almost literally, a k6 threshold (http_req_duration: ['p(95)<200']), which is module 5. Here we only preview it to close the arc: the reason module 5 puts its limits on p(95) and not on avg is exactly what you proved in this lesson —that the average hides the tail and the percentile reveals it—. Choosing your SLO's percentile (p95? p99?) is choosing which fraction of unlucky users you commit to; putting the number on it (200 ms? 500 ms?) is negotiating between what the user tolerates and what the system can give. The metric you learned to compute and read here is the raw material of that decision.

Common mistakes

Writing an SLO over the average. What happens: "the average latency must be < 100 ms" sounds like a target, but it's smoke: a system can meet a 100 ms average while 1 in every 20 users waits 2 seconds (the tail would inflate the average less than you think if the bulk is very fast). Why it happens: the average is the default summary and sneaks into the targets. How to detect it: if your SLO mentions "mean" or "average," it's badly formulated. How to fix it: write the SLO over a percentile (p(95) < X), which does bound the tail's experience.

Reporting only the p50 and thinking it's enough. What happens: someone moves from the average to the p50 (good) but reports only the p50, with no p95, and declares the API healthy —without seeing that the p95 is 10 times the p50—. Why it happens: they learn "use the median" and forget that the median also says nothing about the tail. How to detect it: if your only number is the p50, you're describing the typical and ignoring the one who suffers. How to fix it: always report at least p50 and p95 (and p99 if the tail matters to you). The p50 without the p95 hides the tail as much as the average.

Choosing a very high percentile "just in case" and being unable to meet it. What happens: someone sets their SLO at p(99.9) < 50 ms thinking "stricter is better," and then the team burns out chasing an impossible tail. Why it happens: "strict" gets confused with "correct." How to detect it: if your SLO demands a tail no realistic system gives and costs you more than the business needs, you over-specified. How to fix it: choose the percentile according to what really matters to the user and the business. For a great many services, a sensible p95 is the right point; the p99/p99.9 is reserved for when the extreme tail has a real cost.

Exercises

Exercise 1 — The number that leaves you speechless. In the /quote_slow run, the average was 29.34 ms and 88.2% of the requests were faster than it. (a) Explain in one sentence, for someone non-technical, why that makes the average useless. (b) What number would you use instead to describe the typical user? (c) And to describe the one who has it worst?

See solution

(a) "The 29 ms average is a number that almost 9 in every 10 users surpassed: it doesn't describe 'the normal user,' but a point inflated by the slow minority where almost no one is."
(b) The median (p50 = 11.95 ms): half the users saw that or better. It's the typical's honest number.
(c) The p95 (187.27 ms) —or the p99 (241.47 ms) for the extreme tail—: it describes what the worst 5% (or 1%) suffered. Putting p50 and p95 together tells the complete story; the average, none.

Exercise 2 — The shape of the distribution. For each set of numbers, say whether the distribution has a long tail to the right and how you know from the average/median relationship. (a) avg = 29.34, p50 = 11.95. (b) avg = 100, p50 = 99. (c) avg = 8, p50 = 12.

See solution

(a) Long tail to the right. The average (29.34) is much larger than the median (11.95): the tail of high values pulled the average up. It's /quote_slow's signature.
(b) No tail (compact). avg ≈ p50 (100 ≈ 99): the distribution is symmetric and flat. The average describes it well. (Slow, but consistent.)
(c) Impossible / suspicious data. The average (8) can't be smaller than the median (12) if the tail is to the right; that would indicate a tail to the left, very rare in latencies (it would imply many very low values and a high floor), or an error in the data. In real latencies, avg ≥ p50 almost always.

The mechanical rule: avg > p50 → tail to the right (the normal thing in latencies); avg ≈ p50 → compact; avg < p50 → suspect the data.

Exercise 3 — Choose the SLO's percentile. For each service, propose a percentile (p50, p95, p99, p99.9) for its latency SLO and justify it in one sentence. (a) A recipe blog. (b) A payment gateway. (c) Reservo's /quote API, with moderate traffic. (d) A global bank service with millions of daily requests.

See solution

(a) Recipe blog → p95. It matters that the general experience is good, but a recipe page slow now and then isn't a disaster; the p95 bounds the worst 5% without obsessing over the extreme tail.
(b) Payment gateway → p99 (or higher). Here a slow user can be a lost sale or a failed payment; it's worth committing to the tail (1 in every 100), not just the 5%.
(c) Reservo's /quote with moderate traffic → p95. It's the sensible standard for a typical API: it bounds the 95%'s experience without demanding an expensive tail the business doesn't need.
(d) Global bank, millions of requests → p99.9. At that volume, "1 in every 1000" is thousands of users a day; the very long tail has a real cost and must be watched.

There's no single answer; the key is that the percentile you choose grows with how much a slow user hurts. The more expensive the unlucky user, the higher the percentile you watch.

Summary and next step

In this lesson you saw, in full force, why latency is measured in percentiles and not the average. The number that sums it up: over /quote_slow's distribution with a tail, 88.2% of the requests were faster than the average (29.34 ms) —a value that almost nine in every ten users beat describes no one—, while the median (11.95 ms) described the typical and the p95 (187.27 ms) the one who suffered. You understood why it happens: latencies have a floor but no ceiling, so their distribution is asymmetric, with a tail to the right that drags the average but not the percentiles.

And you closed the arc toward practice: you choose the percentile according to how much the unlucky user matters (p95 for the typical, p99/p99.9 when the tail really hurts), and a latency SLO is always written over a percentile (p(95) < 200 ms), never over the average —which is exactly the shape of module 5's thresholds—. Before the project you should be able to: prove with the 88% datum why the average lies; explain the tailed shape of a latency distribution; and choose and justify an SLO's percentile.

What comes next is putting it all together with your own hands. In lesson 8, the mini-project: you bring up Reservo with its slow endpoint, run the load generator, report real p50/p95/p99, RPS, and % error, write the equivalent k6 summary as content, and interpret in writing why the p95 matters to the user more than the average. It's the whole module, executed by you.

Resources

Google SRE Book — Worrying About Your Tail (latency percentiles) — the canonical argument for why the tail (high percentiles) is watched and not the average; the frame for this whole lesson.
statistics.fmean and statistics.quantiles — Python documentation — the average and the percentiles the measured comparison of this lesson is made with.
k6 — Thresholds over percentiles — how p(95)<200 is written as a limit that makes the test pass or fail; the module-5 preview that closes this arc.
Google SRE Workbook — Implementing SLOs — how latency SLOs are chosen and written over percentiles in practice. Goes deeper into the "from the metric to the SLO" section.

← Previous6. Reading the `k6 run` summary Next →8. Mini-project: measure Reservo's metrics