Module 3: Metrics Latency Throughput Errors

3. p50/p90/p95/p99 with `statistics.quantiles`

Next →

Overview

In lesson 2 it became clear why we look at percentiles instead of the average. This lesson is the mechanics: what exactly p50, p90, p95, and p99 are, how each one is read in a sentence anyone understands, and how they're actually computed from a list of measured latencies using statistics.quantiles from Python's standard library. By the end, a percentile will stop being a blog word and become something you know how to produce with five lines of code and explain without hesitation.

A percentile is a simple idea disguised as a technical term. The p95 of a list of latencies is the value below which 95% of the measurements fall; put in human terms, "95% of the users saw this or something better, and the worst 5% saw this or something worse." The p50 is the median (the user right in the middle); the p99 is the extreme tail (1 in every 100). Computing them is sorting the latencies and finding the value at the corresponding position —or, better, letting statistics.quantiles do the interpolation for you—. We'll do it first on a minimal list of ten latencies, where you can follow the count by hand and see how a single tail value spikes the average but barely moves the median; and then on a real run against Reservo, with thousands of requests.

Connection to the module: lesson 2 gave you the intuition; this one gives you the computation tool; lesson 7 uses it in depth for the average-vs-p95 climax. The q(data, p) helper you build here —the one that wraps statistics.quantiles— is the same one that appears in the load generator throughout the module and in lesson 8's mini-project. In k6 you won't compute percentiles by hand: the tool gives you the p(90) and p(95) columns directly in its summary (lesson 6). But computing them yourself once, with your own hands, is what makes those k6 columns stop being magic.

The exam and the cutoff grade

Think of an exam a thousand students took. The teacher wants to summarize "how the group did." They could give the average of the grades, but that hides a lot: an average of 7 can be "almost everyone got 7" or "half got 10 and half got 4." So instead they use percentiles. They sort the thousand grades from lowest to highest and ask: what grade leaves 50% of the students below? That's the median grade (p50). And 95%? That's the p95: only 5% of the students got more than that grade. Percentiles turn a mountain of numbers into concrete questions about positions: "the student in place 950 of 1000, what did they get?".

With latencies it's identical, only "higher" is worse (slower). You sort all the measured latencies from lowest to highest. The p50 is the latency at the central position: half the requests were as fast or faster, half as slow or slower —the typical user—. The p95 is the latency at the 95% position: 95% of the requests were as fast or faster, and only the worst 5% were slower —the user who starts to suffer—. The p99 is the 99% position: the extreme tail, 1 in every 100. Notice the deliberate asymmetry: we don't pick p50 and then p60, p70, p80... we jump to p90, p95, p99 because what matters to us is the tail —the slow users—, and the tail lives in the high percentiles. The p50 tells us how the typical one is doing; the high percentiles tell us how badly the unlucky one is having it.

The pXX is the value below which XX% of the measurements fall. For latencies: "XX% of the users saw this or better; the worst (100−XX)%, this or worse." p50 = the typical; p95 = the one who suffers (1 in every 20); p99 = the extreme tail (1 in every 100).

statistics.quantiles: how it's actually computed

Python ships everything you need in the standard library: statistics.quantiles. The function splits your data into n equal-sized groups and returns the n − 1 cut points between them. If you ask for n=100 (percentiles), it returns 99 values: the first is the p1, the second the p2, …, the last (number 99) is the p99. To get a specific percentile, you index: the pXX is at position XX − 1 of the returned list (because lists start at 0).

import statistics

# statistics.quantiles(data, n=100) returns 99 cut points: [p1, p2, ..., p99]
def q(data, p):
    """Percentile p (1-99) of data, with the inclusive method."""
    cuts = statistics.quantiles(data, n=100, method="inclusive")
    return cuts[p - 1]   # p50 -> cuts[49], p95 -> cuts[94], p99 -> cuts[98]

A detail worth fixing: statistics.quantiles has two methods, exclusive (the default) and inclusive, and they give slightly different values at the extremes. The exclusive method assumes your data is a sample of a larger population and can extrapolate beyond the observed minimum and maximum; the inclusive one treats your data as the complete population and never leaves the measured range. For latencies of a load test —where your measurements are all the requests you made, not a sample of a theoretical universe— the inclusive method is the most natural, and it's the one we use throughout the guide. The difference is small with lots of data, but it's worth being explicit: we always pass method="inclusive".

Worked example: ten latencies you can follow by hand

Before thousands of requests, let's do it with ten, so you see each number. Imagine you measured ten latencies (in ms): nine fast, between 8 and 14 ms, and one from the tail, of 250 ms —a spike, like the ones produced by a database that sometimes jams—.

import statistics

data = [8, 9, 10, 10, 11, 12, 12, 13, 14, 250]  # 9 fast + 1 from the tail

def q(data, p):
    return statistics.quantiles(data, n=100, method="inclusive")[p - 1]

print("average  (fmean):", round(statistics.fmean(data), 1))
print("median   (p50)  :", statistics.median(data))
print("p90:", round(q(data, 90), 1))
print("p95:", round(q(data, 95), 1))
print("p99:", round(q(data, 99), 1))
print("max:", max(data))

What to expect. The single large value (250) will drag the average well above the nine fast latencies, while the median stays calm in the center. This is the real output:

average  (fmean): 34.9
median   (p50)  : 11.5
p90: 37.6
p95: 143.8
p99: 228.8
max: 250

Stop at the contrast. Nine of the ten requests took 14 ms or less. The typical user (p50) lived 11.5 ms. And yet the average is 34.9 ms: almost triple what the majority lived, pushed on its own by the 250. If you reported "the API takes 34.9 ms on average," you'd be describing a time that none of the ten requests had —neither the fast ones (≤14) nor the slow one (250)—. The average fell into the void. The median (11.5), by contrast, faithfully describes the nine fast requests, and the p95 (143.8) and p99 (228.8) capture that there's an ugly tail. With these five numbers —p50, p90, p95, p99, max— you tell the complete story; with the average alone, you falsify it.

Notice too how the percentiles rise as they approach the tail: p90 = 37.6, but p95 = 143.8 and p99 = 228.8. That abrupt jump between p90 and p95 is the signature of a long tail: when the high percentiles spike while the low ones are close together, you know there's a minority of requests hugely slower than the bulk. It's exactly what happens in production, and what /quote_slow reproduces.

The two methods, up close

So you see the difference between inclusive and exclusive (and why we chose the first), here are the quartiles of those same ten data points with both methods:

statistics.quantiles(data, n=4)                     # exclusive (default)
# -> [9.8, 11.5, 13.2]
statistics.quantiles(data, n=4, method="inclusive") # inclusive
# -> [10.0, 11.5, 12.8]

Both coincide on the median (11.5), but differ on the extreme quartiles (Q1 and Q3): exclusive pushes them a bit further out (9.8 and 13.2) because it assumes there's more population outside the sample; inclusive stays further in (10.0 and 12.8). With ten data points the difference is noticeable; with thousands it blurs. For a load test we use inclusive because our measurements are the complete population of that run, not a sample of something larger.

From ten to thousands: Reservo's real percentiles

Now the same, but on a real run. The generator hits /quote_slow with 2000 requests and 50 concurrent clients, saves the 2000 latencies, and computes the percentiles with the same q helper. This is the real output:

avg (average)       28.64
median (p50)        12.18
p90                 38.59
p95                182.81
p99                240.31
max                257.33

You recognize the signature: p50 and p90 relatively close (12 and 39 ms), and then the jump to p95 = 182.81 and p99 = 240.31 —the long tail of the slow endpoint, which we declared in lesson 1—. With 2000 data points instead of 10, the percentiles are stable and reliable, but the reading is identical to the minimal example's: half the users saw ≤12 ms, and 1 in every 20 saw ≥183 ms. The average (28.64) still falls in no man's land. The only difference between the ten-latency example and this 2000 run is the scale; the technique —statistics.quantiles(data, n=100), indexing [p−1]— is exactly the same.

Common mistakes

Indexing the result of quantiles wrong. What happens: someone does statistics.quantiles(data, n=100)[95] expecting the p95 and gets the p96. Why it happens: the returned list has 99 elements (p1…p99) and indexes from 0, so the pXX is at position XX − 1, not XX. How to detect it: if your "p95" doesn't match what you expect, check the index; [95] is the p96, [94] is the p95. How to fix it: use a helper like q(data, p) that subtracts 1 for you (cuts[p - 1]), and that way you don't get it wrong again.

Forgetting the method and comparing percentiles computed in different ways. What happens: one run uses exclusive (the default) and another inclusive, and comparing them it seems the latency changed when only the method changed. Why it happens: statistics.quantiles uses exclusive by default, and it's easy to forget it in one of the two. How to detect it: small, systematic differences at the extremes between two "identical" runs. How to fix it: fix the method explicitly everywhere (method="inclusive" in this guide) and don't mix it.

Computing percentiles over few data points and trusting them. What happens: someone measures 8 requests and reports a p99 "of 228 ms." With 8 data points, the p99 is almost the maximum and means nothing stable. Why it happens: they forget that a high percentile needs lots of data to be reliable —the p99 asks for at least hundreds of measurements to make sense—. How to detect it: if your p95 or p99 changes drastically between runs, you probably have few data points. How to fix it: measure enough requests (thousands for a stable p99) and distrust high percentiles computed over handfuls of measurements.

Exercises

Exercise 1 — Read the percentiles out loud. For the /quote_slow run (p50 = 12.18 ms, p95 = 182.81 ms, p99 = 240.31 ms), translate each percentile into a sentence a non-technical manager understands. (a) the p50, (b) the p95, (c) the p99.

See solution
  • (a) p50 = 12.18 ms: "Half the users received their quote in 12 milliseconds or less." (The typical user.)
  • (b) p95 = 182.81 ms: "95% of the users received it in 183 milliseconds or less; the worst 5% —1 in every 20— waited more than that." (The one who starts to suffer.)
  • (c) p99 = 240.31 ms: "99% received it in 240 milliseconds or less; the worst 1% —1 in every 100— waited more." (The extreme tail.)

The template: "XX% saw pXX or better; the worst (100−XX)%, that or worse." Note that no sentence mentions the average: to describe the experience, percentiles are used.

Exercise 2 — Compute a percentile by hand and with quantiles. You have these seven latencies already sorted (ms): [10, 12, 14, 15, 18, 22, 400]. (a) What's the median (p50) by eye? (b) What happens to the average because of the 400? (c) Write the statistics.quantiles call that would give you the p90.

See solution
  • (a) With 7 sorted data points, the median is the middle one (position 4 of 7): 15 ms. Six of the seven latencies are between 10 and 22 ms.
  • (b) The 400 spikes the average: (10+12+14+15+18+22+400)/7 ≈ 70.1 ms. That "70 ms average" is more than quadruple the median (15) and larger than six of the seven measurements. The average, again, describes a nonexistent user.
  • (c) statistics.quantiles([10, 12, 14, 15, 18, 22, 400], n=100, method="inclusive")[89] — remember the p90 is at position 90 − 1 = 89.

Exercise 3 — The signature of the tail. For each set of percentiles, say whether the distribution has a long tail (you have to worry about the tail) or is compact (the average would do). (a) p50 = 5, p90 = 8, p95 = 9, p99 = 15. (b) p50 = 12, p90 = 39, p95 = 183, p99 = 240. (c) p50 = 100, p90 = 102, p95 = 103, p99 = 108.

See solution
  • (a) Compact. The percentiles rise smoothly and close together (5 → 8 → 9 → 15); there's no abrupt jump. It's the healthy /quote from lesson 2. The average would describe it well, though it's still worth reporting the p95.
  • (b) Long tail. Abrupt jump between p90 (39) and p95 (183): almost 5 times. It's the signature of a minority of hugely slower requests —the /quote_slow—. You have to look at p95/p99, never the average.
  • (c) Compact (but slow). The percentiles are right up against each other (100 → 108): flat distribution, no tail. It's consistently slow, not sometimes slow. The average wouldn't deceive about the shape, though a 100 ms p50 might be unacceptable in itself.

The rule: look for the jump between percentiles. If p90 → p95 → p99 spike, there's a tail; if they rise together, it's compact.

Summary and next step

In this lesson you turned "percentile" from a word into a tool. A pXX is the value below which XX% of the measurements fall: p50 = the typical user, p95 = the one who suffers (1 in every 20), p99 = the extreme tail (1 in every 100). You computed them for real with statistics.quantiles(data, n=100, method="inclusive"), which returns 99 cut points of which the pXX is at position XX − 1. And you saw the average trap with numbers you can follow by hand: ten latencies where nine were ≤14 ms and one was 250, and the average came out 34.9 —a time no request had—, while the median (11.5) faithfully described the majority.

You learned to recognize the signature of a long tail: when the low percentiles are close together but p95 and p99 spike (as in /quote_slow: p90 = 39, p95 = 183). And you fixed two practical details: indexing [p − 1] and always using method="inclusive" in a load test. Before moving on you should be able to: read a p95 in a sentence for non-technical people; write the quantiles call for any percentile; and distinguish a distribution with a tail from a compact one by looking at the jump between percentiles.

What comes next is the dashboard's second instrument. In lesson 4 we move from latency to throughput: what RPS (requests per second) is, how k6 reports it (http_reqs, iterations), and its —sometimes counterintuitive— relationship with VUs and think time. You'll see with a real sweep why raising virtual users raises throughput only until the system saturates, and after that the only thing that rises is the latency.

Resources

← Previous2. Latency: what it is and why percentiles, not the averageNext →4. Throughput/RPS and its relationship with VUs