Module 3: Metrics Latency Throughput Errors

2. Latency: what it is and why percentiles, not the average

Next →

Overview

Latency is the first instrument on the dashboard and the most important, because it's the one the user feels. Nobody directly perceives how many requests per second your API processes; what they perceive is that when they clicked Quote, the price appeared in 40 milliseconds or in 4 seconds. That wait —the time between the request leaving and the response arriving— is the latency, and this lesson opens it completely: what exactly it measures, how it decomposes inside, the crucial difference between the latency the client sees and the one the server processes, and —the lesson that runs through the whole module— why latency is never summarized with the average.

We start with the concrete. In k6, an HTTP request's latency is called http_req_duration, and it's not an atomic number: it's the sum of three segments (http_req_sending + http_req_waiting + http_req_receiving), where the central segment, http_req_waiting, is the famous TTFB (time to first byte, the time the server takes to start responding). Understanding that anatomy lets you diagnose where the time goes. Then we measure for real, with the Python generator, something that separates whoever understands latency from whoever recites it: that the latency the client observes isn't the same as the server's computation, because the client includes the queue wait. And with those numbers in hand, you deal the first blow to the trap of the average.

Connection to the module: this lesson installs the concept of latency and the intuition of why the average fails; lesson 3 does the mechanical part —computing p50/p90/p95/p99 with statistics.quantiles—; and lesson 7 is the climax, where we see with full numerical force how much the average lies about a distribution with a tail. Here we reuse the load generator and the VUs from module 2 without re-explaining them: if you need to refresh what a VU is, go back to module 2 for a moment. What's new here is what you measure with those VUs.

The elevator's wait time

Imagine you manage an office building and want to know if the elevator "runs well." You measure the wait time of each person who calls it: from when they press the button until the doors open. That's the elevator's latency. Now, that time isn't a single thing inside: there's a segment where the system registers your call (sending), a large segment where the cabin travels to your floor (the work time, the one that really matters), and a segment where the doors open (receiving). If the elevator takes long, knowing which of the three segments the time went into tells you whether the problem is the motor or the doors. That's exactly the anatomy of http_req_duration: sending + server work + receiving.

And here comes the nuance almost everyone overlooks. When you alone call the elevator at three in the morning, the building empty, the cabin arrives in ten seconds: that's the pure service time —how long the mechanism takes when no one else uses it—. But at nine in the morning, with two hundred people calling it on every floor, your wait isn't ten seconds: it's two minutes, because the cabin is busy serving others and you're in a queue. The mechanism didn't get slower —the cabin rises just as fast—; what happened is that now you wait your turn. That distinction —service time (the pure work) vs the time the user sees (work + queue)— is the difference between server latency and client latency, and you'll see it measured in this lesson.

Finally, the naive manager's mistake: reporting "the elevator takes 25 seconds on average." The average sounds reassuring, but it hides that at three in the morning it takes 10 seconds and at nine it takes two minutes. The person waiting two minutes doesn't live an "average of 25 seconds"; they live two minutes, and they're furious. The average is a convenient statistical lie: real as a number, false as a description of what people experience. That's why, for latencies, we look at percentiles.

http_req_duration: the anatomy of a latency

In k6, the latency metric of an HTTP request is http_req_duration. It's the one you'll look at most in the whole guide, and it's worth knowing it's not indivisible. Per k6's official documentation, it decomposes like this:

http_req_duration = http_req_sending + http_req_waiting + http_req_receiving
SegmentWhat it measuresElevator analogy
http_req_sendingTime sending the request data to the server.Registering your call (pressing the button).
http_req_waitingTime waiting for the server's response: the TTFB (time to first byte). It's the real work of the server.The cabin traveling to your floor.
http_req_receivingTime receiving the response data.The doors opening.

(k6 also measures two segments before the request —http_req_blocked, waiting for a TCP connection slot, and http_req_connecting, establishing the connection— but those don't count within http_req_duration.) Of the duration's three segments, the one that almost always dominates and that truly reflects the backend's health is http_req_waiting, the TTFB: it's the time the server took to process the request and start responding. If your total latency is high and the TTFB is the largest part, the bottleneck is in the server (or its database); if the TTFB is low but receiving is high, the problem is transferring a huge response or a slow network. Diagnosing where the time goes is what this anatomy gives you.

Worked example: the client's latency isn't the server's

Let's actually measure the lesson's most important distinction: that the latency the client observes (what the generator measures, and includes the queue wait) can be dozens of times greater than the server's computation (the pure work). We use Reservo's normal /quote —the canonical one, no tricks— and measure it two ways: first with a single client (no queue: it approximates the pure service time) and then with 60 concurrent clients (with a queue: the latency a user lives when there's traffic).

The generator is the same one from module 1, which times client side: it starts the clock just before sending the request and stops it just after receiving the complete response. That end-to-end time.perf_counter() is, by definition, the client's latency —it includes everything: the queue, the transport over localhost, and the server's computation—.

import json, statistics, sys, time, urllib.request
from concurrent.futures import ThreadPoolExecutor

PORT = int(sys.argv[1])
URL = f"http://127.0.0.1:{PORT}/quote"
PAYLOAD = json.dumps({"room": "Focus", "tier": "basic", "hours": 3}).encode()

def one(_):
    """One request. Returns the latency in ms, timed CLIENT side."""
    start = time.perf_counter()
    req = urllib.request.Request(URL, data=PAYLOAD,
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=10) as resp:
        resp.read()
    return (time.perf_counter() - start) * 1000

def q(data, p):
    return statistics.quantiles(data, n=100, method="inclusive")[p - 1]

# 1) almost "pure server": 1 client, no contention
solo = sorted(one(0) for _ in range(300))
# 2) under load: 60 concurrent clients
with ThreadPoolExecutor(max_workers=60) as pool:
    carga = sorted(pool.map(one, range(2000)))

print(f"1 client    (~service): p50 {q(solo, 50):6.3f} ms   p95 {q(solo, 95):6.3f} ms")
print(f"60 clients  (client)  : p50 {q(carga, 50):6.3f} ms   p95 {q(carga, 95):6.3f} ms")

What to expect. With a single client, the latency is sub-millisecond: Reservo computes 2500 * 3 = 7500 in the blink of an eye. With 60 clients at once, the same API, the same computation, takes 30-50 times longer —and that difference isn't server computation, it's queue wait—. This is the real output:

1 client    (~service): p50  0.229 ms   p95  0.330 ms
60 clients  (client)  : p50 10.184 ms   p95 18.376 ms

Read it slowly, because it's one of the central truths of performance testing. The server didn't get slow: processing a quote still costs it fractions of a millisecond (the 0.229 ms of the no-contention case confirm it). What changed is that, with 60 clients competing, each request queues before being served. The client's latency —10 ms median, 18 ms p95— is server work + queue wait, and under load the queue is the big part. That's why measuring a single isolated request with curl tells you the service time, not the latency under load: they're different properties of the same system, and the one that matters to the user is the second.

In k6 terms: http_req_waiting (the TTFB) captures mostly the server's work, but the full http_req_duration —and certainly what the user feels— includes the wait. When in production you see a high p95 with a high TTFB too, the server is saturated; when you see a high p95 with a low TTFB, the request spent time waiting (in a queue, in connection) before the server touched it.

Why the average lies: the first glance

You already have the elevator's intuition; now the first hard datum. Let's take a run of the normal /quote with 30 concurrent clients and look, side by side, at the average and the percentiles. This is the generator's real output (endpoint /quote, 3000 requests, 30 clients):

throughput RPS     5069.7 req/s
errors           0 (0.00%)
-- latency (ms), client side --
min                  0.94
avg (average)        5.88
median (p50)         5.44
p90                  8.45
p95                  9.43
p99                 14.55
max                 34.54

Here the average (5.88 ms) and the median (5.44 ms) almost coincide, and the p95 (9.43 ms) isn't far off. Why? Because /quote is a healthy endpoint, with no long tail: almost all requests take about the same, so the distribution is compact and the average describes it well. Keep this run as the happy case: when the average and the p95 are close, there's no tail to fear.

The problem appears when the distribution has a tail —a few requests much slower than the rest—, which is normal in production (a database spike, a garbage collector that fires, a lock). In that scenario the average and the p95 divorce, and the average starts to lie. To see it, this module uses /quote_slow, the declared slow endpoint. This is its real output (2000 requests, 50 clients):

avg (average)       28.64
median (p50)        12.18
p90                 38.59
p95                182.81
p99                240.31
max                257.33

Look at what happened. The average says 28.64 ms. But the median —the typical user— saw 12.18 ms, less than half the average. And the p95 —the worst 5%— saw 182.81 ms, six times the average. No real user lived "28.64 ms": most lived ~12 ms and a minority suffered ~180 ms. The average is the spot where no one is: it falls in no man's land between the fast bulk and the slow tail, dragged upward by a few enormous values. Reporting "the API takes 28.64 ms on average" describes a user who doesn't exist. This is why we look at percentiles, and lesson 3 teaches you to compute them precisely. For now keep the phrase:

The average of a latency with a tail falls where almost no one is: above the typical user (the p50) and well below the one who suffers (the p95). A latency is described with percentiles —p50 for the typical, p95/p99 for the tail—, never with a single average.

Common mistakes

Measuring with curl once and calling it "the latency." What happens: someone does curl to /quote, sees 2 ms, and reports "the API responds in 2 ms." Why it happens: they confuse the service time (an isolated request, no queue) with the latency under load (with a queue). How to detect it: if your measurement had no concurrency, you measured the rest, not the load. How to fix it: measure with several clients at once, like this lesson's generator —the same API went from 0.23 ms with one client to 18 ms p95 with 60—. The latency that matters to the user is the load one.

Reporting the average latency. What happens: the report says "average latency: 28 ms" and everyone relaxes, without seeing that the p95 is 180 ms. Why it happens: the average is the default summary of almost everything, and for latencies it's exactly the wrong one. How to detect it: if your latency number is an avg and not a percentile, you're describing a user who doesn't exist. How to fix it: report at least p50 and p95 (and p99 if the extreme tail matters to you). The average, for latencies, is ignored.

Confusing http_req_waiting with http_req_duration. What happens: someone optimizes the server looking only at the TTFB (waiting) and doesn't understand why the user keeps complaining. Why it happens: the TTFB is only one segment of the total duration; if the time goes into receiving (huge response) or into the connection wait, lowering the TTFB doesn't help. How to detect it: if http_req_duration is high but http_req_waiting is low, the bottleneck isn't in the server's computation. How to fix it: look at the complete decomposition —sending, waiting, receiving— and attack the segment that dominates, not the one you assume.

Exercises

Exercise 1 — Diagnose by the anatomy. For each case, say where the bottleneck probably is, using the sending + waiting + receiving decomposition. (a) http_req_duration = 900 ms, of which http_req_waiting = 870 ms. (b) http_req_duration = 900 ms, waiting = 40 ms, receiving = 840 ms. (c) http_req_duration = 900 ms, but http_req_blocked (prior) = 850 ms.

See solution
  • (a) The time goes into waiting (TTFB). The server takes long to process and start responding: the bottleneck is in the backend (computation, database, a service it depends on). It's the most common case.
  • (b) The server responds fast (waiting = 40 ms), but transferring the response takes 840 ms (receiving). The bottleneck is in the size of the response or the network: a huge response, or a slow link. Optimizing the server wouldn't help; the payload has to be reduced.
  • (c) The bulk (blocked = 850 ms) is prior to the request and doesn't even count within http_req_duration. It's time waiting for a TCP connection slot: the bottleneck is the connection pool (too few connections for so much concurrency), not the server.

The lesson: "the API is slow" isn't a diagnosis; the decomposition tells you what is slow.

Exercise 2 — Client vs server. In this lesson's run, a single client saw a p50 of 0.229 ms and 60 clients saw a p50 of 10.184 ms, against the same API without changing it. (a) Did the server get ~44 times slower? (b) What causes the difference? (c) Which of the two numbers matters to the user in production?

See solution
  • (a) No. The server computes 2500 * 3 = 7500 just as fast in both cases; its service time didn't change. The single-client case confirms it: 0.229 ms is what the pure computation costs.
  • (b) The queue wait. With 60 clients competing for attention, each request queues before being served. The client's latency is service time + queue wait, and under load the queue dominates.
  • (c) The 60-client one (10 ms p50, 18 ms p95). In production there's concurrent traffic, so the user lives the latency with a queue, not the isolated service time. Measuring with a single client underestimates what people actually experience.

Exercise 3 — Does the average lie here? For each latency distribution (in ms), say whether the average describes it well or lies, comparing it with the p50 and the p95. (a) /quote: avg 5.88, p50 5.44, p95 9.43. (b) /quote_slow: avg 28.64, p50 12.18, p95 182.81. (c) A hypothetical API: avg 100, p50 100, p95 105.

See solution
  • (a) The average describes it well. avg (5.88) ≈ p50 (5.44) and the p95 (9.43) is close: the distribution is compact, with no long tail. When avg ≈ p50 and the p95 doesn't spike, the average is an honest summary.
  • (b) The average lies. avg (28.64) is more than double the p50 (12.18) and six times smaller than the p95 (182.81): there's a long tail dragging the average to no man's land. Here you have to report percentiles, not the average.
  • (c) The average describes it well. avg = p50 = 100 and the p95 (105) is right up against it: flat distribution, no tail. (It's slow —100 ms— but consistent; the average doesn't deceive about the shape.)

The mechanical rule: if avg and p50 almost coincide and the p95 doesn't spike, the average is honest; as soon as avg separates from the p50, there's a tail and the average starts to lie.

Summary and next step

In this lesson you opened the dashboard's first instrument: latency. You learned that in k6 it's called http_req_duration and that it decomposes into sending + waiting + receiving, where waiting is the TTFB —the server's real work— and the key to diagnosing where the time goes. You measured for real the distinction between client latency (what the user sees: work + queue wait) and server latency (the pure computation): the same Reservo went from 0.229 ms with one client to 18 ms p95 with 60, and that difference is queue, not server slowness.

And you dealt the first blow to the module's central trap: the average lies when the latency has a tail. You saw it with numbers —/quote_slow: average 28.64 ms, but the typical user lived 12 ms and the worst 5% lived 183 ms—. The average falls where almost no one is; latency is described with percentiles. Before moving on you should be able to: name the three segments of http_req_duration and what the TTFB is; explain why the client latency exceeds the service time under load; and say why the average of a latency with a tail describes a user who doesn't exist.

What comes next is leaving the intuition and moving to the mechanics. In lesson 3 you learn to compute p50, p90, p95, and p99 for real with statistics.quantiles: what the function returns, how to index each percentile, the difference between the inclusive and exclusive methods, and how a single latency from the tail spikes the average but barely moves the median.

Resources

← Previous1. Module introduction: the test's dashboardNext →3. p50/p90/p95/p99 with `statistics.quantiles`