Module 1: Why Load And Performance Testing

3. The questions a load test answers

Next →

Overview

A load test isn't a ritual you run to get a pretty number; it's an instrument for answering concrete questions about your system. If you don't know what questions it answers, you'll look at its output like someone staring at an airplane cockpit without knowing how to fly: many indicators, no meaning. In this lesson we fix the three questions every load test answers, and that hold both for the Python generator (executed) and for k6 (content): (1) How many concurrent users does the system hold? —its capacity—. (2) What's the p95 latency? —how fast it responds for the bad-percentile user, not the average one—. (3) Where's the breaking point? —beyond what load does latency spike or errors begin—. And along the way we take apart the most expensive mistake of all in measuring performance: trusting the average. The average latency is a half-truth that almost always lies toward optimism, and the percentile is its antidote.

Connection to the module: this lesson is the "what to ask"; the whole of module 3 will be the "how to measure it rigorously" (the metrics in depth: how k6 computes http_req_duration, the p90/p95/p99 percentiles, throughput as http_reqs/s, error rate as http_req_failed). Here we stay with the questions and with the intuition of why the p95 matters, using the real numbers from the generator you already ran in lesson 2. Lesson 4 will show that each of these questions is answered with a different type of test (capacity → stress, etc.). Consider this lesson the mental frame; the rest of the guide is how to fill it in.

The three questions, with an analogy

Imagine you manage a stadium's entrance and want to know if your turnstiles can handle a game day. There are three things you need to find out, and they're exactly the three questions of a load test.

First: how many people can get in at once without chaos forming? Not how many people enter over the whole day (that's accumulated volume), but how many at the same time can be passing through the turnstiles before it jams. That's the question of concurrent capacity. In your API: how many users hitting simultaneously does it hold before degrading? Recall from lesson 2 that load is created by concurrency, not by the total.

Second: how long does one person take to pass the turnstile? And here comes the subtlety that changes everything. If you measure the average, it might give you "12 seconds" and sound fine. But the average hides the unlucky ones: while most pass in 8 seconds, a few —those who arrived when there was a queue— took 90. The p95 ("the time under which 95% of people pass") captures those unlucky ones: it tells you "95% passed in 40 seconds or less," which means 1 in every 20 people waited more than 40 seconds. That's the real experience of your worst 5%, and it's the one people remember and complain about. That's the question of percentile latency.

Third: at what number of people does the system break? You keep raising the flow —500 at once, 1000, 2000— and observe: at first the queue moves smoothly, until at some point it collapses, the wait spikes from seconds to minutes, and some turnstiles start rejecting people. That point has a name: the breaking point. Knowing it tells you how much reserve capacity you have before disaster. That's the question of the limit.

A well-managed stadium knows its three numbers: how many concurrent people it holds, how long the bad percentile takes to pass, and at what flow it breaks. A well-tested API knows the same three.

Question 1: how many concurrent users does it hold?

This is the capacity question, and its unit is concurrency: how many clients hit at once. In k6 it's modeled with VUs (virtual users) —threads that run your script in a loop, each like an independent user hitting the API—; in the Python generator, with the number of concurrent workers (max_workers). The number that matters isn't "how many requests total" but "how many in flight at the same time."

The way to answer it is to raise concurrency in steps and see how far the system responds well. With the generator, contrasting two levels against the same Reservo API:

What to expect — the higher the concurrency, the more requests compete for resources, so throughput rises but latency does too. Real output (two runs):

$ python3.14 load_generator.py http://127.0.0.1:PORT 200 20
requests ............ 200 (concurrency 20)
throughput .......... 4157.5 req/s
latency avg ......... 4.50 ms
latency p95 ......... 17.21 ms

$ python3.14 load_generator.py http://127.0.0.1:PORT 500 50
requests ............ 500 (concurrency 50)
throughput .......... 4856.2 req/s
latency avg ......... 9.60 ms
latency p95 ......... 29.13 ms

(I trimmed the min/max lines to focus the comparison.) Read the trend: going from 20 to 50 concurrent, throughput rose a bit (from ~4157 to ~4856 req/s) but the p95 almost doubled (from 17 to 29 ms). That's typical: as concurrency rises there comes a moment when you no longer get more throughput —the system is saturated— and the only thing that grows is the wait. Useful capacity is the concurrency level where you still meet your latency target. If your target were "p95 < 20 ms," this lab API would give you comfortable capacity at 20 concurrent and already brush the limit at 50.

Question 2: what's the p95 latency? (and why the average lies)

This is the most important and most misunderstood question. "How fast does your API respond?" has many possible answers —the minimum, the average, the median, the p95, the p99, the maximum— and choosing the wrong one makes you lie to yourself. The most common sin is reporting the average.

The problem with the average is that latency distributions aren't symmetric: they have a long tail to the right. Most requests are fast, but a few —those that landed in a moment of contention, or caught the garbage collector, or waited for a connection— are much slower. The average blends everything into a single number that describes neither the majority (which was faster) nor the unlucky ones (which were much slower). Percentiles, by contrast, don't average: they sort and cut. The p95 is the value that leaves 95% of requests below it and 5% above. Put another way: "1 in every 20 users saw a latency worse than this." That's a promise you can keep; the average isn't.

Look at it with the real numbers. In the 50-concurrent run:

What to expect — the average paints a kind picture; the p95 and the maximum reveal the tail the average hides. Real output:

$ python3.14 load_generator.py http://127.0.0.1:PORT 500 50
latency min ......... 2.74 ms
latency avg ......... 9.60 ms
latency max ......... 46.19 ms
latency p95 ......... 29.13 ms

The average says 9.60 ms. Sounds lightning-fast. But look: the p95 is 29.13 ms —three times the average— and the maximum, 46.19 ms. If you promise your boss "the API responds in 9.6 ms," you're hiding that 1 in every 20 requests took 29 ms or more, and some reached 46. For a user, slowness isn't felt on average; it's felt on their concrete request, and if they got the 95th-percentile one, their experience was 29 ms, not 9.6. Multiply that by a page that makes twenty requests to load, and the probability that at least one falls in the slow tail spikes. That's why Google, in its SRE book, insists: measure and set targets in high percentiles (p95, p99), not in averages.

Compare it with the run without contention (concurrency 1), where the average was representative:

$ python3.14 load_generator.py http://127.0.0.1:PORT 100 1
latency avg ......... 0.33 ms
latency p95 ......... 0.35 ms

Here average (0.33) and p95 (0.35) almost coincide: with no tail, the average doesn't lie. The lesson is that the average lies exactly when it matters most —under load, when there's a queue—, which is exactly when you make decisions. That's why in this guide we look at the p95, and why the module 5 thresholds are written on p(95), not on the average.

Of all the ways to summarize latency, the average is the one that misleads most under load, because a long tail of slow requests drags it... downward, hiding the unlucky users. The p95 —"1 in every 20 saw something worse than this"— is the honest measure. Measure and promise in percentiles, not averages.

Question 3: where's the breaking point?

The third question looks for the limit: the load level beyond which the system stops behaving well —latency spikes non-linearly, or errors start appearing (requests that fail, timeouts, 500 responses)—. That point is called the breaking point, and knowing it tells you how much margin you have above your normal load before disaster.

It's found by raising load steadily until something yields. On a lab API like Reservo, running on localhost with plenty of resources, we won't actually "break it" —that'd take an enormous load— but the beginning of the phenomenon does show: as concurrency rises, the p95 grows faster than the load. That accelerating latency growth is the first signal that you're approaching the limit. When, on top of that, the error rate stops being 0% and starts rising, you crossed the breaking point: the system no longer just goes slow, it starts to reject work.

This is the territory of a specific test type —the stress test—, which we'll see in lesson 4, and of the load profiles with increasing ramps, which are module 4. For now keep the idea: a well-designed load test doesn't just tell you "it holds X"; it tells you "it holds comfortably up to X, starts to suffer at Y, and breaks at Z." That map is what lets you size your infrastructure with data instead of faith.

How these three questions look in the k6 summary (content)

When in the coming modules you run k6 (or read its output as content, since it isn't installed here), you'll see that its summary answers exactly these three questions. As a preview —and labeled as content: this block shows the shape of the k6 summary per its official documentation, not a run of this environment—, the key lines are:

// CONTENT (not run here): shape of the k6 summary, see grafana.com/docs/k6
     http_req_duration...: avg=9.6ms  min=2.7ms  med=8ms  max=46ms  p(90)=22ms  p(95)=29ms
     http_req_failed.....: 0.00%   ✓ 0    ✗ 500
     http_reqs...........: 500     4856/s
     vus.................: 50      min=50  max=50

Read it with the three questions in hand: vus answers the first (how many concurrent), http_req_duration with its p(95) answers the second (percentile latency), and http_req_failed (error rate) plus the p(95) spiking answer the third (whether you crossed the breaking point). Notice that the numbers I put in that content summary are consistent with what the Python generator actually measured (average ~9.6 ms, p95 ~29 ms, ~4856 req/s): so it should be —k6 does the same thing as the generator, industrialized. In module 3 we take apart each of these lines.

Common mistakes

Reporting the average latency as if it described the user experience. What happens: someone puts "average latency: 9.6 ms" on a dashboard and everyone relaxes, while 1 in every 20 users suffers 29 ms or more. Why it happens: the average is the default summary, the one everyone computes without thinking. How to detect it: if your latency metric is an average, you're hiding the tail. How to fix it: report and set targets in p95 (and sometimes p99). The average, if at all, as a secondary datum.

Measuring total volume instead of concurrency. What happens: "I made a million requests, it holds up" —but one at a time—. Why it happens: how much total work gets confused with how much simultaneous work. How to detect it: if you can't say how many concurrent users you used, you didn't measure capacity. How to fix it: define and report the concurrency (VUs in k6, max_workers in the generator); it's the variable that creates the load.

Looking for the breaking point in a single fixed-load run. What happens: someone runs 50 concurrent once and says "it didn't break, it's fine." Why it happens: the breaking point only appears by raising the load; a fixed run doesn't look for it. How to detect it: if you never raised the load until something yielded, you didn't find the limit, you only confirmed that 50 is fine. How to fix it: use an increasing profile (a ramp) or several runs at increasing concurrency and observe where the p95 spikes or the error rate stops being zero (the stress test of lesson 4, the profiles of module 4).

Exercises

Exercise 1 — Translate the question into its metric. For each business question, say which of the three —concurrent capacity, p95 latency, or breaking point— answers it. (a) "How many clients can book at once on a Monday morning without the app going slow?" (b) "In how much time do 95% of my users see their quote?" (c) "If we go viral and traffic multiplies by ten, at what point does the server fall over?"

See solution
  • (a) Concurrent capacity. It asks how many simultaneous users it holds while meeting the latency target.
  • (b) p95 latency. "95% of the users" is, literally, the definition of the 95th percentile.
  • (c) Breaking point. It looks for the load level beyond which the system fails; it's found by raising the traffic until it yields (a stress test).

Exercise 2 — The average that lies. A test reports: average 12 ms, p95 90 ms, maximum 300 ms, over 1000 requests. (a) Roughly how many of those 1000 requests took more than 90 ms? (b) A colleague wants to promise the client "we respond in 12 ms." Is it honest? What figure would you promise and why? (c) What does the distance between the average (12) and the maximum (300) tell you?

See solution
  • (a) The p95 leaves 5% above it: approximately 50 requests (5% of 1000) took more than 90 ms.
  • (b) It's not honest: "12 ms" is the average, and it hides that 1 in every 20 users saw 90 ms or more. I'd promise the p95 (90 ms) —or even round it up— because it's a latency I can keep for 95% of people; the average only describes the lucky ones.
  • (c) A huge distance (12 vs 300) indicates a long tail: there are very slow requests, a minority but extreme, dragging the maximum up. It's a sign of contention, garbage-collection spikes, or occasional waits for a resource. It's worth investigating the tail (p99, not just p95).

Exercise 3 — Design the question. Your boss says: "I want to know if Reservo holds up on Black Friday." Rewrite that vague wish as three measurable questions, one for each of the three we saw, putting concrete numbers you invent as targets.

See solution

A reasonable example (the numbers are targets you set according to the business):

  1. Capacity: "Does Reservo sustain 1000 concurrent users quoting, which is our estimated Black Friday peak?"
  2. p95 latency: "Under those 1000 concurrent, does the p95 of /quote stay under 500 ms?"
  3. Breaking point: "If we raise the load above 1000, beyond how many concurrent does the p95 spike or the error rate exceed 1%?"

What matters is the pattern: each vague question ("does it hold up on Black Friday?") becomes actionable when you split it into capacity + percentile latency + limit, each with a target number. Those target numbers are the ones that in module 5 become thresholds.

Summary and next step

In this lesson you fixed the three questions every load test answers: capacity (how many concurrent users does it hold?), percentile latency (what's the p95?), and breaking point (beyond what load does it break?). You saw them with the stadium analogy and with the generator's real numbers: how the p95 almost doubles going from 20 to 50 concurrent, and how the average (9.60 ms) hides a tail that the p95 (29.13 ms) and the maximum (46.19 ms) do reveal.

Above all, you internalized the antidote to the most expensive mistake: the average latency lies under load, because a long tail of slow requests keeps it optimistic while 1 in every 20 users suffers much more. The p95 —"1 in every 20 saw something worse than this"— is the honest measure, and it's the one the rest of the guide's targets and thresholds are built on.

Before moving on you should be able to: name the three questions and the metric each one answers; explain in your own words why the p95 is more honest than the average; and read a triple (average, p95, maximum) and interpret the gap as the distribution's tail.

What comes next is that each of these questions is answered with a different shape of load. In lesson 4 we see the five test types —smoke, load, stress, spike, soak— and what question each one answers: stress looks for the breaking point, spike tests a sudden peak, soak looks for leaks that only appear with hours of load...

Resources

← Previous2. Functional vs load: two distinct questionsNext →4. The types: smoke, load, stress, spike, soak