Module 8: Project Load Test Reservo

5. The run and the metrics (p95/RPS/error)

Next →

Overview

You already have the scenario (lesson 2), the profile (lesson 3), and the thresholds (lesson 4). When you put them together and run the test, a handful of numbers comes out per stage. This lesson is about reading those numbers: the three instruments every load test produces —latency (p50/p95/p99), throughput (RPS), and error rate— and the story they tell together. We run the complete test for real against /quote_cpu and read its table with judgment: why the p95 climbs with the load while the error stays at zero, what a flat RPS means, and why no single instrument gives the verdict. And we write the equivalent k6 summary as content, to see the same numbers in the k6 run format.

Connection to the module: the metrics are the material that lesson 4's threshold judges and that lesson 6 exports to JSON. It fully reuses module 3: what the p95 is and why the average lies, what throughput/RPS is, what the error rate is, and how the k6 summary is read. Here we don't re-explain how a percentile is computed (that's what statistics.quantiles does); we read the metrics the run produces and draw an honest verdict. It's the lesson that teaches you to interpret the run before lesson 6 turns it into an automatic gate.

The three quadrants of the dashboard

A pilot doesn't fly looking at a single instrument. On the dashboard there are at least three quadrants they read together: the speed (am I going fast enough?), the fuel (do I have enough?), and the altitude (am I where I should be?). None alone tells the story: you can go very fast and run out of fuel; you can have a full tank and be falling. The competent pilot reads them together, because each one watches a different way of failing.

A load test has its three quadrants. The latency (p95) is the speed: does it respond fast? The error rate is the fuel: is it failing? The throughput (RPS) is the altitude: how much work does it sustain? And as in the cockpit, none alone gives the verdict. A system with a perfect p95 but 10% error is broken. One with 0% error but a p95 of 2 seconds is unusable. One with good p95 and 0% error but that only holds 5 RPS is useless if you expect 500. Reading the run is reading the three quadrants at once and understanding what way of failing each one watches.

The complete run (executed)

We run the whole test against /quote_cpu —the quote→book scenario, through the smoke→load→stress stages— and read the metrics table. Real output against Reservo:

$ python3.14 loadtest.py http://127.0.0.1:PORT /quote_cpu results.json
LOAD TEST — quote->book scenario against /quote_cpu
profile: smoke(5) -> load(20) -> stress(80) VUs
--------------------------------------------------------------------------
stage     VUs    reqs      RPS      p50      p95      p99   error   checks
--------------------------------------------------------------------------
smoke       5    2336    582.2     8.95    14.13    16.91   0.00%  100.00%
load       20    2920    579.0    34.84    57.23    62.35   0.00%  100.00%
stress     80    3676    587.2   135.48   237.68   249.82   0.00%  100.00%
--------------------------------------------------------------------------

Read it quadrant by quadrant, starting —always— with the error rate (M3: the error first, because a low latency of a failing system is worth nothing):

  • Error rate: 0.00% in the three stages. The system doesn't fail: it responds to all requests. This quadrant is clean. Careful: that does not mean the test passes —the latency can be broken even if the error is zero—. It means the degradation, if any, isn't of availability.
  • Latency (p50/p95/p99). Here's the story. The p50 (the median, the typical user) goes from 8.95 ms to 135.48 ms; the p95 (the tail user) from 14.13 ms to 237.68 ms; the p99 (the worst 1%) from 16.91 ms to 249.82 ms. All three climb with the load, and the stress p95 (237.68 ms) is what matters: it crossed the SLO's terrain. Notice that the p50 and the p95 are close in each stage (135 vs 237 in stress): the distribution doesn't have a very long tail —the CPU work affects all the requests equally, not just a few—. It's a different signature from an endpoint with sporadic timeouts, where the p99 would spike much more than the p50.
  • Throughput (RPS): ~580 req/s, flat. Here's the subtle signal. The RPS barely changes between smoke (582), load (579), and stress (587), even though the VUs multiplied by 16 (from 5 to 80). Why? Because /quote_cpu is CPU-saturated: the server processes approximately the same work per second no matter what, so adding more VUs doesn't increase the throughput —it only lengthens the waiting queue, and that queue is exactly what spikes the p95—. A flat RPS with rising p95 is the signature of saturation: the system reached its ceiling and the extra users just wait.

The combined reading: the system is available (0% error) and is correct (100% checks), but under the peak it gets slow (p95 = 237.68 ms) because it saturated its throughput (~580 RPS is its ceiling with this pricing engine). No single quadrant says it; the three together, yes. That's the interpretation lesson 4's threshold will turn into a FAIL.

The aggregate summary and the k6 one (content)

The threshold doesn't judge a stage: it judges the whole run. These are the aggregate metrics of the three stages together (what the gate evaluates, lesson 4):

aggregate (whole run):  8932 requests
  RPS      583.4 req/s
  p50       42.00 ms
  p95      221.01 ms   <- what the threshold p(95)<200 judges -> FAIL
  p99      244.74 ms
  error     0.00%
  checks  100.00%

The aggregate p95 (221.01 ms) is a bit lower than the peak's (237.68 ms): the fast requests from smoke and load make it cheaper when mixed in (M4). But it still crosses the 200 ms, so the threshold fails. This is how the k6 summary for this test would look —labeled content, faithful to the official k6 run format, not run here—:

     # CONTENT (this is how `k6 run` looks; k6 is not installed)

     scenarios: (100.00%) 1 scenario, 80 max VUs, 10m30s max duration
              * default: Up to 80 looping VUs for 10m30s over 7 stages

     ✓ quote status is 200
     ✓ quote price is correct
     ✓ book status is 200
     ✓ book is confirmed
     ✓ book has booking_id

     ✗ http_req_duration..............: p(95)=221.01ms  (threshold: p(95)<200)
     ✓ http_req_failed................: rate=0.00%      (threshold: rate<0.01)
     ✓ checks.........................: rate=100.00%    (threshold: rate>0.99)

     checks.........................: 100.00% ✓ 22330     ✗ 0
     http_req_duration..............: avg=45ms  min=4ms med=42ms max=280ms p(90)=180ms p(95)=221.01ms
     http_req_failed................: 0.00%   ✓ 0         ✗ 8932
     http_reqs......................: 8932    583.4/s
     iterations.....................: 4466    291.7/s
     vus............................: 80      min=5       max=80
     vus_max........................: 80      min=80      max=80

Map your measured metrics to k6's lines: your p95 = 221.01 is http_req_duration ... p(95)=221.01ms; your error 0.00% is http_req_failed 0.00%; your RPS 583.4 is the rate of http_reqs 583.4/s; your 100% checks are checks 100.00%. The red ✗ next to http_req_duration is the broken threshold —the same FAIL from your table—, and it's what makes k6 run exit with code 99. Note that iterations (4466) is half of http_reqs (8932): each scenario iteration makes two requests (quote and book), as we saw in lesson 2.

Common mistakes

Reading the latency without reading the error first. What happens: a low p95 is celebrated without noticing that 8% of the requests failed. Why it happens: latency is the flashy metric. How to detect it: if your verdict starts with the p95 and not with the error rate, you're reading in the wrong order. How to fix it: read the error rate first —a failing system doesn't have a "good" latency, it has a latency of the requests it did respond to, which is another thing (M3)—. In this run the error was 0%, so the latency is representative; if it had been 8%, the p95 would lie.

Interpreting the flat RPS as a good sign. What happens: it's seen that the RPS stays at ~580 and it's concluded "the throughput is stable, all good." Why it happens: "stable" sounds positive. How to detect it: if the RPS doesn't rise when multiplying the VUs and the p95 climbs, it isn't stability, it's saturation. How to fix it: read the RPS and the p95 together —an RPS that stalls while the p95 grows means the system reached its throughput ceiling and the extra VUs just queue—. That ceiling is a key piece of data: it's how much work Reservo holds with this pricing engine.

Reporting the peak's p95 as the one the threshold evaluates. What happens: "the p95 is 237.68 ms" is said (the stress one) when the gate reports 221.01 (the aggregate). Why it happens: a stage's number is confused with the whole run's. How to detect it: if your "official" p95 doesn't match the gate's, one is a stage's and the other the aggregate. How to fix it: use the per-stage p95 to read the shape (where it bends) and the aggregate for the verdict (what the threshold judges). Both are real; the capstone reports both on purpose (M4).

Exercises

Exercise 1 — The reading order. A run reports: p95 = 45 ms, RPS = 1200, error rate = 12%. (a) In what order do you read these three numbers? (b) What's your verdict and why doesn't the 45 ms p95 save it?

See solution
  • (a) First the error rate (12%), then the checks (if any), and last the latency. Availability rules: there's no point celebrating the speed of a system that fails one in every eight requests.
  • (b) Verdict: fails (a 12% error breaks any reasonable availability SLO, and probably the rate<0.01 threshold). The 45 ms p95 doesn't save it because it's the latency of the requests that did respond —the 88%—; the 12% that failed don't appear in that number. A pretty p95 of a failing system is a mirage: it measures only the lucky users (M3).

Exercise 2 — Flat RPS, what's happening? Between load (20 VUs) and stress (80 VUs), the RPS went from 579 to 587 (almost the same) but the p95 went from 57 to 238 ms. (a) Why didn't the RPS rise if there are four times more VUs? (b) Where did the time of those extra VUs go?

See solution
  • (a) Because /quote_cpu is CPU-saturated and the GIL serializes its work: the server can only process ~580 requests per second, no matter how many clients ask for it. More VUs don't give it more compute capacity; the throughput already hit its ceiling.
  • (b) To the waiting queue. The 60 extra VUs (from 20 to 80) don't get the server to work faster; their requests line up waiting their turn of CPU. That time in line is exactly what spikes the p95 (from 57 to 238 ms). The total work per second is the same; what grows is how long each request waits —and the wait is latency—.

Exercise 3 — Write the verdict. With this lesson's run (aggregate p95 221.01 ms, error 0%, checks 100%) and the SLO p(95)<200, rate<0.01, checks>0.99, write in two or three sentences the honest verdict you'd give the team, reading the three instruments.

See solution

An example of a well-done verdict:

The test fails the latency SLO: the whole run's p95 was 221.01 ms, above the 200 ms threshold, and under the stress stage it reached 237.68 ms. The cause isn't unavailability —the error rate was 0% and the checks 100%, so the system is available and responds correctly— but saturation: /quote_cpu tops its throughput at ~580 RPS, and when pushing the load to 80 VUs the requests queue and the p95 spikes. The system is correct and available, but not fast enough under the expected peak. The gate must block the deploy until the pricing engine's CPU bottleneck is investigated.

The key: cite the three instruments, explain that the problem is latency (not error) and name the cause (throughput saturation), not just the symptom.

Summary and next step

In this lesson you learned to read the run: the three quadrants of the dashboard —latency (p50/p95/p99), throughput (RPS), and error rate— read together, because each one watches a different way of failing. You ran the complete test for real against /quote_cpu and interpreted it: the error at 0% (available), the checks at 100% (correct), but the p95 climbing with the load to 237.68 ms at the peak (slow) because the flat RPS (~580, saturated) reveals that the system hit its throughput ceiling. You saw the aggregate summary (p95 = 221.01 ms, what the threshold judges) and the equivalent k6 summary as content, with the red ✗ of the broken threshold. The underlying lesson: no single instrument gives the verdict; the three together, yes.

You fully reused module 3 (percentiles, throughput, error rate, reading the summary). Before moving on you should be able to: read a metrics table in the correct order (error first); explain what a flat RPS with rising p95 means; and distinguish the per-stage p95 from the aggregate. What comes next, in lesson 6, is to turn this reading into an automatic gate: evaluate the thresholds, exit with an exit code, and export the metrics to a results.json —the green and the red, executed, with their exit codes—.

Resources

← Previous4. The thresholds tied to the SLONext →6. The threshold gate and exporting to JSON