Module 3: Metrics Latency Throughput Errors

4. Throughput/RPS and its relationship with VUs

Next →

Overview

The dashboard's second instrument is throughput: how many requests per second your system processes, the famous RPS (requests per second). While latency measures the experience of one request, throughput measures the flow of the entire system: how much total work it dispatches per unit of time. It's the capacity metric —"how many users can this API serve?"— and the one almost everyone thinks they understand until they measure it, because its relationship with virtual users is subtler than it looks. This lesson takes it apart: what RPS is, how k6 reports it (http_reqs, iterations), and the two counterintuitive truths only seen by measuring.

The first truth: raising VUs doesn't raise throughput indefinitely. There's a natural intuition —"if I add double the virtual users, I process double the requests per second"— and it's false past a certain point. When the system saturates (exhausts its scarcest resource: CPU, threads, connections), adding more VUs no longer gets more RPS out of it; all you achieve is lengthening the queue, that is, raising the latency. You'll see it with a real concurrency sweep over Reservo. The second truth: think time —the pause a real user makes between actions— is what turns "N virtual users" into "a realistic arrival rate." Without think time, a few VUs hammer the API at maximum speed and are nothing like human users; with think time, N VUs produce a predictable RPS you can compute with Little's law. You'll see it measured too.

Connection to the module: latency (lessons 2-3) and throughput (this one) are the two faces of load —time per request vs requests per time—, and they're tied: when the system saturates, throughput flattens and latency spikes, and seeing it in a single table is understanding load. Here we reuse the VUs from module 2. Modeling how the load varies over time (ramps, spikes) is module 4 —the next one—: here the concurrency is fixed in each run and we observe the throughput it produces.

The supermarket checkout

Think of a supermarket with checkout lanes. The throughput is how many customers get paid and out per minute —the store's flow—. The latency is how long each specific customer waits in line until they're checked out. They're different things: the store can have very high throughput (hundreds of customers per minute) while a specific customer waits twenty minutes, if the lines are long.

Now, what happens if you want more throughput and open more lanes? At first it works wonderfully: with one lane you dispatch 6 customers per minute; with two, 12; with three, 18. Throughput rises almost in proportion to the lanes. But there comes a moment when you open the tenth lane and... throughput no longer rises. Why? Because the bottleneck stopped being the lanes: now it's the single aisle everyone enters through, or the central payment system that saturated, or simply that there aren't that many customers waiting. You've reached saturation. From there, opening more lanes (putting more idle cashiers with no customers) doesn't dispatch more people per minute; you just waste cashiers. And if on top of that you force in more customers than the store can process, all you achieve is that the lines get longer —the latency rises— without more people getting out per minute.

A VU is exactly a customer in the store who, as soon as they're checked out, goes back to the end of the line to shop again. Without think time, that customer shops at inhuman speed, nonstop. With think time —a pause of, say, five seconds between one purchase and the next, while they "look at the products"— the customer resembles a real one, and the number of customers per minute they generate drops to something predictable. Think time is what makes "20 virtual users" mean realistic traffic and not a machine gun.

Throughput (RPS) is the system's flow: requests dispatched per second. It rises with the VUs only until saturation; past that point, more VUs only lengthen the queue (raise the latency), not the throughput. Think time turns N VUs into a realistic arrival rate.

RPS in k6: http_reqs and iterations

In the k6 summary, throughput appears in two sibling metrics:

k6 metricWhat it countsHow it's reported
http_reqsTotal number of HTTP requests k6 generated.A total and a rate: 2000 1517.6/s.
iterationsHow many times the VUs ran the default function (the complete script).The same: total and rate /s.

The difference matters. http_reqs counts requests; iterations counts script executions. If your default function makes one request (an http.post to /quote), the two numbers coincide. But if it makes three requests per iteration (a /rooms → /quote → /book flow, like module 6's), then http_reqs will be three times iterations. The rate usually reported as "RPS" is http_reqs's (requests per second); iterations's is useful when you think in users completing a flow per second. In our Python generator, since each task makes a single request, the RPS is simply total_requests / wall_clock_time.

Worked example 1: the concurrency sweep (where Reservo saturates)

Let's see the first truth measured: raising VUs raises RPS only until the system saturates. We run the same /quote, with 2000 requests, raising the concurrency: 1, 5, 15, 30, 60 clients. For each level we measure the RPS and the latency p95, side by side.

import json, statistics, time, urllib.request
from concurrent.futures import ThreadPoolExecutor

def sweep(port, concurrency, total=2000):
    url = f"http://127.0.0.1:{port}/quote"
    payload = json.dumps({"room": "Focus", "tier": "basic", "hours": 3}).encode()

    def one(_):
        s = time.perf_counter()
        req = urllib.request.Request(url, data=payload,
                                     headers={"Content-Type": "application/json"})
        with urllib.request.urlopen(req, timeout=10) as r:
            r.read()
        return (time.perf_counter() - s) * 1000

    t0 = time.perf_counter()
    with ThreadPoolExecutor(max_workers=concurrency) as pool:
        latencies = sorted(pool.map(one, range(total)))
    wall = time.perf_counter() - t0
    rps = total / wall
    p95 = statistics.quantiles(latencies, n=100, method="inclusive")[94]
    print(f"concurrency {concurrency:3d}  ->  RPS {rps:8.1f}   "
          f"avg {statistics.fmean(latencies):6.2f} ms   p95 {p95:7.2f} ms")

for c in (1, 5, 15, 30, 60):
    sweep(PORT, c)

What to expect. The RPS rises from 1 to 5 clients (the API had idle capacity), but soon flattens around a ceiling —Reservo's maximum throughput on this machine— while the latency (avg and p95) keeps rising. This is the real output:

concurrency   1  ->  RPS   4212.6   avg   0.23 ms   p95    0.29 ms
concurrency   5  ->  RPS   5770.5   avg   0.86 ms   p95    1.25 ms
concurrency  15  ->  RPS   5465.7   avg   2.73 ms   p95    3.98 ms
concurrency  30  ->  RPS   5234.2   avg   5.69 ms   p95    9.08 ms
concurrency  60  ->  RPS   5134.5   avg  11.50 ms   p95   19.17 ms

Read the table as a single story. From 1 to 5 clients, the RPS rises (4212 → 5770): there was free capacity. From there, the RPS stops rising —it stays stuck between 5100 and 5800, which is Reservo's ceiling here— no matter how many more clients you add. And while the throughput flattens, look at the latency: the p95 goes from 0.29 ms (1 client) to 19.17 ms (60 clients), multiplying by ~66. That's saturation in the flesh: past the point where the system gives everything it can, each extra VU doesn't buy throughput; it buys latency. The extra clients don't get out of the checkout faster; they just make the line longer.

This has a huge practical consequence. If someone tells you "raise the VUs until you hit 10,000 RPS" and your system saturates at 5,500, no number of VUs will achieve it: all you'll get by raising VUs is a sky-high latency and (soon) errors. The throughput ceiling is a property of the system, not the test. Finding that ceiling —the point where the RPS flattens and the latency takes off— is, in fact, the goal of a stress test (which you saw named in module 1 and will model in module 4).

Worked example 2: think time and Little's law

Now the second truth: think time turns N VUs into a predictable arrival rate. Here each VU makes its request and then sleeps for a fixed time (the think time) before the next, imitating a human who reads the screen between clicks. We fix 20 VUs for 3 seconds and vary the think time: 0, 0.05, 0.1, and 0.2 seconds.

The theory we're going to confirm is Little's law applied to load: the RPS produced by N VUs is, approximately,

RPS ≈ VUs / (service_latency + think_time)

The intuition: each VU completes a cycle every (latency + think_time) seconds, so it makes 1 / (latency + think_time) requests per second, and with N VUs you multiply by N. If Reservo's service latency is ~5 ms (0.005 s), the model predicts the RPS for each think time.

import json, time, threading, urllib.request

def think_run(port, think, vus=20, duration=3.0):
    url = f"http://127.0.0.1:{port}/quote"
    payload = json.dumps({"room": "Focus", "tier": "basic", "hours": 3}).encode()
    count = [0]; lock = threading.Lock(); stop_at = time.perf_counter() + duration

    def vu():
        while time.perf_counter() < stop_at:
            req = urllib.request.Request(url, data=payload,
                                         headers={"Content-Type": "application/json"})
            with urllib.request.urlopen(req, timeout=10) as r:
                r.read()
            with lock:
                count[0] += 1
            if think:
                time.sleep(think)   # the think time: the VU "thinks" between requests

    t0 = time.perf_counter()
    threads = [threading.Thread(target=vu) for _ in range(vus)]
    for t in threads: t.start()
    for t in threads: t.join()
    wall = time.perf_counter() - t0
    rps = count[0] / wall
    model = vus / (0.005 + think)   # Little's law, with service latency ~5 ms
    print(f"think {think:4.2f}s  ->  RPS measured {rps:8.1f}   (model ~{model:7.1f})")

for think in (0.0, 0.05, 0.10, 0.20):
    think_run(PORT, think)

What to expect. Without think time, 20 VUs hammer the API at maximum speed and the RPS is enormous (thousands). As soon as you add think time, the RPS drops to something modest and predictable by Little's law: the larger the think time, the lower the RPS. This is the real output:

think 0.00s  ->  RPS measured   5492.8   (model ~ 4000.0)
think 0.05s  ->  RPS measured    342.2   (model ~  363.6)
think 0.10s  ->  RPS measured    182.0   (model ~  190.5)
think 0.20s  ->  RPS measured     94.9   (model ~   97.6)

Look at the three rows with think time: the model predicts the measured RPS almost perfectly (342 vs 364, 182 vs 190, 95 vs 98). The think time dominates the denominator —0.05 s of think against ~0.005 s of latency—, so each VU makes ~1 request every 0.055 s, and 20 VUs give ~360 RPS. That's the huge usefulness of think time: it lets you reason about how many VUs you need to simulate a given traffic. Want 1000 RPS with users who think 1 second between actions? You need ~1000 VUs (1000 × (0.005 + 1) ≈ 1005). Without think time, that calculation doesn't exist: the VUs run at the machine's speed.

And notice the first row, the one with think time 0: the measured RPS (5492) is higher than the model (4000). Why does the model fail there? Because without think time the 20 VUs saturate the API, and at full saturation the real service latency is no longer the 0.005 s we assumed —it's lower, because the system is maxed out and serves in batches—. It's a reminder of the first truth: without think time, the load enters a saturation regime and the throughput is dictated by the system's ceiling, not a clean formula. Little's model describes the realistic regime (with human pauses) well, not the machine-gun one.

How throughput and latency are read together

The lesson running through both truths is that throughput isn't read alone. A high RPS is good only if the latency under that RPS is acceptable. In the sweep, Reservo gave ~5200 RPS both with 30 clients (p95 = 9 ms) and with 60 (p95 = 19 ms): the same throughput, but double the latency. If someone reports "we hold 5200 RPS" without saying the p95, they hide half the story —at 60 clients that throughput comes with a latency that may already be unacceptable—. That's why, in any load report, the RPS always goes accompanied by the latency (percentiles) at which it was reached, and by the error rate (the next lesson). The three instruments, together.

Common mistakes

Believing more VUs always give more RPS. What happens: the system saturates at 5,500 RPS and someone raises from 60 to 600 VUs expecting 10× the throughput; they get the same RPS with an atrocious latency and, soon, errors. Why it happens: the load you generate (VUs) gets confused with the one the system processes (RPS). How to detect it: if raising VUs flattens the RPS but spikes the latency, you saturated. How to fix it: understand that the throughput ceiling is the system's; past that point, more VUs only buy queue. To raise the ceiling you have to optimize the system (another guide), not the test.

Loading without think time and believing it's realistic. What happens: someone launches 50 VUs without sleep and concludes "my API holds 50 users," when they actually simulated 50 machine guns generating traffic no human would produce. Why it happens: "VU" gets equated with "user," but a real user thinks between clicks. How to detect it: if your VUs have no think time, your RPS per VU is unrealistically high and your "users supported" number is pessimistic and meaningless. How to fix it: add realistic think time (sleep) so N VUs produce a human arrival rate, and use Little's law to size how many VUs you need.

Reporting the RPS without the latency at which it was reached. What happens: "we hold 5200 RPS" sounds like a success headline, but at that load the p95 was 800 ms. Why it happens: throughput is the flashy capacity metric and gets reported alone. How to detect it: if your RPS number doesn't come with a p95 next to it, context is missing. How to fix it: always report "X RPS with p95 of Y ms and Z% error." An RPS without its latency and its error rate doesn't say whether the system was healthy or dying.

Exercises

Exercise 1 — http_reqs vs iterations. A k6 script has a default function that makes a GET /rooms, then a POST /quote, and then a POST /book. The run reports iterations: 2000. (a) What's http_reqs? (b) If the run lasted 4 seconds, what's the RPS (requests/s) and what's the iterations/s rate? (c) Which of the two would you use to say "users completing the flow per second"?

See solution
  • (a) Each iteration makes 3 requests (/rooms, /quote, /book), so http_reqs = 2000 × 3 = 6000.
  • (b) RPS (requests/s) = 6000 / 4 = 1500 req/s. Iterations rate = 2000 / 4 = 500 iterations/s.
  • (c) The iterations one (500/s): each iteration is a user completing the entire quote→book flow. http_reqs counts individual requests, not complete flows.

Exercise 2 — Did it saturate? A sweep gives these results. Concurrency 10 → 3000 RPS, p95 12 ms. Concurrency 40 → 4900 RPS, p95 30 ms. Concurrency 160 → 5000 RPS, p95 220 ms. (a) At what point does the system saturate, approximately? (b) What did you buy going from 40 to 160 VUs? (c) Does it make sense to raise to 640 VUs to "reach 8000 RPS"?

See solution
  • (a) Between 40 and 160 VUs: the RPS goes from 4900 to 5000 (practically the same) while the concurrency quadruples. The throughput ceiling is around 5000 RPS.
  • (b) No throughput (4900 → 5000 is noise) and a lot of latency: the p95 multiplied by ~7 (30 → 220 ms). Past the saturation point, the extra VUs only lengthen the queue.
  • (c) No. The system saturates at ~5000 RPS; no number of VUs will take it to 8000. Raising to 640 VUs would only spike the latency (and soon the errors). For 8000 RPS you have to optimize the system, not add load.

Exercise 3 — Size with Little's law. You want to simulate a traffic where users think 2 seconds between actions, and your API's service latency is ~10 ms (0.01 s). (a) How many requests per second does one VU generate? (b) How many VUs do you need for ~300 RPS? (c) If you removed the think time, would the number of VUs from (b) give 300 RPS?

See solution
  • (a) A VU completes a cycle every 0.01 + 2 = 2.01 s, so it generates 1 / 2.01 ≈ 0.498 requests/s (almost half a request per second).
  • (b) VUs ≈ RPS × (latency + think) = 300 × 2.01 ≈ 603 VUs. (Or, equivalently, 300 / 0.498 ≈ 603.) You need ~600 VUs.
  • (c) No, it would give a lot more (or saturate). Without think time, each VU would make ~1 / 0.01 = 100 requests/s, and 600 VUs would ask for ~60,000 RPS —well above what the system holds—: it would saturate and the latency would explode. Think time is what makes 600 VUs mean 300 realistic RPS.

Summary and next step

In this lesson you opened the second instrument: throughput (RPS), the system's flow. You learned that in k6 it lives in http_reqs (total and per-second requests) and in iterations (script executions), and that they coincide only if each iteration makes one request. And you measured the two counterintuitive truths. The first: raising VUs raises RPS only until saturation; in the sweep, Reservo flattened at ~5200 RPS while the p95 went from 0.29 ms to 19 ms —past the ceiling, each VU buys latency, not throughput—. The second: think time turns N VUs into a realistic rate, predictable with Little's law (RPS ≈ VUs / (latency + think)), which fit the data almost perfectly (182 measured vs 190 from the model).

The underlying lesson is that throughput isn't read alone: a high RPS is worth it only if the latency and the error rate at that RPS are acceptable. Before moving on you should be able to: distinguish http_reqs from iterations; explain what saturation is and why more VUs don't overcome it; and use Little's law to size how many VUs you need for a target RPS with a given think time.

What comes next is the third and last instrument. In lesson 5 comes the error rate (http_req_failed): what counts as a failure, and why a very low latency with 10% errors is a resounding failure —you'll see it measured with the declared flaky endpoint, which responds very fast but returns 500 one in every ten times—.

Resources

← Previous3. p50/p90/p95/p99 with `statistics.quantiles`Next →5. The error rate and why it matters