5. The error rate and why it matters

Overview

The dashboard's third instrument is the easiest to define and the easiest to ignore: the error rate, the percentage of requests that failed. In k6 it's called http_req_failed, and it's a number between 0% and 100%. Its definition holds no mystery; its importance does, because it's the metric that can turn a test that "looks gorgeous" by latency and throughput into a failure. This lesson installs a rule that seems obvious once stated and that nevertheless gets violated all the time: a fast response that returns an error served no one. A very low latency with 10% errors isn't a success with an asterisk; it's a failure, full stop.

The reason is that the three metrics aren't independent: they're three conditions that are met all or the system fails. Latency measures how fast you responded; throughput, how much volume you moved; the error rate, how many of those responses were really the correct answer and not an error. If 10% of your requests returned 500, then your p95 of "10 ms" describes the speed at which you failed one in every ten times —very fast, yes, but a failure—. That's why the error rate is read first: it's the filter that validates (or invalidates) everything else. You'll see it measured harshly using /quote_flaky, the endpoint declared in lesson 1 that responds very fast but returns 500 on ~10% of requests.

Connection to the module: with this lesson you have the three instruments complete —latency (2-3), throughput (4), errors (5)— and you can already read an entire load test, which is exactly what lesson 6 does over the k6 summary. The error rate is also the basis of the second type of threshold you'll see in module 5 (http_req_failed: ['rate<0.01'], "fail the test if more than 1% of requests fail"): here you learn what that rate is; making it gate a deploy is module 5.

The very fast waiter who brings the wrong dish

Imagine a restaurant that prides itself on speed. The waiter is a lightning bolt: takes your order and in ninety seconds you already have a plate on the table. The latency is spectacular. And the restaurant moves a ton of tables per hour —very high throughput—. But there's one detail: one in every ten times, the plate that arrives isn't the one you ordered. You ordered risotto and they bring you, swiftly, a salad you didn't want. Would you say that waiter is "excellent because they're fast"? No. A wrong dish served in ninety seconds isn't better than a wrong dish served in ten minutes: in both cases you didn't eat what you wanted. The speed of a failure doesn't redeem it; it just delivers it faster.

That's exactly an API's error rate. Each request is a diner ordering a dish (a quote, a booking). A successful request is the right plate on the table: status 200 with the correct price. A failed request is the wrong plate or the kitchen saying "I can't": a 500, a timeout, a connection that drops. If your error rate is 10%, one in every ten diners left without their plate —no matter how fast they were brought the error—. And here's the cruel part of the metrics: latency and throughput only count the plates that arrived, correct or not. A 500 that comes out in 5 ms lowers your average latency (it's very fast to generate an error!) and raises your throughput (so many requests per second!). The speed and volume metrics, read alone, reward fast failures. Only the error rate puts them in their place.

The error rate is the filter that validates the other two metrics. A request that fails fast is still a request that failed. A low latency with a high error rate isn't an imperfect success: it's a failure. Read the error rate first; only if it's acceptable do latency and throughput mean anything.

http_req_failed: what counts as an error in k6

In k6, the metric is http_req_failed, and it's a rate: the proportion of failed requests, reported as a percentage. And what counts as "failed"? By default, k6 uses a response callback that considers successful any response with an HTTP status code in the 2xx-3xx range (status < 400), and failed everything else:

Result	Success or failure by default in k6?
200 OK, 201 Created, 301, 304	Success (status < 400).
400 Bad Request, 401, 403, 404	Failure (client error).
500 Internal Server Error, 502, 503	Failure (server error).
Timeout, connection refused, DNS down	Failure (the request didn't even get a response).

Two important nuances. First: by default, k6 looks at only the status code, not the content of the response. A 200 response with the wrong price (incorrect price_cents) counts as a success for http_req_failed, because the status was 200 —verifying that the content is correct under load is a check()'s job, which is module 6—. The error rate captures transport and availability failures (did the server respond, and with a healthy code?), not logic failures (did it respond with the right thing?). Second: in our Python generator we replicate exactly that rule —we mark ok = (resp.status == 200) and count the exceptions (timeouts, dropped connections) as failures—, so our error rate is the direct equivalent of http_req_failed.

Worked example: excellent latency, and still a failure

Let's measure the lesson harshly. We run the generator against /quote_flaky —the declared endpoint that responds fast but returns 500 on ~10% of requests— with 2000 requests and 30 clients, and look at the three metrics together. The generator times each latency, marks whether the status was 200, and at the end reports the RPS, the error rate, and the latency percentiles.

import json, statistics, time, urllib.request
from concurrent.futures import ThreadPoolExecutor

def one(url, payload):
    """Returns (latency_ms, ok). ok = True only if the status was 200."""
    start = time.perf_counter()
    try:
        req = urllib.request.Request(url, data=payload,
                                     headers={"Content-Type": "application/json"})
        with urllib.request.urlopen(req, timeout=10) as resp:
            resp.read()
            ok = (resp.status == 200)
    except Exception:
        ok = False            # timeout, dropped connection, 500... all count as failure
    return (time.perf_counter() - start) * 1000, ok

def load(port, path, total=2000, concurrency=30):
    url = f"http://127.0.0.1:{port}{path}"
    payload = json.dumps({"room": "Focus", "tier": "basic", "hours": 3}).encode()
    latencies, errors = [], 0
    t0 = time.perf_counter()
    with ThreadPoolExecutor(max_workers=concurrency) as pool:
        for latency, ok in pool.map(lambda _: one(url, payload), range(total)):
            latencies.append(latency)
            if not ok:
                errors += 1
    wall = time.perf_counter() - t0
    latencies.sort()
    q = lambda p: statistics.quantiles(latencies, n=100, method="inclusive")[p - 1]
    print(f"endpoint       {path}")
    print(f"RPS            {total / wall:8.1f}")
    print(f"errors         {errors} ({errors / total * 100:.2f}%)")
    print(f"p50            {q(50):8.2f} ms")
    print(f"p95            {q(95):8.2f} ms")

load(PORT, "/quote_flaky")

What to expect. The latency will come out excellent —the endpoint responds fast, both the 200 and the 500— and yet the error rate will be ~10%. This is the real output:

endpoint       /quote_flaky
RPS            4761.1
errors         200 (10.00%)
p50               5.45 ms
p95              10.06 ms

Look at what you have in front of you. A very high RPS (4761), a p50 of 5.45 ms, a p95 of 10.06 ms: by latency and throughput, this run is indistinguishable from a perfectly healthy API. If you only looked at those two metrics, you'd sign off the deploy. But the error rate says 10.00%: 200 of the 2000 requests returned 500. One in every ten users who tried to quote received an error —very fast, granted—. Is this API "fast"? It's fast failing. The p95 of 10 ms describes the speed at which, one in every ten times, you didn't deliver the price. This is the lesson's scenario made a number: gorgeous latency, and a failure.

For the contrast to be total, compare it with the normal /quote under similar load:

endpoint       /quote
RPS            5069.7
errors         0 (0.00%)
p50               5.44 ms
p95               9.43 ms

The latencies are almost identical (p50 5.44 vs 5.45, p95 9.43 vs 10.06) and so is the throughput (5069 vs 4761). By the first two instruments, the two runs are twins. The only thing that separates them is the third instrument: /quote has 0% error and /quote_flaky has 10%. And that difference is everything: the first is a ready API, the second is a broken API that also happens to be fast. Reading the error rate is what lets you tell them apart; without it, you'd have declared them equal.

How to read the error rate alongside the other two

The operative rule is a reading order, and it's worth fixing:

1. First, the error rate. If it's unacceptable (above your threshold —typically something like 1%—), the test already failed, and the latency and throughput are irrelevant: it doesn't matter how fast and in what volume you delivered responses if a large fraction of them were errors. Note the failure and go diagnose why it fails.
2. Only if the error rate is acceptable, read the latency (percentiles) and the throughput (RPS). Now they do mean something, because they describe responses that actually served the user.

This order avoids the module's most common trap: celebrating a low latency that's actually the speed of a failing system. And it explains why in module 5 the error-rate threshold (rate<0.01) is usually the first one written: it's the condition for the test to make sense at all.

Common mistakes

Looking at the latency before the error rate. What happens: the report opens with "p95 of 10 ms, excellent" and the error rate (10%) appears three lines below, with the "fast" verdict already in place. Why it happens: latency is the flashy metric; the error rate is read out of the corner of the eye, if at all. How to detect it: if your conclusion about speed formed before looking at http_req_failed, you formed it on data possibly contaminated by fast failures. How to fix it: reverse the order. Read the error rate first; only if it passes does the latency mean anything.

Believing a fast 500 is "half good" because at least it was fast. What happens: "well, it fails, but at least it fails fast, it doesn't leave the user waiting." Why it happens: the latency intuition (fast = good) gets applied to a result that's a failure. How to detect it: if you're looking for the silver lining of an error, you already got the frame wrong. How to fix it: for the user who wanted their quote, a 500 in 5 ms and a 500 in 5 seconds are the same failure —they didn't get the price—. The speed of an error is no metric consolation.

Confusing the error rate (http_req_failed) with content correctness. What happens: the error rate is 0% and someone concludes "everything came out correct under load," without noticing the API returned 200 with the wrong price_cents. Why it happens: http_req_failed looks at the status code, not the content; a 200 with bad data counts as a success for this metric. How to detect it: if you care that the price is 7500 and not just that the status is 200, http_req_failed doesn't tell you. How to fix it: use check() (module 6) to verify the content under load. The error rate validates the transport; the check validates the logic.

Exercises

Exercise 1 — Success or failure by default? For each response, say whether k6 counts it as a success or a failure in http_req_failed with the default rule. (a) 200 OK with {"price_cents": 9999} (the correct price was 7500). (b) 503 Service Unavailable. (c) A timeout: the server never responded. (d) 404 Not Found.

See solution

• (a) Success. The status is 200 (< 400), so http_req_failed counts it as successful —even though the price is wrong—. The error rate looks at the code, not the content; that wrong price would be caught by a check() (module 6), not this metric.
• (b) Failure. 503 is ≥ 400 (server error): it counts as failed.
• (c) Failure. A timeout got no response; k6 (and our generator) count it as a failure. In the generator it falls into the except and marks ok = False.
• (d) Failure. 404 is ≥ 400 (client error): it counts as failed by default.

Exercise 2 — The correct verdict. You have two runs of the same endpoint. Run X: p95 = 10 ms, RPS = 4761, error = 10%. Run Y: p95 = 40 ms, RPS = 3000, error = 0%. (a) Which is ready for production, if your error threshold is 1%? (b) Why is X, being faster and with more throughput, the worse one? (c) What's misleading about X's p95 and RPS?

See solution

• (a) Y. It has 0% error (under the 1% threshold) and a perfectly acceptable latency (p95 40 ms). X has 10% error, well above 1%: it already failed, even if it's faster.
• (b) Because X's speed and volume describe, in good part, fast failures. A p95 of 10 ms with 10% error means 1 in every 10 users received a very fast 500. Y is a bit slower but works; X is fast but broken.
• (c) That both are inflated by the errors: generating a 500 is faster than computing and returning the real price, so the failures lower X's p95 and raise its RPS. Its speed metrics perversely reward its failures.

Exercise 3 — The reading order. You receive this test summary: p50 = 6 ms, p95 = 12 ms, RPS = 5000, http_req_failed = 8%. (a) What's the first metric you should read and what does it tell you? (b) Should you even look at the p95 to decide? (c) Write the verdict in one sentence.

See solution

• (a) The error rate (http_req_failed = 8%). It tells you that 8 of every 100 requests failed: well above a typical 1% threshold. The test already failed here.
• (b) To decide whether it's ready, no: with 8% error the verdict is already "not ready," and the p95 doesn't change it. The p95 and the RPS are useful later, to diagnose (do the errors come from saturation? from a downed service?), but they don't rescue the decision.
• (c) "Not ready: 8% error rate (threshold 1%); the low latency (p95 12 ms) only describes the speed at which 1 in every 12 requests fails."

Summary and next step

In this lesson you closed the dashboard with the third instrument: the error rate (http_req_failed in k6), the percentage of failed requests. You learned what counts as a failure by default (status ≥ 400, timeouts, dropped connections) and what doesn't (a 200 with wrong content, which is a check's job). And you nailed the central rule with numbers: /quote_flaky gave a p50 of 5.45 ms and a p95 of 10 ms —latency indistinguishable from a healthy API— but 10% error, while /quote gave the same latency with 0%. By the first two instruments they were twins; only the error rate revealed that one was ready and the other broken.

The underlying lesson is the reading order: the error rate first, because it validates (or invalidates) everything else; a fast response that fails served no one, and its speed even inflates the p95 and RPS favorably. Before moving on you should be able to: define http_req_failed and say what counts as an error; explain why a low latency with 10% error is a failure; and state the order in which the three metrics are read.

You now have the three instruments complete: latency, throughput, and errors. What comes next is reading them all together in their native format. In lesson 6 we open the k6 run summary line by line —as labeled content, because k6 isn't installed— and map each line (http_req_duration, http_reqs, http_req_failed, iterations, vus) to the metrics you already computed yourself in Python.

Resources

• k6 — http_req_failed and the response callback — the official reference for the error rate: how a failure is defined by default (status < 400 = success) and how it's customized with setResponseCallback.
• Google SRE Book — Errors (the Four Golden Signals) — errors as one of the four golden signals, and why the error rate is watched alongside (and before) the latency. The frame for the "reading order."
• MDN — HTTP status codes — what each range means (2xx success, 4xx client error, 5xx server error), the basis of the rule for what counts as a failure.
• Module 6 of this guide — Checks, groups, and realistic scenarios — where to verify the content of the response under load (a 200 with the correct price), which the error rate doesn't cover. The boundary between availability and correctness.