Module 6: Checks Groups And Realistic Scenarios

3. A failing check vs an aborting threshold

Overview

There's a confusion almost everyone brings from unit tests and that's worth dismantling once and for all: believing that a failing check stops the test. It doesn't. A failing check records the failure and continues —the iteration finishes its steps, the other VUs keep hitting, the run completes its duration—. This isn't a defect: it's exactly what you want in a load test, where you care about how many of thousands of responses came out wrong, not stopping at the first. But it leaves an open question: if the check stops nothing, who decides that the test failed? Who makes the pipeline go red when too many responses are wrong?

The answer is the threshold, which you met in module 5. And the distinction between the two is the topic of this lesson, because it's the one that organizes the whole verdict of a load test: the check measures, the threshold decides. A check produces a number —the rate of checks that passed, for example 94.10%—. A threshold takes that number and compares it against a limit —"the failed-checks rate must be less than 1%"— and from that comparison a binary verdict comes out: PASS or FAIL, with an exit code (0 or nonzero) that a CI system can read to let a deploy through or block it. The check never aborts; the threshold is the only one that turns failures into a verdict.

Connection to the module: this lesson connects the new piece (the correctness check from lesson 2) with a piece already learned (the threshold from module 5, which here we reuse, not re-explain). k6's check goes as content; its behavior —fails and continues— we actually run with the run of a buggy price check, and the threshold gate —evaluate the rate and return PASS/FAIL + exit code— is also run in Python, as a mirror of k6's threshold. You'll see both things measured: the run completes its iterations despite the failures, and the gate produces a real exit code. Everything labeled as Python was measured with Python 3.14.0.

The smoke detector and the alarm switch

Think of it with two household devices. The smoke detector does one thing: it detects and counts. Every time there's smoke, it lights up and records it. It doesn't put out the fire, doesn't call the firefighters, doesn't cut the power —it only detects—. It's deliberately passive: its job is to observe and warn, not to act. If it detected and acted at once, a bit of kitchen smoke would cut the power to the whole house, and that would be worse than the problem.

The general alarm switch is what acts: when the detector has recorded enough smoke —it crosses a limit—, the alarm fires, the firefighters are called, a decision is made. The detector feeds the alarm with data; the alarm decides when the smoke level warrants an action. Separating the two is what makes the system useful: the detector can be sensitive and count every wisp of smoke without causing chaos, because it isn't the one deciding; and the alarm can have a well-thought-out limit ("if the smoke exceeds X for Y seconds") because it doesn't have to detect, only decide about what the detector counted.

In a load test, the check is the smoke detector: it detects bad responses and counts them, without aborting anything. The threshold is the alarm switch: it takes the check's count and decides, against a limit, whether the test passes or fails. That's why the check never stops the run —it would be the detector cutting the power— and that's why the threshold exists apart —it's the one with the authority to give the verdict—.

The check detects and counts bad responses without aborting anything (the smoke detector). The threshold takes that count and decides, against a limit, whether the test passes or fails, with an exit code (the alarm switch). The check measures; the threshold decides. Separating them is what lets the check be sensitive without causing chaos and the verdict be deliberate.

How the two things look in k6 (content)

In k6, the check and the threshold live in different places in the script, which reinforces that they do different things. The check goes inside the VU's function (it detects for each response); the threshold goes in options (it decides about the total):

// check_vs_threshold.js - the check measures inside the VU; the threshold decides in options.
// SHOWN AS CONTENT: k6 is not installed in this environment.
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  // THE THRESHOLD (decides): the passing-checks rate must be > 99%.
  // If it drops below that, k6 marks the test as FAILED and exits with code != 0.
  thresholds: {
    checks: ['rate>0.99'],
  },
};

const BASE_URL = 'http://localhost:8000';

export default function () {
  const payload = JSON.stringify({ room: 'Focus', tier: 'pro', hours: 3 });
  const params = { headers: { 'Content-Type': 'application/json' } };
  const res = http.post(`${BASE_URL}/quote`, payload, params);

  // THE CHECK (measures): detects and counts, does NOT abort. Continues even if it fails.
  check(res, {
    'status is 200': (r) => r.status === 200,
    'price is correct': (r) => r.json('price_cents') === 6000,  // Focus/pro/3h
  });
}

The two pieces, and how they relate:

The check (inside the VU) evaluates its criteria for each response and feeds an internal k6 metric called checks —the rate of checks that passed—. It never stops the iteration; it only counts.
The threshold checks: ['rate>0.99'] (in options) reads that checks metric and demands its rate be greater than 0.99 (99%). If at the end of the test the rate is lower, k6 marks the test as failed and exits with a nonzero exit code —the signal a CI uses to block a deploy—.

The bridge between the two is the checks metric: the check feeds it, the threshold judges it. A check with no threshold measures but never fails the test; a threshold with nothing to measure makes no sense. Together they're the detector and the alarm.

Seeing the check fail without aborting (executed)

Let's provoke meaningful failures to see the "measure and continue" behavior. Let's run lesson 2's scenario with a variant: the price check has a bug, it forgets the pro discount. That is, for a pro quote it expects the basic price (without the 20% discount), which doesn't match what the API —correctly— returns. The price check should fail on all pro quotes, while the other criteria stay green. The important thing: the run must complete all its iterations despite the failures.

First, the healthy run (no bug), 4 VUs / 3 s / think 100 ms, for a reference. Real output in this environment:

----------------------------------------------------------------
  Scenario quote->book->confirm  ->  4 VUs / 3s / think 100ms
----------------------------------------------------------------
  vus............: 4
  iterations.....: 109   (35.2/s)
  checks.........: 100.00%   (872 of 872)
    quote: status is 200..............:  109 ok / 0    fail
    quote: price is correct...........:  109 ok / 0    fail
    ...
  http_errors....: 0
----------------------------------------------------------------

100% of checks: the healthy API passes the 8 criteria per iteration (872 = 109 × 8). Now the run with the bug in the price check, same configuration:

What to expect. The quote's price check will fail on the pro rows (half the dataset), so the total rate will drop; but the run must complete its ~108 iterations anyway, without stopping. Real output in this environment:

----------------------------------------------------------------
  Scenario quote->book->confirm  ->  4 VUs / 3s / think 100ms  (BUG in the price check)
----------------------------------------------------------------
  vus............: 4
  iterations.....: 108   (34.4/s)
  checks.........: 94.10%   (813 of 864)
    quote: status is 200..............:  108 ok / 0    fail
    quote: price is correct...........:   57 ok / 51   fail   <-- FAILS
    book: status is 200...............:  108 ok / 0    fail
    book: confirmed is true...........:  108 ok / 0    fail
    book: booking_id present..........:  108 ok / 0    fail
    confirm: status is 200............:  108 ok / 0    fail
    confirm: id matches...............:  108 ok / 0    fail
    confirm: price matches quote......:  108 ok / 0    fail
  http_errors....: 0
----------------------------------------------------------------

Read it carefully, because it teaches the key behavior:

iterations: 108 — the run completed its 108 iterations despite the 51 failed checks. It didn't stop at the first failure. This is the essential thing: the check failed many times and the test kept measuring to the end. An assert would have aborted at the first pro.
quote: price is correct: 57 ok / 51 fail — the price check caught the pro quotes. Of 108 iterations, ~57 requested basic rows (they add up) and ~51 requested pro rows (they fail, because the buggy expected value forgot the discount). The check detected and counted, without aborting.
checks: 94.10% (813 of 864) — of 864 total checks (108 × 8 criteria), 813 passed and 51 failed. The percentage dropped from 100% to 94.10%. That number is what the check produces: a measurement. But notice: the run exited successfully anyway. The check measured the problem, but on its own didn't fail the test.

And there's the question that opens the second half: 94.10% of checks is clearly bad, but the test "passed" (completed, no error). Who turns that 94.10% into a FAIL?

Seeing the threshold decide (executed)

The threshold. Reusing the idea from module 5, we write a gate that takes the checks rate and evaluates it against a limit —"less than 1% failed checks"—, returning PASS/FAIL and an exit code, exactly as k6's checks: ['rate>0.99'] threshold would:

# threshold_gate.py - the verdict the check does NOT give: a threshold on the
# failed-checks rate (mirror of the k6 threshold, M5 topic). IS RUN.
import sys
passed, total, max_fail = int(sys.argv[1]), int(sys.argv[2]), float(sys.argv[3])
fail_rate = (total - passed) / total if total else 0.0
ok = fail_rate < max_fail                      # the DECISION
verdict = "PASS" if ok else "FAIL"
print(f"  checks: {passed}/{total}  fail rate: {fail_rate*100:.2f}%  "
      f"limit: <{max_fail*100:.2f}%")
print(f"  threshold checks:['rate>{1-max_fail:.2f}']  ->  {verdict}  (exit {0 if ok else 1})")
sys.exit(0 if ok else 1)                        # exit code a CI reads

We pass it the real numbers from the two runs above and look at the verdict. First the healthy one (872/872), then the buggy one (813/864), against the same 1% limit:

What to expect. The healthy run (0% failures) must pass the limit (exit 0); the buggy one (5.90% failures, well above 1%) must fail it (exit 1). Real output in this environment:

### Healthy run (872/872) against threshold 'less than 1% failed checks':
  checks: 872/872  fail rate: 0.00%  limit: <1.00%
  threshold checks:['rate>0.99']  ->  PASS  (exit 0)
   [real exit code: 0]

### Buggy run (813/864) against the SAME threshold:
  checks: 813/864  fail rate: 5.90%  limit: <1.00%
  threshold checks:['rate>0.99']  ->  FAIL  (exit 1)
   [real exit code: 1]

Here's the decision, measured:

The healthy run → PASS, exit 0. 0% failed checks is below the 1% limit. The threshold gives a green light. A CI that reads exit 0 lets the deploy through.
The buggy run → FAIL, exit 1. 5.90% failed checks exceeds the 1% limit. The threshold gives a red light and the exit code is 1. A CI that reads exit 1 blocks the deploy. Notice: the 94.10% of checks was exactly the same number the buggy run produced; the threshold didn't change it, it only judged it.
The complete bridge. The check measured (94.10%). The threshold decided (FAIL, exit 1). Neither did the check fail the test on its own (the run completed successfully), nor did the threshold measure anything (it only read the check's number). Detector and alarm, each in its role.

This is the verdict of a load test in one sentence: the checks count the bad responses throughout the run without stopping it; at the end, a threshold on the checks rate turns that count into the PASS/FAIL that gates (or not) the deploy. The exit 1 is the mechanical signal that makes a pipeline go red.

Common mistakes

Expecting a failed check to abort the iteration or the test. What happens: someone sees failed checks and is surprised the run completed "as if nothing happened." Why it happens: they bring the assert model, which aborts at the first failure. How to detect it: the summary shows failed checks (94.10%) but the run finished all its iterations and exited successfully. How to fix it: understand that the check measures, doesn't abort. If you need the test to fail, put a threshold on the checks rate. The check is the detector; the threshold is the alarm.

Adding checks but no threshold, and expecting CI to go red. What happens: someone fills the script with checks, some fail, but the pipeline stays green and the deploy passes. Why it happens: the checks measure, but without a threshold nobody turns the measurement into a verdict —k6 exits with code 0 even with failed checks—. How to detect it: checks at 94% and yet exit 0. How to fix it: add thresholds: { checks: ['rate>0.99'] } (or the limit you decide) in options. Without a threshold, the checks are a smoke detector disconnected from any alarm.

Putting the threshold's limit in the wrong place in the script. What happens: someone tries to put the threshold logic inside the VU's function, or the check inside options. Why it happens: they're not clear on the division of labor. How to detect it: the script doesn't compile or doesn't do what's expected. How to fix it: remember where each one lives. The check goes inside the VU's function (measures per response); the threshold goes in options (decides about the total). The place reflects the role: detecting is per response, deciding is about the aggregate.

Exercises

Exercise 1 — Measures or decides? For each statement, say whether it describes a check or a threshold. (a) "Counts how many responses had the correct price." (b) "Makes the test exit with exit code 1 if more than 1% of the responses fail." (c) "Goes inside the VU's function and doesn't abort anything." (d) "Goes in options and produces the pass/fail verdict."

See solution

(a) Check — counts (measures) correct responses.
(b) Threshold — produces the exit code (decides).
(c) Check — lives inside the VU and doesn't abort.
(d) Threshold — lives in options and gives the verdict.

The rule: if it counts and doesn't abort, it's a check; if it decides pass/fail with an exit code, it's a threshold.

Exercise 2 — From the count to the verdict. In the buggy run, the summary gave checks: 94.10% (813 of 864). (a) What's the failed-checks rate? (b) Does it pass or fail against a threshold checks: ['rate>0.99']? (c) And against a looser one, checks: ['rate>0.90']? Justify each one.

See solution

(a) Failed: 864 − 813 = 51 of 864 = 5.90%. (The passed rate is 94.10%.)
(b) Fails. The threshold demands a passed rate > 99% (equivalent to < 1% failures). With 94.10% passed (5.90% failures), it doesn't reach it: FAIL, exit 1.
(c) Passes. The loose threshold demands > 90% passed. With 94.10%, it does exceed 90%: PASS, exit 0. Same check count, different limit, different verdict —which shows the threshold is a policy decision separate from the measurement—.

Exercise 3 — The confused colleague. A colleague says: "I put a price check in my load test. The API returned bad prices about half the time, but the test ran to the end and k6 exited with code 0, so my CI let the deploy through. Is the check broken?" Explain what happened and what they're missing.

See solution

The check isn't broken: it did its job, which is measuring. It detected and counted the bad prices (the checks rate dropped), but a check —by design— doesn't abort or fail the test; it only counts. That's why the run completed and, with nothing else, k6 exited with code 0.

What they're missing is a threshold on the checks rate, for example thresholds: { checks: ['rate>0.99'] } in options. That threshold would read the rate the check produced and, seeing that half failed, would mark the test as failed and exit with a nonzero exit code, which is the signal that makes CI put the pipeline red and block the deploy. The check is the smoke detector; the threshold is the alarm. They need to connect the two.

Summary and next step

In this lesson the division of labor that organizes a load test's verdict became clear: the check measures, the threshold decides. A failing check records and continues —it doesn't abort the iteration or the test—, and you saw it measured: the run with a buggy price check completed its 108 iterations despite 51 failures, with the checks rate at 94.10%. That number is a measurement, not a verdict. The threshold —reused from module 5— is the one that turns it into a verdict: you passed the real numbers to a gate with a 1% limit, and the healthy run came out PASS (exit 0) while the buggy one came out FAIL (exit 1), the signal a CI reads to block a deploy. Smoke detector and alarm: sensitive the one, deliberate the other.

Before moving on you should be able to: explain why a failed check doesn't stop the run and why that's desirable; distinguish what a check measures from what a threshold decides; know where each one lives in the k6 script (check in the VU, threshold in options); and compute whether a given checks rate passes or fails against a limit.

Lesson 4 organizes the scenario we've started to run. When an iteration has several steps —quote, book, confirm—, it's worth wrapping them in named blocks: k6's group()s. You'll see how grouping gives metrics per step (how long quoting took vs booking vs confirming) and tags the checks by group —and you'll measure it in Python, which already reports the scenario's per-group latency—.

Resources

Checks in k6 — the reference for check() and its checks metric, and confirmation that a failed check doesn't abort the test. The "measures" half of this lesson.
Thresholds in k6 — how thresholds: { checks: ['rate>0.99'] } turns the checks rate into a pass/fail verdict with an exit code. The "decides" half. Reused from module 5.
Module 5 of this guide — Thresholds, pass/fail, and SLOs — where the threshold we only reuse here is taught in depth. Review it if you're not clear on how a limit gates a deploy.
sys.exit — Python documentation — how a program returns an exit code (0 or nonzero), the signal a CI reads. The mechanism of the gate you ran.

← Previous2. `check()` for correctness under load Next →4. `group()` to organize the steps (content)