2. What a threshold is and the pass/fail

Overview

A threshold is the smallest and most powerful piece of this module: a rule on a metric that turns a measurement into a binary verdict. "The p95 must be below 200 ms" is a threshold. It's not an aspirational goal or a footnote in a report: it's a condition the test verifies, and from which a single bit of information comes out —pass or fail—. In this lesson we precisely define what a threshold is, how it's read, and why the pass/fail is so valuable; and we build the first executable version of the Python mirror: a function that takes the latencies you actually measured against Reservo and returns True (pass) or False (fail). You see it pass under light load and fail under heavy load, over real numbers. The mechanical consequence of that verdict —turning it into an exit code that makes CI fail— is lesson 4; here we focus on the judgment itself: the rule and the bit it produces.

Connection to the module: lesson 1 explained why we want to judge (going from measuring to gating); this one explains what the judgment is (the threshold) and produces its first executable form. The metric we put under a limit —the p95— comes from module 3; the threshold doesn't reinvent it, it puts a rule on it. In lesson 3 you'll see how k6 declares these same thresholds (as content) for the three key metrics; in lesson 4, how the verdict becomes an exit code. Here we build the heart: the function that looks at a number and says pass or fail.

The inspector with the template

Think of a quality inspector at the end of a screw production line. They're not an artist who "senses" whether a screw is good: they have a template —a metal plate with a hole of an exact diameter—. The screw passes through the hole: good. It doesn't pass: bad. The template turns a continuous property (the diameter, which could be 4.98 mm, 5.01 mm, 5.13 mm...) into a single-bit decision (pass / no pass). The inspector doesn't report "this screw measures 5.13 mm and seems a bit big to me, what do you think?"; they report rejected, and the screw goes to the discard bin. The template is objective (any inspector with the same plate gives the same verdict), fast (one gesture per screw), and actionable (the verdict triggers an action: accept or discard).

A threshold is that template, applied to a performance metric. The continuous metric is the p95 (it could be 9 ms, 199 ms, 247 ms...); the template is the rule p(95) < 200; the verdict is pass/fail. Just like the inspector's plate, a threshold is objective (the same p95 and the same rule always give the same verdict, no opinions), fast (one comparison) and, above all, actionable —the pass/fail is going to trigger something, which in lesson 4 will be stopping or authorizing the deploy—. All this module's power comes from having reduced "is the performance good?" to a bit a machine can read and act on.

The anatomy of a threshold

A threshold has three parts, and it's worth naming them because you'll see them over and over:

1. The metric. What's measured. In this module, three: the latency (p95 of http_req_duration), the error rate (http_req_failed), and the correct-checks rate (checks). The threshold doesn't create the metric —you did that in module 3—; it uses it.
2. The operator and the value —the limit proper. The comparison that defines "good." < 200 (less than 200 ms), < 0.01 (less than 1%), > 0.99 (greater than 99%). The value is a decision (where that 200 comes from is lesson 5); the operator says which side the acceptable is on.
3. The verdict. The bit it produces: True/False, pass/fail, green/red. It's the only thing that comes out of the threshold, and it's the only thing the pipeline needs.

Written as an English sentence, a threshold is always of the form: "the metric must be operator value". "The p95 must be below 200 ms." "The error rate must be below 1%." "The checks rate must be above 99%." If you can say that sentence, you can write the threshold. And if the measured metric meets the sentence, it passes; if not, it fails. There's no third option, and that absence of a third option is exactly the point: it eliminates the "more or less," the "it depends," the "looks fine to me." A threshold doesn't negotiate.

The executable mirror: evaluate_thresholds in Python

Let's build the threshold with our own hands, in Python, so it isn't magic. The evaluate_thresholds function receives the already-measured metrics (the list of latencies, the error rate, the checks rate —all real, measured against Reservo with the techniques of modules 3 and 4—) and applies the three rules. For now it focuses on producing the verdict (it returns True if all pass, False if any fails) and printing PASS/FAIL for each rule. In lesson 4 we'll add the sys.exit that turns that True/False into an exit code.

import statistics


def q(data, p):
    """Percentile p (1-99) with the inclusive method (module 3)."""
    return statistics.quantiles(data, n=100, method="inclusive")[p - 1]


def evaluate_thresholds(latencies, error_rate, checks_rate, p95_limit_ms):
    """Evaluates the measured metrics against the thresholds. Prints PASS/FAIL per
    threshold and returns True only if ALL pass (the gate's verdict)."""
    p95 = q(latencies, 95)

    # Each threshold: (label, passed the threshold?, measured value to show).
    checks = [
        (f"http_req_duration: p(95) < {p95_limit_ms:.0f}ms",
         p95 < p95_limit_ms,
         f"p(95) = {p95:.2f}ms"),
        ("http_req_failed:   rate < 1.00%",
         error_rate < 0.01,
         f"rate  = {error_rate:.2%}"),
        ("checks:            rate > 99.00%",
         checks_rate > 0.99,
         f"rate  = {checks_rate:.2%}"),
    ]

    all_pass = True
    for label, passed, measured in checks:
        status = "PASS" if passed else "FAIL"
        if not passed:
            all_pass = False   # ONE broken threshold is enough to fail everything
        print(f"{label:<38} {measured:<18} {status}")
    return all_pass

Read it like the sentence from before. The first rule says "the p95 must be below p95_limit_ms": it computes p95 = q(latencies, 95) (the real percentile of your latencies) and compares p95 < p95_limit_ms. The second, "the error rate must be below 1%": error_rate < 0.01. The third, "the checks rate must be above 99%": checks_rate > 0.99. Each produces a boolean —its verdict—. And there's a crucial design decision in the line if not passed: all_pass = False: it's enough for ONE threshold to fail for the global verdict to be "fail". A gate is a conjunction, a logical AND: it passes only if all the rules pass. This is just like in the factory —a bottle with the perfect weight but badly capped is discarded anyway—; and it's just like in k6, where if any threshold fails, the whole test fails.

That function is the threshold made code, and the most important thing is that there's nothing magical about it: it's three comparisons and an implicit and. When in lesson 3 you see k6's options.thresholds block, you'll recognize exactly these three rules —k6 writes them more compactly, but they do this—.

Seeing it pass and fail (actually executed)

We wrap evaluate_thresholds in a small runner that first measures (launches load against Reservo, gathers the latencies, counts errors and correct checks) and then evaluates. That runner is threshold_gate.py; we use it in full in lesson 4, here it's enough to see its verdict. Everything that follows is real output, run against the Reservo API on localhost, with /quote_cpu (the endpoint the module declared in lesson 1).

First, light load: 400 requests with 4 concurrent clients. With so little concurrency, /quote_cpu barely suffers GIL contention, so its p95 stays in a few milliseconds.

What to expect — the three rules pass (p95 well below 200 ms, zero errors, checks at 100%); the verdict is PASS:

$ python3.14 threshold_gate.py http://127.0.0.1:PORT /quote_cpu 400 4 200
# /quote_cpu  |  400 requests, concurrency 4
THRESHOLD                              MEASURED           RESULT
----------------------------------------------------------------------
http_req_duration: p(95) < 200ms       p(95) = 9.74ms     PASS
http_req_failed:   rate < 1.00%        rate  = 0.00%      PASS
checks:            rate > 99.00%       rate  = 100.00%    PASS
----------------------------------------------------------------------
GATE: PASS  (exit code 0)

Now, the same rule, the same endpoint, but with heavy load: 2000 requests with 120 concurrent clients. The GIL serializes the CPU work, the queue grows, and the p95 spikes.

What to expect — the p95 crosses 200 ms; that one rule fails, and with that the global verdict is FAIL (the other two keep passing):

$ python3.14 threshold_gate.py http://127.0.0.1:PORT /quote_cpu 2000 120 200
# /quote_cpu  |  2000 requests, concurrency 120
THRESHOLD                              MEASURED           RESULT
----------------------------------------------------------------------
http_req_duration: p(95) < 200ms       p(95) = 246.96ms   FAIL
http_req_failed:   rate < 1.00%        rate  = 0.00%      PASS
checks:            rate > 99.00%       rate  = 100.00%    PASS
----------------------------------------------------------------------
GATE: FAIL  (exit code 1)

Stop at what just happened, because it's the whole lesson. The app didn't change. The rule didn't change (p(95) < 200 in both runs). The only thing that changed was the load: from 4 to 120 concurrent clients. And the verdict went from green to red. That's a threshold doing its job: translating a difference in load into a difference in verdict, with no one interpreting anything. Under light load the app meets the SLO; under heavy load, it doesn't; and the gate says it in one word.

Notice too in the second run: only the latency threshold failed —the error and checks ones stayed green—, and still the whole gate failed. That's the logical AND in action: a gate doesn't average its rules or "round in favor." A single broken rule turns the whole verdict red. It's severe on purpose: quality is a conjunction of conditions, not a score that compensates.

Why the binary is worth so much

You might wonder: don't we lose information by reducing a rich, nuanced p95 to a single bit? Yes, and that's exactly the point. The richness of the p95 (that it was 246.96 ms, with such a distribution, such a tail...) is invaluable when a human investigates —you'll use it in module 7 to find the bottleneck—. But to automatically decide whether the deploy advances, a human isn't in the loop, and a machine can't act on "246.96 ms with a worrying tail": it needs a bit. The threshold is the translator between those two worlds. It saves the complete number for the analysis, and produces the bit for the decision.

That bit is what makes performance automatable. A number requires a human eye to interpret it; a bit requires only an if. By turning "is the performance good?" into a boolean, the threshold lets the answer travel through a pipeline, trigger an action, and protect production with no intervention —at machine speed, on every change, without getting tired—. Losing the nuances at the decision point isn't a defect: it's the condition for being able to decide by machine.

Common mistakes

Confusing a goal with a threshold. What happens: a team says "our goal is p95 below 200 ms" and writes it in a wiki, but the load test doesn't verify it —it only reports the number—. Why it happens: aspiring to something gets confused with enforcing something. How to detect it: if your "goal" can't fail a run (return False, turn the build red), it's a wish, not a threshold. How to fix it: write it as a rule the test evaluates and from which a pass/fail comes out —a comparison in evaluate_thresholds, a line in options.thresholds—.

Believing a gate averages its rules. What happens: someone sees two thresholds green and one red and concludes "two of three, it's fine, it passes." Why it happens: the gate gets thought of as a score. How to detect it: if your verdict logic doesn't fail with a single broken rule, it's wrong. How to fix it: a gate is a logical AND —all_pass starts at True and any not passed sets it to False—. A broken rule fails everything; there's no compensation between thresholds.

Putting the limit on the average instead of the percentile. What happens: someone writes the rule on the average latency ("avg < 200") and the run passes even though 5% of the users suffer seconds of waiting. Why it happens: the average hides the tail (module 3). How to detect it: if your latency threshold uses the average, you're gating the wrong metric. How to fix it: put the limit on a percentile (p95, p99) —it's what describes the suffering user's experience, and it's what k6 and latency SLOs use—.

Exercises

Exercise 1 — Translate to the sentence. Write each of these thresholds as the sentence "the metric must be operator value", and say what metric, operator, and value it has. (a) p(95) < 200. (b) rate < 0.01 on http_req_failed. (c) rate > 0.99 on checks.

Exercise 2 — Emit the verdict. With the three thresholds p(95)<200, error<0.01, checks>0.99, say the global verdict (PASS/FAIL) of each measured run and which rule(s) failed. (a) p95=9.74 ms, error=0.00%, checks=100%. (b) p95=246.96 ms, error=0.00%, checks=100%. (c) p95=150 ms, error=0.00%, checks=98.5%.

Exercise 3 — Add a fourth threshold. You want to add a fourth rule to evaluate_thresholds: "the p99 must be below 500 ms". Write the (label, passed?, measured) tuple you'd add to the checks list, using the q helper. Why would you want to watch the p99 in addition to the p95?

(f"http_req_duration: p(99) < 500ms",
 q(latencies, 99) < 500,
 f"p(99) = {q(latencies, 99):.2f}ms"),

Summary and next step

A threshold is a rule on a metric that produces a binary verdict: pass or fail. It has three parts —the metric (what's measured), the operator and the value (the limit, "the metric must be operator value"), and the verdict (the bit that comes out)— and its value is in being objective, fast, and actionable. You built it with your own hands in Python: evaluate_thresholds takes the actually-measured latencies, error rate, and checks rate, applies the three rules, prints PASS/FAIL for each, and returns True only if all pass —a logical AND, where a single broken rule fails the whole gate—.

And you saw it work: the same app and the same rule (p(95) < 200), green under light load (p95 = 9.74 ms) and red under heavy load (p95 = 246.96 ms), where the only thing that changed was the concurrency. The binary the threshold produces loses the number's nuances on purpose: that loss is what makes performance automatable, because a machine can act on a bit, not on "246.96 ms with a worrying tail."

Before moving on you should be able to: write any threshold as the sentence "the metric must be operator value"; explain why a gate is a logical AND and not an average; and read a run's verdict saying which rule(s) failed. What comes next, in lesson 3, is seeing how k6 declares these same three thresholds —http_req_duration, http_req_failed, checks— in its options.thresholds block (as labeled content), and mapping each one to the Python rule you just wrote. The same inspector's template, in the industrial language.

Resources

• k6 — Thresholds — the official reference: how k6 defines a threshold as a rule the test must meet. The industrial form of the evaluate_thresholds you built here.
• statistics.quantiles — Python documentation — how the q helper computes the real p95 the threshold is applied to. The metric the rule judges.
• Google SRE Book — Service Level Objectives — why a performance condition is expressed as a verifiable rule (an SLO) and not as a wish. The foundation of "a threshold doesn't negotiate."
• k6 — check() — the checks metric (rate of correct verifications) one of the thresholds watches; its in-depth use for verifying correctness under load is module 6.