Module 5: Thresholds Pass Fail And Slos

8. Mini-project: a performance gate

Next →

Overview

It's time to bring the whole module together with your own hands. In this mini-project you build a complete performance gate for the Reservo API: you bring up the server (with the /quote_cpu endpoint the module declared), measure its real metrics under two loads —a light one and a heavy one—, apply an evaluate_thresholds function that emits a pass/fail verdict and exits with the corresponding exit code, and write the equivalent k6 thresholds block as content. The result is the whole module embodied in a deliverable: you'll see the gate green with the light load (all under the limit, exit code 0) and red with the heavy one (the p95 crosses the limit, exit code 1), with the real numbers and exit codes your machine produced. It's not a fill-in-the-blanks exercise: it's the real flow of putting a performance under a gate, from start to finish.

Connection to the module: this capstone synthesizes the seven previous lessons. It uses the threshold and the pass/fail (L2), the three k6 metrics (L3), the exit code that gates CI (L4), the judgment for choosing the limit (L5), the view of the gate as one more in the family (L6) and —optionally— the per-part-of-the-system limits (L7). It's the executable piece that proves you understood the module: not "I know what a threshold is," but "I can set up a performance gate that approves or rejects a deploy with real data." What comes next (module 6) adds the correctness check()s under load and the scenarios; module 7 installs this gate in the complete CI pipeline.

What you'll deliver

Your deliverable has four pieces:

  1. The Reservo API running, with /quote_cpu declared (the canonical server + the lesson-1 addition).
  2. The executable gate (threshold_gate.py): an evaluate_thresholds function that evaluates real p95, error rate, and checks rate against their limits, prints PASS/FAIL for each, and exits with sys.exit(0) or sys.exit(1).
  3. Two real runs: the gate green with light load and red with heavy load, each with its real exit code (echo $?).
  4. The equivalent k6 thresholds block, as labeled content, plus a brief justification of the chosen limit (its SLO).

Step 1 — Bring up the API with /quote_cpu

Start from Reservo's canonical server (module 1, lesson 6) and add the /quote_cpu endpoint this module declared (lesson 1): the CPU_WORK constant, the burn_cpu function, and the branch in do_POST that does the CPU work before responding. Remember the three design decisions: money in integer cents, a large request_queue_size and ThreadingHTTPServer to withstand concurrency, and port 0 to avoid colliding with other processes. Start it and read its port.

What to expect — the server prints the port the operating system assigned it (varies on each start), and responds with the anchor numbers on both endpoints:

$ python3.14 reservo_server.py &
Reservo listening on http://127.0.0.1:57565
$ curl -s -X POST http://127.0.0.1:57565/quote \
    -H 'Content-Type: application/json' -d '{"room":"Focus","tier":"basic","hours":3}'
{"price_cents": 7500}
$ curl -s -X POST http://127.0.0.1:57565/quote_cpu \
    -H 'Content-Type: application/json' -d '{"room":"Focus","tier":"basic","hours":3}'
{"price_cents": 7500}

Both endpoints return 7500 (Focus/basic/3h). The difference isn't in what they respond —the business logic is identical—, but in how long they take under load: /quote_cpu does CPU work the GIL serializes, so its latency degrades with concurrency. That's the target that will make the limit be crossed.

Step 2 — The executable gate

The heart of the deliverable is the evaluate_thresholds function and the main that wraps it. The function applies the three rules and returns the verdict; the main measures, evaluates, and exits with the corresponding code. This is the complete gate (it brings together lessons 2 and 4):

"""Performance gate — measures Reservo and evaluates thresholds with a real exit code."""
import json
import statistics
import sys
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor

BASE_URL = sys.argv[1]
PATH = sys.argv[2]
TOTAL_REQUESTS = int(sys.argv[3])
CONCURRENCY = int(sys.argv[4])
P95_LIMIT_MS = float(sys.argv[5])


def q(data, p):
    return statistics.quantiles(data, n=100, method="inclusive")[p - 1]


def one_request():
    """One request. Returns (latency_ms, ok_error, ok_check)."""
    payload = json.dumps({"room": "Focus", "tier": "basic", "hours": 3}).encode()
    req = urllib.request.Request(
        f"{BASE_URL}{PATH}", data=payload,
        headers={"Content-Type": "application/json"}, method="POST",
    )
    start = time.perf_counter()
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            body = json.loads(resp.read())
        latency_ms = (time.perf_counter() - start) * 1000
        return latency_ms, True, body.get("price_cents") == 7500
    except (urllib.error.URLError, OSError):
        return (time.perf_counter() - start) * 1000, False, False


def measure():
    latencies, errors, checks_ok = [], 0, 0
    with ThreadPoolExecutor(max_workers=CONCURRENCY) as pool:
        futures = [pool.submit(one_request) for _ in range(TOTAL_REQUESTS)]
        for fut in futures:
            latency_ms, ok_error, ok_check = fut.result()
            latencies.append(latency_ms)
            errors += 0 if ok_error else 1
            checks_ok += 1 if ok_check else 0
    return latencies, errors / TOTAL_REQUESTS, checks_ok / TOTAL_REQUESTS


def evaluate_thresholds(latencies, error_rate, checks_rate, p95_limit_ms):
    """Evaluates the metrics against the thresholds. True only if ALL pass."""
    p95 = q(latencies, 95)
    checks = [
        (f"http_req_duration: p(95) < {p95_limit_ms:.0f}ms",
         p95 < p95_limit_ms, f"p(95) = {p95:.2f}ms"),
        ("http_req_failed:   rate < 1.00%",
         error_rate < 0.01, f"rate  = {error_rate:.2%}"),
        ("checks:            rate > 99.00%",
         checks_rate > 0.99, f"rate  = {checks_rate:.2%}"),
    ]
    all_pass = True
    print(f"{'THRESHOLD':<38} {'MEASURED':<18} RESULT")
    print("-" * 70)
    for label, passed, measured in checks:
        if not passed:
            all_pass = False
        print(f"{label:<38} {measured:<18} {'PASS' if passed else 'FAIL'}")
    print("-" * 70)
    return all_pass


def main():
    latencies, error_rate, checks_rate = measure()
    print(f"# {PATH}  |  {TOTAL_REQUESTS} requests, concurrency {CONCURRENCY}")
    if evaluate_thresholds(latencies, error_rate, checks_rate, P95_LIMIT_MS):
        print("GATE: PASS  (exit code 0)")
        sys.exit(0)
    print("GATE: FAIL  (exit code 1)")
    sys.exit(1)


if __name__ == "__main__":
    main()

The three thresholds are your three SLOs made executable: p95 below the limit you pass as an argument, error below 1%, checks above 99%. The all_pass is the logical AND (a broken rule fails everything). And the main translates the verdict into an exit code: True → 0, False → 1.

Step 3 — Run the gate under two loads

Now the part that proves the gate works: run it twice against /quote_cpu, with the same rule (p(95) < 200), changing only the load.

First, light load (400 requests, 4 concurrent). Little GIL contention, low p95, all green.

What to expect — the three rules pass, the gate prints PASS, and echo $? confirms exit code 0:

$ python3.14 threshold_gate.py http://127.0.0.1:57565 /quote_cpu 400 4 200
# /quote_cpu  |  400 requests, concurrency 4
THRESHOLD                              MEASURED           RESULT
----------------------------------------------------------------------
http_req_duration: p(95) < 200ms       p(95) = 9.74ms     PASS
http_req_failed:   rate < 1.00%        rate  = 0.00%      PASS
checks:            rate > 99.00%       rate  = 100.00%    PASS
----------------------------------------------------------------------
GATE: PASS  (exit code 0)
$ echo $?
0

Now, the same rule, the same endpoint, but heavy load (2000 requests, 120 concurrent). The GIL serializes the work, the p95 spikes over the limit, and the gate goes red.

What to expect — the p95 crosses 200 ms; that one rule fails, the gate prints FAIL, and echo $? confirms exit code 1:

$ python3.14 threshold_gate.py http://127.0.0.1:57565 /quote_cpu 2000 120 200
# /quote_cpu  |  2000 requests, concurrency 120
THRESHOLD                              MEASURED           RESULT
----------------------------------------------------------------------
http_req_duration: p(95) < 200ms       p(95) = 246.96ms   FAIL
http_req_failed:   rate < 1.00%        rate  = 0.00%      PASS
checks:            rate > 99.00%       rate  = 100.00%    PASS
----------------------------------------------------------------------
GATE: FAIL  (exit code 1)
$ echo $?
1

Those two runs are the mini-project: the same app, the same rule, a verdict that goes from green to red according to the load, with real exit codes (0 and 1) a pipeline would respect. (The exact numbers vary in each run —they depend on how the operating system distributes the CPU—; what doesn't vary is the story: light passes, heavy fails.)

Step 4 — The equivalent k6 thresholds block

Close the deliverable by writing how this same gate would be declared in k6, as labeled content (k6 isn't installed; this isn't run here). It's the industrial form of your evaluate_thresholds:

// CONTENT (not run here): Reservo's performance gate in k6.
// See grafana.com/docs/k6/latest/using-k6/thresholds/. It would run: k6 run gate.js
import http from "k6/http";
import { check } from "k6";

export const options = {
  vus: 50,
  duration: "30s",
  thresholds: {
    http_req_duration: ["p(95)<200"],   // the p95 below 200 ms (interactive SLO)
    http_req_failed: ["rate<0.01"],     // less than 1% error
    checks: ["rate>0.99"],              // more than 99% correct checks
  },
};

export default function () {
  const url = "http://127.0.0.1:8000/quote_cpu";
  const payload = JSON.stringify({ room: "Focus", tier: "basic", hours: 3 });
  const params = { headers: { "Content-Type": "application/json" } };

  const res = http.post(url, payload, params);

  check(res, {
    "status is 200": (r) => r.status === 200,
    "price_cents is 7500": (r) => r.json("price_cents") === 7500,
  });
}

And the justification of the limit (its SLO), which is what separates a gate with judgment from an arbitrary one:

We gate the quote's p95 at 200 ms because it's a synchronous interaction: the user waits for the response looking at the screen, and the usability reference points place the feeling of "instantaneous" at ~100 ms and the limit before losing attention at ~1 s. A p95 of 200 ms means that 19 of every 20 users perceive the app as snappy. The error limit (1%) and the checks limit (99%) ensure the speed doesn't come at the cost of failed or incorrect responses —a fast but wrong quote is useless—.

With that, if tomorrow a change makes the p95 rise to 260 ms, the gate goes red and the team finds out before deploying. That's a performance under a gate.

Self-assessment rubric

Your deliverable is complete if:

  • The API runs with /quote_cpu declared and returns 7500 on /quote and /quote_cpu (Focus/basic/3h). Money in integer cents, port 0.
  • evaluate_thresholds applies the three rules (p95, error, checks) and returns True only if all pass (logical AND: a broken one fails everything).
  • The gate exits with the correct code: sys.exit(0) on PASS, sys.exit(1) on FAIL. You confirm it with echo $?. (Printing "FAIL" isn't enough: it must exit with ≠ 0.)
  • Two real runs: green with light load (exit 0), red with heavy load (exit 1), with the same rule —the only thing that changes is the load—.
  • The p95 goes on a percentile, not the average (q(latencies, 95), not mean).
  • The k6 thresholds block is written as labeled content and maps one to one with the three Python rules.
  • The limit is justified from the user/business (its SLO), not as a round number.
  • Execution honesty: the Python is run and cited; the k6 goes labeled as content. No git/gh.

Extensions (optional)

If you want to go further:

  • Add the p99. Add a fourth rule p(99) < 500 to evaluate_thresholds (L2 exercise) and gate the extreme tail too.
  • Per-part-of-the-system limits. Use L7's gate_multi.py to gate /quote (strict, 200 ms) and /quote_cpu (loose, 400 ms) with different limits, each according to its SLO, and add abortOnFail to the critical one.
  • Concurrency sweep. Run the gate with increasing concurrencies (4, 20, 60, 120, 200) and find the point where the p95 crosses the limit —the gate's "breaking point"—.
  • The CI chain. Chain gate && echo "deploy" || echo "blocked" (L4) to see the verdict authorize or block a simulated deploy.

Summary and where the module goes next

You built a complete performance gate for Reservo, end to end: you brought up the API with /quote_cpu declared, wrote evaluate_thresholds (three rules, logical AND, verdict), wrapped it in a main that exits with the real exit code (0 on PASS, 1 on FAIL), and ran it under two loads —green with the light (p95 = 9.74 ms, exit 0), red with the heavy (p95 = 246.96 ms, exit 1)— with the same rule, changing only the load. And you wrote the equivalent k6 thresholds block as content, with its limit justified from the SLO. That's a performance under a gate: a latency regression now fails a build just like a broken test.

With this you close the thresholds module. You learned the leap from measuring to judging: what a threshold is and the pass/fail, how k6 declares them, how the exit code fails CI, how to choose the limit with judgment (SLO/SLA), why it's a sibling of coverage gates, and how to abort early and set a limit per part of the system. You have the verdict.

What comes next, in module 6, is strengthening the other half of a realistic load test: the check()s that verify correctness under load (not just that the app is fast, but that it responds well —the correct price_cents— while you hammer it), the group()s to organize, the parametrization of data, and the correlation (extract a booking_id from one response and use it in the next: the quote→book flow). The checks: ['rate>0.99'] threshold you used here watches exactly that checks rate; module 6 produces it in depth. And module 7 takes this gate and installs it in the complete CI pipeline, with the GitHub Actions YAML and the regression analysis. The gate you built is the foundation of all that.

Resources

← Previous7. `abortOnFail` and per-scenario thresholdsNext →1. Module introduction: from hitting a route to a scenario