Module 8: Project Load Test Reservo

6. The threshold gate and exporting to JSON

Overview

So far, measuring and judging were separate steps. In this lesson we join them into the complete gate: a single run that measures the metrics, evaluates them against the thresholds, exports the result to a results.json, and exits with a code a pipeline reads (0 = PASS, 1 = FAIL). It's the piece that turns the test into something automatable. We run it for real in its two faces: green against the healthy build (/quote, p95 under the SLO, exit 0) and red against the heavy pricing engine (/quote_cpu, the stress crosses the p95, exit 1). We see the exported JSON of the red run and check, with a shell chain, how that exit code authorizes or blocks the deploy without anyone looking at a number.

Connection to the module: this lesson joins two modules into the piece that was missing. From module 5, the exit code that gives the verdict teeth (0/1, the pipeline's language). From module 7, exporting the metrics to a file (the run's notebook, which lesson 7 will upload as an artifact and which a regression check could compare). Here we don't re-explain the exit code mechanism or why you export; we use them to close the capstone's gate. Lesson 7 will put this gate in a CI load.yml; lesson 8 will deliver it complete. This is the test's executable heart.

The subway turnstile

At the subway entrance there's a turnstile that does three things in an instant. It reads your card (does it have balance?), it decides (pass / don't pass), and it acts physically: if you have balance, the arms turn and you go through; if not, they lock and you stop. There's no guard interpreting your face or listening to your reasons —the turnstile translates the balance into a mechanical movement the next person in line respects—. And somewhere the record remains: the machine notes your passage, so afterward someone can review how many people entered.

The capstone's gate is that turnstile. It reads the run's metrics, it decides against the thresholds (pass / fail), and it acts with an exit code the pipeline respects: 0 and the arms turn (the deploy advances), 1 and they lock (the deploy is blocked). There's no human interpreting the p95; the number makes it turn or lock. And the record is the results.json it exports: the note of what that run measured, to review later, attach to the build, or compare with future runs. Read, decide, act, record —all four things, in one command—.

The complete gate (the executable run)

This is the heart of loadtest.py: after running the stages and aggregating the metrics, it evaluates the three thresholds, exports the report, and exits with the verdict's code. It's M5's evaluate plus M7's export, joined:

# Fragment of loadtest.py — the gate: evaluate, export, exit with a code.
# (The SLO comes from lesson 4; the aggregate metrics, from lesson 5.)
agg = summarize(whole, whole_wall)              # whole-run metrics
rows = [
    ("http_req_duration: p(95) < 200ms", agg["p95"] < P95_LIMIT_MS),
    ("http_req_failed:   rate < 1.00%",  agg["error_rate"] < ERROR_LIMIT),
    ("checks:            rate > 99.00%", agg["checks_rate"] > CHECKS_MIN),
]
passed = all(ok for _, ok in rows)

# EXPORT (M7): the run's notebook, for the artifact and the comparisons.
report = {"scenario": "quote->book", "quote_path": QUOTE_PATH,
          "slo": {"p95_ms": P95_LIMIT_MS, "error_rate": ERROR_LIMIT, "checks_rate": CHECKS_MIN},
          "aggregate": agg, "stages": per_stage, "passed": passed}
with open(OUT_JSON, "w") as f:
    json.dump(report, f, indent=2)

# EXIT WITH A CODE (M5): the language the pipeline understands.
if passed:
    print("GATE: PASS  (exit code 0)"); sys.exit(0)   # the arms turn
else:
    print("GATE: FAIL  (exit code 1)"); sys.exit(1)   # they lock

Notice the order: export first, exit after. The results.json is written no matter what —you need it whether the run passed or failed, because the record of a failed run is exactly the one you want to review—. And the sys.exit is last: it turns the boolean passed into the number the pipeline reads. True → 0, False → 1.

The green gate (healthy build)

We run the complete test against /quote, the fast canonical endpoint —the healthy build—:

What to expect — the whole run's p95 stays well below the SLO; the three thresholds pass; the JSON is exported and it exits with code 0:

$ python3.14 loadtest.py http://127.0.0.1:PORT /quote green.json
LOAD TEST — quote->book scenario against /quote
profile: smoke(5) -> load(20) -> stress(80) VUs
--------------------------------------------------------------------------
stage     VUs    reqs      RPS      p50      p95      p99   error   checks
--------------------------------------------------------------------------
smoke       5   23452   5861.3     0.79     1.19     1.43   0.00%  100.00%
load       20   26240   5243.1     3.61     5.97     7.34   0.00%  100.00%
stress     80   31340   5208.5    14.54    25.34    31.27   0.00%  100.00%
--------------------------------------------------------------------------

THRESHOLDS (evaluated over the whole run — like k6)
--------------------------------------------------------------------------
THRESHOLD                             MEASURED              RESULT
http_req_duration: p(95) < 200ms      p(95) = 21.37ms       PASS
http_req_failed:   rate < 1.00%       rate  = 0.00%         PASS
checks:            rate > 99.00%      rate  = 100.00%       PASS
--------------------------------------------------------------------------
metrics exported -> green.json
GATE: PASS  (exit code 0)
$ echo $?
0

With the healthy build, even the stress stage (80 VUs) gives a p95 of 25.34 ms, and the aggregate 21.37 ms —far from the SLO's 200 ms—. The three thresholds pass, the gate exits with 0, and the pipeline has a green light to deploy.

The red gate (heavy pricing engine)

Now the same test against /quote_cpu, the pricing engine that does CPU work:

What to expect — under the stress the p95 crosses the SLO; the latency threshold fails; the JSON is exported all the same and it exits with code 1:

$ python3.14 loadtest.py http://127.0.0.1:PORT /quote_cpu red.json
LOAD TEST — quote->book scenario against /quote_cpu
profile: smoke(5) -> load(20) -> stress(80) VUs
--------------------------------------------------------------------------
stage     VUs    reqs      RPS      p50      p95      p99   error   checks
--------------------------------------------------------------------------
smoke       5    2336    583.3     9.07    14.15    17.07   0.00%  100.00%
load       20    2920    582.2    34.42    57.98    62.07   0.00%  100.00%
stress     80    3526    584.3   135.97   239.77   254.80   0.00%  100.00%
--------------------------------------------------------------------------

THRESHOLDS (evaluated over the whole run — like k6)
--------------------------------------------------------------------------
THRESHOLD                             MEASURED              RESULT
http_req_duration: p(95) < 200ms      p(95) = 221.89ms      FAIL
http_req_failed:   rate < 1.00%       rate  = 0.00%         PASS
checks:            rate > 99.00%      rate  = 100.00%       PASS
--------------------------------------------------------------------------
metrics exported -> red.json
GATE: FAIL  (exit code 1)
$ echo $?
1

Here's the complete degradation, caught. Under smoke (p95 14.15) and load (p95 57.98) the system meets the SLO; it's the stress stage (p95 239.77) that breaks it, and the aggregate p95 (221.89 ms) crosses the 200 ms. The latency threshold fails, the gate exits with 1, and the pipeline has a red light. Notice that —just as in M5— the exit code is 1 even though two of the three thresholds passed: one broken rule is enough for the FAIL, and the FAIL is enough for the 1.

The exported JSON (the run's notebook)

Each run wrote its results.json. This is the red run's —the record a pipeline would upload as an artifact and that a regression check could compare (M7)—:

$ cat red.json
{
  "scenario": "quote->book",
  "quote_path": "/quote_cpu",
  "slo": {
    "p95_ms": 200.0,
    "error_rate": 0.01,
    "checks_rate": 0.99
  },
  "aggregate": {
    "reqs": 8782,
    "rps": 583.5,
    "p50": 41.24,
    "p95": 221.89,
    "p99": 249.04,
    "error_rate": 0.0,
    "checks_rate": 1.0
  },
  "stages": [
    { "stage": "smoke",  "vus": 5,  "reqs": 2336, "p95": 14.15,  "error_rate": 0.0, "checks_rate": 1.0 },
    { "stage": "load",   "vus": 20, "reqs": 2920, "p95": 57.98,  "error_rate": 0.0, "checks_rate": 1.0 },
    { "stage": "stress", "vus": 80, "reqs": 3526, "p95": 239.77, "error_rate": 0.0, "checks_rate": 1.0 }
  ],
  "passed": false
}

(Shown abbreviated; the real file also carries rps, p50, and p99 per stage, and the durations.) The JSON keeps everything you need after the terminal closes: the SLO it was judged against, the aggregate and per-stage metrics, and the verdict ("passed": false). It's the run's notebook —the chef writing down today's dish to be able to compare it with tomorrow's (M7)—. Without exporting, the run evaporates when the window closes; with the JSON, a record remains that gets attached to the build, compared with history, and investigated.

The exit code that authorizes or blocks the deploy (executed)

The exit code only matters if something reads it. Without a whole pipeline, the shell itself has the operators CI uses internally: A && B runs B only if A succeeded (code 0); A || B runs B only if A failed. Let's simulate "run the test, and only if it passes, authorize the deploy":

What to expect — with the healthy build the && lets it pass (deploy authorized); with the heavy engine the && short-circuits and the || triggers the block. Real output:

$ python3.14 loadtest.py http://127.0.0.1:PORT /quote green.json > /dev/null \
    && echo "CI: deploy authorized" || echo "CI: deploy BLOCKED"
CI: deploy authorized

$ python3.14 loadtest.py http://127.0.0.1:PORT /quote_cpu red.json > /dev/null \
    && echo "CI: deploy authorized" || echo "CI: deploy BLOCKED"
CI: deploy BLOCKED

End to end: the healthy build passed the gate → code 0 → the && authorized the deploy; the heavy engine failed the gate → code 1 → the && short-circuited and the deploy was blocked. No human looked at a p95. The exit code —the turnstile— did all the work. A real CI pipeline does exactly this, only the "next step" is the real deploy; the load.yml that expresses it is lesson 7.

Common mistakes

Always exiting with 0 (the decorative gate). What happens: the script prints "FAIL" in red but finishes normally, with code 0. The pipeline sees the 0 and deploys anyway. Why it happens: communicating to the human (the print) gets confused with communicating to the machine (the sys.exit). How to detect it: run the gate in a case that should fail and do echo $?; if it says 0, the gate is decorative. How to fix it: the failure branch must call sys.exit(1) (or ≠ 0). The print is for the human; the exit code is for the pipeline (M5).

Exporting only if the run passes. What happens: the JSON is written inside the success branch, so failed runs leave no record. Why it happens: it seems that "only the good is worth saving." How to detect it: if you don't have the results.json of the run that failed, you can't investigate why it failed. How to fix it: export before exiting, no matter what. The record of a failed run is the most valuable —it's the one you review to find the bottleneck—. In the capstone's gate, the json.dump goes before the sys.exit, always.

Believing the shell's && is the CI. What happens: someone sees the chain ... && echo authorized || echo blocked and believes that's "the pipeline." Why it happens: the chain imitates the CI's logic. How to detect it: there's no git/gh here; it's a shell simulation of what a pipeline does. How to fix it: understand that the shell demonstrates the mechanism (an exit code that short-circuits or lets the next step pass); lesson 7's load.yml is the industrial form of the same mechanism, as content. Both use the same language: the exit code.

Exercises

Exercise 1 — Predict the $? and the deploy. For each run, say what echo $? prints and whether the chain && deploy || blocked deploys. (a) /quote with the three thresholds green. (b) /quote_cpu with the p95 red and the other two green. (c) A run with error = 3% (over the 1%) but p95 and checks green.

See solution

(a) $? = 0, deploys (the three PASS → gate PASS → 0 → the && lets it pass).
(b) $? = 1, doesn't deploy (one broken rule is enough for the FAIL → 1 → the && short-circuits, the || blocks).
(c) $? = 1, doesn't deploy (the http_req_failed: rate<0.01 threshold fails with 3% error, even though latency and checks pass; a single broken rule → FAIL → 1 → block).

In all three, the gate looks at the three rules and one broken is enough for the 1.

Exercise 2 — Why export before exiting? The gate writes the results.json right before the sys.exit. (a) What would happen if the json.dump were inside the if passed: (only the success branch)? (b) Why is the JSON of a failed run especially valuable?

See solution

(a) The runs that fail would leave no file: the failure branch's sys.exit(1) would execute without having written anything. Exactly the runs you most want to investigate (the red ones) would be lost.
(b) Because the JSON of a failed run is the evidence of the degradation: it keeps the p95 that crossed, in which stage, with what load, against what SLO. It's what you upload to the build as an artifact, what you share with the team, and what you compare against the baseline to locate when the problem started. A green run confirms all is well; a red one documents what broke —and that documentation only exists if you exported before exiting—.

Exercise 3 — Chain gate and real deploy. Write a shell line that runs the gate against /quote_cpu and, only if it passes, runs a ./deploy.sh (fictional); if the gate fails, that it prints "deploy blocked by performance" without deploying. What operator expresses "only if it passes"?

See solution

python3.14 loadtest.py "http://127.0.0.1:$(cat PORT)" /quote_cpu results.json \
    && ./deploy.sh \
    || echo "deploy blocked by performance"

The && operator expresses "only if it passes": it runs ./deploy.sh only if the gate exited with code 0. Since the run against /quote_cpu fails the p95 threshold and exits with 1, the && short-circuits (doesn't deploy) and the || triggers the block message. It's the same logic a CI pipeline applies between the load-test step and the deploy one —lesson 7's load.yml writes it in GitHub Actions syntax—.

Summary and next step

In this lesson you closed the complete gate: a run that measures, evaluates against the thresholds, exports the result to results.json, and exits with an exit code (0 = PASS, 1 = FAIL) —read, decide, act, record, like the subway turnstile—. You ran it in its two faces: green against the healthy build /quote (aggregate p95 21.37 ms, exit 0) and red against the heavy engine /quote_cpu (the stress takes the p95 to 239.77 ms, aggregate 221.89 ms, exit 1). You saw the exported JSON of the red run —the notebook that keeps the SLO, the metrics, and the verdict—, and checked with the chain && deploy || blocked that the exit code authorizes or blocks the deploy without anyone looking at a number.

You joined module 5 (the exit code) and module 7 (exporting). Before moving on you should be able to: explain why you export before exiting; distinguish a real gate from a decorative one; and chain gate and deploy with &&. What comes next, in lesson 7, is to put this gate in a pipeline: the .github/workflows/load.yml as content, where the threshold is the gate that blocks the deploy —and when it's worth triggering it—.

Resources

k6 — Results output (--out json) — how k6 exports a run's metrics to a file; the industrial equivalent of the results.json the gate exports. The source of the exporting content.
sys.exit — Python documentation — the mechanism with which the gate sets its exit code (0 = PASS, 1 = FAIL); the language the pipeline reads.
json — Python documentation — how the gate serializes the report to results.json with json.dump; the run's notebook.
GitHub Actions — Exit codes and step status — how a step that exits with code ≠ 0 fails the job and stops the pipeline; the other end of the && chain. The complete YAML is lesson 7.

← Previous5. The run and the metrics (p95/RPS/error)Next →7. The CI `load.yml` as content