4. The nonzero exit code that fails CI

Overview

A verdict no one obeys is useless. In lessons 2 and 3 we built the judgment —the threshold that says pass or fail—, but a True/False that stays inside a program stops no deploy. This lesson connects that verdict with the real world through the humblest and most important piece of all automation: the exit code. When a program finishes, it leaves the operating system an integer: 0 means "all good"; any other number means "I failed". That number is the universal language a program speaks to a CI pipeline with. You'll see how sys.exit produces that number in Python, how the shell reads it in the $? variable, how your gate uses it to exit with 0 (pass) or 1 (fail) —actually executed—, and how k6 does exactly the same by exiting with code 99 when a threshold fails. By the end you'll understand why a broken threshold stops a deploy without anyone approving anything: because a nonzero number is all a pipeline needs to turn the build red.

Connection to the module: lessons 2 and 3 produced the verdict (in executed Python and content k6); this one gives it teeth. It's the row left pending in lesson 3's mapping table: the consequence. Here the pass/fail stops being an on-screen decoration and becomes an event a system respects. The generalization of this —that this mechanism is the same one behind coverage gates— is lesson 6; the complete CI pipeline, with its YAML, is module 7. Here we build the gate's mechanism; M7 installs it in the whole pipeline.

The customs inspector's stamp

When you go through customs, the officer checks your papers and does one of two things: they put an "approved" stamp on you and let you continue, or they hold you and you don't advance. What matters isn't what the officer thinks of your papers —that stays in their head—; what matters is the stamp, because the stamp is what the next gate reads. The gate doesn't re-check your papers: it looks at whether you have the stamp. Without a stamp, you don't pass, no matter how good your reasons were.

The exit code is that stamp. Your gate checks the metrics and forms a verdict —that's like the officer thinking—, but what the pipeline reads isn't the internal verdict: it's the number the program leaves when it finishes. 0 is the approved stamp; any other number is "held." The next stage of the pipeline (the deploy) doesn't look at your latencies again: it looks at the previous step's exit code. If it's 0, it continues; if not, it stops. That's why all quality automation rests on this tiny number: it's the stamp the pipeline's gates know how to read.

sys.exit and $?: the number a program leaves

In Python, a program finishes with an exit code, and sys.exit(n) sets it explicitly. The convention is universal across Unix:

• sys.exit(0) — success. "I finished well." It's also what happens if the program ends without calling sys.exit (the default code is 0).
• sys.exit(1) (or any nonzero number) — failure. "Something went wrong."

The shell reads that number in the special variable $? (the exit code of the last command). Let's see it with the smallest possible example: a program that compares a measured p95 against a limit and exits with the corresponding code.

"""Minimal demonstration of sys.exit and the exit code."""
import sys

p95 = 246.96      # measured (heavy load)
limit = 200.0     # the threshold (SLO)

if p95 < limit:
    print(f"p(95)={p95:.2f}ms < {limit:.0f}ms  -> PASS")
    sys.exit(0)   # 0 = success: the pipeline continues
else:
    print(f"p(95)={p95:.2f}ms >= {limit:.0f}ms -> FAIL")
    sys.exit(1)   # !=0 = failure: the pipeline stops

What to expect — since 246.96 isn't less than 200, it enters the else, prints FAIL, and exits with 1; the shell, on asking $?, sees that 1. This is real output:

$ python3.14 exit_demo.py
p(95)=246.96ms >= 200ms -> FAIL
$ echo "the shell saw: \$? = $?"
the shell saw: $? = 1

There's the complete link, executed: the program judged (246.96 ≥ 200 → FAIL), translated it to a number (sys.exit(1)), and the shell received it ($? = 1). That 1 is the "held" stamp. If the p95 had been 150, it would have entered the if, printed PASS, and exited with 0 —the "approved" stamp, $? = 0—. The whole lesson comes down to that translation: verdict → number → something the pipeline reads.

The complete gate, exiting with its real code

Now the real gate, not the toy. threshold_gate.py is lesson 2's evaluate_thresholds wrapped in a main that measures (launches load against Reservo, gathers latencies, counts errors and checks) and then exits with the code corresponding to the verdict:

def main():
    latencies, error_rate, checks_rate = measure()   # measures against Reservo (real)
    print(f"# {PATH}  |  {TOTAL_REQUESTS} requests, concurrency {CONCURRENCY}")
    ok = evaluate_thresholds(latencies, error_rate, checks_rate, P95_LIMIT_MS)
    if ok:
        print("GATE: PASS  (exit code 0)")
        sys.exit(0)     # the True verdict -> code 0 -> the pipeline continues
    else:
        print("GATE: FAIL  (exit code 1)")
        sys.exit(1)     # the False verdict -> code 1 -> the pipeline stops

The last piece of the contract: the boolean ok that evaluate_thresholds returns becomes the exit code. True → sys.exit(0), False → sys.exit(1). Let's run the usual two runs and look, now yes, at the $? each one leaves. First the one that passes (light load on /quote, the fast canonical endpoint):

What to expect — the three rules pass, the gate prints PASS, and the shell receives $? = 0:

$ python3.14 threshold_gate.py http://127.0.0.1:PORT /quote 400 20 200
# /quote  |  400 requests, concurrency 20
THRESHOLD                              MEASURED           RESULT
----------------------------------------------------------------------
http_req_duration: p(95) < 200ms       p(95) = 8.33ms     PASS
http_req_failed:   rate < 1.00%        rate  = 0.00%      PASS
checks:            rate > 99.00%       rate  = 100.00%    PASS
----------------------------------------------------------------------
GATE: PASS  (exit code 0)
$ echo $?
0

And now the one that fails (heavy load on /quote_cpu):

What to expect — the p95 crosses the limit, the gate prints FAIL, and the shell receives $? = 1:

$ python3.14 threshold_gate.py http://127.0.0.1:PORT /quote_cpu 2000 120 200
# /quote_cpu  |  2000 requests, concurrency 120
THRESHOLD                              MEASURED           RESULT
----------------------------------------------------------------------
http_req_duration: p(95) < 200ms       p(95) = 246.96ms   FAIL
http_req_failed:   rate < 1.00%        rate  = 0.00%      PASS
checks:            rate > 99.00%       rate  = 100.00%    PASS
----------------------------------------------------------------------
GATE: FAIL  (exit code 1)
$ echo $?
1

Two verdicts, two real exit codes: 0 when the app meets the SLO, 1 when it doesn't. That number is the only thing the pipeline will look at. Notice that the $? = 1 of the second run came out even though two of the three thresholds passed —a broken rule is enough for the False, and the False is enough for the 1—.

How a number stops a deploy

A CI pipeline (GitHub Actions, GitLab CI, Jenkins...) runs a series of steps, and it has a golden rule: if a step exits with a nonzero code, the step fails, and (by default) the pipeline stops there. It doesn't run the following steps. Since the deploy is usually a step after the tests, a test step that exits with 1 prevents the deploy from happening. That's the whole mechanism: there's no magic, it's the exit-code rule applied to a chain of steps.

We can see it without a whole pipeline, with the shell itself, which has the same operators CI systems use inside. A && B runs B only if A succeeded (code 0); A || B runs B only if A failed (code ≠ 0). Let's simulate "run the gate, and only if it passes, authorize the deploy":

What to expect — when the gate passes, the && lets the next step run (deploy authorized); when it fails, the && short-circuits and the || fires the block message. This is real output:

$ # The gate PASSES: the next step runs
$ python3.14 threshold_gate.py http://127.0.0.1:PORT /quote 300 10 200 > /dev/null \
    && echo "CI: deploy authorized" || echo "CI: deploy BLOCKED"
CI: deploy authorized

$ # The gate FAILS: the && short-circuits, CI stops
$ python3.14 threshold_gate.py http://127.0.0.1:PORT /quote_cpu 2000 120 200 > /dev/null \
    && echo "CI: deploy authorized" || echo "CI: deploy BLOCKED"
CI: deploy BLOCKED

There you have it, end to end: the light load passed the gate → code 0 → the && authorized the deploy; the heavy load failed the gate → code 1 → the && short-circuited and the deploy was blocked. No human looked at a p95. The stamp did all the work. A real CI pipeline does exactly this, only the "next step" is the real deploy and the "block" is that the build goes red and no one can merge. (The concrete YAML that expresses this chain in GitHub Actions is module 7.)

k6 does the same: code 99

k6 follows this same convention, with a detail worth knowing: when one or more thresholds fail, k6 run exits with code 99 (internally it calls it ThresholdsHaveFailed). It's not just any 1: k6 reserves 99 specifically for "the measurements were fine, the test ran completely, but it didn't meet the limits" —it distinguishes it from a 0 (everything passed), a script error, or a setup failure—. For the pipeline the fine distinction doesn't matter: 99 is nonzero, so the step fails and the deploy stops, exactly like your sys.exit(1).

# CONTENT (not run here): this is how k6 would behave. See grafana.com/docs/k6
$ k6 run quote_test.js
     ...
     ✗ http_req_duration..............: p(95)=246.96ms (threshold: p(95)<200)
     ✓ http_req_failed................: 0.00%
     ✓ checks.........................: 100.00%

$ echo $?
99

Lesson 3's red ✗ and this 99 are the two faces of the same event: the threshold failed and because of that the process exited with a nonzero code. That number is what makes a k6 run in your CI pipeline turn the build red. Your Python gate uses 1 and k6 uses 99; both are "nonzero," and that's the only thing the pipeline needs. That a broken performance threshold fails a build is, mechanically, identical to a broken unit test failing it: both exit with code ≠ 0.

Common mistakes

Printing "FAIL" but exiting with code 0. What happens: a script detects the failure, prints a nice red error message... and finishes normally, with code 0. The pipeline sees the 0, thinks everything went well, and deploys anyway. Why it happens: telling the human something failed (the print) gets confused with telling the machine something failed (the sys.exit). How to detect it: run the script and do echo $?; if it says 0 when it should fail, the gate is decorative. How to fix it: make sure the failure branch calls sys.exit(1) (or any ≠ 0). The print is for the human; the exit code is for the pipeline. (This bug is real and silent: the gate "looks" like it works but never blocks anything.)

Letting an uncontrolled exception mask the verdict. What happens: the gate blows up with an exception (e.g. it couldn't connect to the server) and exits with code 1 —the same one it uses for "broken threshold"—, and it's unclear whether the performance failed or the measurement failed. Why it happens: the same code is used for two different things. How to detect it: a FAIL without the thresholds table printed is usually a measurement error, not a broken threshold. How to fix it: reserve sys.exit(1) for "I measured well and a threshold failed"; let infrastructure errors exit with another code (or at least a different message), the way k6 distinguishes the 99 (thresholds) from its other codes.

Assuming the pipeline "knows" what your output means. What happens: someone expects CI to understand a message like "degraded performance" in the text output. Why it happens: the pipeline gets anthropomorphized. How to detect it: if your gate doesn't set an exit code and trusts CI to "read" the text, it won't work. How to fix it: the pipeline doesn't read your text; it reads your exit code. Communicate with it through the only channel it understands: 0 or ≠ 0.

Exercises

Exercise 1 — Predict the $?. For each run, say what echo $? prints right after. (a) The gate with the three rules green. (b) The gate with the p95 red and the other two green. (c) The exit_demo.py with p95 = 150, limit = 200. (d) A k6 run whose http_req_duration doesn't meet the threshold.

Exercise 2 — Fix the decorative gate. This gate prints the failure but the pipeline never blocks it. Why, and how is it fixed?

if not all_pass:
    print("GATE: FAIL")
print("done")

if not all_pass:
    print("GATE: FAIL")
    sys.exit(1)     # <-- this is what blocks the pipeline
print("GATE: PASS")

Exercise 3 — Chain the gate and the deploy. Write a shell line that runs python3.14 threshold_gate.py ... and, only if it passes, runs a ./deploy.sh (fictional); and if the gate fails, prints "deploy blocked" without deploying. What operator expresses the "only if it passes" relationship?

python3.14 threshold_gate.py http://127.0.0.1:PORT /quote 300 10 200 \
    && ./deploy.sh \
    || echo "deploy blocked"

Summary and next step

The exit code is what gives the verdict teeth. When a program finishes it leaves a number: 0 = success, any other = failure, and that number is the universal language a program speaks to a CI pipeline with. In Python you set it with sys.exit(0) / sys.exit(1) and the shell reads it in $?. Your gate turns the verdict's boolean into that code —True → 0, False → 1— and you saw it executed: $? = 0 under light load, $? = 1 under heavy load (even though only one rule failed). A pipeline stops at the first step that exits with code ≠ 0, so a gate that exits with 1 prevents the deploy —you confirmed it with the shell's && deploy || blocked chain, which authorized under light load and blocked under heavy—.

k6 does the same: it exits with code 99 (ThresholdsHaveFailed) when a threshold fails. Python's 1 and k6's 99 are both "nonzero," and that's all the pipeline needs to turn the build red. That's how "performance" becomes as binary and automatable as a unit test: a latency regression fails the build just like a broken test.

Before moving on you should be able to: explain what an exit code is and what $? and the pipeline read; distinguish a real gate (exits with ≠ 0 on failure) from a decorative one (only prints); and chain gate and deploy with &&. What comes next, in lesson 5, is the question we've dodged: where does the limit's number come from? Why 200 ms and not 150 or 500? The answer isn't technical but a business one —the SLO/SLA— and choosing the limit badly makes this whole precise mechanism useless. A perfect gate with an arbitrary limit protects nothing.

Resources

• sys.exit — Python documentation — how a Python program sets its exit code (0 = success, ≠ 0 = failure). The mechanism the gate speaks to the pipeline with.
• k6 — Thresholds (the exit code) — confirms that a broken threshold makes k6 run exit with a nonzero code, which fails the CI step.
• k6 — Error codes (code 99) — k6's source code where ThresholdsHaveFailed = 99 is defined, the exact code k6 returns when a threshold fails. The source of the "99."
• GitHub Actions — Exit codes and step status — how a step that exits with code ≠ 0 fails the job and stops the pipeline. The other end of the stamp. The complete YAML is module 7.