2. What a quality gate is

Description

In the previous lesson you saw a gate break the build and compared it to a smoke detector. Now we're going to take the concept apart calmly, because understanding what exactly a gate is —before memorizing any command— is what later lets you design your own gates, know which to set and which not, and not confuse the tool with a fetish. A quality gate isn't a pytest flag or a line of YAML; it's an idea those tools implement, and the idea is simpler and more powerful than it seems.

By the end of this lesson you'll be able to define a quality gate by its three parts —metric, threshold, and consequence— and explain why all three are necessary. You'll understand the difference between a pipeline that reports (tells you the number) and one that imposes (breaks the build if the number crosses the line), and why that difference changes everything. You'll see that the exit code is the language a gate speaks to CI —the only language CI listens to— and that's why every gate, no matter what it measures, ends up translating into a 0 or a non-zero number. And you'll see that coverage is just one possible metric: any number with a threshold —number of failures, missing tests, binary size— can be turned into a gate with the same recipe.

Connection to the module: this lesson is the theory that holds up the following five. Lesson 1 gave you the intuition (thermometer vs detector); this one makes it precise (the three parts, the exit code). Lesson 3 takes this definition and executes it with --cov-fail-under, seeing the three parts in a real command. Lesson 4 plays with the threshold —what happens when the floor is fixed and the metric drops without crossing it—. Lesson 5 changes the metric —from coverage to "do the smoke tests pass"— demonstrating that the recipe is the same. Lessons 6 and 7 discuss when the consequence helps and when it harms. Everything that comes is variations on the three parts you define here. That's why it's worth installing them well before running one more command.

The subway turnstile, not the "please pay" sign

Imagine a subway station entrance with two possible designs. The first is a sign: "Please pay your fare before passing", with an honesty box beside it. The sign communicates the rule with full clarity. It even measures: if you put a camera, you know how many paid and how many didn't. But it imposes nothing. It depends entirely on each person's good will. Whoever wants to sneak through, sneaks through; the sign doesn't stop them, it just watches them pass.

The second design is a turnstile. It has the same rule —you have to pay to pass— but implements it differently: a physical barrier that doesn't open if you didn't insert your ticket. It doesn't appeal to your good will; it makes sneaking through impossible (or at least, deliberate and hard). The turnstile turns "please pay" from a suggestion into a condition. The rule stopped depending on each person deciding to comply.

A quality gate is the turnstile; a coverage report is the sign. Both "know" the rule —"the code must be 80% covered"—. But the report communicates it and trusts that someone will enforce it; the gate imposes it with a barrier that doesn't open —the red build— if the rule isn't met. And notice the turnstile's anatomy, because it's exactly a gate's: there's a measurement (did you insert the ticket?), a threshold (the ticket has to be valid), and an automatic consequence (the barrier opens or not). None of the three is superfluous. A turnstile without measurement wouldn't know what to do; without a threshold it would let any paper through; without the automatic barrier it would be, again, just a sign.

A quality gate implements a rule with an automatic barrier, not with an appeal to good will. Like the subway turnstile, it has a measurement, a threshold, and a consequence that runs on its own —the barrier that doesn't open—. A report is the sign: it communicates the rule but trusts that someone will enforce it.

The three parts, one by one

Let's take the gate apart into its three pieces, because each module lesson moves one of them.

The metric. It's the number the gate monitors. In a coverage gate, the metric is "what percentage of the code's lines the suite ran". But it could be something else: "how many tests failed" (metric: number of failures), "how many tests marked as critical passed", "how many seconds the suite took", "how many linter warnings there are". The metric has to be something automatically measurable —a machine has to be able to compute it without human judgment—, because the whole point of the gate is that it's imposed without anyone looking. "Is the code elegant?" can't be a gate metric; "is the coverage ≥ 80%?" can.

The threshold. It's the line the metric mustn't cross. In --cov-fail-under=80, the threshold is 80. The threshold is a decision, not a datum: no one tells you it has to be 80; you choose it, and that choice is the most delicate part of the whole gate. Too low, the gate protects from nothing (a threshold of 10% lets almost anything through). Too high, the gate blocks legitimate work and pushes toward tricks to comply with it (100% as a fetish, lesson 7). The correct threshold is a topic for lesson 6, but keep this from now: the threshold is the part of the gate where judgment lives, and choosing it wrong is the most common way a good gate becomes a hindrance.

The consequence. It's what happens when the metric crosses the threshold. In a CI gate, the consequence is always the same in form: the command finishes with a non-zero exit code, which paints the step red and stops the pipeline. The consequence is what separates a gate from a report: without it, you measure but don't impose. And it's automatic by definition —not "someone decides to block the merge", but "the build breaks on its own"—. That automaticity is the tool's entire value: the policy is declared once and applied on every push, without depending on anyone watching.

Put the three together and you have the complete sentence: "if coverage (metric) drops below 80% (threshold), break the build (consequence)". Change the metric and you have a different gate —"if any smoke test (metric) fails, break the build"—. Change the threshold and you adjust its severity. Change the consequence from "break" to "just warn" and —careful— you stop having a gate and go back to having a sign. The whole module is playing with these three knobs.

The exit code: the only language CI listens to

Here's an idea you already grazed in module 2 and that this module makes central: CI doesn't read text, it reads exit codes. When a workflow step runs a command —pytest, coverage report, whatever— the runner doesn't interpret the pretty report the command prints. It looks at a single thing: the integer the command finished with. By universal Unix convention, 0 means "all good" and any other number means "something failed". The runner translates that number to a color: 0 → green, non-zero → red. And a red step stops the job, which is what blocks the merge.

That's why every quality gate, no matter what metric it monitors, ends up doing the same: turning its verdict into an exit code. --cov-fail-under=80 doesn't "warn" that coverage is low; it makes pytest finish with exit code 1 when it is. That's all the magic. The gate is, in essence, a translator: it takes a metric (65.38%), compares it with a threshold (80), and emits the language CI understands (exit code 1 = red). The coverage report you see printed is for you, the human; the exit code is for the machine, CI. Confusing them —believing that because you see the report the gate is working— is lesson 1's mistake: the gate bites only if it emits the correct exit code, and that has to be verified.

A nuance you'll see in lesson 3 and worth anticipating: different tools use different numbers for "failure". pytest (and therefore pytest-cov) uses exit code 1 for "there were failures or the gate wasn't met". The coverage CLI uses exit code 2 specifically for "coverage ended up below fail_under". For CI it doesn't matter which it is —any non-zero number is red—, but for you it matters to know how to read them, because a 2 tells you "it was the coverage gate" and a 1 from pytest could be "it was the gate or a test that failed". The exit code isn't just green/red; it's a little message about what went wrong.

Worked example: the same metric, report vs gate

Let's see, with the Reservo suite, the difference between reporting and imposing, over exactly the same number. First, reporting: we run coverage without any gate, just asking for the report. The suite is in the incomplete state (with cancel_with_refund untested).

python -m pytest --cov=reservo > /dev/null 2>&1; echo "exit code: $?"

What to expect (measured for real on Python 3.14.0):

exit code: 0

Look carefully: exit code 0, green. We asked for the coverage report (--cov=reservo) but didn't put a threshold, so there's no gate. The coverage was 65.38% —low—, but since we didn't ask pytest to impose anything, it finished at 0. It's the sign: it measured, even printed the number, but didn't impose. A CI with this command would be green with 65% coverage, without blinking.

Now, imposing: the same command, the same suite, the same 65.38%, but adding the gate:

python -m pytest --cov=reservo --cov-fail-under=80 > /dev/null 2>&1; echo "exit code: $?"

What to expect (measured for real):

exit code: 1

Exit code 1, red. Nothing changed except that we added --cov-fail-under=80. The coverage is still 65.38%; what changed is that now there's a threshold and a consequence: since 65 < 80, pytest translated that fact to CI's language —exit code 1—. The turnstile appeared. The same number the report showed with indifference, the gate turns into a broken build. That's, in a single comparison, the whole difference between measuring and imposing: it's not in the metric (identical), it's in whether there's a threshold that translates it to an exit code.

Any metric can be a gate

Coverage is the most famous gate, but it's not special. The recipe —metric + threshold + consequence via exit code— applies to any number a machine can compute. It's worth seeing the variety, because it frees you from thinking "quality gate" is a synonym for "coverage":

• Passing-tests gate. Metric: did any tests fail? Threshold: zero failures. Consequence: pytest already implements it on its own —if a test fails, exit code 1—. It's the gate you've had since module 2; the heart of the pipeline is, literally, a "zero failures" gate.
• Marker gate. Metric: did the smoke tests pass? Threshold: all. Consequence: pytest -m smoke finishes at 1 if any fails. It's lesson 5.
• Coverage gate. Metric: % of lines executed. Threshold: 80%. Consequence: --cov-fail-under=80. It's lesson 3.
• Gates outside this module. A linter that breaks the build if there are style errors (metric: number of violations; threshold: zero). A type check with mypy that fails if there are type errors. A security scan that blocks if it finds a vulnerable dependency. They're all the same idea with another metric.

Seeing this list has a purpose: when in lesson 5 we change from coverage to markers, it won't be a new topic, it'll be the same gate with another metric. And when in lesson 6 we discuss whether a gate is worth it, the question will always be the same —does this metric, with this threshold, protect something that really matters?— no matter what the metric is. You learned a recipe, not a command.

Common mistakes

Setting the consequence to "warn" instead of "break", and believing you have a gate. What happens: someone configures coverage to print a yellow warning when it's low, but the build stays green. They believe they set a gate; they set a flashier sign. Why it happens: a warning feels like a consequence, but if it doesn't change the exit code, CI ignores it and the merge proceeds. How to spot it: ask yourself "is the merge really blocked, or does just some text appear?". If the change enters anyway, there's no gate. How to fix it: a gate's consequence has to be a non-zero exit code that stops the pipeline; anything that doesn't break the build is measurement, not imposition. (There's a legitimate place for warning without breaking —when you're introducing a gate gradually— but then be honest: it's a transition sign, not a gate yet.)

Choosing the threshold without thinking, copying it from a tutorial. What happens: someone puts --cov-fail-under=100 because "more is better" or --cov-fail-under=50 because it came in an example, without connecting the number with their project. Why it happens: the threshold is the part that seems like a detail —just a number— when it's actually the gate's central decision. How to spot it: if you can't explain why your threshold is that and not five points more or less, you copied it. How to fix it: the threshold is where judgment lives (lesson 6). An honest starting point is "put it where you are today" —if your coverage is 84%, put the gate at 84 so as not to go backward— and raise it with a ratchet when you improve (lesson 4), instead of chasing an aspirational round number.

Reading the printed report as proof the gate works. What happens: someone sees the coverage report come out in the terminal and concludes "the gate is set", without verifying the exit code. Why it happens: the report is what's visible and striking; the exit code is invisible unless you ask for it with echo $?. How to spot it: as in the worked example, run the command with coverage below the threshold and confirm the exit code is non-zero. If it's 0, you have the report but not the gate. How to fix it: remember that the report is for the human and the exit code for CI; the gate lives in the exit code, not the text. Always verify it bites before trusting it.

Exercises

Exercise 1 — Name the three parts. For the gate pytest -m smoke (which runs only the tests marked smoke and breaks the build if any fails), identify its three parts: (a) the metric, (b) the threshold, (c) the consequence. Then say what you'd change to turn it into a gate that requires the whole suite to pass, not just the smoke.

See solution

The three parts of pytest -m smoke:

• (a) The metric: how many of the tests marked smoke failed? (or, said as a state: did all the smoke pass?).
• (b) The threshold: zero failures among the smoke —all must pass—.
• (c) The consequence: pytest -m smoke finishes with exit code 1 if any smoke fails, which paints the step red and blocks the merge.

To require the whole suite to pass, you change the metric: from "did the smoke pass?" to "did all the tests pass?". In practice it's removing the -m smoke filter and running bare pytest —metric: failures in the whole suite; threshold: zero; consequence: exit code 1 if any test fails—. Note you only touched one knob (the metric); the threshold (zero failures) and the consequence's form (exit code) are identical. That's the advantage of thinking in the three parts: to change a gate, you identify which knob to move.

Exercise 2 — Sign or turnstile. A team says: "We have strict quality control. CI computes the coverage, publishes it in a pull-request comment, and we have the policy that no one should merge with less than 80%." Without running anything, decide: is it a gate (turnstile) or a report (sign)? What's missing, if anything, for it to be a real gate?

See solution

It's a sign, not a turnstile, despite the word "strict". It has a metric (the computed coverage) and even a declared threshold (80%), but it's missing the piece that defines a gate: the automatic consequence. "We have the policy that no one should merge with less than 80%" is a rule that depends on each person complying and on reviewers watching for it —exactly the sign's "please pay"—. The day someone's in a hurry, or a reviewer doesn't notice coverage dropped, the merge with 70% enters without anything preventing it.

What's missing to make it a turnstile: for CI to break the build when coverage drops below 80 —a --cov-fail-under=80 whose non-zero exit code blocks the merge—, or a branch-protection rule that requires that green check before allowing the merge. The distinction is fine but total: publishing the number and trusting discipline is measuring; making the merge impossible below the threshold is imposing. The word "policy" is the clue: a policy a human can skip is a sign; one the machine imposes is a gate.

Exercise 3 — The exit code tells the story. You run two commands on Reservo and note their exit codes. Command A: pytest --cov=reservo → exit 0. Command B: pytest --cov=reservo --cov-fail-under=80 → exit 1. Both measured the same coverage (65.38%). Explain why one gave 0 and the other 1, and what that tells you about where the "gate" lives.

See solution

Both commands measured the same —65.38%— and ran the same tests (all passing). The difference is entirely in the presence of the threshold:

• Command A (--cov=reservo, without --cov-fail-under) asked pytest to report the coverage, but gave it no threshold to impose. Without a threshold there's no line to cross, so pytest has no reason to fail: the tests passed → exit 0. It measured, printed the number, and stayed calm. It's the sign.
• Command B (adding --cov-fail-under=80) gave it a threshold. Now pytest compares 65.38% with 80, sees it's below, and translates that fact to a failure → exit 1. It's the turnstile.

What this reveals: the "gate" doesn't live in the metric (identical in A and B) or the tests (they passed in both). It lives in the threshold that translates to an exit code. Adding --cov-fail-under=80 changed nothing about what was measured; it changed what's done with the measurement —from showing it to imposing it—. That's why a gate is, at bottom, just a threshold connected to an exit code: remove the threshold and it's a report again; remove the translation to an exit code (leave it at "warning") and it's a sign again.

Summary and next step

In this lesson you took the quality gate apart into its three parts: a metric (a number a machine can compute, like coverage), a threshold (the line that mustn't be crossed, the decision where judgment lives), and an automatic consequence (breaking the build via a non-zero exit code). All three are necessary: without a metric there's nothing to measure, without a threshold there's no line, without a consequence you have a sign and not a turnstile. The difference between reporting and imposing isn't in what's measured, but in whether there's a threshold that translates the measurement to an exit code —you verified it by running the same 65.38% with and without a gate: exit 0 vs exit 1—.

It also became clear that the exit code is the only language CI listens to: every gate, whatever it measures, ends up emitting a 0 (green) or a non-zero number (red), and the printed report is for the human, not the machine. And you saw that coverage isn't special: the same recipe —metric + threshold + consequence— turns any measurable number (passing tests, markers, linter violations) into a gate. You learned a reusable recipe, not a stray command.

Before moving on you should be able to: name a gate's three parts and give an example of each; explain why the exit code —and not the report— is where the gate lives; distinguish a report from a gate by whether the consequence breaks the build or just warns; and recognize that coverage is one metric among many possible.

What's next, in lesson 3, is taking this definition and executing it for real, from start to finish: the complete cycle of --cov-fail-under on Reservo —the gate breaking the build with the incomplete suite, you adding the missing test, and the same gate going green—, plus its cousin the coverage CLI with its different exit code and the .coveragerc file that defines what's measured. You installed the theory; time to see it work in your terminal.

Resources

• pytest-cov: configuration options — where --cov-fail-under lives and how it combines with --cov. The direct reference for the coverage gate you'll execute in lesson 3.
• pytest exit codes — the official table of what each exit code means (0 = all passed, 1 = there were failures, and the rest). The language the gate speaks to CI; worth having at hand.
• Coverage.py: fail_under — the threshold in the coverage tool, with its note that it uses its own exit code (2) when it's not reached. We'll contrast it with pytest's in lesson 3.
• About required checks and branch protection — GitHub — how GitHub turns a red check into a truly blocked merge. It's the piece that makes your gate's consequence imposed at the repository level, not just the step.