6. The coverage gate

Description

So far your pipeline goes red for a single reason: a test fails. But there is a way for quality to erode without any test failing: for new code without tests to arrive. A forty-line function that nobody exercises does not break the suite —there is no test that goes red—, it simply is not tested, and the pipeline, blind to what is not measured, lets it pass green. The layer we stack here closes that blindness: the coverage gate, a threshold that breaks the build when the fraction of the code exercised by the tests drops below a minimum.

It is the layer of module 6, and it changes the nature of the pipeline: from "runs the tests" to "demands a floor of quality." You are going to see --cov-fail-under=N break the build for real —coverage 88%, gate at 95% that fails with exit 1, gate at 85% that passes with exit 0—, read the term-missing report that says which lines are missing, and face the most important and worst-understood decision of the topic: where to put the threshold. Because 100% is a fetish, and Reservo is going to demonstrate it to you with a coverage gap that is, in fact, perfectly covered code —by another matrix cell—.

By the end you will know how to put up a coverage gate that breaks the build, read what is missing, and —what separates whoever copies --cov-fail-under=100 from whoever thinks— decide a defensible threshold and distinguish a gap that has to be closed from one that has to be accepted.

Connection with the module: this is the fifth layer, and it leans on all the previous ones. The gate measures on the suite that runs (lesson 2) in a reproducible environment (lesson 3) —without pinned versions, today's 88% would not be comparable with tomorrow's—. And it connects backward with the matrix (lesson 4) in a way that this lesson reveals: the coverage gap of report_pages on your machine is the branch that another cell of the matrix does exercise. Coverage is a tool you already met as a local measure in the fundamentals guide; here it becomes a gate in CI, with the power to block a merge.

The inspector who only checks what they are shown

Think of a building inspector who certifies that a building is safe. They walk the floors the builder shows them —the lobby, three floors of offices—, check that everything is up to code, and sign. But the builder did not show them the basement, nor the attic's electrical installation. The inspector signs "safe" over what they saw, and stays silent about what they did not. If the basement has a fault, their signature does not cover it —they never checked it—, but the stamp says "approved" and everyone assumes they checked everything.

The problem is not the inspector; it is that nobody measures what fraction of the building they really checked. "Approved" without a coverage percentage is a signature that can hide an unchecked basement. The solution is to demand an audited minimum: "do not sign 'safe' if you checked less than 90% of the critical areas." Now the signature has a floor: if the builder hides the basement, the coverage drops below 90%, and the inspector cannot certify until they check it. The threshold turns "I checked what they showed me" into "I checked at least this fraction, or I do not sign."

Test coverage is what fraction of your code the tests exercise, and the coverage gate is that audited minimum. Without a gate, your green suite is the inspector signing over what they were shown: the tests that exist pass, but nothing tells you how much code went unchecked. With a gate —--cov-fail-under=85—, the pipeline refuses to certify green if the exercised fraction drops below the threshold. If someone adds a function without tests (the hidden basement), the coverage drops, the gate breaks the build, and the code does not go in until it is tested. The threshold turns "the tests pass" into "the tests pass and cover at least this."

The coverage gate is the inspector's audited minimum: it is not enough for the tests that exist to pass, at least a certain fraction of the code has to be tested, or the build does not pass. It turns new code without tests from invisible into something that stops the merge.

Measuring coverage: --cov and term-missing

pytest-cov (which is already in your requirements-dev.txt, dragging in coverage) measures the coverage while running the suite. The --cov=reservo flag tells it which package to measure; --cov-branch asks it to also measure branch coverage —not just whether each line executed, but whether each if/else took its two paths, the true and the false—, which is precisely what fills the Branch and BrPart columns of the table; and --cov-report=term-missing prints the report with the missing lines:

python -m pytest --cov=reservo --cov-branch --cov-report=term-missing

What to expect (real output of Reservo's suite on Python 3.14.0):

================================ tests coverage ================================
Name                  Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------------
reservo/__init__.py       0      0      0      0   100%
reservo/calendar.py       9      1      0      0    89%   15
reservo/models.py         9      0      0      0   100%
reservo/pricing.py        5      0      2      0   100%
reservo/refunds.py        7      0      4      0   100%
reservo/reports.py        7      2      2      1    67%   14-16
reservo/schedule.py      20      3      8      2    82%   15, 16->13, 40-41
-----------------------------------------------------------------
TOTAL                    57      6     16      3    88%
========================= 13 passed, 1 skipped in 0.03s =========================

Read the table column by column. Stmts is how many statements the file has; Miss, how many no test exercised; Branch and BrPart, the branches (the if/else) and how many were left half-taken; Cover, the percentage; Missing, the uncovered line numbers. The last row, TOTAL ... 88%, is the number the gate is going to watch.

Stop on two rows that tell the story of this lesson:

• reservo/schedule.py ... 82% ... 15, 16->13, 40-41 — line 15, the branch 16→13, and lines 40-41 are missing. The 40-41 are the cancel function, which no test of the base suite exercises (we left it without a test on purpose). The 16->13 is the branch that skips the canceled bookings. These are real gaps: Reservo code that no test touches. They are closed by writing tests.
• reservo/reports.py ... 67% ... 14-16 — this is the interesting gap, and it is not closed with a test. Lines 14-16 are the else branch of report_pages —the manual fallback for Python < 3.12—. On this machine, which runs 3.14, that branch never executes: Python enters through the if sys.version_info >= (3, 12) and does not even look at the else. It is dead code on 3.14. No test you write on 3.14 can cover those lines, because on 3.14 they are unreachable.

That second row is the key of the module, and we return to it in a moment. First, the gate.

The gate that breaks the build: --cov-fail-under

The --cov-fail-under=N flag turns the measurement into a verdict: if the total coverage is less than N, pytest returns a nonzero exit code —the build breaks—. Look at it with two thresholds, really executed.

Demanding gate, at 95%:

python -m pytest --cov=reservo --cov-branch --cov-fail-under=95

What to expect (real; the coverage is 88%, less than 95):

TOTAL                    57      6     16      3    88%
FAIL Required test coverage of 95% not reached. Total coverage: 87.67%

echo "exit code: $?"

exit code: 1

Exit code 1: the gate broke the build. All the tests passed —13 passed—, but the coverage (87.67%, which rounds to 88% in the table) did not reach the required 95%, so the pipeline goes red. On the runner, with branch protection, that red would block the merge. Notice the message: Required test coverage of 95% not reached —the gate tells you exactly why it failed—.

Reasonable gate, at 85%:

python -m pytest --cov=reservo --cov-branch --cov-fail-under=85

What to expect (real; 88% ≥ 85%):

TOTAL                    57      6     16      3    88%
Required test coverage of 85% reached. Total coverage: 87.67%

echo "exit code: $?"

exit code: 0

Exit code 0: the gate passed. The same suite, the same 88% coverage, but against a threshold of 85% the verdict is green. The difference between the red and the green was not in the code or the tests —they were identical—; it was in where you put the gate. And there is the decision that defines whether your gate helps or gets in the way.

Where to put the threshold: 100% is a fetish

The temptation is to put the gate at 100% —"let everything be tested"—. Reservo demonstrates why that is a mistake, not a virtue. Remember the reports.py row: its else branch (the fallback for Python < 3.12) is unreachable on 3.14. No test you write on your machine can cover those lines. A gate at 100% in the 3.14 cell would be impossible to satisfy —it would fail always, not because of badly tested code, but because of code that in this version does not exist—. Chasing 100% there improves nothing; it only forces you to cheat (exclude lines, write fake tests) to silence a badly calibrated gate.

And here is the twist that connects with the matrix (lesson 4): those lines of reports.py are covered —in the 3.11 cell—. There, sys.version_info >= (3, 12) is false, Python enters through the else, and the fallback branch executes and is measured as covered. The "gap" of 67% on 3.14 is not a quality gap; it is code from another version, which that version's cell exercises. The coverage, looked at cell by cell, will never be 100% in reports.py for a single version, and that is perfectly fine: between the matrix cells, both branches are covered. A gate that demanded 100% per cell would punish Reservo for doing the right thing —supporting several versions with version-specific branches—.

The underlying lesson: coverage is a floor, not a goal. A defensible threshold for Reservo is 85% —below the real 88%, with margin so that a minor change does not break the build over noise, but high enough that adding a forty-line function without tests makes the coverage drop and triggers the gate—. The exact number is chosen with judgment: high so that it catches untested code, not so high that it blows up on legitimately non-coverable code (branches of another version, if __name__ == "__main__", defensive code that cannot be provoked). A mature team puts the gate a little below its current coverage and raises it over time, not pins it at 100% on the first day.

Close the gap vs. lower the gate

When the gate fails, you have two honest paths and one deceptive one. The deceptive one: exclude lines from the count with # pragma: no cover without reason, to inflate the number. The honest ones:

Close the gap —write the missing test—. The base suite does not test cancel (lines 40-41 of schedule.py). If you add a test that books, cancels, and verifies that the slot is available again, those lines are covered. Measured for real, adding that test raises the coverage:

reservo/schedule.py      20      0      8      1    96%   16->13
-----------------------------------------------------------------
TOTAL                    57      3     16      2    93%

From 88% to 93% —schedule.py jumped from 82% to 96%— from a single test that exercises real code that was untested. This is the good use of coverage: it pointed you to a real gap (a function with no test), and you closed it by testing what was missing. Coverage as a diagnostic tool, not as a number to touch up.

Lower (or calibrate) the gate —when the gap is not a defect—. The else branch of reports.py is not closed with a test on 3.14; it is accepted as covered-by-the-matrix. The gate is calibrated so as not to demand the impossible: a threshold of 85% that coexists with that 67% of reports.py, knowing that the 3.11 cell compensates for it. Lowering the gate here is not giving up; it is recognizing that 100% per cell is a fetish that punishes multi-version support.

The discipline: when the gate fails, ask yourself why the coverage is missing. If it is real untested code (cancel), close it with a test. If it is code unreachable in this configuration (the branch of another version), calibrate the gate. Never the third thing —touching up the number— because that turns the inspector into an accomplice.

Common mistakes

Putting the gate at 100% and celebrating it. What happens: someone pins --cov-fail-under=100 feeling rigorous, and the build fails forever because there is legitimately non-coverable code —the else branch of another version, an if __name__ == "__main__", a defensive except that cannot be provoked—. To "fix it," they start writing fake tests or excluding lines without judgment, degrading the suite. Why it happens: 100% sounds like excellence; in reality it is a goal that ignores that part of the code cannot or should not be exercised in every configuration. How to detect it: if your gate forces you to write tests that test nothing real just to raise the number, the gate is badly calibrated. How to fix it: put the gate a little below your real and honest coverage, and raise it over time; treat 100% as suspect, not as a trophy.

Confusing a version gap with a quality gap. What happens: someone sees reservo/reports.py ... 67% in the 3.14 cell and concludes "that function is badly tested," and starts writing tests to cover the else branch —which on 3.14 is unreachable—, wasting time. Why it happens: the coverage table does not distinguish "untested" from "unreachable in this version"; both appear as missing lines. How to detect it: look at which lines are missing; if they are a branch guarded by sys.version_info (or by OS, or by a condition that in this configuration is false), it is code from another configuration, not a quality gap. How to fix it: understand the coverage as a number per configuration, and trust that the matrix covers the branches of each version in the cell where they live. The honest total is that of the complete matrix, not that of a single cell.

Measuring the coverage without pinned environments. What happens: the pipeline measures 88% today and 86% in two weeks without anyone touching the code, because a new version of coverage counts the branches differently, and the gate breaks the build over a ghost. Why it happens: the coverage is a number about an environment, and if the environment floats, the number floats. How to detect it: if the coverage changes without code or test changes, suspect a version of coverage/pytest-cov that moved. How to fix it: the layer of lesson 3 —pinning pytest-cov (which pins coverage)— so that today's 88% is comparable with tomorrow's. A gate is only fair if it measures on firm ground.

Exercises

Exercise 1 — Choose the threshold and defend it. Reservo's real coverage is 88%. A colleague proposes --cov-fail-under=100; another, --cov-fail-under=50. Both are bad for different reasons. Explain why, and propose a defensible threshold with its justification.

See solution

--cov-fail-under=100 is bad for demanding the impossible. Reservo has legitimately non-coverable code in each cell: the else branch of report_pages is unreachable on 3.12+ (and the if one, unreachable on 3.11). A gate at 100% per cell would fail always, not because of badly tested code, but because of branches of another version that in this one do not exist. To silence it, the team would end up writing fake tests or excluding lines without judgment, degrading the suite. 100% sounds rigorous but is a fetish that punishes multi-version support.

--cov-fail-under=50 is bad for protecting nothing. The real coverage is already 88%, far above 50. A gate at 50% would never trigger with the current code, and —worse— would let an enormous drop through: someone could delete half the tests, sink the coverage to 60%, and the gate would still be green. A gate far below the real coverage is decorative: it exists but does not protect.

A defensible threshold: --cov-fail-under=85. It is below the real 88% (with a cushion of ~3 points so that a minor change does not break the build over rounding noise or a branch), but high enough to bite: if someone adds a forty-line function without tests, the coverage would drop below 85 and the gate would trigger. The rule: put the gate a little below your current honest coverage, high to catch untested code, not so high that it blows up on non-coverable code. And raise it over time, as the suite matures, instead of pinning it at 100% on the first day.

Exercise 2 — Close or calibrate. The report shows two gaps: schedule.py line 40-41 (the cancel function, with no test) and reports.py line 14-16 (the else branch of the fallback, in a Python 3.14 run). For each one, say whether you close it with a test or accept it by calibrating the gate, and why.

See solution

schedule.py 40-41 (cancel): closed with a test. It is real and reachable Reservo code that no test exercises —the cancel function exists, works, and on 3.14 (or any version) can be called—. The gap is a genuine defect of the suite: there is untested logic. It is closed by writing the missing test (book, cancel, verify that the slot is available again), which measured raises the coverage from 88% to 93%. This is the good use of the gate: it pointed you to untested code and you closed it by testing it.

reports.py 14-16 (else branch, on 3.14): accepted by calibrating the gate. It is unreachable in this configuration: on 3.14, sys.version_info >= (3, 12) is true, so Python enters through the if and never executes the else. No test written on 3.14 can cover those lines, because on 3.14 they are dead code. It is not a defect of the suite; it is a branch of another version, which the 3.11 cell of the matrix does exercise. It is accepted by putting the gate at a threshold (85%) that coexists with that 67% of reports.py, knowing that the matrix covers the branch where it lives. Trying to "close it" with a test on 3.14 would be wasting time or cheating.

The rule that distinguishes the two cases: ask yourself "can this code be executed in this configuration?". If yes and there is no test (cancel), it is a real gap: close it. If no (branch of another version), it is coverage-per-configuration: calibrate it, trusting the cell that does exercise it.

Exercise 3 — The gate that did not bite. A team has --cov-fail-under=70 and a real coverage of 92%. A colleague submits a pull request that adds a 200-line module with only two trivial tests, sinking the coverage to 74%. The CI passes green. Did the gate fail? What would you change?

See solution

The gate did not fail technically —it did exactly what it was asked: break the build only if the coverage drops below 70%, and 74% ≥ 70%, so it passed—. But it failed in its purpose: it let a large, almost-untested module in (200 lines, two trivial tests) without even a warning. The problem is that the gate was calibrated far below the real coverage (70% when the project lived at 92%), so it had 22 points of cushion to absorb enormous degradations before triggering. A gate like that is almost decorative: it exists, but it allows quality to erode a lot before reacting.

What I would change, two things. First, raise the gate close to the real coverage: with the project at 92%, a threshold of ~90% (a small cushion, not one of 22 points) would make the drop to 74% break the build immediately, forcing the author to test their module before merging. Second, and more powerful, add a diff coverage (patch coverage) check —tools like Codecov or diff-cover measure what fraction of the new lines of the pull request is covered, not just the total—. With that, a 200-line module with two tests would fail for low coverage of the change, even if the project total stayed high, because the degradation dilutes in the total but jumps in the diff. The lesson: a very loose total coverage gate protects poorly; put it close to your real number, and complement with diff coverage so that new untested code does not hide in the average.

Summary and next step

In this lesson you stacked the fifth layer: the coverage gate, which changes the pipeline from "runs the tests" to "demands a floor of quality." You measured Reservo's coverage —88%— with --cov=reservo --cov-branch --cov-report=term-missing, read which lines are missing, and saw --cov-fail-under break the build for real: exit 1 against a gate of 95%, exit 0 against one of 85%, with the same code and the same tests. The difference between the red and the green was not in the code; it was in where you put the gate.

And you faced the underlying decision: 100% is a fetish. The 67% gap in reports.py is not badly tested code, it is the else branch of another version, unreachable on 3.14 and covered by the 3.11 cell of the matrix —the link that ties this layer with lesson 4—. You learned to distinguish the gap that is closed with a test (cancel, which raises the coverage to 93%) from the one that is accepted by calibrating the gate (the version branch), and to never touch up the number. Coverage is a floor and a diagnostic tool, not a goal or a trophy.

Before moving on you should be able to: measure coverage and read term-missing; put up a gate with --cov-fail-under and predict whether it passes or breaks; choose a defensible threshold and argue for it; and distinguish a real quality gap from a configuration gap that the matrix covers.

What follows, in lesson 7, is the last layer before the project: the flaky policy. Your pipeline is already demanding —it runs on several versions, with a coverage gate—, but it is missing an answer to a problem that lesson 5 itself seeded: a test that fails intermittently, sometimes because of the non-deterministic order that the parallelism introduces. A flaky in CI erodes trust like nothing else —if the red sometimes lies, why believe it?—. You are going to see the retry debate really executed (--reruns), the marker quarantine, and why the honest policy is not "retry until it passes" but "patch, ticket, and root fix."

Resources

• coverage.py — the measurement engine behind pytest-cov: how it counts statements and branches, what branch coverage is, and how it is configured in pyproject.toml. The reference for the number the gate watches.
• pytest-cov — the plugin that integrates coverage with pytest: --cov, --cov-report=term-missing, and the --cov-fail-under flag that turns the measurement into a gate. The documentation of what you executed.
• coverage.py: Excluding code from coverage — how to exclude legitimately non-coverable code (# pragma: no cover) with judgment, for the version branches or the defensive code, without touching up the number.
• coverage.py: Branch coverage — how the branches are counted (the Branch and BrPart columns of the table) and why report_pages, with its if/else by version, never gives 100% in a single cell. The technical detail behind the reports.py gap. In the sibling guide testing-fundamentals-and-tdd, coverage is taught as a local diagnostic tool; here we turn it into a CI gate.