Module 6: Quality Gates Coverage Thresholds

1. Module introduction: quality gates

Next →

Description

Up to now your pipeline did something valuable but incomplete: it measured. It ran the Reservo suite on every push, across several Python versions, fast and cached, and gave you back an honest verdict —green if all the tests pass, red if any fails—. That's measuring. But there's a class of degradation no red test gives away, and it's the one this module attacks. Imagine a teammate adds a new function to Reservo —say, the one that cancels a booking and computes the refund— and, in a hurry, pushes it without a single test. What does your pipeline do? Nothing. The suite stays green, because the old tests keep passing; the new code, without tests, simply isn't exercised. No one sees a red, no one finds out. The coverage —the percentage of the code your tests actually run— dropped, silently, and push after push it keeps eroding until one day half the code isn't tested and no one decided it should be that way.

By the end of this lesson you'll understand what a quality gate is and why a CI that only measures isn't enough. A quality gate is a threshold that turns a metric into a merge condition: it doesn't just report "coverage is 65%", it declares "if coverage drops below 80%, I break the build". And when the build breaks —a non-zero exit code— the change doesn't enter the main branch until someone fixes it. You're going to see, executed for real on your machine, the most common gate: pytest --cov=reservo --cov-fail-under=80, breaking the build with exit code 1 when Reservo's coverage is below 80, and going green when you add the missing test. That's this module's leap: from a pipeline that tells you how the quality is to one that demands it.

Connection to the module: this lesson is the map. Here you install the idea —the difference between measuring and imposing— and watch it beat once with a real local demo. Lesson 2 takes the gate concept apart piece by piece: metric, threshold, and consequence, and why the exit code is the language a gate speaks to CI. Lesson 3 is the heart: --cov-fail-under executed for real, with its exit code, and its cousin the coverage CLI. Lesson 4 shows the blind spot of the fixed floor and how a gate can fail on a drop in coverage, not just against a distant floor. Lesson 5 leaves coverage and sets up a marker gate: the job that requires the smoke tests to pass before merging. Lesson 6 is the judgment —when a gate helps and when it gets in the way—. Lesson 7 is the dark side: 100% as a fetish that pushes toward tautological tests, the "green that doesn't verify" turned into policy. And lesson 8, the mini-project, has you set up Reservo's CI coverage gate from start to finish.

A note about the boundary, because this module leans on a sibling guide and doesn't repeat it. Coverage as a local tool —what it is, how to read a report, what an uncovered line means— you learned in module 6 of the testing fundamentals guide. Here we're not going to re-explain what coverage is; we're going to turn it into a gate: from a number you look at in your terminal to a threshold that breaks a build in CI. If you need to refresh how a coverage report is generated and read, that's the place. And the other side of the boundary: flaky tests —the ones that pass sometimes and fail sometimes— are module 7, the one that follows. Here, when a gate turns red, it's for a deterministic and clear reason; inconsistency is the next module's topic.

The building's smoke detector, not the thermometer

Think of two devices hanging from the ceiling of an office building. The first is a thermometer: it measures the temperature and shows it on a little screen. If there's a fire, the thermometer climbs and climbs —28°, 40°, 65°—, and it reflects the truth with full fidelity. But it does nothing else. Someone has to be watching the screen, notice the number went up, understand that means fire, and act. If no one watches, the building burns with a perfectly honest thermometer marking the catastrophe.

The second device is a smoke detector with sprinklers. It also measures —it senses the particles in the air—, but it has a threshold and a consequence: when the smoke passes a certain level, it doesn't just show it, it triggers the alarm and opens the sprinklers. It doesn't depend on someone watching. The measurement crosses a line and something happens automatically. The detector turns "there's smoke" from a datum to be monitored into an action that runs on its own.

Your pipeline, up to this module, was an excellent thermometer. It measured Reservo's coverage and showed it to you —65%, 84%, 91%—, with full honesty. But it depended on you looking at the number, understanding it dropped, and acting. A quality gate is the smoke detector: you put a threshold on the metric —"minimum coverage 80%"— and a consequence —"if it drops below there, the build breaks"—, and from then on CI acts on its own. No one has to be watching the screen. When someone tries to merge a change that sinks the coverage below the line, the gate triggers the alarm —the red build— and the change doesn't enter. The measurement stopped being a datum to monitor and became a condition that's imposed.

A quality gate is a metric plus a threshold plus an automatic consequence. It doesn't just measure the quality (like a thermometer); it breaks the build when the metric crosses the line (like a smoke detector with sprinklers). It turns a number you have to monitor into a condition that's imposed on its own.

Reservo, as we left it — and the function that slipped in without tests

We continue with Reservo, the coworking meeting-room booking system we've been testing since the first module. Pure Python logic: no database, no network, no hidden clocks. Its pieces, in case you need a refresher:

  • Room (id, name, capacity, hourly_cents), Member (id, name, tier: "basic" or "pro"), Booking (with its price_cents field, the start, the end as a half-open range [start, end), and its status).
  • The core functions: price_cents(room, member, hours), refund_cents(booking, price_paid_cents, now), overlaps, is_available, book, and the Calendar that stores the bookings in memory.
  • The anchor numbers, the whole guide's checksum: basic 3 h → 7500, pro 3 h → 6000 (20% discount), basic 1 h → 2500, and the refund on 6000 paid: 6000 if you cancel 72 h ahead (≥ 48 h, 100%), 3000 at 36 h (24–48 h, 50%), 0 at 12 h (< 24 h).

You already have all that tested, with a suite that's green. For this module, Reservo debuted a new function that joins two pieces you already know: cancel_with_refund, which cancels a booking —freeing its slot in the calendar— and, in the same operation, computes how much money is returned to the member per the refund policy. It's a legitimate and useful function. The problem —the one that gives the whole module life— is that it reached the repository without tests.

# reservo/cancellations.py
from datetime import datetime

from reservo.calendar import Calendar
from reservo.models import Booking
from reservo.refunds import refund_cents


def cancel_with_refund(calendar: Calendar, booking: Booking, now: datetime) -> int:
    """Cancel `booking` and return the refund in cents."""
    if booking.status == "cancelled":
        raise ValueError("booking is already cancelled")
    refund = refund_cents(booking, booking.price_cents, now)
    calendar.cancel(booking)
    return refund


def refund_reason(booking: Booking, now: datetime) -> str:
    """Readable refund reason, for the confirmation email."""
    hours_before = (booking.start - now).total_seconds() / 3600
    if hours_before >= 48:
        return "full refund (cancelled 48h or more before start)"
    if hours_before >= 24:
        return "half refund (cancelled 24 to 48h before start)"
    return "no refund (cancelled less than 24h before start)"

Read it carefully, because this code works —it has no visible bugs—. But "works" and "is tested" are two different things, and that distinction is exactly the one a coverage gate makes visible. Right now, Reservo's suite passes entirely green, and yet these two functions are exercised by no test. If tomorrow someone breaks refund_reason —inverts a condition, changes a threshold— the suite would stay green, because there's nothing looking at that function. Coverage is the only thing that gives away this gap, and a coverage gate is the only thing that prevents it.

First contact: the gate that breaks the build

We're going to see the most common gate of all, executed for real. The tool is pytest-cov, a pytest plugin that measures coverage while the suite runs, and its star flag for this module is --cov-fail-under=N: "if the total coverage ends up below N percent, finish with a failure exit code". Let's recall why the exit code matters so much (you saw it in module 2): CI doesn't read the text output, it reads the exit code of the command. A 0 is green; any other number is red and stops the pipeline. --cov-fail-under is, literally, the lever that turns a coverage number into an exit code —and that's why it's a gate—.

The run is on the machine where I write this: Python 3.14.0, pytest 9.1.1, coverage 7.15.2, pytest-cov 7.1.0. The command asks for the reservo package's coverage (--cov=reservo) and puts the gate at 80 (--cov-fail-under=80):

python -m pytest --cov=reservo --cov-fail-under=80

What to expect. With the suite as it stands —with cancel_with_refund and refund_reason without tests—, this comes out, measured for real:

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /private/tmp/reservo-m6
plugins: cov-7.1.0
collected 10 items

tests/test_availability.py ....                                          [ 40%]
tests/test_pricing.py ...                                                [ 70%]
tests/test_refunds.py ...
ERROR: Coverage failure: total of 65 is less than fail-under=80
                                                                         [100%]

================================ tests coverage ================================
_______________ coverage: platform darwin, python 3.14.0-final-0 _______________

Name                       Stmts   Miss  Cover
----------------------------------------------
reservo/__init__.py            0      0   100%
reservo/calendar.py           30      7    77%
reservo/cancellations.py      17     17     0%
reservo/models.py             12      2    83%
reservo/pricing.py            10      1    90%
reservo/refunds.py             9      0   100%
----------------------------------------------
TOTAL                         78     27    65%
FAIL Required test coverage of 80% not reached. Total coverage: 65.38%
============================== 10 passed in 0.03s ==============================

Stop at two lines. The first: 10 passed. The ten tests passed. If this were only the suite, the build would be green and everyone happy. The second: FAIL Required test coverage of 80% not reached. Total coverage: 65.38%, and above, ERROR: Coverage failure: total of 65 is less than fail-under=80. There's the gate speaking. The tests pass, but the total coverage is 65.38% —look at the reservo/cancellations.py row, at 0%: its 17 lines aren't touched by any test—, and since 65 is less than 80, the gate breaks the build. Now let's verify it really broke the build by looking at the exit code, which is the only thing CI would read:

python -m pytest --cov=reservo --cov-fail-under=80 > /dev/null 2>&1; echo "exit code: $?"
exit code: 1

Exit code 1. It's not 0. On the runner, that 1 would paint the step red and stop the merge, exactly as if a test had failed —even though the ten tests passed—. That's a quality gate in action: the metric crossed the line, and the build broke on its own, without anyone having to look at the number and decide. The thermometer became a smoke detector.

And how do you turn off the alarm? Not by dressing up the number: by writing the missing test. In lesson 3 you're going to add the cancel_with_refund test, see the coverage rise to 91.03%, and see the same gate pass with exit code 0. For now, keep the image: the gate doesn't ask you to raise a number, it asks you to test the code you left untested. That's the difference between a gate that helps and a fetish that gets in the way, and it's the thread running through the whole module.

What a quality gate is and what it isn't

It's worth making the anatomy clear before diving in, so the next lessons fall into place.

A quality gate is three things together: a metric (here, the coverage percentage), a threshold (80%), and an automatic consequence (breaking the build —non-zero exit code— if the metric ends up below the threshold). All three are necessary. Without a metric there's nothing to measure. Without a threshold there's no line to cross. And without an automatic consequence you have a thermometer, not a gate: a number someone has to look at and decide about. The gate exists precisely so that no one has to look and decide: the policy is declared once, in the YAML, and CI imposes it on every push.

What a quality gate is not: it's not a real measure of quality. A coverage gate monitors how much code runs when the tests run, and that's a necessary condition —code that never runs was never verified— but not sufficient: code that runs can be falsely verified, with a tautological test that asserts nothing (all of lesson 7 is about this). Confusing "80% coverage" with "80% quality" is the mistake that turns a useful tool into a harmful fetish. The gate tells you which code the tests didn't touch; it doesn't tell you whether the tests that do touch it are worth anything. That judgment is still yours.

And an honesty of the guide, the same as always: the CI workflow runs on a GitHub runner, which we don't have here. So the gate YAML you're going to write and read as content —I show you how it looks and how its log would read—, while the pytest --cov and coverage report runs are real, done locally with Python 3.14.0, coverage 7.15.2, and pytest-cov 7.1.0. When I cite "65.38%" or "exit code 1", I measured that number by executing; when I show a CI log with a red step, that's the honest format of how it would look, not a screenshot of a phantom runner. The gate that breaks the build in your terminal is the same one that would break the build on the runner, because it's the same command reading the same exit code.

Common mistakes

Believing "the green suite" means "the code is tested". What happens: the team sees 10 passed and concludes Reservo is covered, without noticing that a whole function (cancel_with_refund) isn't touched by any test. Why it happens: "green" is a strong signal easy to over-interpret; the eye reads "the tests pass" as "all the code works", when it only means "the code the tests exercise works". How to spot it: look at the coverage, not just the test count. A green suite with 65% coverage is telling you a third of the code never ran in the tests. How to fix it: add a coverage gate, which makes that gap the test count hides visible —and blocking—. Test green and coverage are two different questions: "does what I tested pass?" and "how much did I test?".

Setting a gate but looking at the output instead of the exit code. What happens: someone adds --cov-fail-under=80 to the command, sees the coverage report printed, and believes they now "have the gate", without verifying the command really finishes with a failure exit code when it should. Why it happens: the coverage report is visually striking and gives the sense that "something is happening", but CI doesn't read that text, it reads the exit code. How to spot it: run the command and do echo "exit code: $?" right after; if it's not 1 (or 2, depending on the tool) when coverage is low, the gate doesn't bite. How to fix it: test the gate on purpose with coverage below the threshold and confirm the non-zero exit code, just as you verify a test "bites" by breaking the code. A gate you never saw break the build is a gate you shouldn't trust.

Confusing coverage with quality. What happens: the team treats the coverage percentage as if it were the code's grade, and chases raising it as an end in itself —"we reached 95%, let's go for 100%"—. Why it happens: it's a number, and numbers are easy to chase; it gives the illusion of measurable progress. How to spot it: ask yourself "if I raise this number by writing a test that asserts nothing, does the gate let me?". If the answer is yes (and it is, as you'll see in lesson 7), the number doesn't measure quality, it measures execution. How to fix it: use coverage for what it is —a detector of untested code, a safety floor— and not as a grade. Lessons 6 and 7 develop this distinction, which is the line between a gate that helps and a fetish that harms.

Exercises

Exercise 1 — Thermometer or detector. For each situation, say whether it describes a pipeline that only measures (thermometer) or one with a gate (smoke detector), and explain in one sentence what distinguishes it. (a) "CI prints a coverage report at the end of each run, and the lead reviews it on Fridays." (b) "CI runs pytest --cov=reservo --cov-fail-under=80 and the merge is blocked if coverage drops below 80." (c) "We have a dashboard that graphs coverage week by week."

See solution
  • (a) Thermometer. It measures (prints the report) but doesn't impose: the consequence depends on a human reviewing it and acting. If the lead goes on vacation or doesn't look one Friday, coverage can collapse without anything stopping it. It's honest measurement without a gate.
  • (b) Smoke detector (gate). It has all three pieces: metric (coverage), threshold (80), and automatic consequence (merge blocked via non-zero exit code). No one has to look; CI imposes the line on every push. It's a quality gate.
  • (c) Thermometer (prettier). A dashboard is visualized measurement —very useful for seeing trends—, but it still depends on someone looking at the graph and acting. Graphing the drop doesn't prevent it; it just makes it easier to notice afterward. Without a threshold that breaks the build, there's no gate.

The rule: if the degradation can happen without anything automatic preventing it, it's a thermometer. Only (b) has the consequence that defines a gate.

Exercise 2 — Predict the gate's effect. In the lesson's run, Reservo's coverage was 65.38% with cancel_with_refund untested, and the --cov-fail-under=80 gate broke the build with exit code 1. Without running anything, predict: if a teammate, instead of writing the missing test, deletes cancellations.py from the project (removes the untested function), what would happen to the total coverage and the gate? Is it a good fix?

See solution

The total coverage would rise and the gate would probably pass. Reason: coverage is a percentage —executed lines over total lines—. cancellations.py contributed 17 total lines and 0 executed ones, dragging the average down. By deleting it, those 17 uncovered lines disappear from the denominator, and the percentage of what remains rises (the rest of the package was much better covered). The gate, mechanically, would pass.

But it's a terrible fix, and this is important: raising coverage by removing untested code instead of testing it deceives the gate without fulfilling its purpose. If cancel_with_refund is a function Reservo needs, deleting it so the gate passes is equivalent to turning off the smoke detector instead of putting out the fire. The gate was pointing you to a real gap —useful code without verification—; the correct response is to write the test (lesson 3), not remove the functionality. This exercise illustrates why coverage is a necessary but manipulable condition: the number can be moved for the wrong reasons, and that's why human judgment —"should this code exist and be tested?"— is still irreplaceable.

Exercise 3 — Which module or guide solves this? For each situation, say whether this module (quality gates) solves it or whether it belongs to another module of this guide or to a sibling guide, and name it in one sentence. (a) "I want the build to break if Reservo's coverage drops below 80%." (b) "I don't understand what an 'uncovered' line in a coverage report is." (c) "A Reservo test passes sometimes and fails sometimes without the code changing." (d) "I want to require that the smoke tests pass before allowing a merge."

See solution
  • (a) Break the build if coverage drops below 80% → this module, specifically lesson 3 (--cov-fail-under and the exit code). It's the very definition of a coverage gate.
  • (b) What an "uncovered" line is → the testing fundamentals guide, its module 6 (coverage as a local tool). Here we take for granted how to read a report; what we add is turning it into a gate. If the base concept isn't clear, that's the place to review.
  • (c) A test that passes sometimes and fails sometimes → module 7 (flaky tests in CI). A non-deterministic test is a flaky, and its treatment —retry, quarantine, the failure that only happens in CI— belongs to the next module. A quality gate fails deterministically; a flaky is the opposite.
  • (d) Require the smoke tests to pass before merging → this module, lesson 5 (marker gates). It's a gate that doesn't look at coverage but at a subset of tests, but it's a gate all the same: threshold (they pass), consequence (block the merge).

The mechanical rule: if the question is "how do I make CI impose a minimum?", it's this module. "What is coverage?" is fundamentals M6. "Why is it inconsistent?" is module 7.

Summary and next step

In this lesson you installed the idea that holds up the module: a pipeline that only measures is a thermometer; a quality gate is a smoke detector. Measuring coverage and showing it depends on someone looking at the number and acting; a gate puts a threshold and an automatic consequence on it —breaking the build— so degradation can't happen silently. It's the leap from a CI that tells you how the quality is to one that demands it.

You watched it beat with a real local demo: Reservo debuted cancel_with_refund, a legitimate function that slipped into the repository without tests. The suite still gave 10 passed —green—, but pytest --cov=reservo --cov-fail-under=80 revealed the total coverage was 65.38%, with cancellations.py at 0%, and since 65 < 80, the gate broke the build with exit code 1. The tests passed and still the build turned red, because the metric crossed the line. The guide's honesty also became clear: the coverage runs are real, the gate YAML is content you learn to read and write, and —key— the gate doesn't ask you to dress up a number, it asks you to test the code you left untested.

Before moving on you should be able to: explain in your own words the difference between measuring and imposing quality; name a gate's three pieces (metric, threshold, consequence); say why a green suite doesn't guarantee the code is tested; and explain why the exit code —and not the printed report— is what makes a gate bite.

What's next, in lesson 2, is taking the gate concept apart calmly, before returning to the commands: what distinguishes it from a simple report, why any metric with a threshold can be a gate, and why the exit code is the language a gate speaks to CI. With that conceptual base firm, lesson 3 returns to --cov-fail-under to squeeze it for real —the complete cycle from red to green, executed step by step—.

Resources

  • pytest-cov: fail_under and the coverage report — the documentation of the plugin we use for --cov and --cov-fail-under, the canonical reference for this module. Here we only peek at it; in lesson 3 we open it in depth.
  • Coverage.py: fail_under — the equivalent option in the coverage tool pytest-cov leans on. Note the threshold lives in the [report] configuration; we'll see it with .coveragerc in lesson 3.
  • Coverage.py: introduction and how it's measured — the other side of the boundary: what coverage is and how a report is generated locally, module 6 of the testing fundamentals guide's topic. If "uncovered line" or "statement coverage" don't quite ring a bell, review it before continuing; here we take it for granted.
  • pytest exit codes — the official table of what each pytest exit code means, the language the gate speaks to CI. The 1 you saw (failure) and the 0 (success) are the two faces of every gate.
← Previous8. Mini-project: a fast CI for ReservoNext →2. What a quality gate is