Module 6: Quality Gates Coverage Thresholds

4. Failing on a coverage drop

Next →

Description

The fixed-floor gate you set up in lesson 3 —--cov-fail-under=80— protects against one very concrete thing: coverage dropping below 80%. And it does it well. But it has a blind spot that, if you don't know it, gives you a false sense of security. Imagine Reservo's coverage is 91% —comfortable, well above the floor—. A teammate adds a new function without tests and the coverage drops to 84%. What does the floor-80 gate do? Nothing. It passes green, because 84 is still greater than 80. The coverage dropped seven points —untested code entered— and the gate didn't even notice. The fixed floor watches an absolute line, not the movement; it doesn't distinguish "it was always at 84" from "it just dropped from 91 to 84", and those two situations are very different.

By the end of this lesson you'll understand that blind spot and how to close it. You're going to see, with a real demo, the drop go unnoticed: Reservo's coverage drops from 91.03% to 83.53% when adding a module without tests, and the floor-80 gate passes (exit 0), without seeing the almost-eight-point drop. And you're going to see the response: a gate that fails on the drop, not just against a distant floor. The simplest and most robust form is the ratchet: you put the threshold where you are today —91— instead of a distant round floor, so any drop crosses it. With the gate at 91, that same drop to 83.53% breaks the build (exit 1, total of 84 is less than fail-under=91). The floor stops being a distant line on the ground and becomes a ratchet stuck to your current level: it only goes up, never down.

Connection to the module: this lesson moves one of lesson 2's three knobs —the threshold— and shows that choosing it wrong (a distant floor) leaves a gap that choosing it well (a ratchet at your level) closes. It starts directly from lesson 3's gate: it's the same --cov-fail-under, with a different number and a different philosophy. And it feeds lesson 6, where "where to put the threshold" becomes a full judgment topic. Here the question is specific: how do I make the gate catch a drop, not just a floor? The answer —the ratchet— is one of the most useful ideas in the whole module.

The speedometer that only checks if you exceed 120, not if you braked hard

Imagine a car with a system that fines you if you exceed 120 km/h. It's useful: it keeps you from going dangerously fast. But there's something that system doesn't see. You're going down the highway at 110, calm, and suddenly you slam the brakes to 40 because something distracted you —a sharp brake, dangerous for whoever's behind you—. The system says nothing: 40 doesn't exceed 120, so to it everything's fine. It watches an absolute ceiling, not the sharp changes. A dangerous drop in speed is invisible to it, because all it knows how to do is compare your speed against a fixed line of 120.

A fixed coverage floor is that system. --cov-fail-under=80 watches an absolute line —80—: it stops you if you drop below there. But a drop that doesn't cross that line is invisible to it. Coverage at 91, drops to 84: since 84 > 80, the floor says nothing, just like the speedometer stays silent at the brake from 110 to 40. The problem isn't that the floor is wrong; it's that it measures the wrong thing to detect a drop. A floor measures "are you below the line?"; a drop is "did you go down from where you were?". They're different questions, and the second is the one that catches the untested code that sneaks in while you're still comfortably above the floor.

The solution is to change the question. Instead of comparing against a distant, fixed line, compare against where you were: put the threshold at your current level. If you're at 91, the threshold is 91. Now any drop —to 90, to 84— crosses the threshold and triggers the gate, because the threshold is stuck to your level, not eight points below. And when you improve and rise to 93, you raise the threshold to 93. It's a ratchet: a toothed wheel that only turns one way, that goes up but never down. Your coverage can rise freely, but it can't go backward without breaking the build.

A fixed floor watches an absolute line ("did you drop below 80?") and is blind to a drop that doesn't cross it (from 91 to 84). A ratchet watches the movement ("did you go down from where you were?"): it puts the threshold at your current level, so any drop triggers it. The coverage can rise, but not go backward.

The demo: the drop the floor doesn't see

We're going to reproduce the blind spot for real. We start from lesson 3's green state: Reservo's complete suite, with cancel_with_refund tested, coverage 91.03%, floor-80 gate green. All comfortable.

Now new code enters. Reservo adds a notifications module —the email texts sent when a booking is confirmed or cancelled—. It's legitimate and simple code, but, as happened before with the cancellation, it arrives without tests:

# reservo/notifications.py
from reservo.models import Booking


def booking_confirmed_message(booking: Booking) -> str:
    """Email body when a booking is confirmed."""
    return (
        f"Your booking {booking.id} for {booking.room.name} is confirmed. "
        f"Total: {booking.price_cents} cents."
    )


def booking_cancelled_message(booking: Booking, refund_cents: int) -> str:
    """Email body when a booking is cancelled."""
    if refund_cents > 0:
        return (
            f"Your booking {booking.id} was cancelled. "
            f"You will be refunded {refund_cents} cents."
        )
    return (
        f"Your booking {booking.id} was cancelled. "
        f"No refund applies for this cancellation."
    )

Seven code lines, zero tests. Let's run the floor-80 gate —the same one that was green— to see what it says now, for real, on Python 3.14.0:

python -m pytest --cov=reservo --cov-report=term-missing --cov-fail-under=80

What to expect (real output, measured by executing):

================================ tests coverage ================================
_______________ coverage: platform darwin, python 3.14.0-final-0 _______________

Name                       Stmts   Miss  Cover   Missing
--------------------------------------------------------
reservo/__init__.py            0      0   100%
reservo/calendar.py           30      4    87%   26, 34, 50, 55
reservo/cancellations.py      17      0   100%
reservo/models.py             12      2    83%   34-35
reservo/notifications.py       7      7     0%   3-21
reservo/pricing.py            10      1    90%   14
reservo/refunds.py             9      0   100%
--------------------------------------------------------
TOTAL                         85     14    84%
Required test coverage of 80% reached. Total coverage: 83.53%
============================== 13 passed in 0.03s ==============================

Read the table. The new row, reservo/notifications.py 7 7 0% 3-21: seven lines, all uncovered, 0% —the untested code that just entered—. And the total: TOTAL 85 14 83.53%. The coverage dropped from 91.03% to 83.53% —almost eight points— because untested code entered. But look at the gate's last line: Required test coverage of 80% reached. Total coverage: 83.53%. The gate passed. Let's confirm with the exit code:

python -m pytest --cov=reservo --cov-fail-under=80 > /dev/null 2>&1; echo "exit code: $?"
exit code: 0

Exit code 0. Green. The build would pass, the merge would enter, and no one would find out that the coverage dropped almost eight points and that there's a whole module without a single test. That's the fixed floor's blind spot: 83.53% is still greater than 80, so to the gate "everything's fine", just like the speedometer stays silent at the brake from 110 to 40. The untested code sneaked in above the floor, in the margin between 80 and 91 the gate doesn't watch. Push by push, this is how healthy coverage erodes without any build turning red.

Close the gap: the ratchet at your current level

The cure is to change the gate's question. Instead of "are you below 80?" —a distant line—, ask "did you go down from where you were?" —91—. The simplest way to do it with the tools you already have is to put the threshold at your current level: a ratchet. If the coverage is 91.03%, you put the gate at 91:

python -m pytest --cov=reservo --cov-fail-under=91

Now let's run, with the notifications module still untested inside (coverage 83.53%):

What to expect (real output):

================================ tests coverage ================================
...
TOTAL                         85     14    84%
ERROR: Coverage failure: total of 84 is less than fail-under=91
FAIL Required test coverage of 91% not reached. Total coverage: 83.53%

And the exit code:

python -m pytest --cov=reservo --cov-fail-under=91 > /dev/null 2>&1; echo "exit code: $?"
exit code: 1

Exit code 1. Red. The same drop the floor-80 let through, the ratchet at 91 catches: total of 84 is less than fail-under=91. The logic is simple and powerful: if the threshold is stuck to your current level, any setback crosses it. It doesn't matter that you're still at 83.53% —well above a floor of 80—; what matters is that you dropped from 91, and the ratchet watches exactly that. The untested code no longer has a margin to sneak into: the margin between 80 and 91 the floor ignored, the ratchet covers.

And how do you turn off this alarm? Same as in lesson 3: by writing the missing tests for notifications.py, raising the coverage back to 91 or more, and seeing the ratchet pass. And when the new coverage is, say, 93%, you raise the threshold to 93 —you turn the ratchet one tooth more—, so that from then on you can't even go back to 91. The coverage only rises; the ratchet holds it. That's the healthy cycle: you improve, you raise the threshold, and the new level is protected against the next erosion.

The manual ratchet and the automatic ratchet

Putting the threshold at 91 "by hand" is the ratchet in its simplest form, and for many projects it's enough. But it has a cost: someone has to remember to raise the number every time the coverage improves, editing the workflow. If you forget, the ratchet lags behind —at 91 while the real coverage is 95—, and you again have a margin (91–95) where a drop goes unnoticed. It's a ratchet that works, but that has to be tightened by hand.

There are ways to automate it, and it's worth knowing they exist even if we don't set them up here:

  • Comparing against the pull request's base. Coverage tools like the ones that integrate with GitHub (for example, coverage-reporting services) compute your branch's coverage and compare it automatically against the main branch's, and mark the check red if your change lowers it —without you fixing any number—. It's the real ratchet: the threshold is always "main's coverage", and it updates itself when main improves. The drop is measured against the base, not against a hand-written floor.
  • Saving the number and comparing it in CI. Without an external service, you can save main's coverage in a file (a "badge" or a coverage.json), and on each PR compare the new one against the saved one, failing if it dropped. It's the same ratchet, built with your own pieces.

The guide's honesty applies here: these automations live in the CI workflow —which we don't execute here, there's no runner— and in external services. What we do execute for real, and what you take away as a portable technique, is the ratchet idea and its manual form: --cov-fail-under=<your current level>, which breaks the build on any drop. Start there —it's one line, doesn't depend on anything external, and already closes the distant-floor blind spot—; graduate to the automatic ratchet when the project grows and forgetting to raise the number by hand becomes a real risk.

Floor and ratchet aren't enemies

A clarification so you don't come away with the idea that the fixed floor "is wrong": it isn't. Floor and ratchet answer different questions and often coexist:

  • The fixed floor (--cov-fail-under=80) answers "is the coverage at least acceptable?". It's a minimum-dignity line: below 80, the project is in trouble regardless of the history. Useful as a last-resort net and as a stable number you don't have to touch often.
  • The ratchet (--cov-fail-under=<current level>) answers "did this change make things worse?". It's a defense against erosion: even if you're comfortable above the floor, you can't go backward.

Many mature teams use both: a low, stable fixed floor (never below X) plus a ratchet that prevents drops from the current level. But if you had to choose one to start with, the ratchet catches more real problems, because most erosion happens above the floor —untested code that sneaks in while the coverage is still "acceptable"—. The floor protects you from disaster; the ratchet, from daily carelessness, which is much more frequent. Lesson 6 takes up this decision —which threshold, of which type, a project deserves— as a full judgment topic.

Common mistakes

Believing a fixed floor protects against drops. What happens: the team sets --cov-fail-under=80, sees the coverage at 91%, and assumes it's safe from it dropping. Months later the coverage is at 82% —eroded push by push— and no build ever turned red. Why it happens: a green floor gives the sense of "protected", but it only protects against crossing the line, not against sliding toward it. How to spot it: compare today's coverage with three months ago's; if it dropped without any build failing, the floor was blind to the drop. How to fix it: add a ratchet —the threshold at your current level— that fails on any setback, not just below the floor. The floor watches the ceiling of danger; the ratchet, the ground of your progress.

Lowering the ratchet when it bothers you, instead of writing the test. What happens: the ratchet at 91 breaks the build because untested code entered (83.53%), and someone "fixes" the red by lowering the threshold to 83. The build passes, but you just made the drop the new normal. Why it happens: lowering the number is faster than writing the test, and "well, it's still above 80". How to spot it: if the --cov-fail-under drops in the history, someone loosened the ratchet —which by definition should only go up—. How to fix it: a ratchet that goes down isn't a ratchet, it's a moving floor that legitimizes each erosion. The hard rule: the ratchet's threshold only goes up. When it bothers you, the response is the missing test (raise the coverage back), not the excessive threshold.

Setting the ratchet and forgetting to raise it when improving. What happens: the coverage improves from 91 to 96 over time, but the ratchet stays at 91 because no one raised it. Now there's a five-point margin (91–96) where a drop again goes unnoticed. Why it happens: raising the threshold by hand is an easy step to forget, especially when the coverage improves little by little. How to spot it: if your real coverage is several points above your --cov-fail-under, the ratchet lagged behind and the blind spot reappeared. How to fix it: raise the threshold every time the coverage rises stably —or automate the ratchet (compare against the base) so it adjusts itself—. A ratchet that isn't tightened loosens over time.

Exercises

Exercise 1 — Predict which gate catches the drop. Reservo's coverage was at 91.03%. Untested code enters and it drops to 83.53%. For each gate, say whether it catches the drop (breaks the build) or lets it through, and why: (a) --cov-fail-under=80. (b) --cov-fail-under=91. (c) --cov-fail-under=85. (d) without --cov-fail-under (only --cov=reservo).

See solution
  • (a) --cov-fail-under=80: lets it through. 83.53% ≥ 80, so the gate passes (exit 0). It's the lesson's blind spot: the drop happened above the floor, invisible.
  • (b) --cov-fail-under=91: catches. 83.53% < 91, so it breaks the build (exit 1, total of 84 is less than fail-under=91). The ratchet at the starting level sees any setback.
  • (c) --cov-fail-under=85: catches. 83.53% < 85, so it breaks the build. A ratchet doesn't have to be exactly at the starting level to catch the drop; it's enough for it to be above the level it dropped to. A threshold of 85, though a bit below the real 91, still catches this drop to 83.53%.
  • (d) without --cov-fail-under: lets it through. Without a threshold there's no gate; it only reports 83.53% and finishes at exit 0. The drop isn't even evaluated.

The lesson: the closer the threshold is to your real level, the finer the drop it catches. A floor of 80 only sees drops that reach below 80; a ratchet at 91 sees any drop from 91. (c) shows there's an intermediate range: any threshold above the 83.53% it dropped to catches this drop, but only a threshold at 91 or very close catches every drop from the current level.

Exercise 2 — Design the ratchet cycle. Your project is at 91.03% with the ratchet at 91. You write the missing tests for notifications.py and the coverage rises to 95%. Describe, step by step, what you do with the threshold and why, and what would happen if you forget to do it.

See solution

The healthy ratchet cycle, step by step:

  1. You write the notifications.py tests. The coverage rises from 83.53% back to 91 and, with the new tests also covering previously uncovered code, to 95%. The ratchet at 91 now passes (95 ≥ 91).
  2. You raise the threshold to 95. You change --cov-fail-under=91 to --cov-fail-under=95 in the workflow. You turn the ratchet one tooth: your new level is protected.
  3. You confirm it's still green. 95% ≥ 95, the gate passes. And from now on, any drop below 95 —even to 94— breaks the build.

Why raise the threshold: if you leave it at 91 while the real coverage is 95, a margin reappears (91–95) where a future drop would go unnoticed —you again have a blind spot, smaller but real—. Raising the ratchet to the new level closes that margin.

What happens if you forget: the ratchet stays at 91, and tomorrow someone can add untested code that lowers the coverage from 95 to 92 without breaking the build (92 ≥ 91). You eroded four points for free. That's why step 2 isn't optional: a ratchet only protects the level it's tightened to, and you have to tighten it every time you improve (or automate it by comparing against the base, so it adjusts itself).

Exercise 3 — Floor, ratchet, or both. For each project, recommend a threshold strategy —fixed floor, ratchet, or both— and justify it in two or three sentences. (a) A mature library, coverage stable at 96%, large team, many PRs a day. (b) A new project, coverage 45%, that wants to improve without blocking itself. (c) Reservo today: 91%, small team, wants not to go backward.

See solution
  • (a) Mature library (96%, large team): automatic ratchet, ideally against the base. With many PRs a day and high coverage, the risk is erosion by carelessness —one PR among many that sneaks in with untested code—. A ratchet that compares against main's base and fails on any drop is what protects that 96% without depending on someone watching every PR. A low fixed floor (say 90) as a last-resort net doesn't get in the way, but the ratchet is the one doing the work.
  • (b) New project (45%, wants to improve): gentle ratchet, starting where it is. A high floor (80) would block all work from day one —it's at 45, it can't meet 80 without stopping to write tests for all the old code—. The healthy strategy is a ratchet at 45 ("don't get worse") that rises every time they improve, so the coverage can only grow, without requiring an impossible jump all at once. The ratchet turns "reach 80" into a gradual path instead of a wall.
  • (c) Reservo (91%, small team, don't go backward): manual ratchet at 91. It's exactly the lesson's case. A small team can handle raising the threshold by hand when it improves, without needing automation yet. --cov-fail-under=91 closes the distant-floor blind spot with a single line; a fixed floor of 80 would let through exactly the drops that matter to Reservo. If the team grows, graduate to the automatic ratchet.

The rule running through all three: the threshold must be close to where you are to catch drops, and it must only go up. A distant, fixed floor serves as a last-resort net, but doesn't protect against daily erosion; that's what the ratchet is for.

Summary and next step

In this lesson you discovered the fixed floor's blind spot and how to close it. A floor like --cov-fail-under=80 watches an absolute line and is blind to a drop that doesn't cross it: you saw it for real, when Reservo added notifications.py without tests, the coverage dropped from 91.03% to 83.53% —almost eight points, a whole module untested— and the floor-80 gate passed green (exit 0), like the speedometer that stays silent at a sharp brake because it only watches the 120 ceiling.

The cure is the ratchet: putting the threshold at your current level (91) instead of a distant floor, so any setback crosses it. The same drop the floor ignored, the ratchet at 91 caught —exit 1, total of 84 is less than fail-under=91—. The ratchet only goes up: when you improve, you raise the threshold; when untested code enters, it breaks the build. You saw its manual form (one line, portable, no dependencies) and its automatic forms (comparing against the PR's base), and that floor and ratchet aren't enemies —the floor is the last-resort net, the ratchet the defense against daily erosion—.

Before moving on you should be able to: explain why a fixed floor doesn't catch a drop that stays above it; set a ratchet with --cov-fail-under=<current level> and say why it catches any setback; describe the healthy cycle (you improve → you raise the threshold) and why the ratchet only goes up; and decide between floor, ratchet, or both based on the project.

What's next, in lesson 5, is changing the metric. So far every gate looked at coverage; now you're going to set up a gate that looks at a subset of tests: the marker gate. A CI job that runs pytest -m smoke and requires that critical group of tests to pass before allowing a merge —fast, cheap, and a first line of defense different from coverage—. You'll see it's the same recipe from lesson 2 (metric, threshold, consequence) with the metric changed from "% covered" to "did the smoke pass?".

Resources

← Previous3. `--cov-fail-under` and the exit codeNext →5. Marker gates and smoke tests