Module 6: Quality Gates Coverage Thresholds

6. When the gate helps (and when it gets in the way)

Next →

Description

You already know how to set up gates: coverage (lessons 3 and 4) and marker (lesson 5). This lesson takes the step that separates someone who knows how to set up gates from someone who decides which to set: judgment. Because a gate isn't good or bad in the abstract —it's good or bad for a project, with a threshold, over a metric—. The same 80% coverage gate that saves a team from silent erosion can, in another context or with another threshold, block legitimate work, push toward tricks to comply with it, and become the hindrance everyone learns to work around. Knowing when a gate helps and when it gets in the way is what avoids the two opposite failures: setting no gate (and letting quality erode) or setting bad gates (and having the team hate them and sabotage them).

By the end of this lesson you'll have a clear criterion for deciding. You'll know when a gate helps: when it stops an erosion that no one would otherwise notice, when it makes explicit a decision no one would otherwise make, when it protects something that really can't break. And when it gets in the way: when the threshold is an aspirational number that blocks healthy work, when the metric doesn't measure what matters, when the gate punishes legitimate refactors. You'll take away the honest threshold rule —put it where you are and raise it with a ratchet, not a round number you'd like to have— and the underlying distinction that runs through the whole module: coverage isn't quality; it's a necessary, not sufficient, condition. A well-set gate guards a minimum floor; no gate guarantees that what's above the floor is worth anything.

Connection to the module: this lesson is the hinge between the "how" (lessons 3-5) and the "when it makes sense". It takes up lesson 4's erosion (the case where the gate clearly helps) and anticipates lesson 7 (the case where the 100% fetish clearly gets in the way, pushing tautologies). It's the least technical lesson and the most important for the craft: you learned the tools; here you learn not to use them badly. Without this judgment, it's easy to end up with a CI full of gates no one respects —or without any, leaving the door open to erosion—.

The guard who checks what matters, not the one who checks everything

Imagine two security guards at a building's entrance. The first is sensible: he checks that each person has their badge, stops whoever doesn't have it, and lets whoever does through smoothly. His control protects —no one enters without permission— without getting in anyone legitimate's way. People respect him because his rule makes sense: the badge is exactly what separates who should enter from who shouldn't.

The second guard is a well-intentioned disaster. He checks the badge, yes, but he also measures each person's height, asks their blood type, and requires shined shoes. His rules don't separate anyone dangerous from anyone legitimate —blood type doesn't say whether you have permission to enter—; they just make the line endless. People start to hate him, and worse: they learn to work around him, entering through the back door to avoid the circus. The second guard doesn't just get in the way; he destroys the control, because he teaches everyone to evade it. A rule that gets in the way without protecting isn't neutral: it's worse than no rule, because it trains the team to skip the controls.

A quality gate is a guard. It can be the first —check what really matters, protect without getting in the way, earn respect— or the second —require things that don't separate the good from the bad, make the line endless, and teach the team to work around the control—. The difference isn't in having a gate or not; it's in whether the gate checks the right thing with the right requirement. An 80% coverage gate over critical business code is the first guard. A 100% threshold that pushes toward writing false tests to comply is the second: people learn to "work around" it with tautologies, and the control becomes theater. This is the judgment the lesson teaches you: to be the first guard, not the second.

A gate helps when it checks what really matters with the right requirement —it protects without getting in the way, like the guard who asks for the badge—. It gets in the way when it requires what doesn't separate the good from the bad —it blocks healthy work and teaches the team to work around it, like the guard who measures height—. A gate that gets in the way without protecting is worse than none: it trains people to evade the controls.

When the gate helps

There are three situations where a gate pays off handsomely, and it's worth recognizing them because they're the argument for setting them.

When it stops an erosion no one would notice. It's lesson 4's case, and the strongest. Reservo's coverage was at 91%; untested code entered and it dropped to 83.53%, without any build turning red. Without a gate —a ratchet—, that erosion continues push by push until half the code isn't tested, and no one decided it should be that way: it happened by accumulation of carelessness. A gate turns that invisible slide into a visible red. Here the gate helps unequivocally, because the problem it attacks —silent erosion— is real, common, and otherwise invisible. No one is going to notice by hand that the coverage dropped two points this week; the gate does.

When it makes explicit a decision no one would otherwise make. Without a gate, "merging code without tests" isn't a decision someone consciously makes —it just happens, in a hurry, on a Friday—. With a gate, lowering the coverage requires a deliberate action: either you write the test, or you lower the threshold on purpose (and that stays in the history, visible, debatable). The gate doesn't forbid merging untested code; it forces it to be a choice, not an accident. It turns a silent carelessness into a decision someone has to make with their name on it. That's valuable even when the final decision is "yes, this time we merge without a test, for this reason": at least it was decided, not slid.

When it protects something that really can't break. It's the smoke gate's case (lesson 5). Reservo's pro discount charges real money; if it breaks, a customer pays wrong. A smoke gate that requires that test to pass before merging protects exactly what can't reach production broken. Here the gate helps because what it monitors —the business-critical stuff— justifies the control: the gate's cost (running three tests) is tiny against the cost of what it prevents (charging customers wrong). The proportion between what the gate costs and what it avoids is what makes it worth it.

The thread of the three cases: the gate helps when the problem it attacks is real and otherwise invisible or accidental, and when what it protects justifies the control. Silent erosion, slid decisions, and the business-critical stuff are the three places where a gate earns its place.

When the gate gets in the way

And there are three situations where a gate, however well-intentioned, does more harm than good.

When the threshold is an aspirational number that blocks healthy work. A team with real coverage of 62% puts the gate at 90 "because we want to get there". Result: every PR breaks the build, even the ones that improve the coverage (from 62 to 65 is still < 90). The threshold doesn't reflect where the project is; it reflects where someone would like it to be, and it punishes every legitimate step toward that goal. People start to skip the gate (lowering it by hand, forcing the merge) because complying is impossible without stopping for weeks to write tests for all the old code. An aspirational threshold doesn't motivate; it frustrates and gets worked around. The cure is the honest threshold (below): put it at 62 and raise it with a ratchet.

When the metric doesn't measure what matters. An 80% coverage gate guarantees that 80% of the lines run in the tests. It doesn't guarantee those tests verify anything —a test that runs the line but asserts nothing "covers" it just the same (lesson 7)—. If the team chases the percentage as if it were quality, the gate pushes exactly toward what doesn't matter: raising a number that doesn't measure what it's believed to measure. Here the gate gets in the way not because the metric is useless, but because it's used as if it measured something it doesn't. Coverage measures execution, not verification; confusing them turns the gate into the second guard, requiring blood type instead of the badge.

When it punishes legitimate refactors. A developer deletes dead code —a hundred lines no longer used— and the coverage drops, because those lines, though dead, were "covered" by accident and their disappearance changes the arithmetic; or it raises the denominator in a strange way. A rigid ratchet breaks the build over a change that improved the project (less code, cleaner). A gate that penalizes cleaning up the code teaches the team not to clean it —to accumulate garbage so as not to fight with the gate—. Here judgment rules: a gate must have slack for the changes that lower it for good reasons, or the team will learn to avoid those good changes.

The thread of these three: the gate gets in the way when it requires the impossible (aspirational threshold), measures the wrong thing (execution confused with quality), or punishes the good (healthy refactors). In all three, the team's response is the same and the worst: learning to work around the gate, which turns it into theater.

The honest threshold rule

Out of all the above comes a practical rule that solves most cases: put the threshold where you are, not where you'd like to be, and raise it with a ratchet.

Concretely. Your real coverage is 84%. The honest threshold is 84 (or a touch less, 83, to give slack for roundings and refactors). Not 90 "because it sounds better", not 100 "because it's the ideal". Why where you are? Because a threshold at your current level does the two things you want: it doesn't allow going backward (any drop crosses it, it's lesson 4's ratchet) and it doesn't block healthy work (any PR that maintains or improves the coverage passes). It's the guard who checks the badge: it protects without getting in the way. When the coverage improves to 88 stably, you raise the threshold to 88 —you tighten the ratchet—, and the new level is protected. The coverage rises gradually, each improvement is consolidated, and at no point does the gate require the impossible or allow going backward.

The contrast with the aspirational threshold is total. The aspirational one (90 when you're at 62) blocks everything and gets worked around; the honest one (62 rising with a ratchet) blocks nothing legitimate and rises only when you truly improve. The aspirational one says "get to 90 now"; the honest one says "don't get worse, and improve at your pace". The first is a goal; the second is a mechanism. Goals motivate on a poster; mechanisms protect in CI. For the gate, you want the mechanism.

A concrete number for Reservo, closing the module's thread: Reservo is at 91.03% with the complete suite. The honest threshold is 91 —a ratchet at its real level—, which catches any drop (as you saw in lesson 4) without blocking any work that maintains the coverage. Not 100 (it would push tautologies, lesson 7), not 80 (it would let drops through down to 80). The 91 is where Reservo is, and that's why it's the threshold that protects it without getting in its way.

Coverage isn't quality: the underlying distinction

Beneath all this lesson's judgment there's an idea worth saying without circling: coverage is a necessary, not sufficient, condition of quality. Necessary, because code that runs in no test was never verified —low coverage is a legitimate alarm—. Not sufficient, because code that does run can be falsely verified: a test that calls the function but asserts nothing covers it 100% without checking it does the right thing.

This has a direct consequence for judgment. A coverage gate is good for what coverage does measure —"is there code no test touches?"— and dangerous when treated as what it doesn't measure —"are my tests good?"—. Used as a minimum floor ("let there be no code completely untested"), the gate helps. Used as a grade ("let's reach 95%, 100%"), it pushes the team to raise a number that doesn't reflect what they believe, and there lesson 7's harm begins. The same tool, two uses: one healthy (safety floor) and one toxic (grade to chase).

The rule that goes well with this: use coverage to find the untested code, not to grade the tested code. Lesson 3's term-missing report —the lines no test touches— is coverage at its best use: a map of where tests are missing. The total percentage as a goal to maximize is coverage at its worst use. A gate that says "don't leave code completely untested" (an honest floor or ratchet) is the first guard; one that says "reach 100% whatever it takes" is the second. Lesson 7 shows, with a brutal demo, exactly how the second use perverts the tests.

Common mistakes

Putting the threshold where you'd like to be, not where you are. What happens: real coverage 62%, gate at 90, and now every PR breaks the build —even the ones that improve the coverage—. Why it happens: 90 "sounds like a serious project" and seems motivating. How to spot it: if your gate breaks the build on PRs that raise the coverage, the threshold is above your reality. How to fix it: honest threshold —put it at 62, raise it with a ratchet when you improve—. A threshold that punishes improvements doesn't motivate improvement; it teaches people to work around the gate. The aspirational goal goes in the team conversation, not in the --cov-fail-under.

Treating the coverage percentage as the code's quality. What happens: the team celebrates reaching 95% and concludes "our code is excellent", when part of that 95% is tests that run without verifying. Why it happens: it's a number, and numbers feel objective; it's easy to forget what exactly it measures. How to spot it: ask yourself "if I break a function on purpose, does any test turn red?". If the coverage is high but the answer is sometimes no, your percentage measures execution, not verification. How to fix it: use coverage as a floor ("let there be no untouched code"), not as a grade ("how good the code is"). Test quality is judged by whether they catch bugs (lesson 7), not by the percentage they report.

Setting no gate for fear it'll get in the way. What happens: the team, burned by a bad gate in the past, decides to set none, and the coverage erodes unchecked. Why it happens: "this gate was miscalibrated" gets confused with "gates get in the way". How to spot it: if your coverage drops month over month and there's nothing stopping it, you're missing the gate that does help (the honest ratchet). How to fix it: the problem wasn't the gates, it was the aspirational threshold. An honest gate —ratchet at your current level— protects without getting in the way; it's the first guard. Don't throw out the good guard over the trauma of the bad guard. The answer to a bad gate is a well-calibrated gate, not the absence of gates.

Exercises

Exercise 1 — Helps or gets in the way. For each gate, say whether it probably helps or gets in the way, and why in one or two sentences. (a) Coverage ratchet at 91% over Reservo (real coverage 91%). (b) Coverage gate at 100% over a project at 78%. (c) Smoke gate that requires the pro-discount test to pass before merging. (d) Gate that requires 85% coverage and breaks the build when someone deletes 200 lines of dead code (lowering the coverage to 84%).

See solution
  • (a) Helps. Honest threshold: it's at the real level (91), catches any drop (ratchet) and doesn't block work that maintains the coverage. The first guard: protects without getting in the way.
  • (b) Gets in the way. Aspirational threshold and on top of that the extreme: 100% over a project at 78% blocks every PR and —worse— pushes toward tautologies to close the remaining 22% (lesson 7). The second guard: requires the impossible and teaches people to work around it.
  • (c) Helps. It protects something that really can't break (the pro discount charges real money), at a tiny cost (running a test). The cost/benefit proportion is clearly favorable. First guard.
  • (d) Gets in the way (at that moment). It punishes a legitimate refactor: deleting dead code improved the project, but the gate breaks over the coverage arithmetic. It's the case where judgment must intervene —give slack, or lower the threshold to 84 because the previous 91 included dead lines that no longer exist—. A gate that penalizes cleaning up teaches people not to clean up.

The pattern: (a) and (c) help —honest threshold, proportionate protection—; (b) and (d) get in the way —impossible threshold, punishment of the good—.

Exercise 2 — Fix the aspirational threshold. A team has Reservo at 84% coverage and a gate at --cov-fail-under=95 "because we want quality". Every PR breaks the build, people are frustrated, and two people already forced merges skipping the gate. Diagnose the problem and propose a concrete fix, with numbers.

See solution

Diagnosis: it's the bad guard's aspirational threshold. The real coverage is 84% and the gate requires 95%, so every PR that doesn't raise 11 points at once breaks the build —including the PRs that improve the coverage (from 84 to 87 is still < 95)—. The gate doesn't separate good work from bad; it blocks everything alike. And the classic symptom already appeared: people learned to work around it (force merges), which is worse than having no gate, because now the controls are skipped out of habit.

Concrete fix: lower the threshold to the honest level and turn it into a ratchet.

  1. Set --cov-fail-under=84 (or 83 for slack). Now the gate reflects where the project is: it doesn't break the build on work that maintains the coverage, and it does catch any drop below 84. It stops blocking the legitimate.
  2. Raise the threshold with a ratchet when the coverage improves. When they reach 87 stably, --cov-fail-under=87. The coverage rises gradually, each improvement is consolidated, without requiring the impossible jump.
  3. Leave the 95% as a conversation goal, not a CI one. If the team wants to reach 95, let it be an objective they discuss and plan (writing tests for the weak modules), not a wall that breaks every PR. The goal motivates in planning; the ratchet protects in CI.

With this, the gate goes from hated and evaded guard to respected guard: it protects the real level (you can't go backward) without blocking anyone who works well. And the incentive to force merges disappears, because complying with the gate becomes possible again.

Exercise 3 — The necessary/sufficient distinction. A teammate says: "We have 92% coverage, so our code is 92% well tested and high quality." Correct the claim using the necessary/sufficient distinction, and give a concrete example of how 92% coverage can coexist with tests that verify nothing.

See solution

The claim confuses coverage with test quality, which are different things. The correction, with the distinction:

  • Coverage is necessary but not sufficient. The 92% says 92% of the lines run when the tests run. That's necessary for quality —code that never runs was never verified— but not sufficient: a line running says nothing about whether any test asserts it does the right thing. "92% executed" isn't "92% verified".

  • Concrete example of coexistence: imagine Reservo has a test like this for booking_confirmed_message:

    def test_confirmed_message_runs():
    msg = booking_confirmed_message(booking)
    assert msg is not None

    This test runs the whole function —covers it 100%, raises the percentage— but only asserts that the result "isn't null". If tomorrow someone breaks the message and returns "totally wrong text", the test keeps passing (the wrong text isn't null either). The line is "covered", the percentage looks good, and yet the function is broken and no test notices. That's a tautological test: it contributes coverage without contributing verification.

The correct claim would be: "We have 92% coverage, which means almost all the code runs in the tests —good, there are no big untouched areas—. How much of that is really verified depends on whether our tests assert the correct behavior, which is another question coverage doesn't answer." Lesson 7 shows this trap in action, executed for real.

Summary and next step

In this lesson you took the step from the "how" to the "when": the judgment for deciding which gates to set. A gate is like a guard —it can check what matters and protect without getting in the way (the first guard), or require what doesn't separate the good from the bad, block healthy work, and teach the team to work around it (the second, which is worse than none)—. You saw the three cases where a gate helps —it stops an invisible erosion, makes explicit a slid decision, protects the business-critical stuff— and the three where it gets in the way —aspirational threshold that blocks healthy work, metric confused with quality, punishment of legitimate refactors—.

You take away two underlying rules. The honest threshold one: put it where you are, not where you'd like to be, and raise it with a ratchet —the mechanism that protects without getting in the way, versus the aspirational goal that frustrates and gets worked around—. And the underlying distinction: coverage isn't quality; it's a necessary condition (code that never ran was never verified) but not sufficient (code that ran can be falsely verified). Use coverage to find the untested code, not to grade the tested code.

Before moving on you should be able to: distinguish a gate that helps from one that gets in the way by whether it checks the right thing with the right requirement; apply the honest threshold rule (current level + ratchet) versus the aspirational one; explain why a gate that gets in the way is worse than none; and articulate why coverage is necessary but not sufficient for quality.

What's next, in lesson 7, is the brutal demonstration of the worst use of a gate: 100% as a fetish. When the threshold becomes a goal to maximize whatever it takes, the incentive is perverted and the team writes tautological tests —which run the code to raise the number without verifying anything—. You're going to see, executed for real, a test that raises the coverage to green and that keeps passing when the code is broken on purpose: the "green that doesn't verify" from the fundamentals guide, now turned into the direct product of a misunderstood gate.

Resources

← Previous5. Marker gates and smoke testsNext →7. 100% as a fetish that pushes tautologies