Module 6: Quality Gates Coverage Thresholds

8. Mini-project: a coverage gate for Reservo's CI

Description

The time has come to pull the whole module together into one deliverable. In the previous seven lessons you learned what a quality gate is, how --cov-fail-under implements it with its exit code, how to make it fail on a drop with a ratchet, how to set up a smoke marker gate, when a gate helps and when it gets in the way, and why 100% as a fetish pushes toward tautologies. Now you apply it from start to finish: you set up a coverage gate on Reservo's CI and make it break the build when a test is missing, with the local evidence that it really bites.

This isn't a stray exercise: it's the real work you'd do hardening a project's CI. You're going to produce five concrete deliverables —the YAML workflow with the coverage gate and the smoke gate, the .coveragerc that makes it honest, the local demonstration of the red→green cycle (the gate breaks the build with cancel_with_refund untested, you add the test, the gate passes), the verification that the smoke gate also bites, and a decision note that justifies which threshold Reservo deserves and why not 100%—. All with the guide's honesty: the YAML is content (no runner here), but the pytest --cov runs are real, executed on Python 3.14.0 with coverage 7.15.2 and pytest-cov 7.1.0.

Connection to the module: this lesson closes the arc. Lesson 1 gave you the concept (measuring vs imposing); lesson 2 defined it (metric, threshold, consequence); lesson 3 executed it (the red→green cycle); lesson 4 fine-tuned it (the ratchet against drops); lesson 5 diversified it (the smoke gate); lesson 6 judged it (when it helps); lesson 7 exposed it (the 100% fetish). The mini-project exercises them all at once on Reservo. And it looks ahead: by the end you'll have a CI that imposes quality, not just measures it —which makes module 7's question urgent—: what happens when a test the gate requires is flaky, and passes sometimes and fails sometimes without the code changing? You close the quality gates; module 7 attacks the inconsistency.

The assignment

You're responsible for Reservo's CI. The base pipeline already exists (module 2): it runs the suite on every push. Your job now is to harden it with quality gates, because the team noticed that untested code (cancel_with_refund) slipped in without anything stopping it. Your assignment:

Write the workflow that runs the suite with a coverage gate (--cov-fail-under) that breaks the build when the coverage drops below the threshold.
Configure .coveragerc so the gate measures the reservo package and sees the unimported code.
Demonstrate locally that the gate bites: with cancel_with_refund untested it breaks the build (exit ≠ 0), and adding the test makes it pass (exit 0).
Add a smoke gate as a separate, staged job, and demonstrate it also bites.
Justify, with lessons 6 and 7's criteria, which threshold Reservo deserves and why an honest ratchet, not 100%.

Try each step on your own before looking at the solution. The complete solution is at the end, but the learning is in building it yourself.

Step 1 — The workflow with the coverage gate

Write .github/workflows/tests.yml. It must run on every push and PR, install Python and dependencies, and run pytest --cov=reservo --cov-fail-under=<threshold>. Remember: the gate lives in --cov-fail-under, not in --cov; without the threshold you only report.

Think about it before continuing: which threshold do you set —80 fixed, or Reservo's real level—? Which exit code will make CI break the build?

Step 2 — The `.coveragerc` that doesn't lie

Configure what's measured. Without this, coverage would include the tests and ignore the unimported code (like cancellations.py), giving an inflated and blind number.

Think about it: which option makes coverage discover the package's files even if no test imports them?

Step 3 — Demonstrate the gate bites (red→green)

Run the gate locally, first with cancel_with_refund untested (it must break the build), then adding the test (it must pass). Capture both exit codes: the evidence that the gate really breaks the build when a test is missing is half the deliverable.

Step 4 — The smoke gate, staged

Add a job that runs pytest -m smoke before the complete suite (needs:). Mark the anchor tests, and demonstrate the smoke gate bites by breaking a critical rule.

Step 5 — The decision note

Justify the threshold. 80 fixed or 91 (the real level)? Why not 100? Apply the honest threshold rule and Goodhart's law.

Complete solution

Deliverable 1 — The YAML workflow with the gates

# .github/workflows/tests.yml
name: tests

on: [push, pull_request]

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - name: Check out the code
        uses: actions/checkout@v5

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.14"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Smoke gate (critical tests only)
        run: python -m pytest -m smoke

  coverage-gate:
    needs: smoke          # only runs if the smoke gate passed
    runs-on: ubuntu-latest
    steps:
      - name: Check out the code
        uses: actions/checkout@v5

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.14"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Full suite with coverage gate
        run: python -m pytest --cov=reservo --cov-fail-under=91

The decisions and their why:

--cov-fail-under=91, not 80 or 100. The 91 is Reservo's real level (the complete suite's coverage), an honest ratchet: it catches any drop without blocking work that maintains the coverage, and without pushing tautologies. (Lessons 4, 6, and 7; justified in depth in deliverable 5.)
--cov=reservo measures the code, not the tests. (Lesson 3.)
Two staged jobs with needs: smoke: the smoke gate (fast, critical) runs first; the complete suite with coverage only if smoke passes. Fast feedback on the essentials. (Lesson 5.)
on: [push, pull_request] — the gate protects both pushes and PRs toward the main branch. (Module 2.)

The requirements.txt the workflow installs:

# requirements.txt
pytest==9.1.1
pytest-cov==7.1.0

Reservo is pure stdlib; the only dependencies are pytest and pytest-cov (for the coverage gate). Pinning them is module 3's lesson: deterministic installs so the cell runs the same as you.

Deliverable 2 — The `.coveragerc`

# .coveragerc
[run]
source = reservo

[report]
show_missing = True

source = reservo does two critical things: it limits the measurement to the reservo package (excluding the test files, which would otherwise inflate the number to 100% of themselves) and makes coverage discover all the package's files, imported or not. Without this, cancellations.py —which no test imports until we test it— would be invisible, and the gate would blind itself to exactly the untested code it should catch. This line is what makes the gate honest. (Lesson 3.)

Deliverable 3 — The local demonstration: red → green

This is the evidence that the gate bites, executed for real. The guide's honesty: the YAML is content, but this is real —Python 3.14.0, pytest 9.1.1, coverage 7.15.2, pytest-cov 7.1.0—.

State 1: cancel_with_refund untested → the gate breaks the build.

python -m pytest --cov=reservo --cov-report=term-missing --cov-fail-under=91

What to expect (real output):

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /private/tmp/reservo-m6
plugins: cov-7.1.0
collected 10 items

tests/test_availability.py ....                                          [ 40%]
tests/test_pricing.py ...                                                [ 70%]
tests/test_refunds.py ...
ERROR: Coverage failure: total of 65 is less than fail-under=91
                                                                         [100%]

================================ tests coverage ================================
_______________ coverage: platform darwin, python 3.14.0-final-0 _______________

Name                       Stmts   Miss  Cover   Missing
--------------------------------------------------------
reservo/__init__.py            0      0   100%
reservo/calendar.py           30      7    77%   24, 26, 34, 49-51, 55
reservo/cancellations.py      17     17     0%   7-36
reservo/models.py             12      2    83%   34-35
reservo/pricing.py            10      1    90%   14
reservo/refunds.py             9      0   100%
--------------------------------------------------------
TOTAL                         78     27    65%
FAIL Required test coverage of 91% not reached. Total coverage: 65.38%
============================== 10 passed in 0.03s ==============================

python -m pytest --cov=reservo --cov-fail-under=91 > /dev/null 2>&1; echo "exit code: $?"

exit code: 1

The ten tests pass (10 passed), but cancellations.py is at 0% and the total coverage is 65.38%. Since 65 < 91, the gate breaks the build: exit code 1. On the runner, this 1 would block the merge. The gate did exactly what it was asked: it broke the build because a test is missing.

State 2: I add the missing test → the gate passes.

I write tests/test_cancellations.py that really exercises cancel_with_refund (verifying the refund, that the slot is freed, and that cancelling twice fails —tests that verify, not tautologies, lesson 7—). I run again:

python -m pytest --cov=reservo --cov-report=term-missing --cov-fail-under=91

What to expect (real output):

collected 13 items

tests/test_availability.py ....                                          [ 30%]
tests/test_cancellations.py ...                                          [ 53%]
tests/test_pricing.py ...                                                [ 76%]
tests/test_refunds.py ...                                                [100%]

================================ tests coverage ================================
Name                       Stmts   Miss  Cover   Missing
--------------------------------------------------------
reservo/__init__.py            0      0   100%
reservo/calendar.py           30      4    87%   26, 34, 50, 55
reservo/cancellations.py      17      0   100%
reservo/models.py             12      2    83%   34-35
reservo/pricing.py            10      1    90%   14
reservo/refunds.py             9      0   100%
--------------------------------------------------------
TOTAL                         78      7    91%
Required test coverage of 91% reached. Total coverage: 91.03%
============================== 13 passed in 0.03s ==============================

python -m pytest --cov=reservo --cov-fail-under=91 > /dev/null 2>&1; echo "exit code: $?"

exit code: 0

cancellations.py went to 100%, the total coverage to 91.03%, and the gate passed: exit code 0. The complete cycle, demonstrated: the gate broke the build over a missing test (exit 1), and adding it made it go green (exit 0). We didn't lower the threshold or delete code: we wrote the missing test. That's a quality gate doing its job.

Deliverable 4 — The smoke gate also bites

We register the marker in pytest.ini and mark the three critical anchor tests:

# pytest.ini
[pytest]
markers =
    smoke: fast, critical tests that must pass before any merge (the smoke gate).

With @pytest.mark.smoke on the basic price, the pro discount, and the full refund, the smoke gate runs only those three:

python -m pytest -m smoke

collected 13 items / 10 deselected / 3 selected
======================= 3 passed, 10 deselected in 0.02s =======================

Green (exit 0): 3 selected, 10 deselected. Now we verify it bites, by breaking the pro discount (20% → 25%):

# after changing PRO_DISCOUNT_PERCENT from 20 to 25 in reservo/pricing.py
python -m pytest -m smoke

collected 13 items / 10 deselected / 3 selected

tests/test_pricing.py::test_pro_three_hours FAILED                       [ 66%]
...
>       assert price_cents(focus, pro, 3) == 6000
E       AssertionError: assert 5625 == 6000

FAILED tests/test_pricing.py::test_pro_three_hours - AssertionError: assert 5625 == 6000
================== 1 failed, 2 passed, 10 deselected in 0.03s ==================

python -m pytest -m smoke > /dev/null 2>&1; echo "exit code: $?"

exit code: 1

Exit code 1. The smoke gate caught the pro-price bug (assert 5625 == 6000) in 0.03 s, running only 3 tests. In the staged pipeline, this red would stop everything at the first job —the complete suite with coverage wouldn't even start (needs: smoke)—, saving the time of the expensive run. We restore the discount to 20% and the smoke gate goes back to green (exit 0).

Deliverable 5 — The decision note: which threshold does Reservo deserve?

Reservo deserves an honest ratchet at 91% (its real level), not a floor of 80 or a goal of 100%. The reasoning, with the rules of lessons 4, 6, and 7:

91 instead of 80 (the ratchet against the drop). Reservo's real coverage is 91.03%. A floor of 80 would let drops through: someone could add untested code and lower the coverage to 84% without breaking the build (84 > 80), eroding quality silently (lesson 4). A threshold at 91 —stuck to the real level— catches any setback. It's the ratchet: it only goes up. When the coverage improves to 93 stably, the threshold is raised to 93; it never drops.
91 instead of 100 (avoiding the fetish). Setting the gate at 100% would push the team to close the last 9% with tautological tests —which run without verifying, raising the number without protecting anything (lesson 7)—. The lines missing today (models.py 83%, calendar.py 87%, rare defensive branches) cost more to test for real than they're worth, and chasing them badly would produce worse tests, not better. The honest 91 of tests that bite is worth more than a cardboard 100%. Goodhart's law: the moment 100% is the goal, it stops meaning "well tested".
The threshold is honest, not aspirational (lesson 6). 91 is where Reservo is, so it blocks no PR that maintains the coverage —it's the first guard, it protects without getting in the way—. An aspirational 95 would break the build on healthy work and teach the team to work around the gate.

Conclusion: --cov-fail-under=91, a ratchet at the real level, plus the smoke gate over the three anchor tests. It's what corresponds to what Reservo really risks: silent erosion (which the ratchet stops) and the business-critical stuff (which the smoke protects), without the fraud an aspirational 100% would produce. Coverage is used as a safety floor and a map of what's left to test, not as a grade to maximize.

Common mistakes

Delivering the YAML without the evidence that the gate bites. What happens: someone writes a correct --cov-fail-under but never verifies it really breaks the build when the coverage is low. Why it happens: the YAML "looks fine" and gives the sense of finished work. How to spot it: if you don't have a run with exit 1 (low coverage) and another with exit 0 (after adding the test), you didn't verify the gate bites, you only wrote intentions. How to fix it: run the gate in both states and capture the two exit codes; that evidence is half the deliverable, because the YAML is content and the run is what's real. A gate you never saw break the build is a gate you shouldn't trust.

Setting the threshold at 100% "to maximize quality". What happens: someone sets --cov-fail-under=100 believing it requires the best, and the team closes the last stretch with tautologies. Why it happens: 100% sounds like excellence. How to spot it: if your rare-branch tests are assert x is not None, your 100% is cardboard (lesson 7). How to fix it: honest threshold at the real level (91), which protects without pushing fraud. The correct number is almost never 100; prefer tests that bite to a round percentage.

Forgetting the .coveragerc and measuring the wrong thing. What happens: without source = reservo, the gate includes the tests (inflating the number) and ignores cancellations.py (the untested code), giving a false green. Why it happens: by default coverage measures what runs, and the tests run. How to spot it: if your report has test_*.py rows at 100% and you don't see the untested modules, you're measuring the wrong thing. How to fix it: source = reservo to measure the code and discover the unimported files. A gate blind to the untested code is worse than none: it gives false confidence.

Exercises

Exercise 1 — The gate catches a second gap. After your work, Reservo's coverage is 91% with the gate at 91. A teammate adds notifications.py (the email texts) without tests, lowering the coverage to 83.53%. Without running anything, predict: does the gate at 91 break the build? And a floor-80 gate? Explain the difference and what the teammate should do.

See solution

The gate at 91 (ratchet): breaks the build (exit 1). 83.53% < 91, so --cov-fail-under=91 fails (total of 84 is less than fail-under=91). The ratchet catches the drop: untested code entered and the gate gives it away.
A floor-80 gate: passes (exit 0). 83.53% ≥ 80, so a floor of 80 wouldn't see the drop —lesson 4's blind spot—. The untested code would sneak in above the floor.

The difference is exactly why you chose 91 and not 80: a distant floor lets through drops that a ratchet at the real level catches. The drop from 91 to 83.53% (almost eight points, a whole module untested) is invisible to the floor-80 and red for the ratchet-91.

What the teammate should do: write the notifications.py tests that verify the messages' content (not tautologies like assert msg is not None, but assert "bk-1" in msg and so on), raising the coverage back to 91 or more. Put out the fire, don't lower the threshold. And if the new coverage rises to 93, tighten the ratchet to 93.

Exercise 2 — Detect the YAML with the fake gate. A teammate delivers this workflow "with a coverage gate". It has a problem that makes the gate not bite. Find and fix it.

- name: Run tests with coverage
  run: |
    python -m pytest --cov=reservo || true
    echo "coverage measured"

See solution

There are two problems, and either one makes the gate not bite:

--cov-fail-under is missing. The command has --cov=reservo (which reports the coverage) but not --cov-fail-under=N (which imposes it). Without a threshold there's no gate: it only measures (lesson 2's sign). The coverage could be 10% and the step would pass.
The || true nullifies the exit code. Even if there were a --cov-fail-under, the || true at the end makes the command always finish at exit 0, no matter what pytest returns. || true means "if the command on the left fails, run true (which gives exit 0) instead", erasing the red. The gate's consequence —the non-zero exit code— is neutralized. It's like putting a turnstile and then leaving the barrier open.

Fixed:

- name: Run tests with coverage gate
  run: python -m pytest --cov=reservo --cov-fail-under=91

Now yes: --cov-fail-under=91 sets the threshold (the gate bites if the coverage drops below 91), and without || true the failure exit code reaches CI and breaks the build. The lesson: a gate needs all three parts alive —metric (--cov), threshold (--cov-fail-under), and the consequence intact (the exit code, which || true was killing)—. Any broken link and the gate becomes an ornament.

Exercise 3 — Justify the staging (or not). Your Reservo pipeline has the smoke gate (0.02 s) staged before the complete suite with coverage (0.03 s), with needs: smoke. A teammate says: "Reservo runs in hundredths of a second; the staging just adds latency. Let's run everything in parallel." Are they right for Reservo? Would it change if Reservo had a matrix of three versions that takes 10 minutes?

See solution

For Reservo as it stands, the teammate is right. The staging with needs: smoke makes the complete suite wait for smoke to finish, adding latency in the happy case. That cost is only justified if you avoid something expensive —running the heavy suite when smoke already failed—. With everything in hundredths of a second, there's nothing expensive to avoid: running smoke and the complete suite in parallel finishes sooner, and the staging just adds one job waiting for the other without buying any saving. For a trivially fast suite, parallel is better.

If Reservo had a 10-minute matrix (module 4), the answer inverts: staging wins. Now there is something expensive to protect. If someone breaks the pro discount, the smoke gate catches it in 0.02 s; staged with needs: smoke, the 10-minute matrix never starts, saving those 10 minutes (× the matrix cells, × each broken push). The cost of staging —waiting 0.02 s for smoke to pass in the happy case— is negligible against the 10 minutes you save in the broken case. The rule (lesson 5): stage when what goes behind the gate is expensive and the failures the gate catches are frequent; run in parallel when everything is cheap. The number decides, not the reflex of "staging is always better". For Reservo today: parallel. For Reservo with a big matrix: staged.

Summary and next step

In this mini-project you set up a quality gate on Reservo's CI from start to finish, and —the key requirement— demonstrated that it breaks the build when a test is missing. You delivered the YAML workflow with the coverage gate (--cov-fail-under=91, an honest ratchet) and the staged smoke gate (needs: smoke); the .coveragerc with source = reservo that makes the measurement honest; the local demonstration of the red→green cycle —the gate breaking the build with cancel_with_refund untested (65.38%, exit 1), you adding the test that verifies, and the gate passing (91.03%, exit 0)—; the smoke gate biting the pro-discount bug (assert 5625 == 6000, exit 1); and the decision note that justifies the 91 —ratchet against the drop, not the 80 that would let it through, not the 100% that would push tautologies—.

This closes the module. Now you know how to turn a CI that measures into one that imposes: the difference between measuring and imposing quality, how --cov-fail-under implements it with its exit code, how a ratchet catches the drops a floor ignores, how a smoke gate protects the critical stuff, when a gate helps and when it gets in the way, and why 100% as a fetish produces tests that raise the coverage without catching bugs. Your pipeline no longer just tells you how the quality is; it demands it.

What's next, in module 7, is the ghost that haunts everything you built: flaky tests. A quality gate requires certain tests to pass —the smoke, the whole suite for coverage—. But what happens when a test passes sometimes and fails sometimes without the code changing? That non-deterministic test breaks the gate randomly, erodes trust in CI (the team starts re-running the builds "to see if it passes this time"), and deserves its own treatment: the retry debate, quarantine, and the failure that only happens in CI. You closed the gates that impose quality; module 7 deals with what makes them unreliable when they shouldn't be.

Resources

pytest-cov: --cov-fail-under and options — the reference for the coverage gate you built. Come back to it when you set the gate on your own project.
Coverage.py: configuration with .coveragerc — how source and the [report] options define what's measured and how it's reported. The piece that makes the gate honest.
Build and test Python — GitHub Actions — the official guide to testing Python in CI, with coverage and job examples. The frame of the workflow you wrote.
Job dependencies with needs — GitHub Actions — how to stage the smoke gate before the complete suite. Module 7 (flaky tests) starts from the gates this project left built.

← Previous7. 100% as a fetish that pushes tautologies Next →1. Module introduction: the flaky test in CI

8. Mini-project: a coverage gate for Reservo's CI

Description

The assignment

Step 1 — The workflow with the coverage gate

Step 2 — The .coveragerc that doesn't lie

Step 3 — Demonstrate the gate bites (red→green)

Step 4 — The smoke gate, staged

Step 5 — The decision note

Complete solution

Deliverable 1 — The YAML workflow with the gates

Deliverable 2 — The .coveragerc

Deliverable 3 — The local demonstration: red → green

Deliverable 4 — The smoke gate also bites

Deliverable 5 — The decision note: which threshold does Reservo deserve?

Common mistakes

Exercises

Summary and next step

Resources

Step 2 — The `.coveragerc` that doesn't lie

Deliverable 2 — The `.coveragerc`