7. 100% as a fetish that pushes tautologies

Description

This lesson is the dark side of everything above, and the proof that a misunderstood gate isn't neutral: it does active harm. You already saw that coverage is a necessary but not sufficient condition of quality (lesson 6). Here you'll see what happens when a team forgets that distinction and treats coverage as a goal to maximize whatever it takes —the 100% fetish—. The incentive is perverted in a concrete and demonstrable way: instead of writing tests that verify the code does the right thing, people write tautological tests that only run the code to raise the number. The gate is met, the percentage rises to green, and yet the code is no more protected than before —sometimes, not one bit—.

By the end you'll have seen the trap executed for real, not described. A tautological test —one that calls a Reservo function and only checks that the result "isn't null"— raises that function's coverage and passes green. But when we break the function on purpose —make it return garbage text— that tautological test keeps passing (exit 0), noticing nothing, while a real test that asserts the content catches the bug (exit 1). It's the "green that doesn't verify" from the testing fundamentals guide, now turned into the direct product of a misused coverage gate. You'll take away the lesson that crowns the module: coverage measures what ran, not what was verified, and chasing the number for its own sake produces tests that raise the coverage and don't catch bugs —the worst of both worlds, because they also give a false sense of security—.

Connection to the module: this lesson is the exact counterpart of lesson 3. There, the red gate pushed you to write a test that verifies (it raised the coverage and protected the code) —put out the fire—. Here, the threshold fetish pushes you to write a test that only executes (it raises the coverage and protects nothing) —rip out the detector's battery, but worse, because the detector now seems on—. It closes the module's arc: the gate is a tool, and like every tool, its value depends on how it's used. Used well (lessons 3-6), it catches real erosion; fetishized, it produces the theater this lesson exposes. It's also the strongest connection with the fundamentals guide: the "green that doesn't verify" you learned there is, in CI, the byproduct of a gate turned into a fetish.

The school that teaches for the exam, not for learning

Imagine two schools whose performance is measured by a single number: the percentage of students who pass a standardized exam. The first school takes the number for what it is —an imperfect signal of learning— and really teaches: the students understand, and because they understand, they pass. The number rises because the learning rose. The measurement and what it measures move together.

The second school fetishizes the number. The goal stops being "let them learn" and becomes "let the percentage rise", whatever it takes. How do you raise a pass percentage without teaching? By teaching for the exam: the answers to old exams are memorized, the tricks to guess are drilled, the format is practiced to exhaustion. The students pass —the number rises, the goal is met— but they learned nothing; they just learned to pass that exam. And here's the perverse part: the second school looks, in the number, just as good as the first. The 95% pass rate doesn't distinguish "they learned" from "they memorized the trick". The number was met and the substance was lost, and worse, the number no longer serves to distinguish one school from the other.

This has a name: Goodhart's law —"when a measure becomes a target, it stops being a good measure"—. A metric works as a signal as long as no one manipulates it directly; the moment it becomes the goal, people find ways to move it without moving what it was supposed to measure, and the metric breaks. Coverage is exactly like this. As a signal —"is there untested code?"— it's useful. As a goal —"let's reach 100%"— it becomes manipulable: you can raise it by writing tests that run without verifying, just as you raise a pass percentage by teaching for the exam. The number is met; the substance —tests that catch bugs— is lost. This lesson is that law, demonstrated with Reservo code.

Goodhart's law: when a measure becomes a target, it stops being a good measure. Coverage as a signal ("is there untested code?") is useful; as a goal to maximize ("let's reach 100%") it becomes manipulable —you raise it by writing tests that run without verifying, like a school that teaches for the exam—. The number is met; the protection is lost.

What a tautological test is

A tautology is a statement that's true by its own form, without saying anything about the world: "tomorrow it will rain or it won't rain" is true no matter what, and that's why it informs nothing. A tautological test is the equivalent in code: a test whose assert is true almost no matter what, so it runs the code —and therefore raises the coverage— but doesn't verify that the code does the right thing —and therefore doesn't catch bugs—.

The most common pattern is asserting something trivially true about the result. Reservo has a function that builds the confirmation email text:

# reservo/notifications.py
def booking_confirmed_message(booking: Booking) -> str:
    """Email body when a booking is confirmed."""
    return (
        f"Your booking {booking.id} for {booking.room.name} is confirmed. "
        f"Total: {booking.price_cents} cents."
    )

A tautological test for this function looks like this —and it's tempting, because it's short, always passes, and raises the coverage—:

# tautological: runs the function but asserts nothing about the content
def test_confirmed_message_tautological():
    msg = booking_confirmed_message(b)
    assert msg is not None       # when would it be None? Almost never.
    assert isinstance(msg, str)  # a function that builds an f-string ALWAYS gives a str

Look at the asserts. msg is not None: the function builds an f-string and returns it; it never returns None, so that assertion is true no matter what happens with the content. isinstance(msg, str): an f-string is always a str, so it's also always true. The test runs the whole booking_confirmed_message —covers it—, but checks nothing of what the function should do: it doesn't verify the message mentions the booking, or the room, or the price. It's the tautology: true by its form, blind to the world. And it fulfills exactly its perverse purpose —raising the coverage— without fulfilling a test's real purpose —verifying—.

The demo: the tautological test raises the coverage

Let's first verify that the tautological test really raises the coverage, which is what makes it tempting for whoever chases the number. We run only that test, measuring booking_confirmed_message's coverage, for real on Python 3.14.0:

python -m pytest test_notifications_tautological.py --cov=reservo.notifications --cov-report=term-missing

What to expect (real output):

Name                       Stmts   Miss  Cover   Missing
--------------------------------------------------------
reservo/notifications.py       7      3    57%   16-21
1 passed in 0.01s

The test passes (1 passed) and covers booking_confirmed_message: notifications.py's coverage rises to 57% —the confirmed function's lines, executed; the missing ones, 16-21, are from the other function this test doesn't touch—. For whoever chases the percentage, mission accomplished: they wrote a three-line test, it passed, and the coverage rose. If their goal is "let the number rise", this test is a success. Hold that "success" in mind, because now comes the proof that it's a fraud.

The demo: the tautological test doesn't catch the bug

The question that separates a real test from a tautological one is the fundamentals module's: does the test turn red when the code breaks? A test that doesn't bite when the code is wrong protects nothing, however much coverage it contributes. Let's verify it by breaking booking_confirmed_message on purpose: we make it return garbage text —a totally wrong message that should never reach a customer—:

# reservo/notifications.py, BROKEN on purpose
def booking_confirmed_message(booking: Booking) -> str:
    return "WRONG: totally broken confirmation text"

This code is broken in the worst way: it sends the customer a nonsensical text, without their booking, without their room, without their price. A test worth anything should catch it. We run the tautological test against this broken code:

python -m pytest test_notifications_tautological.py

What to expect (real output):

1 passed in 0.01s

And the exit code:

python -m pytest test_notifications_tautological.py > /dev/null 2>&1; echo "exit code: $?"

exit code: 0

1 passed, exit code 0. The tautological test passes over the broken code. Think about it: booking_confirmed_message now returns "WRONG: totally broken confirmation text", and the test says everything's fine. Why? Because its asserts are still true: "WRONG..." isn't None (msg is not None ✓) and is a str (isinstance(msg, str) ✓). The garbage text meets the two tautologies just like the correct text. The test covered the function, passed green, raised the coverage —and didn't notice the function is catastrophically broken—. It's the "green that doesn't verify" in its purest state: a green that means nothing, because the red was impossible.

The demo: a real test does catch it

Let's contrast with a test that verifies for real —one that asserts the content the function should produce—:

# real test: asserts that the message mentions the booking, the room, and the price
def test_confirmed_message_names_the_booking_and_price():
    msg = booking_confirmed_message(b)
    assert "bk-1" in msg     # the booking
    assert "Focus" in msg    # the room
    assert "7500" in msg     # the price

These asserts aren't tautological: they assert something concrete about the world —that the message contains the booking id, the room name, and the price—. If the function returns the correct text, they pass; if it returns something else, they fail. Let's run this test against the same broken code (booking_confirmed_message returning garbage):

python -m pytest test_notifications_real.py

What to expect (real output):

>       assert "bk-1" in msg
E       AssertionError: assert 'bk-1' in 'WRONG: totally broken confirmation text'
tautdemo/test_notifications_real.py:15: AssertionError

FAILED test_notifications_real.py::test_confirmed_message_names_the_booking_and_price
================================ 1 failed in 0.03s =============================

And the exit code:

python -m pytest test_notifications_real.py > /dev/null 2>&1; echo "exit code: $?"

exit code: 1

1 failed, exit code 1. The real test catches the bug: assert 'bk-1' in 'WRONG: totally broken confirmation text' fails, because the garbage text doesn't contain the booking id. Red. This test verifies, so it bites when the code is wrong.

Put the two demos side by side, because the whole lesson is in the contrast:

Both raise the coverage equally —a coverage gate doesn't distinguish them—. But one protects and the other doesn't. The gate sees the identical number; the reality is opposite. That's exactly the gap through which the 100% fetish inserts the fraud: if your goal is the number, both tests "are worth the same"; if your goal is catching bugs, one is worth it and the other is worse than nothing, because it gives false confidence.

Why the 100% fetish pushes exactly toward this

Connect the mechanism. When the threshold is honest (lesson 6) —a floor or ratchet at your real level—, the gate pushes you to test the untested code: you write tests for the functions no test touches, and since those functions do something, your tests naturally verify that something. The incentive points to verification.

When the threshold becomes a fetish —"we have to reach 100%"—, the incentive twists. The last coverage points are usually the hardest and least valuable: rare error branches, defensive lines, code that almost never runs. Testing that for real is laborious. But the fetish doesn't ask for verification, it asks for the number. And the fastest way to move the number without doing the real work is the tautological test: call the function, assert something trivial, cover the line, raise the percentage. The fetish rewards exactly the shortcut that doesn't verify. The higher and more inflexible the threshold, the stronger the push toward tautology —because closing the last 5% with real tests is expensive, and with tautologies it's free—.

That's why 100% is a particularly toxic fetish. It's not just hard to reach with real tests; it's that the effort of reaching it gets channeled into the false tests, because they're the cheap way to meet the goal. A team with the gate at 100% frequently ends up with more tautologies than a team with the gate at an honest 85% —the first paid the last 15% with fraud; the second didn't have to fake it—. The more ambitious goal produced worse tests. It's Goodhart's law collecting its price: the metric, turned into an extreme target, stopped measuring what mattered and started rewarding its falsification.

How to defend yourself: verify, not just execute

The defense isn't to abandon coverage —it's still a useful signal—; it's to use it well and complement it:

• Honest threshold, not fetish. Floor or ratchet at your real level (lesson 6), not aspirational 100%. It removes the incentive for tautologies at the root: if you don't chase the last point at all costs, you don't need to falsify it.
• Judge the tests by whether they bite, not by what they cover. The fundamentals question: does this test turn red when I break the code? A test that doesn't bite isn't worth it, whatever it covers. The technique of breaking the code on purpose (manual mutation) you used in the demos is how you verify.
• Review the asserts, not just the test's existence. In code review, an assert msg is not None over a function that builds a string should raise an eyebrow: what is it really verifying? A test without a substantive assert is suspected of tautology.
• Coverage points to where tests are missing; you decide whether the test you write verifies. Use the term-missing report to find the untested code (good use), and write for those lines tests that assert real behavior —not tautologies to plug the gap in the number—.

In one sentence: coverage measures what ran, not what was verified, so real protection doesn't come from the percentage but from each test asserting something that would break if the code were wrong. An honest coverage gate tells you where to look; that what you write there really verifies is your responsibility, and no gate can substitute it.

Common mistakes

Writing tests for the number, not for the bug. What happens: someone sees the red gate or the report with an uncovered function and writes the shortest test that covers it —assert result is not None—, without asking what it should verify. The coverage rises, the bug is left netless. Why it happens: covering the line is the apparent objective (what the gate asks for), and verifying is more work. How to spot it: for each test, ask yourself "if I break the function it tests, does this test turn red?". If not, it's tautological. How to fix it: write the assert that asserts the real behavior —what the code should do, not that it "returns something"—. Coverage is a side effect of a good test, not its goal.

Chasing 100% as if it were an achievement. What happens: a team sets the goal at 100% and celebrates reaching it, without noticing the last stretch was closed with tautologies. Why it happens: 100% is round, satisfying, and sounds like excellence. How to spot it: look at the tests that cover the rarest branches; if they're assert x is not None or assert True, the 100% is cardboard. How to fix it: accept that the correct number is almost never 100 —the last branches usually cost more than they're worth to test for real—, and prefer an honest 85% of tests that bite to an inflated 100% of tautologies. Goodhart's law: the moment 100% is the goal, it stops meaning "well tested".

Trusting coverage as if it measured test quality. What happens: "92% coverage" is read as "92% well tested", without distinguishing tests that verify from tests that only execute. Why it happens: coverage is the only number at hand, and it's tempting to let it mean more than it means. How to spot it: this lesson's demo —two tests with the same coverage, one protects and the other doesn't—. If your confidence comes from the percentage and not from having seen your tests bite, it's borrowed confidence. How to fix it: remember that coverage measures execution, not verification. Test quality is verified by breaking the code and seeing which turn red, not by reading a percentage.

Exercises

Exercise 1 — Detect the tautology. For each Reservo test, say whether it's tautological (runs without verifying) or real (verifies), and explain how you know. (a) assert price_cents(focus, pro, 3) == 6000. (b) assert price_cents(focus, pro, 3) is not None. (c) assert isinstance(refund_cents(booking, 6000, now), int). (d) assert refund_cents(booking, 6000, now) == 6000.

Exercise 2 — Turn the tautology into verification. A teammate wrote this test for Reservo's refund_reason (which returns "full refund", "half refund", or "no refund"). It covers the function and passes, but it's tautological. Rewrite it so it verifies for real, and explain what bug yours would catch that theirs wouldn't.

def test_refund_reason():
    reason = refund_reason(booking, datetime(2026, 1, 1, 9))  # 72h before
    assert reason is not None
    assert len(reason) > 0

def test_refund_reason_matches_the_tier():
    # 72h before (>= 48h) -> full refund
    assert "full refund" in refund_reason(booking, datetime(2026, 1, 1, 9))
    # 36h before (24-48h) -> half refund
    assert "half refund" in refund_reason(booking, datetime(2026, 1, 2, 21))
    # 9h before (< 24h) -> no refund
    assert "no refund" in refund_reason(booking, datetime(2026, 1, 4, 0))

Exercise 3 — The 100% dialogue. Your lead says: "Let's raise Reservo's coverage gate to 100%. More coverage is more quality, and 100% is the goal." Using the module's ideas, answer them: why can 100% as a gate produce worse test code than a lower threshold? Give the argument and a concrete Reservo example.

Summary and next step

In this lesson you exposed the worst use of a gate: 100% as a fetish. When coverage stops being a signal and becomes a goal to maximize, Goodhart's law collects its price —"when a measure becomes a target, it stops being a good measure"—: the team, pushed to raise the number, writes tautological tests that run the code without verifying it, like a school that teaches for the exam instead of for learning.

You saw it executed for real, not described. A tautological test —assert msg is not None— covered booking_confirmed_message and passed green, raising the coverage. But when you broke the function (garbage text), that test kept passing (exit 0), blind to the disaster, while a real test that asserts the content —assert "bk-1" in msg— caught the bug (exit 1). Both gave the same coverage; only one protected. That contrast is the whole lesson: coverage measures what ran, not what was verified, and chasing the number for its own sake produces the worst —tests that raise the coverage, don't catch bugs, and on top of that give false confidence—. It's fundamentals' "green that doesn't verify", now as a direct byproduct of a fetishized gate.

Before moving on you should be able to: define a tautological test and detect it by asking "what values would make this assert fail?"; explain why coverage doesn't distinguish a test that verifies from one that only executes; articulate why 100% as a goal pushes toward tautologies (Goodhart); and turn an assert x is not None into a real behavior verification.

What's next, in lesson 8, is the mini-project that closes the module: you set up a coverage gate on Reservo's CI from start to finish, and —the key requirement— you make it break the build when a test is missing. You're going to pull it all together: the YAML workflow with the --cov-fail-under step, the .coveragerc, the local demonstration of the red→green cycle (65.38% exit 1 → you add the test → 91.03% exit 0), the smoke gate as a separate job, and a decision note that justifies —with everything from lessons 6 and 7— which threshold Reservo deserves and why not 100%.

Resources

• Coverage.py: what coverage is and isn't — the official documentation insists that coverage measures executed lines, not verified behavior. The technical basis of the whole lesson.
• pytest.raises: verify exceptions for real — how to test an error branch by verifying the correct exception, instead of "covering" it with a tautology. The tool to make the if hours <= 0: raise ValueError test real.
• Goodhart's law (concept) — "when a measure becomes a target, it stops being a good measure". It's not a pytest concept but a measurement one in general, but it explains exactly why coverage as a goal corrupts. Think of it every time someone proposes maximizing a metric.
• How to write useful assertions in pytest — the reference for assert in pytest; the key against tautologies is that each assertion asserts something that would break if the code were wrong. A test is worth its asserts, not the lines it runs.