6. Why the unit test can't see it

Description

Throughout this module you've seen six divergences, in four families, all with the same outcome: the unit test with the fake passes green and the real system is broken. You might be drawing the wrong conclusion —"I was missing unit tests", "my unit tests were weak", "with more cases I'd have caught it"—. This lesson exists to close that exit off, because it's false and dangerous. The unit test didn't fail to catch the divergence out of carelessness or lack of quantity. It failed because it's structurally unable to catch it. There's no number of unit tests, no ingenuity in the cases, no 100% coverage, that closes this gap, and understanding why is what finally forces you to look toward the contract and integration as the only real way out.

The reason fits in one sentence, and it's worth having clear before the demonstration: in a unit test, the double is at once the subject of the test and the oracle that judges it. The "subject" is the system whose behavior you want to verify; the "oracle" is the source of truth you compare against to decide whether it passed or failed. When you test cancel with the FakeBookingRepository, the fake participates in both things: it's part of the system that runs the scenario and it's the behavior reference the code was written against. You ask: "does my code work with the repository?", and the repository you use to answer is the same fake on whose assumption the code was built. The question and the answer come from the same source. It's circular reasoning, and circular reasoning always "confirms" —not because it's true, but because it can't do otherwise—.

Connection to the module: this lesson gives the theoretical finishing touch to everything before. Lessons 3, 4, and 5 showed what divergences exist; this one explains why none of them are visible from within the unit suite, and why that doesn't change no matter how many unit tests you add. It's the definitive bridge to the solution. If the unit test were able to catch the divergence with enough effort, the answer would be "try harder" and no guide would be needed. The reason the contract (module 3) and integration (module 5) are needed is precisely that the divergence lives in a blind spot the unit test can't reach from any angle. Lesson 7 will put a price on that blind spot; this one demonstrates that the blind spot is inevitable.

Analogy: the answer key the same student wrote

Imagine an exam where the student turns in two things: their answers and the correct-answer key they'll be graded against, both written by them, from the same understanding of the subject. If the student wrongly believes that Australia's capital is Sydney, they'll write "Sydney" in their answer and "Sydney" in their key. At grading time, their answer matches their key: one hundred percent. The exam comes out perfect. And it proves absolutely nothing about whether they know geography, because the error is in both sheets at once: the one that answers and the one that judges share the same false belief, so the grade only measures whether the student is consistent with themselves, not whether they're right. You can ask them a hundred more questions: if they keep grading themselves with their own key, they'll keep getting a hundred, even if half their answers are wrong.

That's exactly a unit test with a double. Your code is the student's answer; the fake is the answer key —and you wrote both, from the same assumption about how the repository behaves—. When the code assumes "get returns None" and the fake confirms "get returns None", the answer matches the key: green. But the green only measures that your code is consistent with your fake, not that your assumption is true. And that's why adding unit tests —asking more questions of the same self-graded exam— doesn't help: each new question is still graded with the key that shares the error. The only way to know whether "Sydney" is correct is to bring an external source of truth —an atlas, a teacher—: someone who doesn't share your assumption. In software, that external source is the real piece (integration) or a battery that compares the fake with the real one (the contract). Without an external source, the exam will grade itself forever, and always pass.

Worked example: a complete suite, blind and green

We're going to demonstrate it with the most honest example possible: a complete and well-made unit suite of the idempotent cancellation. Not a weak one-test suite, but one that covers the happy path in its three refund anchors (72 h, 36 h, 12 h) and the case that matters to us (canceling a nonexistent id). It's the suite a diligent developer would write, and it's all green with the careless fake. At the end, a single integration test with the real SqliteBookingRepository —the same missing-id scenario— to see who tells the truth.

# tests/test_unit_suite_is_blind.py
@pytest.mark.parametrize("hours_before, expected", [
    (72, 6000), (36, 3000), (12, 0)], ids=["72h", "36h", "12h"])
def test_cancel_refund_anchor_with_fake(hours_before, expected):
    repo = BuggyFakeBookingRepository()
    repo.save(a_booking())
    service = make_service(repo, START - timedelta(hours=hours_before))
    assert service.cancel("bk-1") == expected


def test_cancel_missing_returns_zero_with_fake():
    repo = BuggyFakeBookingRepository()
    service = make_service(repo, START)
    assert service.cancel("bk-999") == 0     # green with the fake


# ---- the ONLY integration test, with the real one ----
def test_cancel_missing_returns_zero_with_sqlite():
    repo = SqliteBookingRepository(sqlite3.connect(":memory:"))
    service = make_service(repo, START)
    assert service.cancel("bk-999") == 0     # the real one raises KeyError

What to expect. On my machine (Python 3.14.0, pytest 9.1.1): the four unit tests pass, and only the integration one falls.

python3 -m pytest tests/test_unit_suite_is_blind.py -v

tests/test_unit_suite_is_blind.py::test_cancel_refund_anchor_with_fake[72h] PASSED [ 20%]
tests/test_unit_suite_is_blind.py::test_cancel_refund_anchor_with_fake[36h] PASSED [ 40%]
tests/test_unit_suite_is_blind.py::test_cancel_refund_anchor_with_fake[12h] PASSED [ 60%]
tests/test_unit_suite_is_blind.py::test_cancel_missing_returns_zero_with_fake PASSED [ 80%]
tests/test_unit_suite_is_blind.py::test_cancel_missing_returns_zero_with_sqlite FAILED [100%]
FAILED tests/test_unit_suite_is_blind.py::test_cancel_missing_returns_zero_with_sqlite - KeyError: 'bk-999'
1 failed, 4 passed in 0.04s

Look at the split: 4 green, 1 red, and the only red is the only one that touches the real piece. The four unit tests are good tests —they cover the three refund anchors and the missing-id case, exactly what a reviewer would ask for—, and all four pass. If this suite were all you had, you'd see "all green" and deploy with confidence. The bug —that cancel of a nonexistent id blows up against the real one— is alive, and none of the four unit tests touches it, not even the one that specifically tests the missing id (test_cancel_missing_returns_zero_with_fake). That's the point to nail down: it's not that the missing-case test was missing. The missing-case test exists, is correct, and passes green —because it's graded with the fake, which shares the false assumption—. The only one that tells the truth is the one that brought an external source: the real repository.

Why adding unit tests changes nothing

Let's pause on the most counterintuitive consequence, because it's the one that disarms the "write more tests" reflex. Notice that you already have a unit test dedicated to the missing-id case, and yet the bug slips through. Why? Because that test asks the right question —"does canceling a nonexistent id return 0?"— but asks it of the wrong oracle: the fake, which answers yes because it's built on the same assumption as the code. Adding a second, a tenth, a hundredth unit test of the same case doesn't help, because they all share the oracle. It's the self-graded answer key: a hundred more questions, graded with the same wrong key, give a hundred more correct answers that prove nothing.

This has a precise form worth stating. A unit test verifies the property "the code behaves well given that the double behaves as the double behaves". That final clause is a tautology —the double behaves as it behaves— and that's why the unit test can always pass, no matter how many you write: it never compares the double's behavior with the real one's, because the real one isn't in the room. The gap you're looking for lives between the double and the real one, and a unit test, by definition, only has the double in front of it. It's like looking for the difference between two photos having only one: however much you look at it, the difference isn't in the photo you have, but between the two, and you need the second photo to see it. The second photo is the real piece. No meticulous examination of the first photo replaces it.

From here comes the only way out, and it's twofold. Either you bring the real piece to the test —that's integration, and it's the second photo: you run the scenario against the SqliteBookingRepository and see the difference—. Or you compare the two photos systematically —that's the contract, a battery that runs the same assertions against the fake and the real one and demands that both match, going red where they differ—. Both break the circularity by inserting a source of truth that doesn't share the code's assumption. There's no third way from within the world of unit tests, because from there the oracle is always the double. That's why this module inevitably flows into modules 3 and 5: not as a luxury, but as the only way to see what the unit test can't.

The coverage misunderstanding

It's worth disabling a defense people wield here: "but I have 100% coverage". Coverage measures which lines of your code ran during the tests. In our blind suite, cancel's coverage could be very high —the happy path traverses almost all its lines—. And it's useless against this divergence, for two reasons. First: the line that fails isn't in your code, it's in the SqliteBookingRepository's get, which your unit tests never run because they use the fake; your unit suite's coverage doesn't even look at that file. Second, and more fundamental: coverage measures execution, not oracle correctness. You can run the line if booking is None: return 0 at 100% —the unit test traverses it— and still prove nothing about the real system, because you ran it against an oracle that shares your error. A line covered by a self-graded test is falsely covered. Coverage tells you what you looked at; it doesn't tell you whether what you used to judge was the truth. Against divergence, coverage gives a false sense of completeness: test green, coverage green, and the bug intact.

Common mistakes

Responding to a divergence by writing more unit tests. What happens: once a divergence is discovered, the team resolves to "cover" that case better with more unit tests. Why it happens: "more tests" is the trained reflex to any bug. How to detect it: if the new tests use the same double, they're graded with the same oracle and all will pass, without touching the bug —you saw it: there was already a unit test for the missing case and it didn't help—. How to fix it: faced with a divergence, don't add unit tests; add a test with an external source of truth —an integration test against the real one, or a contract comparing fake and real—. The number of unit tests is orthogonal to the divergence.

Trusting coverage as proof that "everything is tested". What happens: someone sees 100% coverage and concludes there are no uncovered bugs left. Why it happens: a high, green number feels like completeness. How to detect it: coverage doesn't include the real piece's code your unit tests don't run (the SqliteBookingRepository), and it doesn't measure whether the oracle (the double) tells the truth. 100% coverage with doubles is 100% of "I ran my lines against my assumptions". How to fix it: use coverage for what it's good at (finding code no test touches) and not for what it can't (guaranteeing that the seams with the real thing are verified). That's what the contract and integration are for, which coverage doesn't measure.

Believing the problem is the quality of this fake, not the structure. What happens: someone concludes "the bug was because of a careless fake; with a well-written fake, the unit test would have caught it". Why it happens: in the get→None example, the careless fake was the visible culprit. How to detect it: remember the datetime divergence (module 1) and the types one (lesson 4): there the fake was impeccable and yet the unit test saw nothing, because the dict doesn't serialize. The circularity doesn't depend on whether the fake is well or poorly written; it depends on the unit test's oracle being the fake. How to fix it: don't chase the "perfect fake" as a cure —even the perfect fake is an oracle that shares the assumptions you wrote it with—. The cure is an external source of truth, which no fake, however good, can be from within the unit test.

Exercises

Exercise 1 — Identify the oracle. For each test, say what acts as the oracle (the source of truth it's judged against) and whether that oracle is external to the system under test or shares its assumptions: (a) assert price_cents(FOCUS, ANA, 3) == 6000, on the pure function; (b) cancel of a missing id with the careless fake, expecting 0; (c) cancel of a missing id with the real SqliteBookingRepository, expecting it to raise.

Exercise 2 — The missing-case test that wasn't enough. In the blind suite, test_cancel_missing_returns_zero_with_fake tests exactly the case that fails in production (canceling a nonexistent id) and still passes green. Explain why having the right test for the right case wasn't enough, and what that specific test was missing.

Exercise 3 — Design the test that would see it. Without implementing the full contract (module 3), describe the minimal test that would break the circularity for lesson 5's uniqueness divergence (the fake accepts a double booking, the real one rejects it). Say what external oracle it uses and why that makes it able to catch the bug.

def test_double_booking_is_rejected_by_real_repo():
    repo = SqliteBookingRepositoryUnique(sqlite3.connect(":memory:"))
    repo.save(a_booking("bk-1"))                 # focus, same time
    with pytest.raises(sqlite3.IntegrityError):  # the external oracle: the real engine
        repo.save(a_booking("bk-2"))             # focus, same time -> must fail

Summary and next step

In this lesson you understood why the unit test's blindness to divergence isn't an accident cured with effort, but a structural property. In a unit test, the double is at once the subject of the test and the oracle that judges it —the student's answer and their own answer key—, so the green only measures that the code is consistent with the fake, not that the fake's assumption is true. You saw it with a complete and well-made suite: four green unit tests, including the missing-case one, and a single red integration test, the only one with an external source of truth. And you disabled two reflexes: adding unit tests doesn't help (they share the oracle) and coverage doesn't measure oracle correctness (lines green, bug intact). The only way out is to bring an external source: the real piece (integration) or a fake-vs-real comparison (contract).

Before moving on you should be able to: state why the unit test is unable to see the divergence, in terms of subject and oracle; explain why having the test for the right case wasn't enough; and identify, in a given test, whether its oracle is external or complicit.

You now know the bug is invisible from the unit suite and will stay invisible however much you grow it. What's missing is putting a price on it. Lesson 7 does the math the business pays when one of these lies crosses the blind spot and reaches production: the raw traceback of the incident —a member who gets a 500 error where they expected a "refund 0"—, and the three dimensions of the cost: late detection, blast radius, and confusing debugging. The green that gave the go-ahead to the deploy has a bill, and we're going to read it.

Resources

• pytest documentation — parametrizing tests — the technique with which the blind suite covers the three refund anchors in a single test; useful for building complete unit suites like the one that, even complete, doesn't see the divergence.
• coverage.py documentation — the tool that measures which lines your tests run; reading what it measures (execution) and what it doesn't (oracle correctness, unexecuted real-piece code) is understanding why 100% doesn't protect against the divergence.
• Martin Fowler — Self Testing Code and the role of the oracle — the backdrop on what makes a test informative; the conceptual frame behind "the oracle must be independent of the subject".
• testing-fundamentals-and-tdd-guide — the sister guide on what makes a test good and on coverage as a tool and not a goal; the natural review if "oracle" or "coverage" felt shaky.