Module 2: The Double That Lied

7. The cost in production

Next →

Description

Up to now the module has treated the divergence as a technical phenomenon: why it happens, in how many forms, why the unit test doesn't see it. What's missing is the part the business cares about and that justifies all the effort of the disciplines to come: how much it costs when one of these lies crosses the blind spot and reaches production. Because if the cost were trivial —an ugly log, a rare case with no consequences— we could shrug and carry on. It isn't. The divergence that passes the unit test green and blows up with the real piece has a concrete bill, and in this lesson we're going to read it line by line, starting with the incident's raw traceback and continuing through the three dimensions that make this kind of bug among the most expensive there are.

The three dimensions are these, and we'll go through them with the get→None case from lesson 3. First: late detection. The unit test's green wasn't neutral; it was an active signal of "go ahead, deploy". The bug didn't slip in despite the tests: it slipped in with the tests' blessing, which is worse, because nobody suspects what came out green. Second: blast radius. A bug that reaches production doesn't affect a test; it affects every user who goes through that route, in real time, until someone notices and reverts it. Third: confusing debugging. These bugs tend to blow up far from their cause —a KeyError in the repository because of an assumption made in the service—, and the team loses hours looking for the problem where it isn't, because the symptom and the cause live in different pieces. Late detection, wide radius, confusing debugging: all three multiply.

Connection to the module: this lesson closes the module's arc by putting a price on the problem the previous six described. Lessons 2 to 5 showed the divergences; lesson 6 explained why they're invisible to the unit test; this one explains why that invisibility is expensive and not just inconvenient. It's the economic justification of modules 3 to 7: the contract and integration have a cost —they have to be written and maintained—, and this lesson demonstrates that that cost is a tiny fraction of what the bug they prevent costs. When in module 3 I ask you to write a contract battery, I want you to remember this bill and understand that it isn't bureaucracy: it's cheap insurance against an expensive claim.

Analogy: the printing error

Imagine a typo —a single wrong letter— on a poster. If you catch it in the draft, on your screen, you fix it in two seconds and nobody finds out: cost, zero. If it slips past and you catch it in the printing proof, before the print run, you reprint one sheet: cost, a few cents and a while. If it slips all the way past printing a hundred thousand posters and pasting them all over the city, the typo now lives on every corner: you have to reprint everything, send a team to peel and repaste, endure the memes from people who photographed the error, and explain to the client why their brand came out misspelled on the main avenue. The same wrong letter costs zero, cents, or a fortune, and the only thing that changed is at what stage you caught it. The later, the more expensive.

A double's divergence is that typo, and the stages are the same. Caught in a contract, on your machine, it's the draft: a local red test, you fix it in minutes, nobody finds out —cost, almost zero—. Caught in an integration test in CI, before deploying, it's the printing proof: the pipeline goes red, you don't deploy, you fix and retry —cost, a while—. Caught in production, with the real piece and real users, it's the papered-over city: every member who goes through the broken route hits the error, someone has to notice it, diagnose it, revert the deploy, and explain the incident —cost, a fortune in time, trust, and money—. This lesson is about the third stage, the expensive one, so you understand why it's so worth catching the typo in the first. The contract and integration are nothing more than "reviewing the poster before printing a hundred thousand".

The incident: the raw traceback

Let's reconstruct the moment. The idempotent cancellation feature —the one from lesson 3, with the guard if booking is None: return 0— passed its unit tests green and was deployed. In production, the repository isn't the fake: it's the real SqliteBookingRepository. A member opens an old link, to a booking that was already deleted, and clicks "Cancel". This is what the system does, captured as it would appear in the server log:

# prod_incident.py — the production handler, with the real repository
repo = SqliteBookingRepository(sqlite3.connect(":memory:"))   # production: real
service = IdempotentBookingService(
    FixedClock(datetime(2026, 3, 1, 9)), StubPaymentGateway(ok=True),
    SpyEmailSender(), repo)

# The member cancels a booking that no longer exists (double-click / stale link).
refund = service.cancel("bk-deleted-yesterday")
print(f"Refund: {refund}")   # never gets here

What to expect. On my machine (Python 3.14.0), running the script directly, the raw traceback —the one that would end up in your production error log—:

Traceback (most recent call last):
  File "/tmp/prod_incident.py", line 14, in <module>
    refund = service.cancel("bk-deleted-yesterday")
  File "/tmp/reservo/idempotent.py", line 19, in cancel
    booking = self._repo.get(booking_id)
  File "/tmp/reservo/sqlite_repo.py", line 50, in get
    raise KeyError(booking_id)       # the real one RAISES if it doesn't exist
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'bk-deleted-yesterday'

That uncaught KeyError is, in a real web application, a 500 error: the server responds "internal error", the member sees a broken screen instead of a friendly "there was nothing to cancel, refund 0", and the event is written to the error log. What the developer designed as the gentlest case of the system —canceling something nonexistent, which should have been a calm no-op— became the case that takes down the server. The idempotent feature, whose reason for being was to calmly handle the double-click and the old link, does exactly the opposite in production: it turns those benign cases into failures. With this concrete image —the traceback in the log, the 500 on the member's screen— let's break down the three dimensions of the cost.

Dimension one: late detection

The first thing to understand is that the unit test's green wasn't a detection that failed; it was a green light that turned on. When the suite passed, it didn't say "I found no problems"; it said "go ahead, this is fine, deploy with confidence". The deploy didn't happen despite the tests: it happened because the tests authorized it. And there's what's insidious about late detection: nobody audits what came out green. Code with a red unit test is stopped and reviewed; code with a green unit test passes without anyone looking twice. The divergence takes advantage of exactly that trust: it disguises itself as green, gets the stamp of approval, and crosses into production through the front door, with permission.

The cost of detecting late isn't linear, it's exponential, and it's the printing-error lesson. In the draft (local contract), the divergence is a red test you fix before committing: minutes, zero users affected, zero people involved other than you. In the printing proof (integration in CI), it's a red pipeline that stops the deploy: a while, zero users, maybe a conversation with the team. In production, it's an incident: users affected in real time, someone on call woken up, a diagnosis under pressure, a rollback, a post-mortem, and the team's confidence in its suite eroded. The same divergence —a character, .get() instead of [...]— costs minutes or costs a whole day of several people, and the only thing that changes is at what stage it was caught. Detecting late is multiplying the cost by the number of stages the typo managed to cross without anyone seeing it.

Dimension two: blast radius

A unit test that fails affects one thing: the test. A bug in production affects everyone who goes through the broken route, in real time, until someone stops it. That's the difference in radius, and it's enormous. In our case, each member who opens an old link or double-clicks "Cancel" —which isn't a rare case; it's among the most common in any application with shared links and buttons pressed twice— gets a 500 error. Not one; all of them, while the broken deploy is alive. If the bug is in production for an hour before someone notices and reverts, the radius is "everyone who tried to cancel something nonexistent in that hour". It could be dozens, hundreds, thousands, depending on the traffic.

And the radius isn't measured only in users; it's measured in consequences per user. Here the damage is a visible error and a frustrated action —bad, but recoverable: the member retries or writes to support—. In other divergences the radius is worse and quieter. Think of lesson 5's uniqueness one: if the fake let through the belief that "there can be no double bookings" and the real one didn't have the constraint, the radius isn't a visible error but corrupted data —two members with the same room at the same time, discovered only when both show up for the meeting—. That radius is more expensive because the damage is already done in the data when you detect it, and cleaning it up (deciding who keeps the room, compensating the other) costs much more than reverting a deploy. The radius rule: a production bug damages in proportion to the traffic that touches it and to how irreversible its effect is, and both scale with the time the bug spends alive.

Dimension three: confusing debugging

The third dimension is the one that eats up people's hours, and it's born from a property of these bugs: the symptom and the cause live in different pieces. Look at the traceback again. The error —the KeyError— blows up in sqlite_repo.py, in the repository's get. But the repository is doing exactly the right thing: raising when it doesn't find, just as its contract demands. The real cause of the bug isn't there; it's in idempotent.py, in the guard if booking is None that assumed a wrong contract, written by another person, in another file, maybe weeks earlier. The traceback takes you to the scene of the symptom, not to the scene of the crime.

Put yourself in the shoes of whoever gets the alert at midnight. They see a KeyError in the repository. Their first instinct, reasonable, is to suspect the repository: "is the get wrong? why does it raise?". But the get is impeccable —however you look at it, it does the right thing—. They may spend a good while defending the repository's innocence before looking up and asking who calls that get and with what expectation. And even when they get to idempotent.py, the guard if booking is None: return 0 looks fine: it's defensive, it's clear. It takes the conceptual leap —"this guard assumes get returns None, but the real repo raises"— to understand that the bug is a disagreement between two correct pieces, not a defect in one. That leap is hard precisely because no piece is wrong on its own; the error is the crack between them, and cracks don't show up in tracebacks. This kind of debugging —where each piece you examine turns out innocent— is the one that burns time and patience, and it's typical of divergence bugs precisely because, as we saw in lesson 3, they're nobody's fault: they're a disagreement nobody had the responsibility to detect.

The final tally, and why the contract is cheap

Let's add it up. A wrong character in a fake produced: a production incident (late detection), 500 errors for every member who canceled something nonexistent while it lasted (blast radius), and a debugging session that starts by blaming the innocent piece (confusing debugging). Translate that into what it really costs: on-call engineering time, diagnosis time for several people, a rollback, a post-mortem, frustrated members, and —the hardest cost to recover— a bit of the team's confidence in its own test suite, because "it was all green and it still broke" is a phrase that corrodes.

Now compare with what it would have cost to catch the same typo in the draft. A contract battery that asserts "get of a missing id raises", run against the fake and the real one: fifteen minutes to write, it runs in hundredths of a second, and it goes red on the developer's machine —against the careless fake— before any commit. The bug never reaches CI, much less production. The arithmetic is overwhelming: minutes of prevention against a person-day of claim, and that's without counting the trust. That's why the contract and integration aren't a luxury of meticulous teams or process bureaucracy: they're the most favorable economic calculation there is in testing. You pay pennies of prevention so you don't pay a fortune of incident. With this bill read, module 3 stops being "another technique to learn" and becomes the obvious thing: the cheap way to never read this traceback in your log again.

Common mistakes

Dismissing the case as "rare" and deprioritizing it. What happens: someone sees "canceling a nonexistent id" and thinks "that hardly happens, I'll fix it when I have time". Why it happens: the case sounds like an improbable edge. How to detect it: old links and double-clicks are among the most common in real applications; "canceling something that's no longer there" happens all the time. And even if it were rare, a rare case that takes down a 500 is still a 500. How to fix it: measure the real frequency before calling a case rare, and remember that a bug's severity is frequency × damage; a high damage (500 error, corrupted data) deserves attention even if the frequency seems low.

Treating the incident as a person's failure, not the process's. What happens: after the post-mortem, someone concludes "whoever wrote the fake was careless, let them be more careful". Why it happens: looking for an individual culprit is more comfortable than changing the process. How to detect it: lesson 6 already demonstrated that no individual care closes this gap —the datetime fake was impeccable and still diverged—. If your only defense against the next divergence is "let people be careful", you have no defense. How to fix it: divergence incidents are prevented with mechanisms (contract, integration in CI), not with reprimands. The correct post-mortem doesn't end in "be more careful" but in "we added a contract for BookingRepository that runs in CI".

Believing the rollback closes the incident. What happens: the deploy is reverted, the 500 disappears, and the case is considered closed. Why it happens: the symptom is gone, so it seems resolved. How to detect it: the rollback removes the bug from production, but the divergence is still there —the fake still returns None, the guard still assumes wrong—, ready to come back on the next attempt to deploy the feature. And if there was corrupted data (like in the uniqueness divergence), the rollback doesn't clean it. How to fix it: the rollback is first aid, not a cure. Truly closing the incident is repairing the divergence (fixing the guard or the repo's contract), adding the mechanism that would have caught it (contract/integration), and cleaning any data the bug damaged while it was alive.

Exercises

Exercise 1 — Place the stage and the cost. For each moment when the get→None divergence is caught, say which "printing stage" you're at and estimate the relative cost: (a) a local contract goes red before the commit; (b) an integration test in CI stops the deploy; (c) the 500 error appears in the production log an hour after the deploy.

See solution
  • (a) The draft. Minimal cost. The red contract appears on your machine before sharing anything; you fix it in minutes, zero users affected, zero people involved other than you. It's catching the typo on the screen: nobody finds out it existed.
  • (b) The printing proof. Low cost. The red pipeline stops the deploy before it reaches users; it costs a while and maybe a team conversation, but no member sees the bug. It's catching the typo on the proof sheet: you reprint one page, not a hundred thousand.
  • (c) The papered-over city. High cost. The bug is already in production: users affected in real time during that hour, someone on call, diagnosis under pressure, rollback, post-mortem. It's the typo on a hundred thousand pasted posters. The same divergence, but caught at the most expensive stage.

The quantitative moral: the cost grows with the stage, not with the bug's complexity. A trivial bug caught late costs more than a complex bug caught early. That's why moving detection toward the draft (contract) is the highest-return investment in testing.

Exercise 2 — Compare the radii. Two divergences reach production: the get→None one (500 error when canceling a nonexistent id) and the uniqueness one (the system accepted double bookings because the constraint was missing). Compare their blast radii on two axes: how many users it affects and how reversible the damage is. Which is worse and why?

See solution

The get→None one: affects every member who cancels something nonexistent while the deploy is alive. The damage per user is a visible 500 error and a frustrated action: annoying, but reversible —the member retries after the rollback, no data was damaged—. The radius is "users in the broken window", and once the deploy is reverted, the problem disappears without residue.

The uniqueness one: affects every pair of members who booked the same room at the same time while the constraint was missing. The damage per user is corrupted data: two bookings that shouldn't coexist, and that the system accepted as valid. It's irreversible on its own —the code rollback doesn't delete the double bookings already written; someone has to find them, decide who keeps the room, compensate the other, and fix the data by hand—. And it's silent: nobody sees an error; the bug is discovered when two members show up for the same meeting, maybe days later.

Which is worse: the uniqueness one, for two reasons. Its damage is irreversible (corrupted data that survives the rollback) and silent (there's no error to trigger an alert, so it can live much longer before being detected, widening the radius). A 500 error at least screams; a double booking stays quiet corrupting the state. The lesson: divergence bugs that corrupt data silently are more expensive than those that fail loudly, even though the latter look more dramatic.

Exercise 3 — Write the correct post-mortem. The get→None incident was closed with "the developer who wrote the fake should be more careful with .get() vs [...]". Explain why that closure is insufficient and draft the actions a correct post-mortem would list.

See solution

Why "be more careful" is insufficient: it's an action on a person, not on the system, and lesson 6 already demonstrated that no individual care closes the divergence gap —the datetime one came from an impeccable fake—. "Be more careful" doesn't prevent the next divergence from another cause (the real one gains a constraint, a type changes); it only postpones the next incident until someone, inevitably, makes a mistake again or the system changes. A post-mortem that ends in a reprimand leaves the team no more protected than before.

The actions of a correct post-mortem, all on the process:

  1. Repair the concrete divergence: decide get's real contract (raises for a missing id) and fix the idempotent feature to respect it (try/except KeyError, not if booking is None), and fix the careless fake to honor the contract ([...] instead of .get()).
  2. Add the mechanism that would have caught it: a contract battery for BookingRepository that asserts its behavior agreements (including "get of a missing id raises") and runs against the fake and the real one in CI, so that any future divergence goes red before the deploy.
  3. Add integration coverage at the highest-risk seams (the repository), to test the edges against the real piece, not just against the fake.
  4. Check whether there was damaged data during the incident window and clean it (here there wasn't, but it's part of the checklist).
  5. Share the learning: document that "all green" with doubles doesn't guarantee healthy integration, so the team doesn't read the green as a guarantee it isn't again.

The common thread: each action installs a mechanism or repairs a disagreement, none asks for "more care". That way the next developer —or the next schema change— hits a local red test, not a production incident.

Summary and next step

In this lesson you put a price on the module's problem. You read the incident's raw traceback —an uncaught KeyError that in a web app is a 500 error, in the case the idempotent feature was supposed to handle most calmly— and broke down the three dimensions that make this kind of bug expensive: late detection (the green wasn't neutral, it was a green light to the deploy, and detecting late multiplies the cost like the typo on a hundred thousand posters), blast radius (every user of the broken route, with damage proportional to the traffic and to how irreversible the effect is), and confusing debugging (the symptom blows up in the innocent piece, far from the cause, because the bug is a disagreement between two correct pieces). And you did the math that justifies everything to come: minutes of contract against a person-day of incident.

Before moving on you should be able to: explain why the unit test's green was an active authorization and not a failed detection; compare the radius of two divergences by users affected and reversibility; and draft a post-mortem that installs mechanisms instead of asking for care.

With this, the module has given you the complete problem: what the divergence is, why it's inevitable, in how many forms it appears, why the unit test doesn't see it, and how much it costs. What's left is for you to put it into practice yourself. Lesson 8 is the mini-project: we give you a Reservo feature with a planted divergence, and your job is to write the unit test that stays green, the integration one that goes red, and the diagnosis of why the green was lying —plus propose, without implementing it yet, the shape of the contract that would have caught it—. It's the final rehearsal before module 3 gives you, at last, the solution.

Resources

← Previous6. Why the unit test can't see itNext →8. Mini-project: find the divergence the unit test doesn't reveal