Module 4: Verifying The Contract From Both Sides

5. Catching a provider breaking change

Next →

Description

We arrive at the golden payoff. Everything before —the two sides, the battery against both, the transitivity guarantee— pointed to this moment: using the contract to catch an incompatible change before deploying it. A breaking change is a modification in a component that breaks a promise another component depends on. Here the component that changes is the provider —the SqliteBookingRepository— and the promise it breaks is a contract clause. The scenario is the most common and the most dangerous: the change seems harmless. It's not an obvious error; it's a reasonable "simplification" a colleague would make in good faith, that compiles, and that passes all the tests that don't exercise exactly that case.

The concrete change is the one we've anticipated since module 2: get, instead of raising when the id doesn't exist, now returns None. One line. raise KeyError(booking_id) becomes return None. To whoever writes it, it looks cleaner —"I return None if I don't find it, like many APIs"—. But the contract promised that get of a missing id raises, and there's a real consumer —BookingService.cancel— relying on that promise. Without a contract, the change is deployed and blows up in production, far and late. With a contract, you run the battery and the [sqlite] side goes red on the exact clause, on your machine, in two hundredths of a second. This lesson is that hunt, with real output of both moments: the crime (what happens in production without a contract) and the arrest (the red that catches it before).

Connection to the module: lessons 2, 3, and 4 built the machinery; this one fires it. It's the demonstration that the contract isn't pretty documentation, but an active deploy safety net. Its mirror lesson is 6: there the one who breaks the agreement isn't the provider (which fails a promise) but the consumer (which relies on something unpromised). Together, the two cover the two ways to break a contract. Lesson 7 will close by explaining who owns the agreement and why.

Analogy: the screw supplier who changes the thread

Imagine a factory that assembles bicycles. An external supplier sends it screws of a standard thread —say, metric M5—, and the entire assembly line is designed around that thread: the nuts, the tools, the holes. The agreement with the supplier says "M5 thread". One day, the supplier decides to "improve" its screws to an M6 thread, more resistant. For them it's a step forward; they don't warn anyone, because "a screw is still a screw". The change is, for their catalog, harmless.

Without quality control at receiving, those M6 screws enter the line, and the disaster appears late and far: not at receiving, but mid-assembly, when an M5 nut doesn't thread, or —worse— when it threads by force, the bicycle ships, and a wheel comes loose on the street with a customer on it. The symptom is miles from the cause: no one on the street suspects the screw supplier. Now put an inspector at receiving with a caliper and the agreement: each batch of screws is measured against "M5 thread" before entering the line. The M6 batch is rejected at the door, with an exact diagnosis —"incorrect thread: M5 expected"—, and never reaches a bicycle.

The screw supplier is the provider's team; the M5 thread is the contract clause; changing to M6 is the get that returns None; the bicycle that comes loose on the street is BookingService.cancel blowing up in production; and the inspector with the caliper at receiving is the contract battery you run before the deploy. The contract doesn't prevent the supplier from changing the thread; it prevents the changed thread from entering the line unseen.

Worked example, part 1: the crime (production without a contract)

Let's first see what happens if the change is deployed without the contract catching it. The provider's team edits SqliteBookingRepository.get. Before:

    def get(self, booking_id):
        row = self._conn.execute(
            f"SELECT {SELECT_COLUMNS} FROM bookings WHERE id = ?",
            (booking_id,),
        ).fetchone()
        if row is None:
            raise KeyError(booking_id)   # honors the contract: missing id -> raises
        return _row_to_booking(row)

After the "harmless" change:

    def get(self, booking_id):
        row = self._conn.execute(
            f"SELECT {SELECT_COLUMNS} FROM bookings WHERE id = ?",
            (booking_id,),
        ).fetchone()
        if row is None:
            return None          # BREAKING CHANGE: it used to raise KeyError
        return _row_to_booking(row)

A single different line. Now, in production, someone asks to cancel a booking that no longer exists (a double-click, an old id, whatever). BookingService.cancel starts like this:

    def cancel(self, booking_id) -> int:
        booking = self._repo.get(booking_id)         # it used to raise; now it returns None
        now = self._clock.now()
        refund = refund_cents(booking, booking.price_cents, now)   # <-- booking is None
        ...

cancel trusts that get raises for a missing id —we documented it in lesson 2—. With the provider changed, get doesn't raise: it returns None, cancel carries on with booking = None, and blows up further down. Let's run that scenario exactly as it would occur:

What to expect. On my machine (Python 3.14.0):

python3 demo_prod.py     # cancel of a missing id, with the provider already changed
Traceback (most recent call last):
  File "/tmp/demo_prod.py", line 13, in <module>
    service.cancel("does-not-exist")
  File "/tmp/m4work/reservo/services.py", line 37, in cancel
    refund = refund_cents(booking, booking.price_cents, now)
AttributeError: 'NoneType' object has no attribute 'price_cents'

Look at the error carefully, because its shape is the moral. It doesn't say "the repository broke its contract". It doesn't mention get, or the change, or the missing id. It says AttributeError: 'NoneType' object has no attribute 'price_cents', on the refund_cents line, inside cancel —three steps after the real problem—. The symptom is far from the cause: whoever debugs this in production will see cancel failing on price_cents and will lose time suspecting the refund logic, the computation, everything but the repository's get that, three lines earlier and silently, returned None instead of raising. That's the cost of a breaking change reaching production: a confusing, distant, and late error, with an uphill trace to a cause the message doesn't name.

Worked example, part 2: the arrest (the contract catches it before the deploy)

Now let's rewind to the correct moment: the change is made in the provider, but before merging and deploying, we run the contract battery. It's the same battery from lesson 4 —without touching a line of the tests—; the only thing that changed is the provider's implementation.

python3 -m pytest tests/test_repository_contract.py -v
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 8 items

tests/test_repository_contract.py::test_save_then_get_returns_the_same_booking[fake] PASSED [ 12%]
tests/test_repository_contract.py::test_save_then_get_returns_the_same_booking[sqlite] PASSED [ 25%]
tests/test_repository_contract.py::test_get_of_a_missing_id_raises[fake] PASSED [ 37%]
tests/test_repository_contract.py::test_get_of_a_missing_id_raises[sqlite] FAILED [ 50%]
tests/test_repository_contract.py::test_saving_the_same_id_twice_updates_not_duplicates[fake] PASSED [ 62%]
tests/test_repository_contract.py::test_saving_the_same_id_twice_updates_not_duplicates[sqlite] PASSED [ 75%]
tests/test_repository_contract.py::test_find_by_room_returns_only_that_rooms_bookings[fake] PASSED [ 87%]
tests/test_repository_contract.py::test_find_by_room_returns_only_that_rooms_bookings[sqlite] PASSED [100%]

=================================== FAILURES ===================================
___________________ test_get_of_a_missing_id_raises[sqlite] ____________________

repo = <reservo.sqlite_repo.SqliteBookingRepository object at 0x105ebb750>

    def test_get_of_a_missing_id_raises(repo):
>       with pytest.raises(KeyError):
             ^^^^^^^^^^^^^^^^^^^^^^^
E       Failed: DID NOT RAISE KeyError

tests/test_repository_contract.py:41: Failed
=========================== short test summary info ============================
FAILED tests/test_repository_contract.py::test_get_of_a_missing_id_raises[sqlite]
========================= 1 failed, 7 passed in 0.02s ==========================

1 failed, 7 passed. Compare this red with the production AttributeError: they're from another planet. Here the message is surgical. First, which clause broke: test_get_of_a_missing_id_raises —"get of a missing id raises"—. Second, who broke it: the [sqlite] bracket, the real provider; the [fake] of the same clause stays green, because the fake didn't change. Third, how: Failed: DID NOT RAISE KeyError —the provider should have raised and didn't—. With that you know exactly what to review (SQLite's get), what the contract expected (a KeyError), and that neither the consumer nor the fake is to blame. And all this happened before merging: the breaking change never reached production, never blew up a cancel, never confused anyone with a distant AttributeError.

Notice the asymmetry of the red: only the [sqlite] failed. That detail is pure diagnosis. That [fake] is green and [sqlite] is red on the same clause tells you, unambiguously, "the one that deviated from the contract is the real provider, not the contract itself or the fake". If both sides had failed, you'd suspect the test or the contract; that only one fails points a finger at the implementation that changed.

Why the contract sees what a unit test doesn't

You might ask: why didn't the team's unit test suite catch this? The answer is module 1's gap, and seeing it here closes the circle. BookingService's unit tests use the FakeBookingRepository, and the fake didn't change: its get still raises KeyError for a missing id. So cancel's unit tests still see a get that raises, still pass, and have no way of knowing that the other provider —the real one, production's— stopped raising. The breaking change lives exactly in the divergence between the fake and the real one, and no test that uses only the fake can see it.

The contract sees what the unit test doesn't because the contract runs against the provider that changed. The clause test_get_of_a_missing_id_raises[sqlite] exercises the real SqliteBookingRepository, not the fake, so when the real one stops raising, that line notices. It's the same lesson from module 1 —"cross the seam with the real piece"—, now systematized: you don't cross the seam once by hand, but on every run of the contract, for each clause, automatically. The contract is the net that makes "testing against the real thing" not depend on someone remembering to do it.

Shifting the risk to the left

There's an underlying idea worth naming, because it's the why of the whole module: moving the discovery of the error to the left in time. Draw a change's life from left to right: you write it, you review it, the tests run, it's merged, it's deployed, it runs in production. The further to the right you discover a bug, the more expensive it is: in production it costs an incident, affected users, an urgent trace; in review it costs a comment; in your local suite it costs two hundredths of a second and a clear red.

The get breaking change is the same bug in both of this lesson's examples; the only thing that changes is where it's discovered. In part 1 it's discovered in production (far right): expensive, confusing, late. In part 2 it's discovered when running the local battery (far left): cheap, clear, immediate. The contract is the tool that pushes the discovery to the left —from the incident to the red test—. It doesn't stop people from making breaking changes (that's human and inevitable); it makes breaking changes get found early, when fixing them is trivial. That is, in one sentence, contract testing's reason for being: shifting the risk to the left.

Common mistakes

Seeing the red and "fixing the test" instead of the provider. What happens: someone sees test_get_of_a_missing_id_raises[sqlite] red and changes the test to accept None, making it green. Why it happens: a red feels like an annoying test, and "making it pass" seems like progress. How to detect it: if your fix for the red consisted of weakening the contract's assertion instead of touching the provider, you inverted the relationship —you let the implementation rule over the contract—. How to fix it: the contract is the agreement; the red says the provider violated it. The correct decision is either to revert the provider's change (have it raise again) or, if the team deliberately decides that get should now return None, renegotiate the contract with the consumer —change the clause and adapt BookingService.cancel to the new behavior, at the same time—. Never quietly loosen the test to cover the red: that reintroduces the bug and turns off the alarm.

Believing a change "that compiles and passes the unit tests" is safe. What happens: the get change compiles, BookingService's unit tests (with the fake) stay green, and it's deployed with confidence. Why it happens: "compiles and the tests pass" is the usual safety criterion. How to detect it: if the tests that passed use only the fake at the seam that changed, they didn't test the change —they tested the fake, which didn't change—. How to fix it: for a change in a provider, the safety criterion isn't "the unit tests pass", but "the provider's contract passes", because the contract is the only thing that runs against the implementation you touched. Add the contract battery to what you run before merging repository changes.

Not running the contract after a provider change. What happens: someone edits SqliteBookingRepository and doesn't run the contract battery, trusting the rest of the suite. Why it happens: the contract "was already green", so re-running it seems unnecessary. How to detect it: if your flow doesn't run the contract battery on every provider change, a breaking change can slip in just as the M6 screw batch would slip in without an inspector at receiving. How to fix it: the battery only protects if it runs again. Part 2's arrest happened because someone ran the battery after the change; if they hadn't, part 1's crime would have run its course. The contract is an inspector, and an inspector who doesn't check the new batch is useless.

Exercises

Exercise 1 — Read the red. The failure says test_get_of_a_missing_id_raises[sqlite] ... Failed: DID NOT RAISE KeyError. Break down that message into the three things it tells you, and explain what information it would give you —or take away— if the bracket said [fake] instead of [sqlite], or if both sides failed.

See solution

The three things the message says:

  1. Which clause broke: test_get_of_a_missing_id_raises → "get of a missing id must raise". You know exactly which contract promise was violated.
  2. Who broke it: the [sqlite] bracket → the real provider. The culprit is the SQLite implementation, not the fake or the consumer.
  3. How: DID NOT RAISE KeyError → the provider should have raised KeyError and raised nothing (returned None). The exact technical symptom.

If the bracket said [fake]: it would tell you it was the fake that stopped raising —maybe someone "synced" the double wrong—. It would be module 2's bug (the fake that lies), not a breaking change of the real provider. Different cause, different place to look.

If both sides failed ([fake] and [sqlite]): you'd suspect the test or the contract, not an implementation —because it's rare for the two implementations to break the same way at once independently—. That only one side fails is what points a finger at the implementation that changed; that both fail suggests the assertion itself is written wrong or that you renegotiated the contract halfway. The bracket isn't decoration: it's half the diagnosis.

Exercise 2 — Another breaking change, another clause. The provider's team now "optimizes" find_by_room: by carelessness, the query loses its WHERE room_id = ? and ends up as a bare SELECT ... FROM bookings. Without running anything, predict: which contract clause would go red, on which side, with what difference of values, and why the [fake] one would stay green?

See solution

test_find_by_room_returns_only_that_rooms_bookings[sqlite] would go red. That clause saves two bookings —bk-1 in focus and bk-2 in studio— and expects find_by_room("focus") to return only {"bk-1"}. Without the WHERE room_id = ?, the query returns all the table's rows, so find_by_room("focus") returns both bookings and the set of ids is {"bk-1", "bk-2"}. The assertion assert ids == {"bk-1"} fails with a message showing the slipped-in element: Extra items in the left set: 'bk-2'. The promise "returns only that room's bookings" broke because the filter disappeared.

The [fake] side would stay green because the FakeBookingRepository.find_by_room filters in Python with its list comprehension ([b for b in self._store.values() if b.room_id == room_id]), and that filter didn't change. The fake still returns only focus's. Again the asymmetry is the diagnosis: [fake] green + [sqlite] red = "the real provider broke a promise the fake still fulfills". And again, a one-line breaking change (losing the WHERE) that "compiles and passes the fake's unit tests" but that the contract catches before the deploy, pointing to the exact clause and the extra datum that slipped in.

Exercise 3 — When the change is intentional. Suppose the team decides, deliberately and for good reasons, that get should return None for a missing id (to align with another API). The contract goes red. Explain what the correct sequence of steps is to make that change without leaving a bug, and why "just changing the test to accept None" isn't that sequence.

See solution

The correct sequence treats the change for what it is: a renegotiation of the contract between the consumer and the provider, not a unilateral adjustment of the provider. The steps:

  1. Agree on the new contract with the consumer side. The contract's owner is the consumer (lesson 7); if get is going to return None, you have to look at all the consumers that depended on it raising —starting with BookingService.cancel— and decide how they adapt.
  2. Change the contract clause to reflect the new agreement: test_get_of_a_missing_id_returns_none instead of ..._raises, asserting assert repo.get("does-not-exist") is None.
  3. Adapt the consumers at the same time. cancel can no longer trust a KeyError; now it must check if booking is None: raise SomeError(...) explicitly before using booking.price_cents. Without this step, part 1's AttributeError comes back.
  4. Make the provider fulfill the new clause (have it return None) and run the battery: green on both sides, with the new promise.

Why "just changing the test to accept None" is not that sequence: that shortcut does step 2 (loosens the clause) but skips 1 and 3. The result is a green contract and a broken consumer: cancel still expects a KeyError that no longer arrives, and the production AttributeError reappears —only now without any alarm, because you turned off the only one that detected it—. Changing the contract is legitimate; changing it without adapting whoever depended on the old promise is how you introduce a bug with the blessing of a green suite. The red isn't the enemy: it's the list of who needs to be warned.

Summary and next step

In this lesson you fired the machinery and collected the golden payoff: catching a provider breaking change before deploying it. You saw the same bug —the get that returns None instead of raising— in its two possible destinations. In production, without a contract, it's an AttributeError: 'NoneType' object has no attribute 'price_cents' inside cancel, far from the cause, confusing and late: the crime. Before the deploy, with a contract, it's a surgical red —test_get_of_a_missing_id_raises[sqlite], DID NOT RAISE KeyError, with [fake] intact— that names the clause, the guilty provider, and the exact symptom in two hundredths of a second: the arrest. With the screw supplier and their changed thread you understood that the contract doesn't prevent the provider from changing, but prevents the change from entering the line unseen. And you named the underlying idea: shifting the risk to the left, from the expensive incident to the cheap red test.

Before moving on you should be able to: distinguish the production error (distant, confusing) from the contract red (surgical, immediate); explain why the unit test with the fake didn't catch the change and the contract did; read the [sqlite]/[fake] bracket as a diagnosis; and describe the correct sequence for an intentional provider change without leaving a bug.

This lesson caught the provider that breaks a promise. Its mirror is the consumer that relies on a promise that was never made. Lesson 6 takes that case: BookingService assumes that find_by_room comes ordered, something the contract doesn't promise, and you'll see the technique for catching that over-assumption —running the consumer against a provider that returns a different legal order— with its own real output, red and green.

Resources

← Previous4. The same battery against the fake and the real oneNext →6. Catching a consumer over-assumption