7. Fix determinism, do not retry

Description

For six lessons you have been accumulating tools to survive the flaky: the retry unblocks them, quarantine isolates them, reproduction puts them in your hand. They all share a limit we have already named several times and that now it is time to face head-on: none fixes anything. The retry retries a flaky that stays as flaky as before; quarantine hides a flaky that stays alive; reproducing gives you a bug you have not yet touched. They are triage —painkillers that keep you functioning— and triage without a cure is a sentence to repeat it forever. This lesson is the cure: fixing determinism at its root, removing the source of non-determinism that makes the test dance. When a test is deterministic, it needs neither retry nor quarantine, because it no longer fails at random. It is where the flaky stops being flaky.

The thesis is simple and powerful: every flaky has, at its bottom, a source of non-determinism —the clock, the order, the shared state, the randomness without a seed, the network—. Curing the flaky is identifying that source and removing it, not learning to live with it. You are going to see the two most common cures executed for real on the two flaky of the module: injecting the clock kills the should_audit flaky (and along the way makes the other branch probable, previously untestable), and a fixture that gives a fresh Calendar per test kills the order flaky (the order stops mattering). Both, run many times, come out stable —the flicker disappears—.

Connection with the module: this is the conceptual closing lesson, the one that gives meaning to the previous ones. The retry (3) and quarantine (4) are triage; reproduction (5–6) is the step prior to the cure; and this lesson is the cure. Lesson 8, the mini-project, has you decide between triage and cure facing a flaky that blocks Reservo's CI —and this lesson is the one that tells you the cure is always the correct answer in the medium term, even though the triage unblocks you today—. The diagnosis that precedes the cure (isolating the exact source) leans on the sibling guide; here the sources are evident and we go straight to removing them.

Level the floor instead of getting used to tripping

Imagine an office where there is a loose tile at the entrance. Every so often, someone steps on it wrong and trips. The "solutions" a team can adopt are revealing:

• Put up a "watch out for the tile" sign. It helps those who read it, but the danger is still there and the newcomers trip just the same. (It is quarantine: you mark the problem, you do not remove it.)
• Ask everyone to "step carefully there." It works sometimes; in a hurry, people forget and trip. (It is the retry: you retry crossing until you make it.)
• Level the tile. Once done, the problem disappears forever: nobody trips, no sign or care is needed, and the newcomers do not even know there was a danger.

The first two are symptom management —living with the loose tile—. The third is the cure —removing the cause—. And notice the asymmetry of effort: leveling the tile costs a while once; putting up signs and asking for care costs forever, because the danger never goes away. The triage seems cheaper ("just a sign") but is paid on every future trip; the cure seems more expensive ("you have to lift the tile") but is paid only once.

A flaky is the loose tile. The retry and quarantine are the sign and the "step carefully" —useful to not smash your face today, useless for the danger to disappear—. Fixing determinism is leveling the tile: you remove the source of non-determinism, and the flaky does not come back, needs no sign, and the next person who comes to the project will not even know there was a problem. This module taught you to put up signs and ask for care because sometimes you have to cross the entrance now; but the goal, always, is to level the tile.

Every flaky hides a source of non-determinism: the clock, the order, the shared state, the randomness without a seed, the network. Retry and quarantine live with it; fixing determinism removes it. The cure is not learning to cross the loose tile carefully —it is leveling it, so that nobody trips again.

Cure one: inject the clock (kill the clock flaky)

The should_audit flaky has a very clear source of non-determinism: the function reads the wall clock (datetime.now()) on its own, so its result depends on the instant of the call. The cure is not to retry until the clock lands well —that is crossing the tile carefully—; it is to take away from the function the power to read the clock on its own and, instead, inject the instant from outside. That injection point is the seam that should_audit already had: the now=None parameter.

Remember the signature:

# reservo/audit.py
def should_audit(now=None) -> bool:
    if now is None:
        now = datetime.now()          # <- the source of non-determinism
    return now.microsecond % 2 == 0

The flaky was born because the test called should_audit() without an argument, letting the function fall into datetime.now(). The cure, on the test side, is to pass it a fixed now: a frozen instant that you choose, with a known microsecond. Thus the test stops asking "what time is it now?" (non-deterministic) and asks "what does should_audit do with this concrete instant?" (deterministic).

# demo_fixed/test_audit_fixed.py
from datetime import datetime

from reservo.audit import should_audit

# The frozen clock that BEFORE was hidden: now it is an explicit datum
# of the test. even microsecond (123456) -> should_audit returns True ALWAYS.
FROZEN_EVEN = datetime(2026, 8, 1, 9, 0, 0, 123_456)
FROZEN_ODD = datetime(2026, 8, 1, 9, 0, 0, 123_457)


def test_booking_is_audited_when_microsecond_is_even():
    # Deterministic: we inject the clock. It no longer depends on the moment of the run.
    assert should_audit(FROZEN_EVEN) is True


def test_booking_is_skipped_when_microsecond_is_odd():
    # And now we can test the OTHER branch, impossible to fix with the real clock.
    assert should_audit(FROZEN_ODD) is False

Notice the second test, test_booking_is_skipped_when_microsecond_is_odd. Injecting the clock not only cured the flaky: it enabled testing the branch that was previously untestable. With the real clock, you could not force an odd microsecond at will —you depended on luck—, so the "odd → not audited" branch was never tested reliably. With the injected now, you choose FROZEN_ODD and verify that branch with certainty. The seam that cures the flaky is the same one that gives you real coverage: a deterministic test is not only stable, it is more complete.

Worked example 1: the clock flaky, now stable run after run

The proof that a cure worked is that the flicker disappears. Let us run the two deterministic tests eight times in a row —just as we did with the flaky in lesson 1, when it danced between 1 failed and 8 passed—. With Python 3.14.0 and pytest 9.1.1, measured by executing the eight runs:

for i in $(seq 1 8); do python -m pytest demo_fixed/test_audit_fixed.py -q; done

What to expect. Eight identical runs, all green, without a single flicker:

2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s

Eight 2 passed, without exception. Compare with lesson 1, where the same class of test gave 1 failed one out of every two times. The tile is leveled: the clock no longer rules, so the result is a pure function of the inputs (FROZEN_EVEN, FROZEN_ODD) that you control. This test no longer needs --reruns, nor @pytest.mark.flaky, nor quarantine —it does not fail at random because it does not depend on any randomness—. And in verbose:

collecting ... collected 2 items

demo_fixed/test_audit_fixed.py::test_booking_is_audited_when_microsecond_is_even PASSED [ 50%]
demo_fixed/test_audit_fixed.py::test_booking_is_skipped_when_microsecond_is_odd PASSED [100%]

============================== 2 passed in 0.01s ==============================

The two branches of should_audit tested, deterministic, both green, always. That is curing a flaky: not that it passes this time, but that it cannot fail at random ever.

An honest nuance: injecting the clock cures the test. If you also want the production function to be deterministic (that in production it not sample by clock, which is a dubious idea in itself), the root fix would be to change the sampling strategy for a deterministic one —for example, sampling according to a stable hash of the booking's id, not according to the clock—. That is a redesign of the feature, beyond the scope of the module; here the lesson is the seam that makes the test deterministic, which is what removes the flaky from the CI.

Cure two: isolate the state (kill the order flaky)

The order flaky of lesson 5 had another source of non-determinism: a Calendar shared at module level between two tests, so the verdict of one depended on whether the other ran before. The cure is not to force a fixed order or retry —that is crossing the tile carefully—; it is to remove the shared state, giving each test its own fresh Calendar. Without common state, the order stops mattering: each test starts from scratch, isolated from its neighbors.

The pytest tool for this is the fixture: a function marked with @pytest.fixture that produces a new object, and that pytest runs once for each test that requests it. Each test receives its own instance, freshly created, without a trace of what the others did.

# demo_fixed/test_isolated_calendar.py
from datetime import datetime, timedelta

import pytest

from reservo.models import Room, Member
from reservo.calendar import Calendar, book, is_available

focus = Room(id="r1", name="Focus", capacity=1, hourly_cents=2500)
ana = Member(id="m1", name="Ana", tier="basic")
start = datetime(2026, 8, 1, 9, 0)


@pytest.fixture
def calendar():
    # Each test receives a NEW Calendar. Without shared state, without order.
    return Calendar()


def test_a_focus_free_at_nine(calendar):
    assert is_available(calendar, "r1", start, start + timedelta(hours=1)) is True


def test_b_book_focus(calendar):
    b = book(calendar, focus, ana, start, start + timedelta(hours=3))
    assert b.price_cents == 7500

Compare with the sick version of lesson 5. There, shared = Calendar() lived at module level, a single one for the two tests. Here, calendar is a fixture, and each test receives it as a parameter (def test_a_focus_free_at_nine(calendar)): pytest calls the calendar() function again for each test, so test_a gets an empty Calendar and test_b gets another empty Calendar, independent. When test_b books Focus, it mutates its calendar, not test_a's. The shared state disappeared, and with it, the order dependency.

Worked example 2: the order flaky, now indifferent to the order

The proof of this cure is that the failure that lesson 5 provoked by changing the order no longer happens in any order. Let us run the two isolated tests in definition order and in reverse order —the one that previously turned everything red—. With Python 3.14.0 and pytest 9.1.1, measured by executing:

# definition order
python -m pytest demo_fixed/test_isolated_calendar.py -v
# reverse order (the one that broke before)
python -m pytest \
  "demo_fixed/test_isolated_calendar.py::test_b_book_focus" \
  "demo_fixed/test_isolated_calendar.py::test_a_focus_free_at_nine" -v

What to expect. The two orders, green:

# --- definition order ---
demo_fixed/test_isolated_calendar.py::test_a_focus_free_at_nine PASSED   [ 50%]
demo_fixed/test_isolated_calendar.py::test_b_book_focus PASSED           [100%]
============================== 2 passed in 0.01s ==============================

# --- reverse order (the one that broke before) ---
demo_fixed/test_isolated_calendar.py::test_b_book_focus PASSED           [ 50%]
demo_fixed/test_isolated_calendar.py::test_a_focus_free_at_nine PASSED   [100%]
============================== 2 passed in 0.01s ==============================

2 passed in both orders. In lesson 5, reversing the order gave 1 failed, 1 passed —test_a found Focus already booked by test_b—. Now, with the fixture, whether test_a runs before or after, it always receives a freshly created empty Calendar, so Focus is always free for it, and test_b books in its own calendar without affecting anyone. The order stopped mattering because the shared state —the loose tile— disappeared. This flaky, which the retry could not rescue (lesson 6), was cured at its root by isolation.

Notice something that ties the module together: this flaky was of the kind that --reruns did not save (lesson 6), because its cause was structural, not chance. The structural cure —isolating the state— is the only one that works for it. It is the confirmation of the thesis: the tool that cures a flaky depends on its source of non-determinism, and for the state/order, that tool is isolation, never the retry.

The map of cures by source of non-determinism

The two cures you executed are cases of a general principle. Each source of non-determinism has its cure, and they all share the form: remove the dependency on the uncontrolled world, and inject/isolate in its place.

• Clock (datetime.now(), time.time()): inject the instant as a parameter or fixture; use a fixed clock. (Libraries like freezegun automate freezing the clock for a whole function.) You did it with should_audit(now=...).
• Order / shared state (module variables, singletons, common files): isolate with fixtures that give a fresh object per test; never share mutable state between tests. You did it with the calendar fixture.
• Randomness (random, UUIDs, generated data): fix the seed (random.seed(0)) or inject the value; make the result reproducible.
• Network / external services (APIs, databases): use test doubles (mocks/fakes) for the unit suite; isolate the real integration tests from the fast gate, or give them a legitimate retry (the "first ATM" case of lesson 3).
• Concurrency / limited resources (timeouts, ports, temporaries): give each test its own resource (tmp_path), avoid timeouts based on real wall-clock time, do not assume hardware speed.

The unified pattern: a deterministic test does not depend on anything it does not control from the test itself. If its result can change without you changing its input —because the clock advanced, because another test ran before, because the random die landed differently—, there is a source of non-determinism to inject or isolate. Curing the flaky is closing that door to the uncontrolled world.

Common mistakes

Treating the triage as a destination. What happens: the team puts --reruns or xfail on a flaky "for now," and that "for now" becomes permanent —the flaky lives in triage forever, never gets cured—. Why it happens: the triage unblocks, and once unblocked the urgency to cure disappears. How to detect it: if you have a flaky with a retry or in quarantine for months without an attempt at a fix, the triage became a destination. How to fix it: each triage carries a ticket with the pending cure (lesson 4), and the cure —inject the clock, isolate the state— is scheduled, not postponed indefinitely. The "watch out for the tile" sign was never the solution; leveling the tile was.

Curing the symptom without finding the source. What happens: someone "fixes" the order flaky by fixing the order of the tests (-p no:randomly, or renaming them so they run in a certain order) instead of isolating the state. Why it happens: fixing the order makes the suite pass today, and it looks like a fix. How to detect it: if your fix depends on the tests running in a certain order, you did not remove the shared state —you only hid its symptom, and it will return with parallelism or a change of order—. How to fix it: find the source (the shared state) and remove it (the fixture). A test that needs to run in a certain order is still fragile, even though it passes today. The cure attacks the cause, it does not accommodate the symptom.

Believing that "it passed eight times" proves determinism without understanding why. What happens: someone runs the fixed test several times, sees green, and considers it cured without understanding what source they removed. Why it happens: the repeated green reassures. How to detect it: if you cannot name the source of non-determinism you eliminated (the clock, the state) and how you eliminated it (injection, isolation), you do not know whether you cured or got lucky. How to fix it: the cure is understood, not guessed —you must be able to point to the line that introduced the chance/state and the line that closed it—. Eight greens are evidence of the cure, not the cure; the cure is the seam or the fixture, and knowing why they work is what lets you apply them to the next flaky.

Exercises

Exercise 1 — Cure by source. For each flaky, name the source of non-determinism and the concrete cure. (a) A test that does assert booking.id == "b1" where the id is generated with uuid4(). (b) A test that compares datetime.now().date() with an expected date. (c) Two tests that read and write the same counter in a module global variable.

See solution

• (a) Source: randomness (UUID). uuid4() generates a different random id each time, so == "b1" is impossible to satisfy stably. Cure: inject the id (pass it as a parameter or use a fixture/factory that assigns predictable ids), or if the id does not matter for the test, do not assert it —assert another deterministic property—. Making the value reproducible is the cure.
• (b) Source: clock. datetime.now().date() is today's date, which changes every day —the test passes today and fails tomorrow—. Cure: inject the date (a fixed now, as you did with should_audit), or use freezegun to freeze the clock. The test must ask for a date you control, not for "today."
• (c) Source: shared state (module global variable). The shared counter makes the result of each test depend on what ran before —the order flaky—. Cure: isolate with a fixture that gives a fresh counter per test, or encapsulate the counter in an object that each test creates anew. Without mutable global state, there is no order dependency.

The pattern: name the source (clock/randomness/state), then inject it or isolate it. The cure is always to close the door to the uncontrolled world.

Exercise 2 — The seam that gives coverage. By injecting the clock in should_audit, you not only cured the flaky but were able to add test_booking_is_skipped_when_microsecond_is_odd. Explain why that second branch was impossible to test reliably with the real clock, and what this says about the relationship between determinism and coverage.

See solution

With the real clock, should_audit() returns True or False according to the parity of the microsecond at the instant of the call, and that instant you do not control: it is chance. To test the "odd → False" branch reliably, you would need to guarantee that the call lands on an odd microsecond —impossible with the real clock, you could only wait to get lucky, and a test that depends on luck to exercise a branch is precisely a flaky—. In practice, that branch was left untested stably: some runs touched it, others did not, and none guaranteed it.

By injecting the clock (should_audit(FROZEN_ODD)), you choose the microsecond, so you can force the odd branch at will and assert it with certainty. The seam that cured the flaky is the same one that gave you deterministic access to both branches.

What it says about determinism and coverage: non-determinism not only causes flaky, it also prevents coverage. Code that depends on the clock/chance has branches you cannot exercise at will, so you cannot test them reliably. Making it deterministic —injecting the dependency— is a prerequisite for both stability and complete coverage. A good design for tests (with seams) gives you both things at once: stable tests and reachable branches.

Exercise 3 — Symptom vs. cause. A colleague "fixes" the order flaky of the shared Calendar by adding @pytest.mark.run(order=1) to test_a to force it to run first, and the suite passes. Argue why this is accommodating the symptom, not curing the cause, and what would happen in CI with parallelism.

See solution

Forcing test_a to run first makes the suite pass today, but does not remove the shared state —the module-level shared = Calendar() is still there, and test_b still mutates it—. It only hid the symptom by tying the order by force. The underlying problem, that two tests depend on a common state, persists intact.

What would happen in CI with parallelism (-n auto, from module 5): with pytest-xdist, the tests are distributed among several processes, and the forced order within one process does not guarantee the global order —worse, if test_a and test_b land in different processes that share... actually with xdist each process has its own module, but the general point holds: any change in how the tests are distributed or run can re-expose the dependency—. And even if the order were respected, the design is still fragile: a test that needs to run in a certain order is a test that assumes things about its neighbors, exactly what a test must not do. The day someone adds a test_c that also touches shared, or the order changes for any reason, the flaky returns.

The real cure is the fixture: giving each test a fresh Calendar eliminates the shared state, and then the order stops mattering —not because you forced it, but because there is no longer anything that depends on it—. Accommodating the symptom (forcing the order) is fragile and temporary; removing the cause (isolating the state) is robust and permanent. Level the tile, do not ask everyone to step in a certain order.

Summary and next step

In this lesson you reached the cure, the one that gives meaning to the whole module: fixing determinism at its root, removing the source of non-determinism instead of living with it. The retry and quarantine are the "watch out for the tile" sign and the "step carefully" —they keep you standing today—; fixing determinism is leveling the tile, so that the flaky does not come back and nobody trips.

You executed the two most common cures on the two flaky of the module. Injecting the clock (the now= seam of should_audit) killed the clock flaky: the deterministic tests came out 2 passed eight times in a row, without a flicker, and along the way you enabled testing the odd branch, previously untestable —determinism gives stability and coverage—. A fixture that gives a fresh Calendar per test killed the order flaky: 2 passed in definition order and in reverse, because without shared state the order stopped mattering —exactly the flaky the retry could not rescue—. And you saw the map of cures by source: clock → inject, state → isolate, chance → seed, network → double, resources → give each test its own.

Before moving on you should be able to: name the source of non-determinism of a flaky and its cure; inject a clock and isolate state with a fixture; explain why determinism enables coverage in addition to stability; and distinguish accommodating the symptom (forcing the order) from curing the cause (isolating the state).

What follows, in lesson 8, is the mini-project: a flaky blocks Reservo's CI and stops three PRs, and you decide —retry, quarantine, or fix— with an honest decision matrix, and you execute the decision with real evidence. The whole module converges there: you will know that triage unblocks today and the cure resolves forever, and you will have to justify, for this flaky, which one corresponds and why. It is Reservo's tile, and it is your turn to decide whether you put up a sign or level it.

Resources

• Fixtures — pytest documentation — the reference for the cure of the order flaky: fixtures that give a fresh object per test to isolate the state. Read about the scope of the fixtures (why the default function is what guarantees a new object per test).
• tmp_path — pytest documentation — the fixture that gives each test its own temporary directory, the cure of the shared-resources flaky (temporary files that two tests step on). The same isolation principle as the calendar fixture.
• freezegun — PyPI — the library that freezes datetime.now() for a whole test function, automating the injection of the clock when the now= seam does not exist or is uncomfortable. The cure of the clock flaky, industrialized.
• datetime — Python documentation — the source of non-determinism we inject: understanding that now() reads the wall clock on each call is understanding why injecting it cures the flaky. The now= seam is what takes that power away from the function.