8. Mini-project: a flaky test blocks Reservo's CI

Description

The time has come to bring the whole module together into a real decision. The seven previous lessons gave you the diagnosis (what a flaky is, why it is toxic in CI), the triage tools (retry, quarantine), the hunt for the CI-only (classify, reproduce), and the cure (fixing determinism). Now you exercise them all facing a concrete situation: the should_audit flaky blocks Reservo's CI and stops three pull requests, and you have to decide what to do —retry, quarantine, or fix— and justify the choice with an honest decision matrix and real evidence.

This is not a test-writing exercise: it is the judgment work that separates a team that drowns in flaky from one that manages them. The same tool (retry, quarantine, fix) is correct or disastrous depending on the context —the urgency, the cause of the flaky, who is blocked—, and choosing well requires understanding the trade-off of each one. You are going to produce four deliverables: the diagnosis of the flaky (what it is, what causes it, how serious), the decision matrix (the three options with their cost and benefit for this case), the execution of the decision with really measured evidence, and a policy note so the team does not repeat the problem. All with the honesty of the guide: the CI YAML is content, but every pytest output —RERUN, xfailed, the stable suite— is real, executed in Python 3.14.0.

Connection with the module: this lesson closes the arc. Lesson 1 gave you the flaky; lesson 2, its toxicity; lesson 3, the retry; lesson 4, the quarantine; lesson 5, the CI-only; lesson 6, the reproduction; lesson 7, the cure. The mini-project exercises them all at once on a decision a tech lead really makes. And it looks forward: by the end, you will know how to manage the flaky in your pipeline, the last piece before the capstone of the guide (module 8), where you assemble Reservo's complete pipeline —suite, matrix, coverage, cache, and yes, a flaky policy—.

The scenario

It is 4 in the afternoon on a Thursday. The Reservo team has three open PRs that need to merge before the end of the week:

• Ana's PR: changes the text of a confirmation email. Urgent (marketing is waiting for it), does not touch auditing.
• Bruno's PR: adds a new room to the catalog. Important, does not touch auditing.
• Carla's PR: fixes a real refund bug. Critical, does not touch auditing.

All three have the CI in red. The failure, in all three, is the same:

FAILED tests/test_audit.py::test_new_booking_is_audited - assert False is True
1 failed, 7 passed in 0.02s

None of the three touched should_audit. The audit flaky —the one that looks at the wall clock— landed on an odd microsecond in each PR's run, and since the gate does not distinguish "red from a bug" from "red from a flaky," all three are blocked. It is damage one of lesson 2, live: a flaky in the shared gate stops three people who did nothing wrong. You have to unstick this —today— and, in addition, resolve the flaky so it does not happen again. Your work:

1. Diagnose the flaky: what it is, what its source of non-determinism is, how serious.
2. Build the decision matrix: the three options (retry, quarantine, fix) with their cost and benefit for this case.
3. Execute the decision, with real evidence (the pytest output that backs it).
4. Write the policy note for the team, so the next flaky is handled without drama.

Try each step on your own before looking at the solution. The learning is in deciding yourself, not in reading the decision.

Step 1 — Diagnose the flaky

Before choosing a tool, understand the enemy. Run the flaky several times and answer: is it really flaky (does it flicker without changing the code)? What is its source of non-determinism? Does the retry rescue it (a hint of whether it is chance or structure)? How often does it fail?

Think about it before continuing: is the cause of this flaky chance (clock/network) or structure (order/state)? That question decides which tools can even work.

Step 2 — Build the decision matrix

For this flaky, in this situation (three urgent PRs blocked on a Thursday at 4), evaluate the three options. For each one: does it unblock today? does it cure the flaky? what risk does it bring? Remember from lesson 3 that the retry is a painkiller, from lesson 4 that quarantine isolates with a ticket, and from lesson 7 that only the fix cures.

Think about it: does any option unblock and cure at once? Or do you have to combine one of triage (for today) with the cure (so it does not come back)?

Step 3 — Execute the decision with evidence

Once the option (or the combination) is chosen, execute it and capture the real output that backs it. If you choose to fix, show the stable suite; if you combine triage + cure, show both pieces of evidence.

Step 4 — Write the policy note

A paragraph the team can paste into its CONTRIBUTING.md: how to treat a flaky next time, so as not to improvise under pressure.

---

Complete solution

Deliverable 1 — Diagnosis of the flaky

I run the flaky several times to confirm it flickers, and with --reruns to read its cause. With Python 3.14.0, pytest 9.1.1, and rerunfailures 16.4:

python -m pytest tests/test_audit.py::test_new_booking_is_audited --reruns 3 -v

In a real run where the first attempt landed odd and the retry even:

tests/test_audit.py::test_new_booking_is_audited RERUN                   [100%]
tests/test_audit.py::test_new_booking_is_audited PASSED                  [100%]
========================== 1 passed, 1 rerun in 0.01s ==========================

The diagnosis, point by point:

• Is it flaky? Yes. Run many times without touching the code, it gives 1 failed, 7 passed one out of every two times and 8 passed the other. Inconsistency without a change of input: the mark of the flaky (lesson 1).
• Source of non-determinism? The wall clock. should_audit() without an argument falls into datetime.now() and decides by the parity of the microsecond (now.microsecond % 2 == 0). The test test_new_booking_is_audited calls should_audit() without injecting now, tying its verdict to the instant of the run.
• Chance or structure? Chance. The RERUN followed by PASSED confirms it: the retry does rescue it, because each retry re-reads the clock and re-throws the die (lesson 6). If it were structural —order/state—, the retry would fail identically in all the attempts. That it helps tells me it is chance from the clock.
• Severity? It fails ~50% of the runs. It is among the worst: so frequent that it constantly blocks the gate. A 1% flaky is tolerated with a retry; a 50% one is a barricade.

Conclusion of the diagnosis: clock flaky, chance, ~50% failure, cause in the test (not in Reservo's business logic). The price and refund logic is healthy —the seven greens confirm it—; the problem is a test that asks a non-deterministic question.

Deliverable 2 — The decision matrix

For this flaky (clock, chance, 50%) in this situation (three urgent PRs blocked, Thursday 4 pm):

Option	Unblocks today?	Cures the flaky?	Risk / cost	Verdict for this case
Retry (--reruns)	Yes, almost always (94% with --reruns 3; the retry rescues because it is chance)	No	Retries the whole suite; can cover real bugs; perpetuates the flaky if it stays	Useful as immediate triage, but global is crude
Quarantine (xfail/flaky marker)	Yes, completely (takes the flaky out of the gate; all three PRs pass)	No	Must carry a ticket and be temporary; the flaky stays alive	Good surgical triage: isolates only the culprit
Fix (inject the clock)	Not immediately for today's PRs (you have to write the fix, review it, merge it)	Yes, at its root	Takes a bit of work now	The only cure; indispensable in the medium term

The reading of the matrix for this case: no single option is enough. The fix cures but does not unblock right today (Ana's, Bruno's, and Carla's three PRs cannot wait for the fix to pass through review). The retry and quarantine unblock today but do not cure. That is why the correct decision is a combination: surgical triage now to unstick the three PRs, and the fix in parallel as the real resolution, with the quarantine tied to a ticket so it does not get forgotten.

Concretely: put the flaky in quarantine with xfail + ticket (unblocks the three PRs immediately, surgically, without retrying the whole suite or covering other possible reds) and open the fix PR right away (inject the clock, the cure of lesson 7). When the fix merges, the quarantine is removed. The quarantine is the "watch out for the tile" sign we put up while someone —today— levels the tile.

Why quarantine and not global retry for the triage? Because the global retry (addopts = --reruns 2) would retry the whole suite, including Reservo's six deterministic anchors and any real bug that failed honestly —right when Carla's PR fixes a refund bug and I do not want a global retry to mask a regression there—. The quarantine with xfail is surgical: it isolates only test_new_booking_is_audited and lets the rest of the suite fail honestly. In a situation with a critical refund PR at stake, that aim matters.

Deliverable 3 — Execution of the decision, with evidence

Part A — the quarantine (today's triage). I mark only the flaky with xfail, with its ticket, strict=False so that neither a failure nor a pass breaks the gate:

# tests/test_audit.py
import pytest

from reservo.audit import should_audit


@pytest.mark.xfail(
    reason="clock flaky — ticket RES-412; fix in PR #128 (inject now)",
    strict=False,
)
def test_new_booking_is_audited():
    assert should_audit() is True

With the quarantine in place, the gate is unblocked however the flaky lands. I verify both faces, measured by executing with -rxX:

# --- run where the flaky failed -> XFAIL (expected, does not break the gate) ---
demo_quarantine/test_xfail_quarantine.py x                               [100%]
XFAIL ... - clock flaky — ticket RES-412; fix in PR #128 (inject now)
1 xfailed in 0.01s

# --- run where the flaky passed -> XPASS (unexpected, also does not break the gate) ---
demo_quarantine/test_xfail_quarantine.py X                               [100%]
XPASS ... - clock flaky — ticket RES-412; fix in PR #128 (inject now)
1 xpassed in 0.01s

1 xfailed or 1 xpassed, and in both the gate stays green. Ana's, Bruno's, and Carla's PRs can now merge —the flaky stopped blocking them—, and the flaky is still visible (with its ticket RES-412 and the reference to the fix PR) in every run. Nobody hid it; it was isolated with follow-up.

Part B — the fix (the real cure, PR #128). In parallel, I write the cure of lesson 7: inject the clock. The test stops asking "what time is it now?" and asks "what does should_audit do with this instant?":

# tests/test_audit.py (curated version, replaces the flaky)
from datetime import datetime

from reservo.audit import should_audit

FROZEN_EVEN = datetime(2026, 8, 1, 9, 0, 0, 123_456)   # even microsecond
FROZEN_ODD = datetime(2026, 8, 1, 9, 0, 0, 123_457)    # odd microsecond


def test_booking_is_audited_when_microsecond_is_even():
    assert should_audit(FROZEN_EVEN) is True


def test_booking_is_skipped_when_microsecond_is_odd():
    assert should_audit(FROZEN_ODD) is False

The proof that the fix cures: I run the deterministic tests eight times in a row and none flickers. Measured by executing:

2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s
2 passed in 0.00s

Eight 2 passed, without a single red —compare with the 1 failed of one out of every two runs of the original flaky—. The tile is leveled. When PR #128 merges, I remove the xfail quarantine: the flaky no longer exists, so there is nothing to isolate. And bonus (lesson 7): the fix enabled test_booking_is_skipped_when_microsecond_is_odd, the odd branch that with the real clock was untestable —determinism gave me coverage in addition to stability—.

The complete sequence, then: xfail quarantine with a ticket (today, 4 pm → the three PRs merge) → PR #128 with the cure (this afternoon/tomorrow) → merge of the fix → remove the quarantine. Triage to unblock, cure to resolve, and the ticket that ties one to the other so the triage does not become permanent.

Deliverable 4 — The policy note

To paste into Reservo's CONTRIBUTING.md:

Flaky test policy. A flaky test (passes sometimes, fails sometimes without the code changing) is never "fixed" by re-running the CI until it passes: that leaves it alive and trains the team to distrust the red. Facing a flaky that blocks the gate: (1) Diagnose its source —chance (clock, network) or structure (order, shared state)?—; a quick hint: if --reruns rescues it, it is chance; if it fails identically in the retries, it is structure. (2) Unblock today with surgical quarantine: @pytest.mark.xfail(reason="... ticket X", strict=False) on only that test —never global retry, which masks the whole suite—. The quarantine always carries a ticket and is temporary. (3) Cure at its root in a PR: remove the source of non-determinism —inject the clock (now= parameter/freezegun), isolate the state with a fixture, seed the randomness—. (4) Remove the quarantine when the cure merges. We review the xfail list every sprint: zero is the goal; an xfail older than a month is an alarm. The retry (--reruns) is an emergency painkiller, with a ticket, never a destination.

Common mistakes

Choosing the tool without diagnosing the cause. What happens: someone puts --reruns 5 global "because there's a flaky," without seeing that —had it been an order flaky— the retry would not have helped, or —as here— that there was a critical refund PR that a global retry could mask. Why it happens: under pressure the first tool is grabbed. How to detect it: if you chose before knowing whether the flaky is chance or structure, you chose blindly. How to fix it: step 1 (diagnosis) goes before step 2 (decision); the cause determines which tools can work and which are dangerous in the context.

Staying in the triage and never opening the fix PR. What happens: the quarantine is put in place, the PRs merge, the urgency goes away, and the fix is never written —the "temporary" xfail turns six months old—. Why it happens: once unblocked, the pressure to cure disappears. How to detect it: if your quarantine does not have a fix PR already open or scheduled, it is going to become permanent. How to fix it: the quarantine and the fix PR are opened together, tied by the ticket; the policy note includes the per-sprint review precisely to catch the xfail that age. Triage without a scheduled cure is denial with an open date.

Using global retry when a critical PR is at stake. What happens: addopts = --reruns 2 is enabled to calm the flaky, without noticing that it retries the whole suite —including Carla's PR that fixes a refund bug—, so if that fix had an intermittent regression, the global retry could mask it. Why it happens: the global retry is one line and "fixes" the visible flaky. How to detect it: if your triage affects tests that are not the flaky, it is too broad. How to fix it: surgical triage —xfail/flaky on the guilty test, not global --reruns—; the aim matters especially when there are critical changes that need an honest gate.

Exercises

Exercise 1 — Change the context, change the decision. The same clock flaky, but now the context is different: it is Monday morning, there are no urgent PRs blocked, and you have all week. Does your decision change with respect to the one in the mini-project? Justify.

See solution

Yes, it changes —and this is precisely the lesson that the correct tool depends on the context—. In the mini-project, the urgency (three PRs blocked on a Thursday 4 pm) forced an immediate triage (quarantine) plus the cure in parallel. A Monday without urgency, there is nothing to unblock in a hurry, so you can go straight to the cure without quarantine: you open PR #128 (inject the clock), it is reviewed calmly, it merges, and the flaky disappears —without going through xfail—.

Why is it better to skip the triage when there is no urgency? Because the quarantine, even well done, is debt: an isolated test you have to remember to remove. If you can cure directly, you avoid creating that debt. The quarantine exists to unblock under pressure; without pressure, it is an unnecessary detour. The decision is not dictated only by the flaky (which is the same), but by the flaky plus the context: high urgency → triage + cure; low urgency → direct cure.

The moral: there is no universal answer to "retry, quarantine, or fix?". There is an answer for this flaky at this moment. Diagnose the cause (decide what can work) and read the context (decide what is advisable).

Exercise 2 — The flaky the retry does not save. Suppose the flaky that blocks the three PRs were not the clock one, but the order one of lesson 5 (the shared Calendar). Redo the decision matrix: does the retry still serve as triage? And the quarantine? What is the cure?

See solution

The matrix changes in one key cell: the retry no longer serves as triage, because this flaky is of structure, not chance. As we saw in lesson 6, --reruns 3 on the order flaky gives RERUN, RERUN, RERUN, FAILED —it fails identically in all the retries, because the Calendar stays booked between attempts—. The retry unblocks nothing here; it would be useless.

• Retry: ❌ does not unblock (fails identically in the retries; the state does not revert) and ❌ does not cure. Discarded even as triage.
• Quarantine (xfail on the order-dependent test): ✅ unblocks (takes it out of the gate) but ❌ does not cure. It is still valid triage for today.
• Fix (fixture that gives a fresh Calendar per test): ✅ cures at its root —the order stops mattering—. It is lesson 7, the cure of the order flaky.

The decision: since the retry is out, today's triage is only quarantine (with a ticket), and the cure is the fixture. The sequence: xfail with a ticket → PR with the fixture → merge → remove the quarantine. And a diagnostic lesson that this change underscores: the cause of the flaky filters the available triage tools. For the chance flaky (clock), retry and quarantine serve as triage; for the structure one (order), only the quarantine —the retry is useless—. That is why step 1 (diagnosing chance vs. structure) is the first thing: it discards options before you consider them.

Exercise 3 — Defend the quarantine before an impatient boss. Your tech lead says: "Let's not waste time with xfail and tickets. Put --reruns 3 global in the pytest.ini, let it retry everything, and let's move on —we'll fix the flaky someday—." Give three reasons, anchored in the module, to prefer the surgical quarantine + scheduled fix over the indefinite global retry.

See solution

Three reasons:

1. The global retry masks the whole suite, not just the flaky (lesson 3). --reruns 3 in addopts retries Reservo's six anchors and any real bug that fails honestly —right when there is a critical refund PR (Carla) that needs an honest gate—. If that fix had an intermittent regression, the global retry could rescue it and let it through. The quarantine with xfail is surgical: it isolates only the flaky and lets the rest of the suite really fail.

2. "We'll fix it someday" is how a flaky lives forever (lessons 2 and 4). A global retry without a ticket has no date or owner; the urgency disappears once unblocked and nobody returns. The quarantine with a ticket (RES-412) and the fix PR opened today tie the triage to a concrete, scheduled cure. The ticket is what turns "someday" into "PR #128, this sprint."

3. The global retry trains the toxic reflex and hides the debt (lesson 2). With the whole suite retrying in silence, the team loses sight of how many flaky it has and whether they get worse —the X rerun becomes background noise that nobody counts—. The quarantine keeps the flaky visible (it appears as xfailed/xpassed with its ticket in every run), so the debt is in sight and can be managed. A visible problem with a ticket gets fixed; one masked globally rots.

The honest closing: it is not that the retry is forbidden —it is a legitimate emergency painkiller—; it is that the global and indefinite retry combines the worst (masks everything, without a ticket, without visibility). The surgical quarantine unblocks just as fast, without those costs, and with a scheduled cure. We unblock today and resolve —not one thing at the cost of the other—.

Summary and next step

In this mini-project you exercised the whole module on a real decision: the should_audit flaky was blocking three urgent Reservo PRs, and you had to choose —and justify— between retry, quarantine, or fix. You diagnosed the flaky (clock, chance, ~50%, cause in the test; the retry rescues it → chance, not structure). You built the decision matrix and saw that no single option is enough: the fix cures but does not unblock today, the triage unblocks but does not cure. You executed the combination with real evidence —xfail quarantine with a ticket to unstick the three PRs immediately (1 xfailed/1 xpassed, gate green), and the fix by clock injection as the cure, stable in eight runs (2 passed ×8)—. And you wrote the policy so the team handles the next flaky without improvising.

The underlying lesson, the one that closes the module: the correct tool —retry, quarantine, fix— depends on the cause of the flaky (chance vs. structure filters what can work) and on the context (the urgency filters what is advisable). The triage unblocks today; the cure resolves forever; and a ticket that ties one to the other is what prevents the painkiller from becoming the diet. Level the tile, always —and put up the sign only while someone levels it—.

Before closing you should be able to: diagnose a flaky (cause, chance vs. structure, severity); build an honest decision matrix for a flaky in its context; execute the decision combining surgical triage with a scheduled cure; and write a flaky policy for a team.

What follows is module 8, the capstone of the whole guide: assembling the complete CI pipeline for Reservo —the suite running on every push, the version matrix (module 4), the cache and parallelism (module 5), the coverage gate (module 6), and the flaky policy you just designed (module 7)—. Everything you learned, together, in a single workflow that protects Reservo's main branch without becoming a bottleneck. The flaky, which started by breaking trust in the traffic light, ends as one more piece of a pipeline the team does believe.

Resources

• Flaky tests — pytest documentation — the complete frame this mini-project exercises: diagnose, isolate, and fix at the root, in that order. Return to it to see that the triage→cure sequence is the official recommendation, not an opinion of the guide.
• pytest-rerunfailures — GitHub repository — the reference for the retry and the @pytest.mark.flaky marker, the triage options of the decision matrix. The authors insist, like this lesson, that the retry is temporary.
• xfail and strict — pytest documentation — the surgical quarantine tool of the decision: xfail(strict=False) with reason/ticket. Read why strict=False is the right one for a flaky that sometimes passes.
• Fixtures — pytest documentation and freezegun — PyPI — the two root cures of exercise 2 and the mini-project: the fixture that isolates the state (order flaky) and the freezing of the clock (clock flaky). Module 8 (capstone) integrates the flaky policy you designed here into Reservo's complete pipeline.