Module 8: Project A Ci Pipeline For Reservo

7. The flaky policy

Next →

Description

Your pipeline is already serious: it runs on every push, on three versions, with cache, parallelism, and a coverage gate. But it is missing an answer to a problem that lesson 5 itself seeded and that erodes trust like no other: the flaky test —the one that sometimes passes and sometimes fails without the code changing—. An honest red says "there is a bug, fix it." A flaky red says "maybe there is a bug, or maybe not, run it again and we will see." And the day the team learns that the red sometimes lies, it stops believing all the reds —including the true ones—. A flaky is not a test that fails; it is a test that poisons the signal of the whole pipeline.

It is the layer of module 7, and the most judgment-based of all, because the easy tool —retry until it passes— is also the easy trap. You are going to see the retry debate really executed: the run without retry (1 failed) and with --reruns 2 (RERUN→PASSED), the per-test retry with @pytest.mark.flaky, and the quarantine with a marker (-m "not flaky"). And you are going to understand why the retry, used without discipline, does not fix the flaky: it hides it, and hiding a flaky can hide a real bug. The honest policy is not "retry until it passes"; it is "temporary patch, ticket, and root fix."

By the end you will know how to configure retries and quarantine, recognize the triggers of a flaky (order, parallelism, resources, time), and —what separates whoever silences a flaky from whoever resolves it— apply a policy that keeps the pipeline fast without letting the trust erode.

Connection with the module: this is the sixth and last layer before the project, and it closes a circle opened in lesson 5. The parallelism (-n auto) you added for speed changes the order of execution, and that change is one of the classic triggers of flaky: a test that assumed running after another is uncovered when xdist distributes them. You accelerated there; you manage the side effect here. And it marks an important boundary: this lesson teaches you to contain a flaky in CI (retry, quarantine, policy), but diagnosing why a test is non-deterministic —hunting the root cause— is the topic of the sibling guide test-failure-diagnosis. Here you decide what to do with the flaky; there you find out why it is one.

The car alarm that howls for no reason

Think of a car whose alarm goes off on its own. The first week, every time it howls, you run out to see if it is being stolen. There is never anyone: it goes off with the wind, with a passing truck, with nothing. By the third week, when the alarm sounds, you do not even turn —"it's the alarm again, it goes off on its own"—. And then, one night, someone really tries to steal it, the alarm howls, and nobody turns, because nobody believes it. The alarm that went off for no reason was not only annoying: it destroyed its own ability to warn of a real danger.

The problem is not that the alarm sounds; it is that it sounds when there is no danger, and those false alarms train everyone to ignore it. A useful alarm has to be reliable: when it sounds, there is something. The serious owner does not lower its volume to not hear it (that makes it useless in another way); they fix the sensor that goes off with the wind, so that each howl means danger again.

A flaky test is that alarm. When it fails without a bug —because of the order, because of a resource, because of the clock—, it is a false alarm. And the false alarms train the team to ignore the reds: "the CI is red again, hit re-run." The day a red does signal a real bug, nobody looks at it, because the pipeline already lost its credibility. The temptation is to lower the volume —retry until it passes, so as not to hear the red—, but that makes the alarm useless in another way: now it does not even warn of the real dangers. The serious answer is to fix the sensor: find why the test is non-deterministic and make it deterministic, so that each red means something again.

A flaky is an alarm that howls without danger: it trains the team to ignore the reds, and thus destroys the signal of the whole pipeline. Retrying until it passes is lowering its volume; the serious answer is fixing the sensor —making the test deterministic—.

The retry: the easy tool (and its trap)

pytest-rerunfailures (already in your requirements-dev.txt) retries the tests that fail. The --reruns N flag retries up to N times before declaring the definitive failure. Let us see it with a real flaky: a test that fails the first time and passes on the retry (models a test that depends on a resource that is "not ready yet" on the first attempt).

Without retries, the flaky breaks the build:

python -m pytest flaky_demo/

What to expect (real):

=========================== short test summary info ============================
FAILED flaky_demo/test_flaky_confirmation.py::test_booking_confirmation_is_sent - AssertionError: the confirmation service did not respond on the first try
========================= 1 failed, 1 passed in 0.03s =========================

Red: 1 failed. Now with --reruns 2:

python -m pytest --reruns 2 -v flaky_demo/

What to expect (real):

collected 2 items

flaky_demo/test_flaky_confirmation.py::test_booking_confirmation_is_sent RERUN [ 50%]
flaky_demo/test_flaky_confirmation.py::test_booking_confirmation_is_sent PASSED [ 50%]
flaky_demo/test_flaky_confirmation.py::test_pricing_is_deterministic PASSED [100%]

========================== 2 passed, 1 rerun in 0.02s ==========================

Green: 2 passed, 1 rerun. Notice the RERUN: the test failed, pytest-rerunfailures ran it again, and the second time it passed. The build stays green and the summary betrays the retry (1 rerun). It seems like the perfect solution —the flaky no longer breaks the build—, and here is the trap.

The retry does not fix the flaky; it hides it. The test is still non-deterministic —it still fails the first time—, only now the pipeline disguises it by retrying. And disguising has a dangerous cost: if that test ever fails because of a real bug —not because of its usual flakiness, but because the code really broke—, the retry will also hide it, giving you a green over a bug. You lowered the alarm's volume: it no longer bothers you, but it no longer warns of the thieves either. The 1 rerun in the summary is the hint that something was disguised; a team that ignores those hints accumulates retried flaky until the pipeline is a theater of greens that mean nothing.

The retry with discipline: per test, not global

A global --reruns 2 retries all the tests, including the healthy and deterministic ones —so a real bug in a good test would also be retried and could sneak through—. The disciplined way is to retry only the test you know is flaky, leaving the others with their honest red on the first failure. pytest-rerunfailures allows it with a per-test marker:

import pytest

@pytest.mark.flaky(reruns=2)
def test_external_calendar_sync():
    ...  # this test hits an external resource that sometimes takes a while

Now only test_external_calendar_sync is retried; the rest of the suite fails on the first try if something breaks. Really executed:

python -m pytest -v flaky_demo/test_flaky_marked.py

What to expect (real):

flaky_demo/test_flaky_marked.py::test_external_calendar_sync RERUN         [ 50%]
flaky_demo/test_flaky_marked.py::test_external_calendar_sync PASSED        [ 50%]
flaky_demo/test_flaky_marked.py::test_core_pricing_stays_green PASSED      [100%]

========================== 2 passed, 1 rerun in 0.02s ==========================

test_external_calendar_sync was retried (RERUN→PASSED); test_core_pricing_stays_green passed on the first try, without retries —it keeps its honest red if it ever breaks—. This is the difference between lowering the volume of the whole house and putting a patch on the alarm that fails: the per-test retry contains the flaky without anesthetizing the rest of the suite. It is still a patch —the test is still flaky—, but a localized and explicit patch, that also flags the test as suspect for whoever reads the code.

The quarantine: isolate the flaky from the main gate

When a flaky is annoying enough or slow to fix, the next tool is quarantine: taking it out of the main run —the one that decides whether the merge passes— without deleting it, so that it does not block the team while it is investigated. It is done with a marker and a filter. The test is marked:

@pytest.mark.flaky(reruns=2)   # the marker already flags it; the filter isolates it
def test_external_calendar_sync():
    ...

And the pipeline's main run excludes it with -m "not flaky":

python -m pytest -m "not flaky" -v flaky_demo/test_flaky_marked.py

What to expect (real):

collected 2 items / 1 deselected / 1 selected

flaky_demo/test_flaky_marked.py::test_core_pricing_stays_green PASSED      [100%]

======================= 1 passed, 1 deselected in 0.01s ========================

1 deselected: the flaky was left out of the main run —it cannot break the build, neither with its flakiness nor by retrying itself—, and test_core_pricing_stays_green runs normally. The quarantine turns "this flaky blocks the whole team every time it howls" into "this flaky is isolated, does not block the merge, and there is a ticket to fix it." It is usually combined with a second run —non-blocking— that does include the flaky (-m "flaky"), to keep watching them without their stopping the main branch: if a quarantined flaky starts failing always (not intermittently), that is a sign of a real bug that the quarantine helped to isolate, not to hide.

The quarantine is a patch, like the retry —the test is still flaky—, but an honest and visible patch: the deselected in the log screams "there are tests outside the main gate," and the marker in the code names them. What is dangerous is not putting a flaky in quarantine; it is leaving it there forever, forgotten, until the "temporary quarantine" is a graveyard of tests that nobody fixes.

The triggers: why a test is flaky (and why only in CI)

Containing a flaky is this lesson; diagnosing it in depth is the sibling guide. But it is worth recognizing the classic triggers, because many explain the most disconcerting pattern —the flaky that only fails in CI, never on your machine—:

  • The execution order. A test that leaves state —a file, a global variable, a row in a test DB— and another that assumes that state (or its absence). In series, they always run in the same order and the coupling is not seen; with -n auto (lesson 5), xdist distributes them into different workers and the order changes, uncovering the failure. This is the flaky that the pipeline's parallelism creates.
  • Shared resources. Two tests that use the same port, the same temporary file, the same table. On your machine, one alone; in parallel CI, they collide.
  • Time and races. A test that assumes something takes "little" (a short sleep, a tight timeout). Your laptop is fast; the CI runner, loaded and shared, is slower, so the timeout that was ample on your machine falls short there. The flaky that only the slow CI uncovers.
  • Randomness and dates. A test that uses random without a seed, or real datetime.now(), or the order of a dict/set that it assumes is stable. It passes almost always, fails in 1% of the runs.

Reservo is pure and deterministic logic —that is why its suite is not flaky—, but as soon as a project touches the world (network, disk, time, concurrency), the flaky appear, and CI —parallel, slow, clean on each run— is where they are uncovered most. The practical lesson: when a test fails only in CI, suspect first the order (did the parallelism introduce it?), the resources, and the time, before the business code.

The honest policy: patch, ticket, root fix

Bring the pieces together into a policy, because the capstone is evaluated by the method. Facing a flaky:

  1. Contain the bleeding (temporary patch). Per-test retry (@pytest.mark.flaky) or quarantine (-m "not flaky"), so that the flaky stops blocking the team today. Never a global --reruns that anesthetizes the whole suite.
  2. Open a ticket. The patch is temporary only if there is a commitment to fix it. Without a ticket, the patch is permanent and the quarantine is a graveyard. The flaky marker in the code must point to an issue.
  3. Fix the sensor (root). Make the test deterministic: isolate its state (fixtures that clean), fix its resources (unique ports, per-test temporary files), control the time (inject the clock instead of real now()), seed the randomness (random.seed). The how of that diagnosis is the sibling guide; the that it has to be done is this policy.
  4. Remove the patch. When the test is deterministic, take it out of quarantine and remove the reruns. A fixed flaky becomes a normal test again with its honest red.

What this policy is not: "put --reruns 3 on it and forget about it." That is lowering the alarm's volume forever. The difference between a team that uses retry as a patch-with-ticket and one that uses it as a permanent silencer is the difference between a pipeline in which the red means something and one in which nobody looks at the reds. The retry is an aspirin: it relieves the symptom so you can function while you cure the disease, not a cure.

Common mistakes

Using global --reruns as a permanent policy. What happens: a team, fed up with flaky, puts --reruns 3 global in the pipeline and leaves it forever. Now all the tests are retried, so not only the flaky are hidden —the real bugs are hidden too: a test that starts failing because of a genuine bug is retried and, if it passes one out of three times, sneaks in green. Why it happens: it is one line, it fixes the annoying red immediately, and it feels like a solution. How to detect it: if your pipeline retries everything and you have gone months without fixing a single flaky at the root, the retry became a silencer. How to fix it: retry per test (never global), with a ticket, as a temporary patch; and a discipline of fixing the root that empties the quarantine instead of filling it.

Letting the quarantine become a graveyard. What happens: the flaky are marked and excluded with -m "not flaky", but nobody ever fixes them, so the quarantine grows —ten, twenty tests outside the main gate— until a real portion of the suite no longer protects anything. Why it happens: the quarantine removes the immediate pain (the flaky no longer blocks), and without pain there is no urgency to fix. How to detect it: count the tests in quarantine and look at their tickets; if there are many and the tickets have gone months untouched, it is a graveyard. How to fix it: each test in quarantine needs a ticket with an owner and a date; review the list periodically; and treat a quarantined flaky that starts failing always as the real bug that the quarantine helped to isolate. The quarantine is a waiting room, not a tomb.

Blaming the business code when the flaky is infrastructure. What happens: a test starts failing intermittently only in CI, and the team reviews the business logic over and over —the price calculation, the refund rule— without finding anything, because the code is fine. The flaky is because of the order that the parallelism changed, or a shared resource, or the slow runner. Why it happens: "the test fails" makes you think first of what the test tests, not of the conditions in which it runs. How to detect it: if the failure appears only in CI (not on your machine), or only with -n auto (not in series), or only sometimes, the trigger is almost surely infrastructure —order, resources, time—, not business. How to fix it: reproduce the CI conditions (run in series vs. parallel, with -p no:randomly or forcing an order, in a clean environment) to isolate the trigger; the in-depth diagnosis is the sibling guide. Start with how the test runs, not with what it tests.

Exercises

Exercise 1 — Global vs. per test. A colleague fixes a flaky by putting --reruns 2 global in the pipeline's pytest. Explain what risk this introduces for the healthy tests of the suite, and rewrite the solution so that only the flaky test is retried.

See solution

The risk: a global --reruns 2 retries all the tests, not just the flaky. That means that a healthy test —deterministic, that today gives an honest red when something breaks— would also be retried if it started failing because of a real bug. If that bug is intermittent (a new race, say) or if the retry passes by chance, the global retry would hide it, giving you a green over a genuine bug. You lowered the volume of the whole house to silence one alarm: now no alarm warns. The global retry turns every test of the suite into potentially flaky-tolerant, eroding the signal of all of them.

The disciplined solution: retry only the test you know is flaky, with the per-test marker, leaving the rest with their red on the first try:

import pytest

@pytest.mark.flaky(reruns=2)
def test_external_calendar_sync():
    ...  # the only one that retries

def test_core_pricing_stays_green():
    ...  # no marker: fails on the first try if it breaks, as it should

And in the pipeline, python -m pytest without global --reruns —the marker does the retry only where it was placed—. Thus, the flaky is contained without anesthetizing the suite: test_external_calendar_sync is retried, test_core_pricing_stays_green keeps its honest red. The rule: the retry is applied to the test that needs it, never to the whole suite.

Exercise 2 — The flaky the parallelism created. In lesson 5 you added -n auto. A week later, a test that had been green for months starts failing in CI ~1 in 5 runs, always with a FileNotFoundError about a temporary file. In series (pytest without -n) it never fails. Explain which trigger is most likely and what you would do, distinguishing what is of this guide from what is of the sibling guide.

See solution

The most likely trigger is a shared resource uncovered by the change of order that the parallelism introduced. The strong hint is "in series it never fails, with -n auto it does, and intermittently": that points to two tests using the same temporary file (same name/path), and in series they ran in an order in which one created it before the other read it, so the coupling was never seen. With -n auto, xdist distributes them into different workers that run at once, so sometimes the test that reads the file executes before (or in parallel with) the one that creates it, and the FileNotFoundError jumps. The parallelism did not cause the bug —the coupling (two tests sharing a file with a fixed name) already existed—; it revealed it by breaking the order that hid it. It is exactly the circle that lesson 5 seeded: you accelerated, and the acceleration uncovered a flaky.

What I would do, separating the two guides: of this guide (containment + policy) — contain the flaky so it does not block the team today: quarantine (-m "not flaky") or per-test retry, with a ticket opened immediately; never global --reruns. Of the sibling guide (test-failure-diagnosis, the root) — hunt and fix the cause: give each test its own temporary file (a pytest tmp_path, unique per test, instead of a fixed shared name), so that no test depends on another's file. Once the test is deterministic under -n auto, it is taken out of quarantine and becomes normal again. The boundary: here I decide what to do with the flaky (contain, ticket, root fix as a commitment); the how of the diagnosis —reproduce, isolate the resource, correct the fixture— is the sibling guide.

Exercise 3 — Write Reservo's flaky policy. Reservo today is deterministic and has no flaky, but the team wants a written policy for when they appear (they are going to integrate an external payments API that sometimes takes a while). Write the policy in four or five lines, as it would go in the project's README.

See solution

A reasonable policy for the README:

Flaky test policy. A test that fails intermittently without code changes is treated like this: (1) Contain — it is marked with @pytest.mark.flaky(reruns=2) to retry it per test (never global --reruns), or put in quarantine with @pytest.mark.flaky + the main run with -m "not flaky" if it is very annoying; the main run decides the merge, a second non-blocking run (-m "flaky") keeps watching them. (2) Ticket — every marked flaky opens an issue with an owner and a date; without a ticket, the patch is not accepted. (3) Root — it is fixed by making the test deterministic (fixtures that isolate state, unique per-test resources, injected clock, fixed seed); the diagnosis follows the failure-diagnosis guide. (4) Clean up — once the test is fixed, the reruns is removed and it comes out of quarantine. A quarantined flaky that starts failing always is treated as a real bug. Never is global retry used as a permanent solution: that hides bugs, it does not fix flaky.

What makes this policy good: it recognizes the retry as a temporary patch with a ticket, not as a cure; it explicitly forbids the global --reruns (the most common mistake); it distinguishes containing (this guide) from diagnosing the root (the sibling guide); and it closes the cycle with "clean up," so that the quarantine does not become a graveyard. It is the difference between a team whose red means something and one that retries until the pipeline is theater. For the concrete case the team mentions —the payments API that sometimes takes a while—, the per-test retry is the correct starting patch, while the root is fixed (probably isolating the external call with a test double, so that the suite does not depend on the real latency of a third party).

Summary and next step

In this lesson you stacked the sixth and last layer: the flaky policy, the answer to the test that sometimes passes and sometimes fails and that erodes trust in the whole pipeline like an alarm that howls without danger. You saw the retry debate really executed —without retries (1 failed), with --reruns 2 (RERUN→PASSED, 2 passed, 1 rerun)— and understood its trap: the retry does not fix the flaky, it hides it, and hiding it can hide a real bug. You learned the retry with discipline (@pytest.mark.flaky per test, never global) and the quarantine (-m "not flaky", 1 deselected) to isolate a flaky from the main gate without deleting it.

You recognized the triggers —order (which the parallelism of lesson 5 changes), shared resources, time, randomness— and why they explain the flaky that only fails in CI. And you assembled the honest policy: temporary patch (per-test retry or quarantine) + ticket + root fix + clean up, with the clear boundary that containing is this guide and diagnosing the root is the sibling guide test-failure-diagnosis. The underlying lesson: the retry is an aspirin, not a cure, and a pipeline whose reds sometimes lie is a pipeline in which nobody looks at the reds.

Before moving on you should be able to: configure per-test retry and quarantine; explain why the global retry is dangerous; recognize the triggers of a flaky (above all the order that the parallelism uncovers); and write a policy that contains the flaky without letting the trust erode.

What follows, in lesson 8, is the project: you bring together the six layers —base, reproducibility, matrix, speed, gate, flaky— into a single tests.yml, you set up the local parity, and you are evaluated by the method of your decisions. And that lesson closes the guide, with the review of the eight modules and the path toward the ecosystem's sibling guides. The rehearsal of each section is over; the concert has arrived.

Resources

  • pytest-rerunfailures — the retry plugin: --reruns, --reruns-delay, the per-test @pytest.mark.flaky(reruns=N) marker. The reference for what you executed, with the warning (in its own README) to use it with judgment.
  • Working with custom markers — pytest documentation — how to declare and filter markers (-m "not flaky"), the mechanism of the quarantine. It explains the marker registration in pyproject.toml that avoids the unknown-marker warning.
  • How to manage flaky tests — Google Testing Blog — how Google treats flaky at scale: quarantine, flakiness-rate measurement, and why the retry without a root fix does not scale. The backing for the "patch + ticket + root" policy.
  • Sibling guide test-failure-diagnosis — where it is diagnosed why a test is non-deterministic (reproduce, isolate the trigger, correct the fixture). This lesson taught you to contain the flaky; there you learn to hunt its cause. The boundary between the two guides, made explicit.
← Previous6. The coverage gateNext →8. Project: a CI pipeline for Reservo