Module 7: Flaky Tests In Ci

3. The retry debate: `--reruns`

Next →

Description

When a flaky blocks everyone's CI (lesson 2), the team reaches almost by instinct for a tool that promises immediate relief: the automatic retry. The idea is tempting in its simplicity: if the test fails, let pytest run it again on its own, two or three times; if in any of those attempts it passes, count it as green and unblock the PR. Nobody has to hit "re-run" by hand, the pipeline heals itself, and life goes on. This lesson installs that tool —pytest-rerunfailures with the --reruns N flag— and runs it for real on the should_audit flaky, so you see with your own eyes what it does, and then it gives you the honest debate: when the retry is a legitimate painkiller and when it is, simply, an automatic way to lie to yourself in green.

You are going to see the real output with the RERUN marker: a run where --reruns 3 retries the flaky and ends 1 passed, 1 rerun —the retry rescued it— and another where not even three retries suffice and it ends 1 failed, 3 rerun —the retry lost—. Those two outputs, measured by executing, are the debate in the flesh: the retry sometimes turns a red into green without the code changing, and that is precisely its virtue and its poison at the same time.

Connection with the module: lesson 2 established that re-running blindly is toxic. This lesson does not contradict it: it refines it. The automated retry is re-running, but with a difference —it is explicit, bounded, and visible in the log (RERUN)—, and that difference makes it tolerable as an emergency triage without making it acceptable as a cure. Lesson 4 offers the more disciplined alternative (quarantine, which isolates instead of retrying the whole suite), and lesson 7 gives the only real cure (fixing determinism). The retry is the first link of that chain, and it has to be understood well —including its danger— so as not to stay stuck in it.

The "retry" button of the ATM

Think of an ATM that sometimes, due to a network hiccup, rejects your withdrawal with an "operation failed, try again." You hit "retry," and the second time it works. The retry button is useful and honest when the failure is genuinely transient —a network packet that got lost, a momentary latency—: retrying does not hide any real problem, it just gets around a passing stumble that will not matter again.

But imagine another ATM that rejects your withdrawal because you really do not have funds. You hit "retry" and... sometimes it works, because the balance the ATM reads is out of sync and on the second attempt it reads an old number that is enough. Retrying "solved" the problem —it gave you the money—, but it solved nothing: the real problem (your balance, the desync) is still there, and now it is hidden behind a successful retry. The day the desync plays the other way, or the bank reconciles the accounts, the problem reappears, worse and later.

The test retry is exactly that button. Retrying a flaky test for a genuinely transient cause —a network that blipped in an integration test— is like the first ATM: legitimate, it gets around a passing stumble. Retrying a test that fails because of a real intermittent bug —an order problem, badly handled state, a race condition— is like the second ATM: the green you get hides a problem that is still alive and will reappear worse. And the retry, on its own, cannot distinguish one case from the other: it retries the same, and hands you the green the same. Distinguishing them is your job, not its.

The retry (--reruns) reruns a failing test and counts it as green if any attempt passes. It is legitimate for a genuinely transient failure; it is dangerous for a real intermittent bug, because it turns it into a lying green. The tool does not distinguish which is which —you do have to do that—.

Install and run the retry, for real

pytest-rerunfailures is a pytest plugin that adds the ability to retry failed tests. It is installed with pip in the same venv where you run pytest:

pip install pytest-rerunfailures

Once installed, pytest gains the --reruns N flag: "if a test fails, run it again up to N more times; if any passes, count it as passed." You do not have to touch the test code for the global mode; the flag applies to the whole suite. Let us verify it got installed, because the plugin appears in the pytest header —an honest way to confirm it is active—:

python -m pytest --version

On the machine where I write this, with the plugin installed, the header of any run now includes the line plugins: rerunfailures-16.4. That plugins: is your confirmation that the retry is available; if it does not appear, the --reruns flag would give an "unknown option" error.

Worked example 1: --reruns 3 rescues the flaky

Let us run only the flaky, with --reruns 3, in verbose mode (-v) to see each attempt on its own line. The flag says "if test_new_booking_is_audited fails, retry up to 3 more times." With Python 3.14.0, pytest 9.1.1, and rerunfailures 16.4, this is a real run where the first attempt landed on an odd microsecond (failed) and the retry landed on an even one (passed):

python -m pytest tests/test_audit.py::test_new_booking_is_audited --reruns 3 -v

What to expect.

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- /private/tmp/reservo-m7/.venv/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/reservo-m7
plugins: rerunfailures-16.4
collecting ... collected 1 item

tests/test_audit.py::test_new_booking_is_audited RERUN                   [100%]
tests/test_audit.py::test_new_booking_is_audited PASSED                  [100%]

========================== 1 passed, 1 rerun in 0.01s ==========================

Read it line by line. The first attempt failed, and instead of reporting FAILED, pytest printed RERUN —the plugin's mark: "this failed, I'm going to retry it"—. The second attempt landed on an even microsecond, should_audit() returned True, and the test passed: PASSED. The final summary counts it as 1 passed, 1 rerun: one passed test (for all gate purposes, green) and one retry consumed. The PR is unblocked. Nobody touched the code. The flaky is as flaky as before —only this time the die landed well on the second throw—.

That 1 rerun in the summary is important: it is the visible footprint that there was a retry. Unlike hitting "re-run" by hand (which leaves no trace of the previous attempt in the log), the automated retry leaves a record that the test needed to be retried. That record is what separates a disciplined retry —one you know happened and can count— from a blind re-run that gets lost in oblivion.

Worked example 2: when not even three retries suffice

The retry does not guarantee the green. If the flaky fails in the four attempts (the original plus the three retries), the test is reported as failed. With a flaky that fails ~50% of the time, having all four throws land badly has probability ≈ 6% —little, but it happens—. Here is a real run where it occurred:

python -m pytest tests/test_audit.py::test_new_booking_is_audited --reruns 3 -v

What to expect.

collecting ... collected 1 item

tests/test_audit.py::test_new_booking_is_audited RERUN                   [100%]
tests/test_audit.py::test_new_booking_is_audited RERUN                   [100%]
tests/test_audit.py::test_new_booking_is_audited RERUN                   [100%]
tests/test_audit.py::test_new_booking_is_audited FAILED                  [100%]

=================================== FAILURES ===================================
_________________________ test_new_booking_is_audited __________________________

    def test_new_booking_is_audited():
        # Author's intent: "a new booking gets audited".
        # BUG: should_audit() without an argument reads the real clock, and returns
        # True only when the microsecond is even (~half the time). This test is FLAKY
        # by construction: sometimes it passes, sometimes it fails, without touching the code.
>       assert should_audit() is True
E       assert False is True
E        +  where False = should_audit()

tests/test_audit.py:11: AssertionError
=========================== short test summary info ============================
FAILED tests/test_audit.py::test_new_booking_is_audited - assert False is True
========================== 1 failed, 3 rerun in 0.03s ==========================

Three RERUN in a row —the three retries— and at the end FAILED: all four throws landed on an odd microsecond. The summary: 1 failed, 3 rerun. The retry used up its three chances and lost. The PR is still blocked.

This output teaches something crucial about the retry: it is a bet, not a guarantee. With --reruns 3 on a 50% flaky, you unblock 94% of the time and stay blocked 6%. Going up to --reruns 10 raises the probability of green (99.9%), but notice what you are doing: the more retries you allow, the more aggressively you turn reds into greens, and the more deeply you bury any real problem hiding behind the flaky. The number of reruns is a knob that changes how much you lie in green, not whether you lie.

The debate, with the two outputs in hand

Now that you have seen the retry rescue and lose, the honest debate.

In favor: the retry unblocks the team without manual work. Damage one of lesson 2 —the flaky blocks everyone's PR— is real and urgent. A team with a live flaky and no retry loses hours hitting "re-run" by hand, with the frustration and the toxic reflex that trains. The retry automates that unblock: it is explicit (it is in the YAML, not in everyone's instinct), bounded (N attempts, not infinite), and visible (RERUN in the log, X rerun in the summary). If your flaky is a genuinely transient failure —the integration test whose network blips once in a thousand— retrying is the correct answer, not a patch: there is nothing to fix in the code, just a passing stumble to get around. Like the first ATM.

Against: the retry hides real bugs and perpetuates the flaky. If your flaky is not a transient stumble but a real intermittent bug —a race condition, shared state, a rounding error that depends on the order—, the retry is the second ATM: it gives you the green and hides the problem, which is still alive and will reappear worse. And even when the cause is "harmless" like our clock, the retry perpetuates the flaky: each green-by-retry is one less reason to fix it at its root, so the flaky lives forever, consuming reruns and eroding trust in the background. The retry relieves the acute symptom (today's block) in exchange for making the disease chronic (the flaky that never gets cured).

The verdict of the industry, and of this guide: the retry is a painkiller, not an antibiotic. It is justified as triage —to unblock the team today while you do the real work— and only if it comes with two commitments: (1) a ticket that records the flaky to fix it, and (2) visibility that the retry happened, so that "green" does not erase the debt. A retry without a ticket or visibility is not triage, it is automated denial. And under no circumstances is the retry the goal: the goal is that the test not need to be retried (lesson 7).

How to enable it: CLI and configuration

There are two ways to turn on the retry, and the choice says a lot about your intention.

By CLI, for a one-off run: pytest --reruns 3. Useful when you retry on purpose and consciously —"today I need to unblock, I run with reruns once"—. It does not stick to the suite; the next run without the flag does not retry.

By configuration, for the whole suite, always: in pytest.ini (or pyproject.toml), with addopts:

# pytest.ini
[pytest]
addopts = --reruns 2

With that, every pytest run retries up to 2 times, without writing the flag. I verified it by executing: with that pytest.ini, a normal run of the flaky prints configfile: pytest.ini in the header and retries on its own, ending —in a real run— 1 passed, 1 rerun. It is convenient... and dangerous. A global --reruns hidden in the config retries everything, including the tests that should fail honestly. It masks new flaky before you notice them, and it makes the retry invisible to whoever does not read the config. If you are going to use global retry, let it be a documented team decision, with low reruns (1–2), and never as a substitute for fixing. Preferable is the per-test retry (the quarantine of lesson 4), which retries only the marked flaky and lets the rest of the suite fail honestly.

Common mistakes

Raising the reruns until the flaky "disappears." What happens: --reruns 3 does not always give green, so someone raises it to --reruns 10 "to be sure." Why it happens: more retries = more green, and green gets confused with healthy. How to detect it: if you are raising the number of reruns to cover a flaky, you are measuring how much you want to lie, not fixing anything. How to fix it: the number of reruns should never grow to accommodate a flaky; if --reruns 2 is not enough as triage, the flaky is serious enough to put it in quarantine (lesson 4) and fix it (lesson 7), not to retry it harder.

Global retry in addopts without the team knowing. What happens: someone puts --reruns 2 in pytest.ini to calm a flaky, and now the whole suite retries in silence, including real bugs that were failing honestly. Why it happens: it is a one-line fix that "makes the problem disappear" for everyone. How to detect it: review addopts; if there is a --reruns that nobody remembers having discussed, it is invisible retry. How to fix it: global retry is a team decision, documented and with minimal reruns; for a specific flaky, prefer the per-test retry (@pytest.mark.flaky, lesson 4), which is visible and bounded to the culprit.

Treating 1 passed, 1 rerun the same as 1 passed. What happens: the summary says 1 passed, 1 rerun and the developer reads only "passed," ignoring the 1 rerun. Why it happens: the green reassures and the rerun is fine print. How to detect it: if your suite reports reruns and nobody looks at them, you are wasting the only honest signal the retry gives you. How to fix it: the X rerun is information, not decoration —each rerun is a flaky that needed a push—. Count the reruns, record which test caused them, and treat them as pending debt. A 1 passed, 1 rerun means "it passed, but I had to retry": it is not the same as "it passed clean."

Exercises

Exercise 1 — Predict the summary. You run pytest tests/test_audit.py::test_new_booking_is_audited --reruns 3. The flaky fails ~50% of the time, independently on each attempt. (a) What summary do you see if the first attempt passes? (b) And if the first fails and the second passes? (c) And if the four attempts fail? Write the summary of each case.

See solution
  • (a) First attempt passes: there is no retry —the retry only acts on failures—. Summary: 1 passed (without any rerun). The --reruns 3 was available but not used.
  • (b) The first fails, the second passes: one retry, successful. Summary: 1 passed, 1 rerun (with a RERUN line and then PASSED in verbose mode). It is the run of worked example 1.
  • (c) The four fail: the original plus the three retries, all red. Summary: 1 failed, 3 rerun (three RERUN lines and then FAILED). It is the run of worked example 2.

The key: the X rerun counts retries consumed, not total attempts. A test that passes on the first try has 0 reruns; one that passes on the second attempt has 1 rerun; one that exhausts the three retries and fails has 3 reruns. And passed/failed reflects the final result after all allowed retries.

Exercise 2 — Legitimate retry or retry-lie? For each flaky, say whether retrying it is a legitimate painkiller (triage) or a retry-lie that hides a bug, and why. (a) An integration test that calls an external API and fails 1 in 500 times due to a network timeout. (b) test_new_booking_is_audited, which fails ~50% by reading the clock. (c) A test that fails intermittently because two tests share a temporary file and sometimes one deletes it before the other reads it.

See solution
  • (a) Legitimate painkiller (with nuances). A genuinely transient network timeout, 1/500, is the first ATM: there is no bug in your code, just a network stumble that will not matter again. Retrying is defensible here —although the ideal is to isolate that integration test from the fast gate and not depend on the network in the unit suite—. The retry gets around a real transient failure without hiding anything of yours.
  • (b) Retry-lie (perpetuates the flaky). There is nothing transient: should_audit() looks at the clock by design, and retrying only throws the die again. The green you get does not correspond to anything having been solved; the flaky is identical. It is acceptable triage only with a ticket and a fix plan (lesson 7), never an end: the cause is a non-determinism that has to be removed, not waited out.
  • (c) Retry-lie that hides a real bug. The shared temporary file is a race condition / shared state: a real bug (second ATM). Retrying gives you the green and buries the bug, which will reappear —worse— when the order or the parallelism changes (lessons 5–6). Here the retry is actively harmful: it removes the signal that would have led you to isolate the state (lesson 7).

The rule to distinguish: ask "is the cause of the failure a genuinely passing external stumble, or a source of non-determinism in my code/tests?" The former can be retried; the latter has to be fixed, and the retry only hides it.

Exercise 3 — The retry with conditions. Your team decides to use retry as triage while it fixes the audit flaky. Write the two conditions the lesson requires for that retry to be disciplined and not automated denial, and explain what bad consequence each one prevents.

See solution

The two conditions:

  1. A ticket that records the flaky. It prevents the retry from becoming permanent by forgetfulness: without a ticket, "we'll retry while we fix it" turns into "we retry it forever," because nothing forces a return. The ticket is the explicit commitment that the retry is temporary and that there is pending work (the fix of lesson 7). Without it, the painkiller becomes the diet.

  2. Visibility that the retry happened (the X rerun in the summary, counted and reviewed). It prevents "green" from erasing the debt: if nobody looks at the reruns, the team loses sight of the fact that there is a live flaky, and with that also loses the ability to notice if it worsens or if a new flaky was added. Visibility keeps the flaky on the radar even though the retry unblocks it.

Together they turn the retry from "hiding the problem automatically" into "unblocking today without losing sight of the need to cure." The consequence both prevent, in one sentence: that the triage disguise itself as a solution and the flaky live forever under a comfortable green.

Summary and next step

In this lesson you installed pytest-rerunfailures and ran --reruns 3 for real on the should_audit flaky. You saw the retry rescue it —RERUN followed by PASSED, summary 1 passed, 1 rerun— and lose —three RERUN and FAILED, summary 1 failed, 3 rerun—, and with those two outputs in hand you had the honest debate: the retry unblocks the team without manual work (in favor) but hides real bugs and perpetuates the flaky (against). Like the retry button of the ATM: legitimate before a transient stumble, dangerous before a real problem still alive behind the green.

The verdict: the retry is a painkiller, not an antibiotic. It is justified as triage —with a ticket and visibility of the reruns—, never as a cure or a goal. You learned to enable it by CLI (--reruns 3, conscious and one-off) and by config (addopts, global and dangerous), and why the per-test retry is preferable to the global one.

Before moving on you should be able to: install and run --reruns N; read a summary with X rerun and distinguish 1 passed, 1 rerun from 1 passed; argue when retrying is legitimate and when it is a retry-lie; and name the two conditions (ticket + visibility) that make the retry disciplined.

What follows, in lesson 4, is the more surgical alternative to the global retry: quarantine. Instead of retrying the whole suite blindly, you mark only the flaky —with @pytest.mark.flaky to retry it per test, or with xfail to take it out of the gate while you investigate it— letting the rest of the suite fail honestly. You are going to see both executed, with their real xfailed/xpassed, and the discipline that makes them healthy: always with a ticket, always temporary.

Resources

  • pytest-rerunfailures — GitHub repository — the plugin's canonical documentation: --reruns, --reruns-delay, the @pytest.mark.flaky marker, and the authors' own warnings about not using it to hide bugs. Read it to see the full options that here we only glimpse.
  • pytest-rerunfailures — PyPI — the installation page (pip install pytest-rerunfailures) and the compatibility matrix with pytest versions. Confirms it runs with pytest 9.1.1, the one of this guide.
  • Configuration: addopts — pytest documentation — how to put default flags in pytest.ini/pyproject.toml, including the global --reruns this lesson warns to use carefully. Useful to understand what your suite runs even if you do not write it on the command line.
  • Flaky tests — pytest documentation — return to the section on retries: pytest itself frames the retry as temporary triage, not as a solution, exactly the verdict of this lesson.
← Previous2. Why the flaky is toxic in CINext →4. Quarantine: mark and isolate