Module 7: Flaky Tests In Ci

6. Reproducing a CI-only failure on your machine

Next →

Description

Lesson 5 gave you the diagnosis: a failure that only happens in CI is a real bug that needs a runner condition —a different order, parallelism, a different TZ, an absent file— to manifest. But a diagnosis without reproduction is a hypothesis. This lesson teaches you to turn the hypothesis into certainty: reproduce the CI-only on your machine, blowing yourself the wind that the runner blows, to see the red appear in your terminal at will. A bug you reproduce is a bug you can fix (lesson 7) and verify; one you only see on the runner is a ghost you chase blindly.

The technique is not new —it is the method of module 3 (reproducing a CI failure locally: read the log, match the layers, run the same command)—, but sharpened for the CI flaky, where the "layer" that has to be matched is not a dependency version but an execution condition: the order, the parallelism, the time zone. You are going to see, executed for real, how forcing CI's order reproduces the red of the shared Calendar, and how the retry —which rescued the clock flaky— does not rescue this order one. We will mark the boundary with precision: here we get as far as reproducing the CI-only in the CI context; the in-depth diagnosis of the flaky (isolating the minimal pair, freezing the clock, hunting the line) is the sibling guide.

Connection with the module: lesson 5 classified the causes of the CI-only; this one reproduces them. It is the bridge between "I know what wind blows" and "I can fix it" (lesson 7): without reproducing, you cannot verify that your fix worked —you would fix blindly and hope CI proves you right, a very slow cycle—. Reproducing locally closes that cycle in seconds. And it connects backward with module 3, whose method we reuse, and sideways with test-failure-diagnosis-guide, where the deep diagnosis continues.

The mechanic who asks "make it happen again"

When you take the car to the mechanic for a strange noise, the first thing a good mechanic does is not open the engine: it is to ask you to reproduce the noise. "Does it sound when braking? When turning left? Cold or hot?". They get in the car with you and recreate the conditions —brake, turn, accelerate— until the noise appears there, with them listening. Only then do they start to diagnose, because now they have the problem in hand, repeatable, and can test whether their fix silences it.

A mechanic who cannot reproduce the noise is lost: they can change parts at random, hand you the car, and have the noise persist —because they never confirmed they were touching the right part nor that their change worked—. Reproduction is not an optional step before the fix; it is what makes possible a verifiable fix. Without it, "fixing" is guessing.

Reproducing a CI-only is asking your machine to "make the noise happen again." The noise is the runner's red. The conditions —brake, turn— are the winds of the catalog: the order, the TZ, the parallelism. You recreate those conditions in your terminal until the red appears with you watching, repeatable. And then —only then— you fix, and verify that your fix silences it by running the same reproduction and seeing it go green. Without reproducing, you would change code at random and wait for the runner to prove you right a push later: the slowest and most frustrating cycle in engineering.

Reproducing a CI-only is recreating on your machine the runner's condition —the order, the TZ, the parallelism— until the red appears at will. It is what turns a ghost you only see on the runner into a bug you can fix and verify in seconds. Without reproduction, fixing is guessing.

The method, sharpened for the CI flaky

The method of module 3 transfers almost as-is, with step 0 (read the log) identical and the "match layers" steps reinterpreted as "match execution conditions."

Step 0 — read the CI log and extract the condition. The log tells you which wind blew. Look for: the order in which the tests ran (if the runner prints the list, or if it uses pytest-randomly, the seed: Using --randomly-seed=1234), whether it ran in parallel (-n auto/-n 4 in the command), the time zone of the runner (almost always UTC; the header or the env says it), and the file or variable the error mentions (FileNotFoundError: datos/x.csv). That is your "does it sound when braking or when turning?".

Step 1 — match the condition on your machine. According to what the log revealed, you blow that wind locally. The tools, one per cause:

  • Order: name the tests in CI's order (pytest a::t2 a::t1), or install pytest-randomly and reuse the seed from the log (pytest -p randomly --randomly-seed=1234), or disable randomization to fix an order with -p no:randomly.
  • Parallelism: run with pytest-xdist (pytest -n 2) to recreate the runner's simultaneous execution.
  • Time zone/locale: prepend TZ=UTC to the command (TZ=UTC pytest ...), as in module 3.
  • Absent file: move or temporarily rename the local file you suspect (mv datos/fixture.csv /tmp/), so your machine is as "bare" as the runner.

Step 2 — run and observe. You run with the matched condition and look to see whether the red appears. If it appears, you reproduced: you have the bug in hand. If not, the condition you matched was not the right wind —go back to step 0 and try another, one at a time (the golden rule of module 3: change one condition at a time, to know which one it was)—.

Step 3 — the handoff. With the CI-only reproduced, the module ends its part: you know what condition triggers it and you have it repeatable. From here, two paths: if the cause is clear (the module-level shared), you go straight to fixing determinism (lesson 7); if you need to hunt why exactly the code produces the wrong value —isolate the minimal pair of tests, put a breakpoint(), reduce the case—, that is the in-depth diagnosis of test-failure-diagnosis-guide.

Worked example 1: reproduce the order flaky by forcing CI's order

Let us apply the method to the CI-only of the shared Calendar of lesson 5. Suppose the CI log showed that test_b_book_focus ran before test_a_focus_free_at_nine (or a randomization seed that produces that order), and that test_a failed with assert False is True.

Step 0 — condition: the reverse of the definition order (test_b before test_a).

Step 1 — match the order: we name the two tests in CI's order, explicitly.

Step 2 — run and observe. With Python 3.14.0 and pytest 9.1.1, measured by executing:

python -m pytest \
  "demo_ci_only/test_shared_calendar.py::test_b_book_focus" \
  "demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine" -v

What to expect.

collecting ... collected 2 items

demo_ci_only/test_shared_calendar.py::test_b_book_focus PASSED           [ 50%]
demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine FAILED   [100%]

=================================== FAILURES ===================================
__________________________ test_a_focus_free_at_nine ___________________________

    def test_a_focus_free_at_nine():
        # Assumes an EMPTY calendar. True only if it runs BEFORE test_b.
>       assert is_available(shared, "r1", start, start + timedelta(hours=1)) is True
E       AssertionError: assert False is True

demo_ci_only/test_shared_calendar.py:16: AssertionError
=========================== short test summary info ============================
FAILED demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine - Asse...
========================= 1 failed, 1 passed in 0.01s ==========================

Reproduced. The runner's red appeared in your terminal, at will, just by matching the order. Now you know —you do not believe— that the cause is the order with shared state: you changed that single condition and the color moved, whereas in your usual order (lesson 5) it went green. You have the "noise" sounding with you listening. The mechanic can start working.

A useful shortcut for step 1 when you do not know CI's exact order: install pytest-randomly (pip install pytest-randomly) and run the suite several times —it randomizes the order on each run, so sooner or later it produces the bad order and reproduces the failure—. When it reproduces it, the plugin prints the seed (Using --randomly-seed=NNNN); save it and reuse it (--randomly-seed=NNNN) to reproduce that exact order at will. It is blowing all the order winds until you hit the one that makes the house creak, and then fixing it.

Worked example 2: the retry does NOT reproduce nor rescue the order flaky

In lesson 5 we noted that the retry does not save an order flaky. Let us verify it by executing, because it is a key piece of the diagnosis. We run the same reproduction —CI's order— but now with --reruns 3, hoping (mistakenly) that retrying rescues test_a:

python -m pytest --reruns 3 \
  "demo_ci_only/test_shared_calendar.py::test_b_book_focus" \
  "demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine" -v

What to expect.

collecting ... collected 2 items

demo_ci_only/test_shared_calendar.py::test_b_book_focus PASSED           [ 50%]
demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine RERUN    [100%]
demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine RERUN    [100%]
demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine RERUN    [100%]
demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine FAILED   [100%]

=================================== FAILURES ===================================
__________________________ test_a_focus_free_at_nine ___________________________
...
E       AssertionError: assert False is True
demo_ci_only/test_shared_calendar.py:16: AssertionError
=========================== short test summary info ============================
FAILED demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine - Asse...
===================== 1 failed, 1 passed, 3 rerun in 0.02s =====================

Look at the three RERUN followed by FAILED: the retry retried test_a three times and all three it failed again. Summary: 1 failed, 1 passed, 3 rerun. Why? Because test_b already booked Focus in the shared, and that state does not revert between retries —each rerun of test_a runs with the calendar just as occupied, so it finds is_available at False over and over—. Unlike the clock flaky (where each retry re-threw the die of chance and sometimes came out green), here the result is deterministic given the state, and the state is the same in the four attempts.

This output is doubly valuable. First, it confirms the rule: the retry rescues chance flaky, not state/order. Second, it is a diagnostic tool: if you put --reruns on a flaky and it does not help —it fails identically in all the retries—, you have just learned that its cause is not chance but state or order. A flaky the retry does not calm is telling you "my problem is structural, fix my determinism (lesson 7), do not retry me."

The boundary: how far this module goes

It is worth being precise about where this lesson ends, because reproducing is not diagnosing-in-depth nor fixing.

This module takes you as far as having the CI-only reproduced on your machine —the "noise sounding with you listening"—, with the runner's condition identified (here, the order with shared state) and repeatable at will. That is the handoff point, and it forks:

  • If the cause is evident from the reproduction —as here, where the module-level shared jumps out— you go straight to fixing determinism (lesson 7): a fixture that gives a fresh Calendar per test, and the order stops mattering.
  • If you need to understand why exactly —isolate the minimal pair of tests that interacts, put a debugger, reduce the case to the essential, freeze variables one by one— that trade is the sibling guide test-failure-diagnosis-guide. Here we do not enter; our job was to give you the reproducible CI-only that that guide needs as a starting point, in the CI context.

The boundary is the same as module 3's, applied to the flaky: reproducing closes the gap between your machine and the runner; what to do with the reproduced failure forks between "the cause is clear, fix it" (lesson 7) and "it has to be diagnosed in depth" (sibling guide). Without the reproduction, neither of the two branches is possible —that is why reproducing is the step that unblocks everything else—.

Common mistakes

Trying to fix without reproducing first. What happens: a CI-only comes out, the developer believes they know the cause, changes code, pushes, and waits for CI to prove them right —a ten-minute cycle per attempt, blindly—. Why it happens: reproducing locally seems like a detour when "I already know what it is." How to detect it: if your way to verify a CI-only fix is to push and wait for the runner, you are not reproducing, you are guessing with a very slow cycle. How to fix it: reproduce locally first (seconds per attempt), fix, verify locally that it now passes, and only then push. The mechanic does not hand you the car hoping the noise is gone; they silence it with them listening.

Matching several conditions at once. What happens: to reproduce fast, someone forces the order, the TZ, and the parallelism all at once, reproduces the red, and does not know which of the three was the cause. Why it happens: the rush to see the red. How to detect it: if you reproduced by changing three things, you know that one was it, not which one. How to fix it: the golden rule of module 3 —change one condition at a time—. Match the order, run; if it does not reproduce, revert and try the TZ; and so on. Each run gives you clean information about one condition, instead of an ambiguous datum that you will have to untangle to fix the right cause.

Confusing "I did not reproduce" with "it is irreproducible." What happens: someone forces the order, it stays green, and concludes "it's a CI thing, it can't be reproduced." Why it happens: it is assumed that the order was the only possible cause. How to detect it: if you gave up after trying a single condition from the catalog, you did not exhaust it. How to fix it: run through the five winds —order, parallelism (-n 2), TZ, absent file, and if it applies, resources— one at a time. Almost every CI-only is reproducible; "it can't be" almost always means "I have not yet blown the right wind." Step 0 —reading the log well— is what tells you which wind to try first.

Exercises

Exercise 1 — Choose the reproduction tool. For each diagnosed CI-only, write the command (or the action) with which you would reproduce it on your machine. (a) The log shows that CI ran with pytest-randomly and Using --randomly-seed=4242, and a test failed. (b) The log shows -n auto and two tests that write to the same file fail intermittently. (c) The error is assert 15 == 21, a difference of 6 hours, and the runner runs in UTC.

See solution
  • (a) Reuse the exact seed from the log to fix that order: pytest -p randomly --randomly-seed=4242. The seed reproduces the exact order that produced the red; it is the most faithful way to match the order condition when CI randomizes.
  • (b) Recreate the parallelism: pytest -n 2 (or -n auto). Running in two processes makes the two tests execute at once and step on the file, just as on the runner. You may have to run it several times, because the race condition is probabilistic.
  • (c) Force the runner's zone: TZ=UTC pytest ... (the pattern of module 3). The difference of exactly 6 hours points to your UTC-6 against the runner's UTC; prepending TZ=UTC matches that layer.

The discipline: each wind has its reproduction tool —--randomly-seed for order, -n for parallelism, TZ= for zone—. Step 0 (reading the log) is what tells you which to use; applying it is matching the condition.

Exercise 2 — Interpret the retry that does not help. You run a flaky with --reruns 3 and see RERUN, RERUN, RERUN, FAILED —it fails in the four attempts, identical—. Another flaky, with the same --reruns 3, you see RERUN, PASSED —it passes on the second attempt—. What does each pattern tell you about the cause of each flaky, and what module tool does each one call for?

See solution
  • RERUN, RERUN, RERUN, FAILED (fails identically in the four): the cause is structural —shared state or order—, not chance. The retry does not change the result because the state that causes the failure does not revert between attempts (like the already-booked Calendar). It calls for: reproducing the condition (this lesson, forcing the order/parallelism) and fixing determinism (lesson 7, isolating the state with a fixture). The retry is useless here.
  • RERUN, PASSED (passes on retry): the cause is chance —the clock, the network, something that re-rolls on each attempt—. The retry sometimes lands well. It calls for: as triage, the retry or the per-retry quarantine (@pytest.mark.flaky, lesson 4) can unblock; as a cure, fixing the source of chance (lesson 7, injecting the clock). The retry at least works as a patch, even if it does not cure.

The diagnostic lesson: the behavior of the test under --reruns reveals its cause. If the retry does not help, it is structural (state/order); if it helps, it is chance. That distinction decides which module tool to apply.

Exercise 3 — The slow cycle vs. the fast one. A colleague fixes a CI-only like this: changes code, pushes, waits 8 minutes for CI to run, sees it is still red, changes something else, pushes, waits 8 minutes... They are on their fourth attempt and one hour in. Describe the cycle the lesson proposes instead and why it is faster, even though "reproducing locally" seems like an extra step.

See solution

The cycle the lesson proposes: (1) reproduce the CI-only locally by forcing the runner's condition —once, maybe a couple of minutes of reading the log and trying the order/TZ/-n—; (2) with the red repeatable in your terminal, fix and run the reproduction again —seconds per attempt—; (3) when the local reproduction goes green, push once with confidence.

Why it is faster even though it seems like an extra step: the colleague pays 8 minutes per attempt and makes several attempts blindly —their feedback cycle is the runner, the slowest possible—. The cycle of the lesson pays the reproduction only once and then iterates against the local terminal, whose feedback is seconds. Four blind attempts = one hour; four local attempts = a couple of minutes plus the reproduction setup. "Reproducing locally" is not an extra step: it is what replaces the 8-minute cycle with a seconds-long one. The mechanic who reproduces the noise with them listening fixes it in one visit; the one who changes parts and asks you to "come back if it still sounds" makes you come back four times.

Summary and next step

In this lesson you turned the diagnosis of the CI-only into reproduction: blowing yourself the runner's wind —the order, the TZ, the parallelism— until the red appears in your terminal at will, like the mechanic who recreates the noise before fixing. You reused the method of module 3, sharpened for execution conditions: read the log (step 0), match the condition with the right tool (--randomly-seed for order, -n for parallelism, TZ= for zone, move the absent file), run and observe, one condition at a time.

You saw it executed: forcing CI's order reproduced the red of the shared Calendar (1 failed, 1 passed), and --reruns 3 did not rescue it —three RERUN and FAILED, because the state does not revert between retries—, confirming that the retry saves chance flaky, not order, and serving as a diagnostic tool. And you marked the boundary: here you get as far as reproducing; the in-depth diagnosis is the sibling guide, and the cure is lesson 7.

Before moving on you should be able to: reproduce a CI-only by forcing the runner's condition with the appropriate tool; interpret the behavior under --reruns as a hint of the cause (chance vs. structure); explain why the local reproduction cycle is faster than iterating against the runner; and say where this module ends and where the diagnosis guide begins.

What follows, in lesson 7, is the real cure. Retry and quarantine unblocked you; reproducing gave you the bug in hand. None of that fixed anything. Lesson 7 fixes determinism at its root: injecting the clock (the now= seam of should_audit) to kill the clock flaky, and a fixture that gives a fresh Calendar per test to kill the order flaky —both executed, both stable run after run—. It is where the flaky stops being flaky.

Resources

  • pytest-randomly — PyPI — the plugin that randomizes the order of the tests and, crucially, prints and accepts a seed (--randomly-seed) to reproduce an exact order. The central tool to reproduce an order flaky at will.
  • pytest-xdist — documentation — how to run in parallel with -n to reproduce locally the runner's parallelism and expose race conditions over shared resources.
  • Selecting tests by node — pytest documentation — how to name tests with file::test and in what order, the technique of worked example 1 to force CI's order without plugins.
  • Reproducing a CI failure locally — module 3 of this guide — the base method this lesson reuses (read the log, match the layers one at a time, run the same command). The in-depth diagnosis of the flaky continues in the sibling guide test-failure-diagnosis-guide.
← Previous5. The failure that only happens in CINext →7. Fix determinism, do not retry