5. The failure that only happens in CI

Description

Up to here we worked a flaky that flickers the same everywhere: should_audit looks at the clock, and that clock advances the same on your laptop as on the runner. But there exists a whole family of failures with a different and more disconcerting signature: green on your machine, always; red in CI, sometimes or always. You run the suite in your terminal a hundred times and it passes all hundred; you push it, the runner runs it, and it comes out red. You do not understand: "it works on my machine" —the phrase this whole ecosystem exists to banish—. This lesson explains why CI sees red where you see green, with a catalog of the real causes, and demonstrates the most common of all with two Reservo tests that share a Calendar: they pass in one order and fail in the other.

The key to understanding the CI-only is to stop thinking "the CI is broken" and start asking "what does the runner have that my machine does not?". Because there is always a concrete difference —the runner runs the tests in a different order, or parallelizes them, or has a different time zone, or is missing a file that exists in your folder— and that difference is the one that exposes a non-determinism that your environment, by pure luck, kept hidden. The CI does not invent the bug: it reveals it, because it tests your code under conditions that you do not unintentionally reproduce.

Connection with the module: lessons 1–4 dealt with the flaky visible everywhere (the clock) and how to manage it (retry, quarantine). This lesson opens the second half: the flaky that hides from you and only shows itself in CI. Lesson 6 teaches you to reproduce it on your machine —forcing the runner's condition— leaning on the method of module 3. And lesson 7 cures it by fixing determinism. Here the work is light diagnosis: naming the cause. The boundary with the sibling guide test-failure-diagnosis-guide is sharp —the in-depth diagnosis of a flaky lives there—; here we classify the causes in the CI context to know which condition to reproduce.

The house that only creaks when the north wind blows

A homeowner hears an unsettling creak in the roof. They call an inspector. The inspector comes on a calm day, checks everything, hears nothing: "your house is perfect, sir." He leaves. That night, the creak returns. The owner goes crazy: is he mad?, is the inspector incompetent?

Neither of the two. The creak only happens when the north wind blows, which exerts a force on a badly fixed beam. On the day of the inspection there was no wind, so the condition that triggers the creak was not present, and the house —truly— did not creak. The problem is real and it is in the beam; it just needs a specific environmental condition to manifest, and that condition was not there on the day the inspector measured.

Your machine is the inspector on the windless day. You run the suite under your conditions —your tests in your order, one after another, in your time zone, with all your files present— and it does not creak: green. The CI runner is the night with the north wind: it runs in another order, or in parallel, in UTC, without the files you have loose in your folder. That different condition exerts force on a "badly fixed beam" of your code —a test that assumes an order, a shared state, a time zone— and then it does creak: red. The bug was there the whole time; your environment, by luck, never blew the wind that wakes it.

Understanding this changes your reaction. Instead of "the CI is broken" (blaming the inspector) or "it's a mystery" (giving up), you ask: what wind blows in CI that does not blow on my machine? And since the possible winds are few and known, the question has an answer.

A failure that only happens in CI is not a broken CI nor a mystery: it is a real bug that needs a specific runner condition to manifest —a different order, parallelism, a different time zone, an absent file—, a condition that your machine, by luck, does not reproduce. The CI does not invent the bug; it blows the wind that reveals it.

The catalog of winds: why CI differs from your machine

The "north winds" of CI are a handful, and knowing them turns the hunt for a CI-only into a checklist. These are the five most common.

1. The execution order. Your machine runs the tests in one order (usually the definition order, file by file). CI can run them in another order —because it uses pytest-randomly to vary them on purpose, because it distributes the suite among several processes, or simply because it collects the files in another order—. If two tests share state and one depends on running before the other, your order leaves them green and CI's order turns them red. It is the cause we will demonstrate.

2. The runner's parallelism. If the CI uses pytest-xdist (-n auto, from module 5) to run the suite in several processes at once, tests that on your machine ran in series —one finishes before the other starts— now run simultaneously. Any shared resource —a temporary file, a port, a table, a global variable— that in series gave no problem, in parallel produces a race condition. On your machine, without -n, you never see it.

3. The time zone and the locale. CI runners almost always run in UTC and in the C/en_US.UTF-8 locale. Your machine is in your zone (say America/Mexico_City, UTC-6) and your locale. A test that compares hours, formats dates, or depends on the decimal separator or the order of names, passes in your zone/locale and fails in CI's. (This is precisely the terrain of the pytz of module 3, now seen as a source of environment flaky.)

4. A file or resource that exists on your machine and not in CI. You have a config.local.json, a test datum, an environment variable exported in your .zshrc, a folder with open permissions —things that live on your machine and are not in the repo—. Your test finds them; the runner, which only has what is committed, does not. The test passes for you and fails in CI with a FileNotFoundError or a KeyError that in your terminal is impossible to reproduce... until you delete that local file.

5. Limited resources of the runner. The runner has less CPU, less RAM, slower disks than your laptop. A test with a tight timeout —"this must respond in 100 ms"— passes on your fast machine and fails on the slow runner, which needed 130 ms. The code did not change; what changed is how long it takes, and the test tied its verdict to a time that is only met on fast hardware.

Notice the common thread: in all five, the bug already existed in your code (a fragile test, a shared state, a hidden dependency) and CI only contributes the condition that manifests it. That is why the correct reaction is never "fix the CI" but "identify which runner condition revealed my bug, and reproduce it" (lesson 6) to then fix determinism (lesson 7).

The reigning cause: order and shared state

Of the five, the most common and the most instructive is order with shared state. We are going to build it in Reservo, execute it, and watch it flicker according to the order —the classic CI-only—.

The original sin is a Calendar shared at module level between two tests:

# demo_ci_only/test_shared_calendar.py
from datetime import datetime, timedelta

from reservo.models import Room, Member
from reservo.calendar import Calendar, book, is_available

# TRAP: a Calendar shared at module level. If a test mutates it, the
# next one sees it mutated. The result depends on the execution ORDER.
shared = Calendar()
focus = Room(id="r1", name="Focus", capacity=1, hourly_cents=2500)
ana = Member(id="m1", name="Ana", tier="basic")
start = datetime(2026, 8, 1, 9, 0)


def test_a_focus_free_at_nine():
    # Assumes an EMPTY calendar. True only if it runs BEFORE test_b.
    assert is_available(shared, "r1", start, start + timedelta(hours=1)) is True


def test_b_book_focus():
    b = book(shared, focus, ana, start, start + timedelta(hours=3))
    assert b.price_cents == 7500

Read it carefully. test_a_focus_free_at_nine asserts that the Focus room is free at 9 —true if the calendar is empty—. test_b_book_focus books Focus from 9 to 12 —which mutates the shared forever—. Since the two share the same shared, the result of test_a depends on whether it runs before or after test_b. Before: the calendar is empty, the room is free, green. After: test_b already booked, the room is occupied, is_available returns False, and test_a fails. The verdict of test_a does not depend on its code —it depends on who ran first—.

Worked example: green in your order, red in CI's order

On your machine, pytest runs the tests in definition order: test_a first (empty calendar, passes), test_b after (books, passes). All green. With Python 3.14.0 and pytest 9.1.1, measured by executing:

python -m pytest demo_ci_only/test_shared_calendar.py -v

What to expect (your order: a, then b → green).

collecting ... collected 2 items

demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine PASSED   [ 50%]
demo_ci_only/test_shared_calendar.py::test_b_book_focus PASSED           [100%]

============================== 2 passed in 0.01s ==============================

2 passed. On your machine this is solid as a rock —you run it a hundred times and it passes a hundred—, because the definition order always puts test_a before test_b. Here there is no wind: the inspector hears nothing.

Now let us simulate CI's order. On the runner, for any of the reasons in the catalog (randomization, distribution among processes, a different collection), the tests can run in reverse: test_b first. We can force that order locally by naming the tests explicitly in reverse order —this is exactly what lesson 6 formalizes as a reproduction technique—:

python -m pytest \
  "demo_ci_only/test_shared_calendar.py::test_b_book_focus" \
  "demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine" -v

What to expect (CI's order: b, then a → red).

collecting ... collected 2 items

demo_ci_only/test_shared_calendar.py::test_b_book_focus PASSED           [ 50%]
demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine FAILED   [100%]

=================================== FAILURES ===================================
__________________________ test_a_focus_free_at_nine ___________________________

    def test_a_focus_free_at_nine():
        # Assumes an EMPTY calendar. True only if it runs BEFORE test_b.
>       assert is_available(shared, "r1", start, start + timedelta(hours=1)) is True
E       AssertionError: assert False is True
E        +  where False = is_available(<reservo.calendar.Calendar object at 0x102010ad0>, 'r1', datetime.datetime(2026, 8, 1, 9, 0), (datetime.datetime(2026, 8, 1, 9, 0) + datetime.timedelta(seconds=3600)))

demo_ci_only/test_shared_calendar.py:16: AssertionError
=========================== short test summary info ============================
FAILED demo_ci_only/test_shared_calendar.py::test_a_focus_free_at_nine - Asse...
========================= 1 failed, 1 passed in 0.01s ==========================

There it is, the CI-only, reproduced. The same code, the same two tests, and only the order changed: test_b ran first, booked Focus, and when it was test_a's turn the room was already occupied —is_available returned False, and assert False is True failed—. In your order, green; in CI's order, red. Neither the inspector nor you were crazy: the house creaks only when the north wind blows, and "the wind" was the reverse order.

Stop on what this means for the diagnosis. When you see a CI-only, the question is not "why does the CI lie?" but "what order/condition did the runner use?". Here, if the CI log showed test_b running before test_a (or a randomization seed), you would have the wind identified. The cause is the module-level shared, and the fix —a fixture that gives a fresh Calendar to each test— is lesson 7. For now, what matters is the classification: this is an order flaky from shared state, the reigning cause of the CI-only.

Why the retry does not save an order flaky

Here a preview that ties this topic with lesson 3 is worthwhile. The retry (--reruns) reruns the failed test, in the same session, with the state as it was left. For the clock flaky, that worked: each retry re-reads the clock and re-throws the die. But for the order flaky, the retry reruns test_a with the shared already mutated by test_b —the calendar is still occupied— so the retry fails again, and again, and again. The retry cannot rescue an order flaky, because it does not revert the state that caused it; it only repeats the test in the same poisoned environment.

We will see this executed in lesson 6, but note it now: the retry rescues chance flaky (clock, network), not state flaky (order, shared resources). It is one more reason not to treat the retry as a universal cure, and a diagnostic hint: if --reruns does not help, suspect shared state or order, not chance.

Common mistakes

Blaming the CI instead of looking for the condition. What happens: a CI-only comes out and the developer concludes "the runner is misconfigured" or "it's a GitHub thing," and opens an infrastructure ticket instead of looking at their code. Why it happens: "it works on my machine" pushes to blame the foreign environment. How to detect it: if your explanation of a CI-only does not name a concrete runner condition (order, TZ, parallelism, file), you did not diagnose, you just blamed. How to fix it: assume the bug is yours and that CI revealed it; run through the catalog of five winds and find which one blows on the runner and not on your machine. It is almost always one of those five.

Assuming that "it passes a hundred times on my machine" proves the test is solid. What happens: someone runs the suite many times on their laptop, always green, and declares the test reliable —ignoring that their laptop always uses the same order, the same TZ, without parallelism—. Why it happens: repetition in one environment gives false confidence. How to detect it: if your hundred runs were all under the same conditions (same order, same TZ, without -n), you did not prove robustness, you proved a single combination a hundred times. How to fix it: vary the conditions on purpose —run with -p randomly, with -n 2, with TZ=UTC— to blow yourself the winds of CI (lesson 6) before the runner blows them for you.

Confusing an order CI-only with the clock flaky and treating them the same. What happens: the team throws every flaky into the same bag and applies --reruns to all of them. For the order one, --reruns does not help (the state is still poisoned), and the team concludes "this flaky is incurable." Why it happens: the cause of the flaky is not distinguished. How to detect it: if --reruns rescues some flaky and not others, you have different causes —chance (rescuable) vs. state/order (not rescuable by retry)—. How to fix it: classify by cause before choosing the tool. The order one is fixed by isolating the state (lesson 7), not by retrying; that the retry does not save it is the hint that its cause is structural, not chance.

Exercises

Exercise 1 — Identify the wind. For each CI-only, say which of the five causes in the catalog is the most probable and what runner condition you would reproduce. (a) A test that formats a date as text passes on your machine and fails in CI showing the time six hours different. (b) A test that reads datos/fixture.csv passes in your folder and in CI fails with FileNotFoundError. (c) Two tests that write to the same temporary file pass in series on your machine and fail intermittently in CI, which runs with -n auto.

See solution

• (a) Time zone (cause 3). A difference of exactly six hours is the signature of your America/Mexico_City (UTC-6) against the runner's UTC. You would reproduce it by forcing TZ=UTC when running the test on your machine (lesson 6). The root fix: not depending on the system's zone —making the zone explicit in the code—.
• (b) Absent file (cause 4). The fixture.csv exists in your folder but is not committed (or is in a .gitignore), so the runner does not have it. You would reproduce it by deleting/moving that file on your machine and running the test. The fix: commit the fixture or generate it in the test.
• (c) Parallelism with shared resource (cause 2). The -n auto makes the two tests run at once and step on the temporary file —a race condition that in series does not appear—. You would reproduce it by running with pytest -n 2 on your machine. The fix: give each test its own temporary file (a tmp_path fixture), do not share.

The common discipline: each CI-only has a concrete, reproducible runner condition. Naming it is the diagnosis; forcing it on your machine (lesson 6) is the reproduction; removing the dependency on it (lesson 7) is the cure.

Exercise 2 — Predict according to the order. The two tests of test_shared_calendar.py share a Calendar. For each execution order, say which tests pass and which fail, and why. (a) test_a, then test_b. (b) test_b, then test_a. (c) Only test_a, without test_b in the run.

See solution

• (a) test_a → test_b: both pass (2 passed). test_a runs with the empty calendar (Focus free → True, passes). Then test_b books Focus (passes). It is the definition order, the one of your machine.
• (b) test_b → test_a: test_b passes, test_a fails (1 failed, 1 passed). test_b books Focus first (passes). Then test_a finds the calendar with Focus already occupied (is_available → False), and assert False is True fails. It is the CI order of the worked example.
• (c) Only test_a: passes (1 passed). Without test_b in the run, nobody booked Focus, the calendar is empty, the room is free. This reveals that test_a in isolation is correct —the problem is not test_a itself, but its dependence on the state that test_b leaves—.

The lesson: the verdict of test_a is not a function of test_a; it is a function of what ran before. A test whose result depends on its neighbors is a fragile test, and CI —with its different order— is the one that gives it away.

Exercise 3 — The wind the retry does not calm. A colleague puts @pytest.mark.flaky(reruns=5) on test_a_focus_free_at_nine to calm the CI-only, and complains that "not even with five retries does it pass in CI." Explain why the retry does not save it, and what they should have done instead.

See solution

The retry does not save it because this flaky is of state/order, not chance. When in CI test_b runs first and books Focus, the shared is left mutated —Focus occupied— for the rest of the session. Each of the five retries of test_a runs again with that same poisoned shared, finds Focus just as occupied, and fails again. The retry does not revert the state; it only repeats the test in the already contaminated environment. Unlike the clock flaky (where each retry re-throws the die of chance), here there is no die to re-throw: the result is deterministic given the state, and the state does not change between retries.

What they should have done: (1) classify the flaky —the fact that --reruns does not help is the hint that the cause is structural (state/order), not chance—; (2) reproduce it by forcing CI's order on their machine (lesson 6); (3) fix determinism by giving each test a fresh Calendar with a fixture (lesson 7), which eliminates the shared state and makes the order stop mattering. The retry was the wrong tool because the problem was not bad luck, it was a design that tied two tests through a common state.

Summary and next step

In this lesson you opened the second half of the module: the failure that only happens in CI —green on your machine always, red on the runner—. It is not a broken CI nor a mystery, but a real bug that needs a specific runner condition to manifest, like the house that only creaks with the north wind. You learned the catalog of five winds —execution order, parallelism, time zone/locale, absent file, limited resources— and that in all five the bug already lived in your code; CI only contributes the condition that reveals it.

And you saw the reigning cause —order with shared state— executed for real: two Reservo tests that share a Calendar pass in your order (2 passed) and, when CI's order is forced (test_b first), the second fails (1 failed, 1 passed) because it finds the room already booked. The same code, different order, different verdict. And you noted a key hint: the retry does not rescue an order flaky, because it does not revert the state that caused it —if --reruns does not help, suspect state, not chance—.

Before moving on you should be able to: name the five causes of the CI-only and the runner condition that reproduces each one; explain why "it passes a hundred times on my machine" does not prove robustness; predict the verdict of two tests with shared state according to their order; and argue why the retry does not save an order flaky.

What follows, in lesson 6, is turning that diagnosis into action: reproducing a CI-only on your machine. Instead of waiting for the runner to blow the wind, you blow it yourself —forcing CI's order, the TZ, the parallelism with -n, the randomization— to see the red appear in your terminal at will. With the failure reproduced locally, you can already fix it (lesson 7). It is the method of module 3, now sharpened for the CI flaky.

Resources

• Flaky tests — pytest documentation — the section on test order and shared state: pytest explains that tests must be independent of the order, exactly the sin of the shared Calendar of this lesson.
• pytest-xdist — documentation — the parallelism plugin (-n auto) of module 5, here seen as a source of CI-only: running in parallel exposes shared resources that in series give no problem. We will use it to reproduce in lesson 6.
• pytest-randomly — PyPI — the plugin that randomizes the order of the tests on purpose, to hunt order dependencies like the one of this lesson before CI hunts them for you. Lesson 6 uses it to reproduce.
• Default environment variables in runners — GitHub Actions — where to see the runner's configuration (time zone, system) that differs from your machine; the source of the TZ/locale "winds" of the catalog.