1. Module introduction: the flaky test in CI

Description

Over six modules you built a pipeline that makes a very concrete promise: every time someone pushes code, the Reservo suite runs on its own and tells you the truth. Green: the code is healthy. Red: there is something to fix. That promise —red means bug— is the entire reason a CI is worth having. It is what lets you merge with confidence, what protects the main branch, what turns "I hope it works" into "the guardian verified it." This module is about the disease that attacks that promise from within: the flaky test.

A flaky test is one that sometimes passes and sometimes fails without anyone touching the code. You run it now and it goes green; you run it ten seconds later and it goes red; you run it again and green once more. The code is the same, the test is the same, and the verdict changes. Its close relative is the test that only fails in CI —green on your machine, always; red on the runner, sometimes or always—. Both do exactly the same damage: they break the "red = bug" contract. When a red can be a real bug or "the usual flaky," you can no longer read it. And a traffic light that sometimes lies in red is not a traffic light, it is noise with lights.

By the end of this lesson you will know what a flaky is, why in the CI context it is more toxic than an honest failure, and you will see it beat for real: Reservo ships a function that looks at the wall clock and decides differently on every run, so the same suite —without changing a single character— will give you 8 passed one time and 1 failed, 7 passed the next. That flicker, executed in your terminal, is the whole problem of the module in miniature.

Connection with the module: this lesson is the map. Here you install the idea —what a flaky is, why it poisons CI— and you watch it flicker once. Lesson 2 dissects why it is especially toxic in CI and not just annoying. Lesson 3 opens the retry debate (--reruns), executed for real: you will see the RERUN rescue the flaky, and the argument for and against. Lesson 4 teaches quarantine —isolating the flaky from the gate, not re-running blindly—. Lesson 5 catalogs the failure that only happens in CI and its causes. Lesson 6 reproduces it on your machine. And lesson 7 closes with the only real cure: fix determinism, do not retry. Lesson 8, the mini-project, has you decide —retry, quarantine, or fix— facing a flaky that blocks Reservo's CI.

A boundary note, because this module leans on its siblings and does not repeat them. Diagnosing a flaky in depth —reproducing it reliably, isolating the pair of tests that interact, freezing the clock to hunt the guilty line— is the trade of the sibling guide test-failure-diagnosis-guide. Here the focus is the flaky in the CI context: how it blocks or unblocks the team, the retry debate, quarantine, and the CI-only. When we reach "reproduce the failure on your machine" we will lean on the method of module 3 (reproducing a CI failure locally) and at that point we will bridge to the diagnosis guide. The coverage gates were module 6; here we do not return to them.

The smoke detector that goes off when you toast bread

Imagine a smoke detector in a restaurant kitchen. Its job is one and it is sacred: to sound when there is fire. One day it starts sounding also when someone toasts bread, or when the steam from a pot rises. Not always —sometimes the bread toasts in silence, sometimes it sounds—, but often enough to become a nuisance. What does the kitchen team do? At first, every time it sounds, someone runs to check. After the tenth false alarm, they stop running. After the twentieth, someone stands on a chair and takes out the battery "until we stop cooking so close." And one night, for real, a pan catches fire. The detector does not sound —it has no battery— or it sounds and nobody turns, because "it's the bread again."

The detector did not fail by sounding too much. It failed by becoming unpredictable, and with that it destroyed the only thing that made it useful: that people believed it. A detector that never sounds is useless. One that sounds with real smoke is a treasure. But one that sounds sometimes with smoke and sometimes with bread is worse than having none at all, because it teaches the team to ignore it right when it matters most.

A flaky test is that detector. Its alarm is the red of the CI. When the red appears sometimes with a real bug and sometimes with "the usual bread," the team learns to ignore it —to re-run without looking, to merge "because it's surely the flaky"—. And the day the red is a real fire, nobody turns anymore. That is why this module does not treat the flaky as a minor technical nuisance: it treats it as what it is, a direct attack on the trust that makes a CI worth anything.

A flaky test passes sometimes and fails sometimes without the code changing. In CI its poison is not the red itself, but that it breaks the "red = bug" contract: it trains the team to distrust the traffic light, and a traffic light you do not trust protects nothing.

Reservo, as we left it — and a function that betrays the exact time

We continue with Reservo, the room-booking system of a coworking space that we have been testing since the first module. It is pure Python logic: no database, no network, no hidden clocks... until today. Its pieces, in case you need a refresher:

• Room (id, name, capacity, hourly_cents), Member (id, name, tier: "basic" or "pro"), Booking (with its price_cents field, the start, the end as a half-open range [start, end), and its status).
• The core functions: price_cents(room, member, hours), refund_cents(booking, price_paid_cents, now), overlaps, is_available, book, and the Calendar that keeps the bookings in memory.
• The anchor numbers, the checksum of the whole guide: basic 3 h → 7500, pro 3 h → 6000 (20% discount), and the refund on 6000 paid: 6000 if you cancel 72 h before (≥ 48 h, 100%), 3000 at 36 h (24–48 h, 50%), 0 at 12 h (< 24 h).

You already have all that tested, and all of it is deterministic: give it the same inputs, it gives you the same result, always. A test of price_cents(focus, ana, 3) == 7500 passes today, tomorrow, and on any runner on the planet, because nothing in that function depends on when you run it.

For this module we add to Reservo something that does depend on when you run it, because we need a real flaky to study it. The team decided not to record all bookings in the audit log —there would be millions—, but to sample roughly half. And someone implemented that sampling in the worst possible way: by looking at the wall clock.

# reservo/audit.py
from datetime import datetime


def should_audit(now=None) -> bool:
    """Decide whether a booking is recorded in the audit sample.

    NAIVE IMPLEMENTATION — and that is precisely the lesson of the module. To
    avoid recording ALL bookings, it samples ~half by looking at the parity of
    the microsecond of the wall clock: even -> audited, odd -> skipped.
    Since the clock advances on each read, the parity changes from one call to
    the next: the result is NOT deterministic. A test that calls
    should_audit() without injecting `now` will be flaky by construction.

    The `now` parameter is the seam: if you pass it, the function becomes
    a pure function of its input and its result is reproducible.
    """
    if now is None:
        now = datetime.now()
    return now.microsecond % 2 == 0

Stop on the last line: return now.microsecond % 2 == 0. now.microsecond is the microsecond part of the current instant —a number between 0 and 999999 that changes every time you read the clock—. % 2 == 0 asks whether it is even. Since the exact microsecond in which you run the function is, for all practical purposes, chance, the answer is "even or odd" with roughly the same probability, and it changes on every call. That is the heart of the flaky: a decision made by reading the wall clock, without anyone asking, is not reproducible.

Notice also the now=None parameter. That is the seam —the point through which, later on (lesson 7), we will inject a fixed clock to cure the flaky at its root—. For now ignore it: the problem begins when a test calls should_audit() without passing now, letting the function read the real clock. That test is a flaky.

First contact: the same suite, two verdicts

Here is the test that a well-meaning developer wrote for the new feature. Read it: it looks perfectly reasonable.

# tests/test_audit.py
from datetime import datetime

from reservo.audit import should_audit


def test_new_booking_is_audited():
    # Author's intent: "a new booking gets audited".
    # BUG: should_audit() without an argument reads the real clock, and returns
    # True only when the microsecond is even (~half the time). This test is FLAKY
    # by construction: sometimes it passes, sometimes it fails, without touching the code.
    assert should_audit() is True

The author wanted to assert "a new booking gets audited," which sounds like a business truth. But by calling should_audit() without an argument, they let the function read the clock, and tied the test's verdict to the parity of the microsecond in which it runs. When the microsecond is even, should_audit() returns True and the test passes. When it is odd, it returns False and the test fails with assert False is True. There is no bug in price_cents, nor in refund_cents, nor in anything a Reservo customer cares about: the bug is in the test, which asked a non-deterministic question.

Worked example: run the same suite several times and watch the verdict flicker

We are going to run Reservo's full suite —the three price tests, the three refund tests, and the new audit one— on the machine where I write this, with Python 3.14.0 and pytest 9.1.1. We will run it several times in a row, without changing a single line between runs, to see the flaky in action. We use -q (--quiet), which summarizes each run in one line.

python -m pytest -q

What to expect. Not one output, but two different ones, depending on the microsecond in which each run lands. In a run where the clock was on an odd microsecond, this is what came out, measured for real:

>       assert should_audit() is True
E       assert False is True
E        +  where False = should_audit()

tests/test_audit.py:11: AssertionError
=========================== short test summary info ============================
FAILED tests/test_audit.py::test_new_booking_is_audited - assert False is True
1 failed, 7 passed in 0.02s

Read the summary: 1 failed, 7 passed. Seven green tests —the six Reservo anchors plus the deterministic half of auditing that we will see later— and one red, the flaky, which landed on an odd microsecond. Now, without touching anything, run exactly the same command again, on an even microsecond:

=========================== short test summary info ============================
...
8 passed in 0.01s

8 passed. All green. The same command, the same code, the same test —and the verdict changed from red to green only because the clock advanced a few microseconds between one run and the other. This is a flaky, live, in your terminal. If you run it six times in a row you will see something like failed, passed, passed, passed, passed, passed —sometimes red, almost always green, with no pattern that depends on anything you can control from the code—.

Stop to feel what this does to a CI. Imagine that run is the gate of a pull request. You open the PR, the suite runs, it comes out red: 1 failed. You look at the failure, you do not understand —your change did not touch auditing—, you re-run the CI, and now it comes out green: 8 passed. What did you learn? That the red "was not for real." And the next time you see a red, your first instinct will no longer be "I'm going to fix the bug" but "I'm going to re-run, it's surely the flaky." There, exactly there, trust in your pipeline started to die.

What a flaky is and what it is not

It is worth sharpening the definition before moving on, because not every intermittent red is a flaky and not every flaky looks the same.

A flaky is a test whose verdict is not a deterministic function of the code under test. Give it the same code and the same test, and sometimes it says one thing and sometimes another. The cause is always a source of non-determinism that slipped into the test or the code: the wall clock (our case), the execution order relative to other tests, state shared between tests, randomness without a fixed seed, a network call that sometimes takes too long, concurrency. In this module we work the two most common sources in CI: the clock (this should_audit) and the order/shared state (two Reservo tests that step on each other, in lesson 5).

What a flaky is not: it is not a test that always fails —that is an honest failure, a reproducible bug, and it is fixed like any other—. Nor is it a test that fails on one Python version and passes on another consistently —that is a version incompatibility, which is module 4—. The mark of the flaky is inconsistency without a change of input: same code, same environment, a result that dances. And its cousin, the CI-only, has its own mark: green on your machine, red on the runner, because the runner has a condition you do not (a different order, a different TZ, a different parallelism) —sometimes intermittently, sometimes always-red-in-CI-always-green-locally, which still counts as flaky for the team because the verdict depends on where it runs, not on the code—.

And an honesty of the guide, the usual one: the CI workflow runs on a GitHub runner that we do not have here, so the YAML we write and read as content. But the flaky is real and executed: should_audit looks at your machine's clock just as it would look at the runner's, and the 1 failed / 8 passed flicker I measured by running pytest for real, it is not a mock-up. When I cite an output with RERUN or a count, that number came from executing the command.

Common mistakes

Believing that "it passes most of the time" is the same as "it passes." What happens: a flaky test goes green nine out of ten runs, and the team treats it as healthy —"it almost always passes, it must be fine"—. Why it happens: the frequent green deceives; "probable" gets confused with "reliable." How to detect it: if you cannot run the test a hundred times in a row and get a hundred greens, it is not reliable, it is a flaky on a good streak. How to fix it: a test either is deterministic or it is not; "almost always" is the definition of the problem, not a mitigating factor. Treat it as flaky from the first flicker.

Looking for the bug in the production code when the flaky is in the test. What happens: the CI fails in test_new_booking_is_audited, and the developer starts reviewing Reservo's audit logic looking for what is wrong —when should_audit does exactly what its (bad) implementation says—. Why it happens: it is assumed that a red points to a production bug, not to a badly written test. How to detect it: if the test passes and fails without the code changing, the non-determinism is in the test or in its interaction with the clock/order, not in a classic bug. How to fix it: first ask yourself "does this test ask a deterministic question?"; if it depends on the clock or the order, there is the problem.

Re-running the CI until it comes out green and merging. What happens: the red appears, the developer hits "re-run" two or three times, it comes out green, and merges without more ado. Why it happens: it is what unblocks the PR right now, and under delivery pressure it feels rational. How to detect it: if your usual way to "fix" a red is to run it again, you are treating the symptom and leaving the flaky alive for the next one. How to fix it: re-running can be an emergency triage (lesson 3), but never the answer; the flaky has to be recorded, put in quarantine with a ticket (lesson 4), and fixed at its root (lesson 7). Re-running and forgetting is how a flaky survives months.

Exercises

Exercise 1 — Flaky or not flaky. For each test, say whether it is flaky, an honest failure, or a version incompatibility, and why in one sentence. (a) assert price_cents(focus, ana, 3) == 7500 fails always, on every machine, with assert 6000 == 7500. (b) assert should_audit() is True passes sometimes and fails others on the same machine without changing the code. (c) assert list(itertools.batched("ab", 1)) fails on Python 3.11 and passes on 3.12+, always the same.

See solution

• (a) Honest failure. It fails always, with the same inputs, on every machine: that is a reproducible bug (here, price_cents is returning 6000 where the anchor says 7500). It is fixed like any deterministic bug: you find the cause and correct it. It is not flaky —there is no inconsistency without a change of input—.
• (b) Flaky. Same code, same machine, a verdict that dances between pass and fail: the exact mark of the flaky. The cause is the source of non-determinism (the clock, via should_audit() without now).
• (c) Version incompatibility. It fails on 3.11 and passes on 3.12+ consistently —give it the same version and it always gives the same—. That is not flaky, it is a reproducible environment difference, and its terrain is module 4 (the matrix), not this one.

The rule: flaky = inconsistent without changing the input. If by fixing the input (including the version) the result becomes constant, it is not flaky; it is a failure or an incompatibility.

Exercise 2 — The analogy at the table. The smoke detector that sounds with toasted bread taught the kitchen team to ignore it. Map each element to its equivalent in CI: (a) the smoke detector, (b) that it sounds with the bread, (c) taking out its battery, (d) the real fire that nobody attends.

See solution

• (a) The smoke detector → the CI (the test suite as a gate). Its job is to sound (red) when there is fire (a bug). It is valuable only if the team believes it.
• (b) That it sounds with the bread → the flaky test. An alarm that fires without real fire: a red that does not correspond to a bug. Not always, but often enough to become noise.
• (c) Taking out its battery → re-running without looking / merging ignoring the red / disabling the gate. The human reaction to false alarms: turn off the signal to stop suffering it —and with that, be left without protection—.
• (d) The real fire that nobody attends → a real bug that passes disguised as flaky. The day the red was a bug, the team re-runs it or ignores it "because it's surely the flaky," and the bug reaches production. The flaky not only wastes time: it makes the real reds go unnoticed.

The shared moral: the problem is not that the alarm sounds too much, it is that it becomes unpredictable and with that it loses the only property that made it useful —that people believe it—.

Exercise 3 — Which lesson solves this? For each situation, say which lesson of this module addresses it. (a) "The flaky blocks everyone's PR and I want to unblock now, even if I don't fix it today." (b) "I want to take the flaky out of the gate while I investigate, without deleting it or hiding it." (c) "This test passes on my machine always and only fails in CI." (d) "I want the flaky to stop being flaky, at its root."

See solution

• (a) Unblock now, even if I don't fix it today → lesson 3 (the retry debate, --reruns). Retrying is the immediate unblocking tool; the lesson gives you its real output and, above all, its cost: it can cover a bug.
• (b) Take it out of the gate while I investigate, visible → lesson 4 (quarantine). @pytest.mark.flaky or xfail(strict=False) with a ticket isolate it without deleting the evidence.
• (c) Green locally, red only in CI → lessons 5 and 6 (the failure that only happens in CI and how to reproduce it). There is the catalog of causes and the technique to force the runner's condition on your machine.
• (d) That it stop being flaky, at its root → lesson 7 (fixing determinism). Retry and quarantine are triage; the cure is to remove the source of non-determinism —inject the clock, isolate the state—.

The mechanical rule: "unblock now" is retry (3), "isolate to investigate" is quarantine (4), "only fails over there" is CI-only (5–6), "cure" is determinism (7).

Summary and next step

In this lesson you installed the idea that holds up the module: a flaky test passes sometimes and fails sometimes without the code changing, and its poison is not the red itself, but that it breaks the "red = bug" contract and trains the team to distrust the traffic light —the smoke detector that sounds with the bread until they take out its battery—. A CI you do not trust protects nothing.

You saw it beat with a real local demo: Reservo shipped should_audit, an audit sampling that looks at the parity of the wall clock's microsecond, and test_new_booking_is_audited calls it without injecting now. You ran the same suite several times and saw the verdict flicker —1 failed, 7 passed on an odd microsecond, 8 passed on an even one—, without touching a single line between runs. The boundary was also clear: the in-depth diagnosis of a flaky is the sibling guide; here the focus is the flaky in the CI context.

Before moving on you should be able to: define in your own words what a flaky is and what distinguishes it from an honest failure or a version incompatibility; explain why an intermittent red damages trust in the CI; and point out, in test_new_booking_is_audited, exactly which line introduces the non-determinism (the should_audit() call without now).

What follows, in lesson 2, is to cut open why this damage is worse in CI than on your machine. A local flaky is a nuisance of yours; a flaky in CI is a nuisance of everyone's, because the gate is shared —it blocks the whole team's PRs, invites re-running instead of reading, and makes real bugs pass in disguise—. We are going to measure that cost with precision.

Resources

• Flaky tests — pytest documentation — the official pytest page on what a flaky is, why it hurts, and what categories there are. It is the conceptual reference for the whole module; read it to see that this problem is recognized and studied, not an oddity of yours.
• datetime.now() — Python documentation — the source of non-determinism of should_audit: it reads the wall clock at the instant of the call. Lesson 7 replaces it with an injected now; here it is the root of the flaky.
• pytest-rerunfailures — PyPI — the retry plugin we will run for real in lesson 3. Glance at it now to have the name; in lesson 3 we install it and run it on this same flaky.
• About continuous integration — GitHub Actions — the CI context where the flaky does its worst damage: a shared gate that runs on every push. Return to it to remember why the CI red was supposed to mean something.