Module 7: Flaky Tests In Ci

2. Why the flaky is toxic in CI

Next →

Description

In lesson 1 you saw a flaky flicker: the same Reservo suite giving 1 failed, 7 passed and then 8 passed, without touching the code. Maybe you thought: "annoying, yes, but I run again and move on." That thought —perfectly reasonable on your machine— is exactly the one that turns the flaky into a poison in CI. This lesson is about why the same flaky test that on your laptop is a minor nuisance, in a shared pipeline is a team problem that erodes trust, wastes money, and lets real bugs through.

The difference is in one word: shared. Your machine is yours; if a test flickers on you, you suffer it and move on. The CI gate is everyone's: it runs on each person's PR, blocks each person's merge, and its red is seen by all. A flaky there is not your nuisance, it is that of twenty people at once. And since the red blocks, the team's natural reaction —"run it again"— not only fails to fix the flaky but institutionalizes it: it teaches people to re-run the red instead of reading it, and the day the red is a real bug, nobody believes it anymore.

By the end of this lesson you will be able to explain, with precision, the three damages a flaky does specifically in CI —it blocks everyone, it trains distrust, it hides real bugs—, put a cost on them, and understand why "re-run until it passes" is the trap that aggravates them. You will see, executed for real, the run where a single flaky dyes red an otherwise green suite: a 1 failed, 7 passed that in a gate means "nobody merges until this goes green."

Connection with the module: lesson 1 gave you the what (a flaky flickers and breaks the red=bug contract). This one gives you the why it hurts so much in CI, which is the justification for everything that follows. Because the flaky is toxic and blocks, lesson 3 discusses the retry as an emergency unblock; because re-running blindly is bad, lesson 4 offers quarantine as a disciplined alternative; because a part of these flaky only appear in CI, lessons 5 and 6 hunt them; and because none of that cures, lesson 7 fixes determinism. Understanding the damage is what will make you choose well among those tools in the mini-project.

The broken traffic light that nobody trusts

Think of an intersection with a traffic light. Its entire value depends on one thing: that it is predictable. Green, you go; red, you stop. Every driver builds their behavior on that promise, and that is why the intersection works without a police officer directing it.

Now imagine that traffic light breaks in a particular way: sometimes it turns red without anyone coming from the other side. Not always —most of the time it works—, but often enough for the drivers to notice. What happens? At first everyone stops at each red, obedient. But after stopping ten times for nothing, they start doing something very dangerous: they look at the red and decide on their own whether they have to stop or not. "It turned red but nobody's coming, I'll go." And it works... until the day someone is coming from the other side, the driver assumes "another false red" and goes. The crash was not caused by the traffic light turning red. It was caused by it becoming unreliable, and with that forcing each driver to substitute the clear rule ("red = stop") for their own fallible judgment ("red = let's see if it's real").

A flaky in CI is that traffic light. The red of the pipeline was supposed to be a clear rule for the whole team: red, don't merge. When the red starts lying sometimes, each developer stops obeying the rule and starts judging each red —"is it the flaky or is it real?"—. And since judging is fallible and takes effort, most opt for the shortcut: re-run until it goes green, or merge "because it was surely the flaky." There the traffic light stopped directing the traffic. And the crash —a real bug that passes the gate— is a matter of time.

On your machine, a flaky is your problem that you solve by running again. In CI, the gate is shared: the flaky blocks everyone, and the natural reaction —re-run until green— trains the team to disobey the red. The damage is not the red; it is that the red stops meaning something.

Damage one: it blocks everyone's PR, not just yours

The first damage is the most immediate and the easiest to measure. On your machine, if a test flickers on you, the cost is yours and it is small: you run again and keep working. In CI, the flaky lives in the gate —the door every change must cross to merge—, and that gate is shared.

Imagine the should_audit flaky in the Reservo suite, with a branch-protection rule that says "no merging to main with CI in red" (a very healthy rule, which module 6 defended). Now:

  • Ana opens a PR that changes the text of an email. It does not touch auditing at all. The CI runs, should_audit lands on an odd microsecond, and her PR comes out red. Ana cannot merge.
  • Bruno opens a PR that adds a new room. He does not touch auditing either. Same flaky, same red. Bruno cannot merge.
  • Carla's PR, which fixes a real bug and is urgent, comes out red for the same reason. Carla cannot merge.

None of the three touched the flaky code. None has a bug. And the three are blocked by the same test that flickers, each one wondering what they did wrong —when they did nothing—. A flaky on your machine costs you thirty seconds; a flaky in the gate costs the whole team, multiplied by each PR that runs while the flaky lives. That is the first cost, and it is money and time: reruns that consume billed minutes, PRs stopped, people waiting.

Worked example: a flaky dyes a green suite red

Let us see the exact moment when this happens. Reservo's suite has eight tests: six solid anchors (price and refund), the deterministic half of auditing, and the flaky. We run the full suite —as the gate would— in a run where the flaky lands on an odd microsecond. With Python 3.14.0 and pytest 9.1.1, measured by executing:

python -m pytest -q

What to expect.

>       assert should_audit() is True
E       assert False is True
E        +  where False = should_audit()

tests/test_audit.py:11: AssertionError
=========================== short test summary info ============================
FAILED tests/test_audit.py::test_new_booking_is_audited - assert False is True
1 failed, 7 passed in 0.02s

Read that 1 failed, 7 passed with gate eyes. Seven green tests —the six Reservo anchor numbers, perfectly healthy, plus the deterministic part of auditing— and the whole gate is red. It does not matter that 87% of the suite passes: a single red test turns the check red, and the branch-protection rule does not distinguish "red from a bug" from "red from a flaky." For GitHub, for the rule, for the merge button, it is a red and that's it. Every PR that runs against this state is blocked by a test that has nothing to do with anyone's work.

And here is the cruel detail: the red is honest in its form —the test really failed, assert False is True really happened— but lying in its meaning —there is no audit bug to fix—. That combination, a real red that does not correspond to a real problem, is what makes the flaky so hard to ignore and so easy to misinterpret.

Damage two: it trains the team to distrust the red

The second damage is slower and much more expensive, because it is not technical but cultural. When a flaky blocks PRs, the team looks for the fastest way out, and it is almost always the same: re-run the CI until it comes out green. The first time you do it with guilt. The fifth, with naturalness. The twentieth, it is a reflex: you see red, you hit "re-run," you don't even read the failure.

That reflex is the poison. Because it does not distinguish. The muscle you learned —"red → re-run"— fires the same way at the usual flaky as at a new red that is a bug. You have trained your team, without meaning to, to treat every red as noise. And with that you destroyed the only thing the CI contributed: a verdict you trust without thinking. Now every red requires a judgment —"is it the flaky or is it real?"— and that judgment is slow, fallible, and under delivery pressure almost always resolves to "it must be the flaky, I'll merge."

Notice the asymmetry that makes it so toxic. A test that always fails is honest: it forces you to look at it, you cannot ignore it, and you fix it. A flaky is dishonest: it gives you enough greens for re-running to "work," and thus teaches you not to look at it. The flaky is worse than an always-red test, because the always-red forces you into the correct action (fix) while the flaky seduces you toward the incorrect one (re-run and forget). A single flaky tolerated for a few months reconfigures how a whole team reads its CI, and that reconfiguration is not reversed by turning off the flaky: the distrust, once learned, stays.

Damage three: it hides real bugs

The third damage is the gravest, because it is the one the flaky existed to prevent. When the team learns that "red can be the flaky," it starts attributing any intermittent red to the known flaky. And some of those reds are real bugs.

Think of the mechanics. You have an audit flaky that everyone knows flickers. One day, Diego merges a change that, without his noticing, breaks refund_cents in a rare case: the 72 h refund sometimes returns 5900 instead of 6000 due to a rounding error that depends on the order in which the bookings are processed. Another PR's CI runs, test_full_refund_72h_before comes out red. That PR's developer looks at the red, thinks "another flaky," re-runs, and by chance this time it passes. Merges. The refund bug, which the CI did catch, was buried under the assumption that "every weird red is the flaky."

This is the outcome the traffic-light analogy predicted: the crash. The flaky not only wastes time and erodes trust; it camouflages real bugs, because it gives the team a comfortable explanation ("it's the flaky") to dismiss reds that deserved attention. A bug that the CI caught but the team ignored out of habit is worse than a bug the CI never caught: in the second case you were missing a test; in the first you had it and wasted it. The flaky turns your good tests into background noise that masks the true alarms.

Why "run it again" is the trap, not the solution

It is worth being explicit about why the natural shortcut aggravates the three damages instead of relieving them, because under pressure "re-run" always feels like the sensible option.

Re-running fixes nothing: the flaky stays exactly as flaky after it passed by chance. The only thing that changed is that this particular PR got unblocked —at the cost of leaving the flaky alive for the next PR, and the next, and the next—. It is bailing water from a boat with a hole: it keeps you afloat for a while, but the hole is still there and everyone has to keep bailing.

Worse: re-running trains the wrong reflex (damage two) and camouflages real bugs (damage three). Every time you re-run and it passes, you reinforce the lesson "the red was not for real," and that lesson is applied indiscriminately to the next red, be it flaky or bug. Re-running is not neutral: it actively worsens the culture of reading the CI.

This does not mean that retrying is always forbidden. It means that retrying is, at most, an emergency triage —a way to unblock the team today while the real work of recording, isolating, and fixing the flaky is done—, never an end. Lesson 3 studies the retry with exactly that lens: useful as a painkiller, dangerous as a cure, and always accompanied by a plan to eliminate the flaky at its root. The difference between a team that survives its flaky and one that drowns in them is not whether they retry —sometimes you have to—, it is whether retrying comes with a ticket and a date or is the endpoint.

Common mistakes

Measuring the cost of the flaky only in your time. What happens: someone defends leaving a flaky alive with "it's a test, I run it again and that's it, thirty seconds." Why it happens: it is thought from one's own machine, where the cost is individual and small. How to detect it: ask "does this test run on the shared gate?"; if so, multiply those thirty seconds by every team PR that gets blocked, plus the billed reruns, plus the erosion of trust. How to fix it: account for the CI cost as a team cost, not a personal one. A flaky in the gate does not cost thirty seconds; it costs thirty seconds × people × frequency, plus a cultural damage that has no small price.

Treating "it passes 90%" as acceptable for a gate. What happens: it is argued that a flaky that passes nine out of ten times "barely bothers." Why it happens: the 90% sounds high. How to detect it: if the gate runs, say, fifty times a day across all PRs, a flaky with a 10% failure blocks five times a day someone who did nothing wrong —and each block invites a rerun that reinforces the toxic reflex—. How to fix it: for a shared gate, the standard is not "it almost always passes" but "it always passes"; any spurious failure rate, however low it seems, is amplified by the volume of the gate and by the cultural damage.

Confusing "the red is real" with "the problem is real." What happens: the team sees assert False is True —a genuine failure, with traceback and all— and concludes there is an audit bug to fix, and loses hours looking for it. Why it happens: a red with a traceback looks identical to a bug; the form is honest. How to detect it: if the same test passes on re-run without changing anything, the red is real as an event but does not point to a production bug —it points to a non-deterministic test—. How to fix it: before hunting a production bug, confirm that the test asks a deterministic question; if it depends on the clock/order, the "bug" is in the test, and the fix is lesson 7, not an expedition through Reservo's logic.

Exercises

Exercise 1 — Put the cost in numbers. The should_audit flaky fails 10% of the gate runs. The team opens 30 PRs a day, and each PR runs the CI an average of 2 times (initial push + one adjustment). Each CI rerun costs 3 minutes of runner. Estimate (a) how many gate runs a day fail because of the flaky, and (b) the runner minutes a day spent only on the reruns that those failures cause (assume one rerun per failure). And in one sentence, name the cost that does not appear in those numbers.

See solution
  • (a) Runs that fail because of the flaky: 30 PRs × 2 runs = 60 gate runs a day; at 10% spurious failure, ≈ 6 runs a day come out red because of the flaky (with no real bug).
  • (b) Rerun minutes: each of those 6 failures causes at least one rerun of 3 minutes → 18 minutes a day of runner burned only on re-running because of the flaky, ≈ 90 minutes a week, ≈ 6.5 hours of runner a month thrown away on a single badly written test.
  • The cost that does not appear: the cultural damage —each of those 6 daily reruns trains the team to re-run the red instead of reading it, and that, besides eroding trust, makes some real bug pass disguised as flaky—. That cost does not fit in a spreadsheet, and it is usually the most expensive of all.

The lesson: even ignoring the cultural damage, the measurable cost of a 10% flaky in an active gate is already hours of runner a month. "I run it again and that's it" ignores that that "it" is paid by many people, many times.

Exercise 2 — Worse than always-red. A colleague says: "I prefer a flaky that passes 90% to a test that always fails; at least the flaky lets me move forward sometimes." Argue why, for the team's health, a 90% flaky can be worse than a test that always fails.

See solution

A test that always fails is honest: it does not let you move forward until you look at it, so it forces you into the correct action —fixing it (or deleting it if it no longer applies)—. You cannot ignore it, and that is why it poisons nothing: it is an acute pain that resolves quickly.

A 90% flaky is dishonest precisely by letting you move forward "sometimes." Those occasional greens make "re-running" work enough to become the habit, and thus the flaky survives —nobody fixes it because nobody is forced to look at it—. Worse, it trains the "red → re-run" reflex that later applies to real reds (damages two and three). The always-red is cured in hours because it blocks; the 90% flaky lives for months because it almost does not block, and in those months it reconfigures how the team reads its CI.

The paradox: the property your colleague values in the flaky —"it lets me move forward sometimes"— is exactly the one that makes it toxic and persistent. A pain that lets you function is a pain you do not attend.

Exercise 3 — The disguised bug. Reservo has a known audit flaky. One day, test_full_refund_72h_before (a solid anchor that always passed) starts failing intermittently after a merge by someone else. The team, out of habit, attributes it to the "known flaky" and re-runs until it passes. What signal should have stopped that attribution, and what should the team have done?

See solution

The signal that should have stopped them: the test that now flickers is not the known flaky. The known flaky is test_new_booking_is_audited; the one that started failing is test_full_refund_72h_before, a refund anchor that had always passed deterministically. Attributing its red to the "audit flaky" is an identity error: they are different tests, with different causes. A deterministic anchor that suddenly becomes intermittent after a merge is the signature of a new bug introduced by that merge —probably shared state or order, given that it is intermittent—, not of the preexisting flaky.

What the team should have done: (1) not throw all intermittent reds into the same "it's the flaky" bag; identify which test flickers. (2) Note that a stable anchor becoming flaky coincides with a merge → suspect that merge. (3) Reproduce the new flaky (lessons 5–6) and fix its determinism or revert the merge. This is damage three live: the known flaky gave the perfect cover for a real refund bug to pass as "more of the same." The discipline that avoids it: every flaky is tracked by identity and with a ticket (lesson 4), so that "it's the flaky" is a verifiable claim, not a pocket excuse.

Summary and next step

In this lesson you measured why the same flaky that on your machine is a minor nuisance, in CI is a team poison. The difference is one word —shared— and from it come three damages: the flaky blocks everyone's PR (you saw 1 failed, 7 passed turn a 87%-green gate red, stopping those who did not touch the flaky code); it trains the team to distrust the red (the "red → re-run" reflex applied indiscriminately); and it hides real bugs (genuine reds dismissed as "the usual flaky"). The traffic light that sometimes turns red for no reason does not fail because of the red, it fails by becoming unreliable and forcing everyone to judge instead of obey.

It was also clear why "run it again" is the trap: it fixes nothing, reinforces the toxic reflex, and camouflages bugs. Retrying can be an emergency triage, never an end —and always with a ticket and a plan—.

Before moving on you should be able to: name the three damages of the flaky in CI and explain why they are of the team and not personal; argue why a 90% flaky can be worse than an always-red; and detect when "it's the flaky" is an excuse that hides a real bug.

What follows, in lesson 3, is to look head-on at the tool the team reaches for first when the flaky blocks: the retry. We are going to install pytest-rerunfailures and run --reruns 3 for real on this same flaky —you will see the RERUN rescue it sometimes and fall short others— to have, with the real output in hand, the honest debate: when retrying unblocks and when, simply, it lies to you in green.

Resources

← Previous1. Module introduction: the flaky test in CINext →3. The retry debate: `--reruns`