4. Quarantine: mark and isolate

Description

The global retry of lesson 3 has an aiming problem: it retries the whole suite. If you enable --reruns 2, not only the audit flaky gets retried —so do the six Reservo anchors, which are deterministic and should never need a second attempt, and any real bug that failed honestly—. It is like prescribing an antibiotic to the whole family because one has an infection. This lesson teaches the surgical alternative: quarantine, which isolates only the flaky test and leaves the rest of the suite in peace, with its honest red intact.

Quarantine is a medical term borrowed with precision: you separate the sick one from the healthy group —not to cure them right there, but so they do not infect while you study them—. In tests, putting a flaky in quarantine means taking it out of the gate —so its flicker stops blocking everyone's PRs— without deleting it or hiding it, leaving it visible and tracked while it is investigated and fixed. You are going to see two ways to do it, both executed for real: @pytest.mark.flaky(reruns=N), which retries only that test; and @pytest.mark.xfail(reason=..., strict=False), which declares it "expected to fail" so that its result —fail or pass— does not break the build, and remains as xfailed or xpassed with its reason in plain sight.

Connection with the module: lesson 3 gave you the retry as emergency triage and its danger (retrying blindly). This lesson refines the triage: instead of retrying everything, it isolates the culprit, with aim and with visibility. The discipline you learn here —quarantine always with a ticket, always temporary, always reviewed— is what prevents the triage from rotting into permanent denial. And it prepares lesson 7, the cure: quarantine buys time to investigate and fix determinism; it does not replace it. A flaky in eternal quarantine is a flaky that won.

The sick passenger who does not get off the plane

Imagine a long flight where a passenger starts coughing badly. There are three possible responses, and only one is sensible.

The first: ignore it and carry on as if nothing. The risk is that it infects the whole cabin —the equivalent of leaving the flaky in the gate, blocking and confusing everyone—.

The second: pretend it does not exist, cover it up, tell it to be quiet. This is worse: the passenger is still sick and now nobody knows it, and when the problem explodes (an emergency medical landing) nobody was prepared. In tests, this is deleting the test or commenting it out without leaving a trace: the flaky "disappears," but so does the evidence that there was a problem, and the bug it may have hidden is left unwatched.

The third, the sensible one: move the passenger to a separate row, with attention. You do not get them off the plane (you do not delete the test), you do not leave them infecting (you do not leave it in the gate). You separate them, mark them —"this one needs follow-up"—, and note their case to attend to it. They are still on board, still visible, but no longer put the rest at risk while they are investigated.

That third response is the quarantine of a flaky. Neither ignoring (leaving it blocking), nor hiding (deleting it). Isolating with visibility: taking it out of the gate so it does not infect, marking it so everyone knows it is under observation, and recording it with a ticket so it gets attended to. The difference between hiding and isolating-to-investigate is the whole difference between denying the problem and managing it.

Putting a flaky in quarantine is isolating it from the gate —so it stops blocking everyone— without deleting it or hiding it. It remains visible, marked, and with a ticket, while it is investigated and fixed. It is not a cure: it is separating the sick one so it does not infect while you study it.

Form one: @pytest.mark.flaky — retry only that test

pytest-rerunfailures (the same plugin from lesson 3) offers, besides the global --reruns, a per-test marker: @pytest.mark.flaky(reruns=N). Placed on top of a test, it says "retry this one up to N times; the rest of the suite, don't even touch it." It is the aim that the global retry lacked.

# demo_quarantine/test_marker_flaky.py
import pytest

from reservo.audit import should_audit


@pytest.mark.flaky(reruns=3)
def test_audited_marked_flaky():
    # Only THIS test retries (per-test marker), without global --reruns.
    assert should_audit() is True

The difference from lesson 3 is one of scope: here there is no --reruns on the command line. The retry lives in the marker, stuck to the guilty test, and only to it. You run the normal suite —pytest, without flags— and the six Reservo anchors run once each, as they should; only test_audited_marked_flaky retries if it fails.

Worked example 1: the marker retries only its test

Let us run the marked test, without any reruns flag, in verbose mode. With Python 3.14.0, pytest 9.1.1, and rerunfailures 16.4, in a real run where the first attempt landed odd:

python -m pytest demo_quarantine/test_marker_flaky.py -v

What to expect.

collecting ... collected 1 item

demo_quarantine/test_marker_flaky.py::test_audited_marked_flaky RERUN    [100%]
demo_quarantine/test_marker_flaky.py::test_audited_marked_flaky PASSED   [100%]

========================== 1 passed, 1 rerun in 0.01s ==========================

RERUN and then PASSED, summary 1 passed, 1 rerun —the same as in lesson 3, but without having written --reruns—. The retry came from the marker. If this suite also had a Reservo anchor, that anchor would have run only once, without retries, because the marker does not reach it. That is the advantage: the retry is documented in the test, visible to whoever reads it, and bounded to the only one that needs it. Nobody who opens the file can ignore that this test is a recognized flaky —the marker screams it—.

The marker is a legitimate form of quarantine when the flaky is tolerably rare and its green-by-retry does not hide a serious bug. But notice its limit: retrying, even per test, is still retrying. If the test exhausts its reruns and fails, it blocks the gate again. For a flaky you want to take out of the gate completely while you investigate it —so that not even a bad day of retries blocks it—, there is a more forceful tool: xfail.

Form two: xfail — take it out of the gate completely

@pytest.mark.xfail is a native pytest marker (no plugin needed) that declares: "this test is expected to fail." With that declaration, pytest changes how it counts the result:

• If the test fails, it does not report it as FAILED (which would break the build) but as XFAIL ("expected failure": it failed, as we expected, all in order).
• If the test passes, it reports it as XPASS ("unexpectedly passed": it passed, against expectations).

The key for quarantine is the strict parameter. With strict=False (what we want for a flaky), neither XFAIL nor XPASS breaks the build —the flaky is completely out of the gate, pass or fail—. With strict=True, an XPASS would break the build (useful for other cases, but not for a flaky, where you do not want "it passed by chance" to block you).

# demo_quarantine/test_xfail_quarantine.py
import pytest

from reservo.audit import should_audit


@pytest.mark.xfail(
    reason="flaky under investigation — ticket RES-412; does NOT block the gate",
    strict=False,
)
def test_audited_quarantined():
    # In quarantine: fail or pass, it does NOT break the build. It stays VISIBLE as xfail/xpass.
    assert should_audit() is True

Notice the reason. It is not decoration: it is the label of the quarantined passenger. It says why it is isolated (it is flaky, under investigation) and where to track it (the RES-412 ticket). An xfail without a reason is a hidden test without explanation —denial—; an xfail with a reason and a ticket is disciplined quarantine —isolation with follow-up—.

Worked example 2: xfail in its two outcomes

Since the flaky flickers, a test under xfail(strict=False) lands sometimes on XFAIL (when it fails) and others on XPASS (when it passes). Let us see both, measured by executing, with -rxX so the summary lists the reasons. First, a run where the flaky failed (→ XFAIL, the expected):

python -m pytest demo_quarantine/test_xfail_quarantine.py -rxX

What to expect (the flaky failed → XFAIL).

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 1 item

demo_quarantine/test_xfail_quarantine.py x                               [100%]

=========================== short test summary info ============================
XFAIL demo_quarantine/test_xfail_quarantine.py::test_audited_quarantined - flaky under investigation — ticket RES-412; does NOT block the gate
============================== 1 xfailed in 0.01s ==============================

And now a run where the same flaky passed (→ XPASS, unexpected but, with strict=False, harmless):

What to expect (the flaky passed → XPASS).

demo_quarantine/test_xfail_quarantine.py X                               [100%]

=================================== XPASSES ====================================
=========================== short test summary info ============================
XPASS demo_quarantine/test_xfail_quarantine.py::test_audited_quarantined - flaky under investigation — ticket RES-412; does NOT block the gate
============================== 1 xpassed in 0.01s ==============================

The decisive thing: in both cases, the final summary is green for the gate. 1 xfailed and 1 xpassed are states that do not break the build (with strict=False). The flaky no longer blocks anyone —however it lands—, and yet it is still visible: the lowercase x (xfail) or uppercase X (xpass) on the progress line, and the XFAIL/XPASS in the summary with its reason and its ticket. The passenger is in their separate row, marked, without infecting, and everyone can see they are there.

Compare with skip. You might think of @pytest.mark.skip to take the flaky out of the gate, and it works as far as not blocking. But skip is more opaque: the test does not run, so you lose all information about whether it is still flaky, whether it started passing always (a sign that maybe it already fixed itself), or whether it degenerated into always-red (a sign of a new bug). xfail(strict=False) does run the test and shows you the result (xfail vs xpass), just without letting it break the build. For a flaky under investigation, xfail observes; skip blinds. Prefer xfail while you want to watch the flaky, and reserve skip for tests that simply must not run (for example, one that depends on an absent resource).

The discipline that separates quarantine from dumpster

Quarantine is healthy only if it respects three rules. Without them, it degenerates into a dumpster of tests that nobody looks at —the worst of all worlds, because it accumulates hidden flaky that no longer protect anything—.

Rule 1: always with a ticket. Each test in quarantine carries, in its reason, the reference to a ticket that tracks it (like RES-412). The ticket is the promise that the quarantine is temporary and that there is pending work. Without a ticket, "in quarantine" becomes "forgotten forever."

Rule 2: always temporary, with a date or review. A quarantine without an expiration is a sentence. The team must periodically review the tests in quarantine —were they fixed?, are they still flaky?, did any degenerate into a real bug?— and take them out (fixed) or escalate them (if they got worse). A flaky in xfail for a year is not quarantine, it is a dead test pretending to be alive.

Rule 3: count the quarantine. The number of tests in quarantine is a health metric of the suite. Zero is ideal. A handful, tolerable if they are under active investigation. Dozens, an alarm: it means the team isolates flaky faster than it fixes them, and the suite is rotting. Count your xfail/flaky and watch that the list does not grow without going down.

Quarantine, well done, is an honest management tool: it acknowledges that you cannot fix everything today, unblocks the team without lying, and keeps the problem in sight with a plan. Badly done —without a ticket, without an expiration, without a count— it is just an elegant way to sweep the trash under the rug, with the aggravating factor that the rug looks clean (the gate is green) while the trash grows underneath.

Common mistakes

Using xfail(strict=True) for a flaky. What happens: someone marks the flaky with xfail but leaves strict=True (or does not know that is the default in some configs), and then when the flaky passes (XPASS), the build breaks with "unexpectedly passed." Why it happens: strict is subtle and its default depends on the config. How to detect it: if your test in quarantine breaks the build when passing, you have strict=True. How to fix it: for a flaky —which by definition sometimes passes— use strict=False, so that neither the failure nor the expected pass break anything. strict=True is for another case (marking a known bug that must keep failing until you fix it), not for a flaky.

Deleting or commenting the test instead of putting it in quarantine. What happens: fed up with the flaky, someone deletes test_new_booking_is_audited or comments it out. The gate goes green and everyone is happy. Why it happens: deleting is the fastest unblock of all. How to detect it: if a test "disappeared" from the history without a fix that justifies it, it was hidden, not solved. How to fix it: deleting kills the evidence —if the flaky was hiding a real bug, now there is nothing to watch it—. Quarantine with xfail and a ticket keeps the test alive and visible; the sick passenger is moved to a different row, not thrown off the plane.

Leaving the quarantine forever. What happens: the xfail with "ticket RES-412" has been there for eight months, the ticket is closed-by-inactivity, and nobody remembers why. Why it happens: quarantine unblocks today, and without a periodic review the "temporary" becomes permanent by inertia. How to detect it: review the dates of your xfail; any one older than a few weeks without movement is suspicious. How to fix it: rule 2 —quarantine with an expiration and review—; schedule a recurring review of the isolated tests, and treat the quarantine list as debt that must go down, not as a filing cabinet that only grows. A cured flaky (lesson 7) comes out of quarantine; a forgotten one turns it into a dumpster.

Exercises

Exercise 1 — Read the outcomes. A test under @pytest.mark.xfail(strict=False) is run four times in the gate. The results of the progress line are: x, X, x, x. (a) How many times did the test fail and how many did it pass? (b) How many of those four runs broke the build? (c) What does the pattern tell you about the state of the flaky?

See solution

• (a) The lowercase x is XFAIL (the test failed, as expected); the uppercase X is XPASS (the test passed, unexpected). The pattern x, X, x, x means it failed 3 times and passed 1 time.
• (b) None. With strict=False, neither XFAIL nor XPASS breaks the build. The four runs left the gate green —that is the whole point of quarantine: the flaky no longer blocks anyone, however it lands—.
• (c) The flaky is still alive and still flickering (3 failures, 1 pass): it did not fix itself nor degenerate into always-red. If you had seen X, X, X, X (always xpass), it would be a sign that maybe it now always passes —a candidate to take out of quarantine and verify—. If you had seen x, x, x, x (always xfail), it would be a sign that it stopped being flaky and became always-red —a possible new deterministic bug, to investigate—. The value of xfail over skip: it lets you see these signals because the test keeps running.

Exercise 2 — Marker vs. xfail. For each situation, say whether @pytest.mark.flaky(reruns=3) or @pytest.mark.xfail(strict=False) is preferable, and why. (a) A flaky that fails 1 in 50 times and almost never blocks, but occasionally bothers. (b) A flaky that fails ~50% and blocks constantly, which you need out of the gate while you investigate its cause in depth.

See solution

• (a) @pytest.mark.flaky(reruns=3). A 1/50 flaky almost never fails, so a couple of per-test retries rescue it practically always (failing twice in a row is 1/2500) without taking it out of the gate. The test stays inside the gate and keeps really verifying its claim most of the time; the marker only absorbs the rare stumble. Taking it out with xfail would be excessive —you would lose the real verification over a failure that almost never happens—.
• (b) @pytest.mark.xfail(strict=False). A 50% flaky that blocks constantly is not tamed with retries —even --reruns 3 leaves it blocked 6% of the time, and you would be retrying half a suite of runs—. You need to take it completely out of the gate while you investigate, and for that xfail(strict=False) is forceful: fail or pass, it does not block, and it stays visible with its ticket. It is real quarantine, not the retry.

The rule: flaky(reruns=N) for the rare stumble you want to absorb without taking out of the gate; xfail(strict=False) for the serious flaky you need out of the gate while you cure it. Both are triage; lesson 7 is the cure that takes both out.

Exercise 3 — Healthy quarantine or dumpster. You review a team's suite and find this in three tests. Classify each one as healthy quarantine or dumpster, and say what it is missing or has in excess. (a) @pytest.mark.xfail(reason="flaky, RES-412, review in sprint 14", strict=False), added a week ago. (b) @pytest.mark.xfail(strict=False) without a reason, without a date, in the history for 10 months. (c) The test simply commented out with # this fails sometimes, I left it off.

See solution

• (a) Healthy quarantine. It has the three things: a ticket (RES-412), a review plan (sprint 14), and strict=False so as not to break the build. It is the passenger moved to a different row, labeled and with follow-up. It is missing nothing; you just have to fulfill the promised review.
• (b) Dumpster disguised as quarantine. strict=False takes it out of the gate, yes, but without a reason, without a ticket, and 10 months old: nobody knows why it is isolated or when it will be attended to. It is a dead test pretending to be under observation. It is missing a ticket, a reason, and —above all— a review that cures it (lesson 7) or escalates it. The age is the alarm.
• (c) Pure dumpster / hidden test. Commenting the test out takes it out of the gate by killing all visibility: it does not run, is not counted, is not tracked, and if it was hiding a real bug, nothing watches it. It has the comment in excess and is missing being an xfail with a ticket (to stay visible) or a real fix. It is throwing the passenger off the plane.

The unified criterion: healthy quarantine = out of the gate + visible + with a ticket + temporary. Remove any of those four and you have a dumpster. (b) and (c) fail in visibility and/or follow-up; (a) fulfills them.

Summary and next step

In this lesson you learned quarantine: isolating a flaky from the gate without deleting it or hiding it, like the sick passenger who is moved to a different row —neither ignored nor thrown off the plane, but separated, marked, and with follow-up—. You saw two forms executed for real: @pytest.mark.flaky(reruns=3), which retries only that test (output RERUN → PASSED, 1 passed, 1 rerun, without global --reruns); and @pytest.mark.xfail(reason=..., strict=False), which takes it out of the gate completely —whether it lands on XFAIL (failed, expected) or XPASS (passed, unexpected), neither breaks the build—, leaving it visible with its reason and its ticket. And why xfail observes where skip blinds.

Above all, the discipline that separates quarantine from dumpster: always with a ticket, always temporary, always counted. A well-done quarantine unblocks the team without lying and keeps the problem in sight with a plan; badly done, it hides trash under a rug that looks clean.

Before moving on you should be able to: mark a flaky with flaky(reruns=N) and with xfail(strict=False); read xfailed/xpassed and know why neither breaks the build; choose between the marker and xfail according to the severity of the flaky; and apply the three rules of healthy quarantine.

What follows, in lesson 5, is the other half of the module: the flaky that only happens in CI. Up to here we worked a flaky that flickers the same on your machine and on the runner (the clock). But many flaky are green on your laptop, always, and only turn red in CI —because of the different execution order, the runner's parallelism, its time zone, an absent file—. You are going to see the catalog of those causes and, with a real demo of two Reservo tests that share a Calendar, the classic CI-only: green in one order, red in another.

Resources

• xfail and skip — pytest documentation — the official reference for @pytest.mark.xfail, the strict parameter, and the difference from skip. Read the xfail section: it explains exactly why strict=False is the right one for a flaky.
• pytest-rerunfailures — GitHub repository — the documentation of the per-test @pytest.mark.flaky(reruns=N) marker, the quarantine-by-retry of this lesson. The authors also warn about not leaving it permanent, rule 2 of quarantine.
• -r for the test summary — pytest documentation — how -rxX makes the summary list the XFAIL and XPASS with their reason, the visibility that makes a quarantine auditable. Without it, the isolated tests are lost from sight.
• Flaky tests — pytest documentation — the conceptual frame: pytest recommends isolating and tracking flaky, not ignoring or deleting them. The quarantine of this lesson is that recommendation put into practice.