Module 5: Fast Ci Caching And Parallelism

6. Isolation: the requirement for parallelizing

Next →

Description

Lesson 4 gave you parallelism and a fivefold speedup; lesson 5 gave you the way to apply it where it hurts. Both assumed, silently, something we now have to make explicit and demonstrate: that the tests can be distributed across workers without getting in each other's way. That assumption doesn't always hold. There's a class of test that passes perfectly serially —one after another, in a single process— and breaks as soon as you distribute it, not by chance but deterministically, because it depends on something another test left behind. Isolation is the property that guarantees this doesn't happen, and it's the precondition for parallelism: without it, -n auto doesn't speed up, it corrupts.

By the end you'll understand why tests must be independent to run in parallel, and you'll see it executed for real: a Reservo test with a Calendar shared between tests passes serially (3 passed) and fails under -n 3 (2 failed, with assert 1 == 2 and assert 1 == 3), because each worker is a separate process with its own memory and the global state doesn't travel between them. You'll understand exactly why it fails, and you'll fix it: replace the shared state with a fixture that gives each test a fresh Calendar, so it passes serially and in parallel. And you'll fix a critical distinction: this failure is not a flaky —it fails every time you parallelize, not sometimes—; it's an isolation problem, and its remedy is from here, not from module 7.

Connection to the module: this is the hinge lesson. Lesson 4 started parallelism; lesson 6 explains what requirement your suite must meet for that parallelism to work. It's also the one most tied to the concept of shared state and order that runs through all of testing: a well-isolated test doesn't depend on the order it runs in or on what another test did. Boundary, again, because it's easy to get confused: the failure you'll see is deterministic under -n —it always happens—, so it's an isolation defect (this module). A flaky is an intermittent failure with no apparent cause —sometimes yes, sometimes no—, and its treatment (retry, quarantine) is module 7. "Every time I parallelize" ≠ "sometimes for no reason". Don't confuse the requirement you learn here with the problem dealt with there.

The cooks who shared the same cutting board

Imagine two cooks who share a single cutting board. Working in turns —one chops, finishes, cleans, and passes the board to the other— the system works: each uses the board when it's their turn, finds what the previous one left, and everything flows. It's slow (they go one at a time), but correct.

Now put the two cooks to work at the same time, each at their own station, and tell them to keep using "the board". The problem is obvious: there's only one board, and they can't both use it at once. If you give each a board so they can work in parallel, a subtler problem appears: the second cook's recipe said "use the onion that's already chopped on the board" —taking for granted the first left it there—, but now the second has their own board, empty, without the first's onion. Their step fails, not because they cook badly, but because they depended on a state another left on a shared resource, and once separated that state is no longer there.

That is, exactly, the trap of parallelism in tests. Running serially is one cook using the board in turns: each test finds what the previous one left in the global state, and everything passes. Running in parallel with pytest-xdist is giving each worker its own station with its own board: each worker is a separate Python process, with its own memory, its own global variables. A test that said "use what the previous test left in the global variable" now runs in a worker where that variable is empty —the "previous test" ran in another worker, in another memory—, and it fails. It doesn't fail by chance: it fails every time you distribute it, because the dependency it assumed is never met between distinct processes.

Each pytest-xdist worker is a separate process with its own memory. A test that depends on the state another test left in a global variable passes serially (same memory, in turns) and fails in parallel (distinct memories). Isolation —each test building its own world— is what makes distributing possible.

The anti-pattern: a shared Calendar

Let's see it in Reservo, with an example that's a real and common mistake. Someone writes three tests for the Calendar, and —to "save" creating one in each test— uses a single module-level Calendar, shared by all three:

# isolation_demo/test_shared_calendar.py
from datetime import datetime, timedelta

from reservo.calendar import Calendar
from reservo.models import Room, Member
from reservo.schedule import book

ROOM = Room(id="r1", name="Focus", capacity=4, hourly_cents=2500)
PRO = Member(id="m2", name="Ben", tier="pro")
DAY = datetime(2026, 8, 1, 9, 0)

# The mistake: ONE calendar reused by each test, instead of a fresh one each time.
shared_cal = Calendar()


def test_first_booking_is_recorded():
    book(shared_cal, ROOM, PRO, DAY, DAY + timedelta(hours=1), hours=1)
    assert len(shared_cal.all_bookings()) == 1


def test_second_booking_is_recorded():
    book(shared_cal, ROOM, PRO, DAY + timedelta(hours=1), DAY + timedelta(hours=2), hours=1)
    assert len(shared_cal.all_bookings()) == 2


def test_third_booking_is_recorded():
    book(shared_cal, ROOM, PRO, DAY + timedelta(hours=2), DAY + timedelta(hours=3), hours=1)
    assert len(shared_cal.all_bookings()) == 3

Notice the trap. shared_cal = Calendar() is created once, on module import, and the three tests reuse it. Each test adds a booking and asserts how many there are in total: the first expects 1, the second expects 2, the third expects 3. But the second only reaches 2 if the first ran before in the same process and left its booking in shared_cal; the third only reaches 3 if the two previous ones ran before, right there. Each test depends on the state the previous one left in the shared variable. It's the onion the first cook left on the board.

Serially it works, because pytest runs the tests in order, one after another, in a single process —a single board, in turns—:

python -m pytest isolation_demo/test_shared_calendar.py

What to expect (real output):

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /private/tmp/reservo-m5
configfile: pyproject.toml
plugins: xdist-3.8.0
collected 3 items

isolation_demo/test_shared_calendar.py ...                               [100%]

============================== 3 passed in 0.01s ===============================

3 passed. Green. The first leaves a booking, the second finds it and adds its own (2), the third finds the two and adds its own (3). Everything adds up... as long as there's a single board.

The failure under -n: the real demo

Now run exactly the same three tests, without changing a line, distributed across three workers with -n 3:

python -m pytest isolation_demo/test_shared_calendar.py -n 3

What to expect (real output):

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /private/tmp/reservo-m5
configfile: pyproject.toml
plugins: xdist-3.8.0
created: 3/3 workers
3 workers [3 items]

.FF                                                                      [100%]
=================================== FAILURES ===================================
________________________ test_third_booking_is_recorded ________________________
[gw2] darwin -- Python 3.14.0 /private/tmp/reservo-m5/.venv/bin/python

    def test_third_booking_is_recorded():
        book(shared_cal, ROOM, PRO, DAY + timedelta(hours=2), DAY + timedelta(hours=3), hours=1)
>       assert len(shared_cal.all_bookings()) == 3
E       AssertionError: assert 1 == 3
E        +  where 1 = len([Booking(id='bk-1', ...)])

isolation_demo/test_shared_calendar.py:35: AssertionError
_______________________ test_second_booking_is_recorded ________________________
[gw1] darwin -- Python 3.14.0 /private/tmp/reservo-m5/.venv/bin/python

    def test_second_booking_is_recorded():
        book(shared_cal, ROOM, PRO, DAY + timedelta(hours=1), DAY + timedelta(hours=2), hours=1)
>       assert len(shared_cal.all_bookings()) == 2
E       AssertionError: assert 1 == 2
E        +  where 1 = len([Booking(id='bk-1', ...)])

isolation_demo/test_shared_calendar.py:30: AssertionError
=========================== short test summary info ============================
FAILED isolation_demo/test_shared_calendar.py::test_third_booking_is_recorded - AssertionError: assert 1 == 3
FAILED isolation_demo/test_shared_calendar.py::test_second_booking_is_recorded - AssertionError: assert 1 == 2
========================= 2 failed, 1 passed in 0.28s ==========================

Read the verdict: 2 failed, 1 passed. The same three tests that serially gave 3 passed now break under -n 3. And read how they break, because it tells the whole story:

  • test_second_booking_is_recorded: AssertionError: assert 1 == 2. It expected to find 2 bookings in shared_cal, but found 1 —the one it added itself—. The first test's booking wasn't there, because the first test ran in another worker (gw0), with another copy of shared_cal.
  • test_third_booking_is_recorded: AssertionError: assert 1 == 3. It expected 3, found 1 —only its own—. The two previous ones ran in other workers, in other memories.

The [gw1] and [gw2] labels are the definitive clue: pytest-xdist distributes each test to a different worker (gw0, gw1, gw2 are the "group workers"). Each worker imported the module on its own, so each one executed shared_cal = Calendar() separately and has its own empty calendar. When test_second runs in gw1, its shared_cal only has the booking it added —one—, not the first test's, which lives in gw0's memory. The assert 1 == 2 is the exact portrait of the onion that's not on the second cook's board.

And something crucial: this isn't chance. Run -n 3 ten times and it'll fail all ten, because the dependency between tests is never met when each lives in a distinct process. It's a deterministic failure of parallelism, caused by a poorly isolated test. It's not a flaky.

The fix: a fixture that gives a fresh world

The cure is to remove the shared state: have each test build its own Calendar, without depending on any other. pytest has the exact tool, a fixture: a function that prepares something fresh for each test that asks for it.

# isolation_demo/test_isolated_calendar.py
from datetime import datetime, timedelta

import pytest

from reservo.calendar import Calendar
from reservo.models import Room, Member
from reservo.schedule import book

ROOM = Room(id="r1", name="Focus", capacity=4, hourly_cents=2500)
PRO = Member(id="m2", name="Ben", tier="pro")
DAY = datetime(2026, 8, 1, 9, 0)


@pytest.fixture
def cal():
    # A new, empty Calendar for EACH test.
    return Calendar()


def test_one_booking_is_recorded(cal):
    book(cal, ROOM, PRO, DAY, DAY + timedelta(hours=1), hours=1)
    assert len(cal.all_bookings()) == 1


def test_two_bookings_are_recorded(cal):
    book(cal, ROOM, PRO, DAY, DAY + timedelta(hours=1), hours=1)
    book(cal, ROOM, PRO, DAY + timedelta(hours=1), DAY + timedelta(hours=2), hours=1)
    assert len(cal.all_bookings()) == 2


def test_three_bookings_are_recorded(cal):
    for i in range(3):
        book(cal, ROOM, PRO, DAY + timedelta(hours=i), DAY + timedelta(hours=i + 1), hours=1)
    assert len(cal.all_bookings()) == 3

Look at the two differences, which are the whole lesson. First: the cal fixture —marked with @pytest.fixture— returns a new Calendar() each time a test asks for it (by putting it as a parameter: def test_...(cal)). There's no module-level shared_cal; each test receives its own calendar, freshly created, empty. Second: each test builds all the state it needs. The one that wants to test two bookings makes the two bookings itself, in its own cal, and asserts 2; it doesn't take for granted that another test left the first. Each test is self-sufficient: its own cutting board, its own onion.

Serially it passes, as before. And now, under -n 3:

python -m pytest isolation_demo/test_isolated_calendar.py -n 3

What to expect (real output):

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /private/tmp/reservo-m5
configfile: pyproject.toml
plugins: xdist-3.8.0
created: 3/3 workers
3 workers [3 items]

...                                                                      [100%]
============================== 3 passed in 0.24s ===============================

3 passed, in parallel. Now it doesn't matter which worker each test falls in: each creates its own fresh Calendar and verifies what it built itself, without depending on anyone. Distributing them across three processes no longer breaks anything, because there was no state to distribute. That's isolation: a test that passes the same serially as in parallel, in any order, in any worker, because it shares a board with no one.

Why isolation is the precondition, not a luxury

It's worth saying it directly: isolation isn't an optimization of parallelism, it's its requirement. You can't "parallelize and then, if anything, isolate". If your tests share state, -n auto doesn't speed them up —it corrupts them, as you just saw—, and the only sane way out is to isolate them first.

The good news is that isolation is, moreover, a property you want anyway, parallelism or no. A test that depends on what another left is fragile for many reasons beyond xdist: it breaks if you reorder the tests, if you delete one in the middle, if you run one alone to debug it (and suddenly "it fails alone" because it was missing the state another gave it). Well-isolated tests are more robust, easier to read (each tells its complete story), and easier to debug (they fail for their own cause, not a neighbor's). Parallelism simply exposes without mercy the lack of isolation that was already a latent problem. -n auto is, among other things, an excellent detector of poorly isolated tests.

That's why the module's order is this: first parallelism (lesson 4), which gives you the speedup and, along the way, the detector; then isolation (this lesson), which is the requirement to collect that speedup without breaking anything. If -n auto discovers tests that fail, don't turn off parallelism: thank it for the diagnosis and isolate.

Common mistakes

Blaming xdist ("parallelism is unstable") instead of the poorly isolated test. What happens: someone turns on -n auto, sees red tests that passed serially, and concludes "parallel tests aren't reliable, better run them serially". Why it happens: it's easier to blame the new tool than to suspect your own code. How to spot it: if the failure is deterministic —it happens every time you parallelize, with assert 1 == 2 or similar over state that's not there—, it's not the tool, it's a test that shares state. How to fix it: isolate the test (fixture with fresh state per test) instead of turning off parallelism. Going back to serial hides the problem; it doesn't fix it, and on top of that you give up the speedup.

Confusing this deterministic failure with a flaky. What happens: someone sees the red under -n and catalogs it as flaky —"it fails sometimes"—, and goes looking for the solution in module 7's retry. Why it happens: "it fails in some runs and not others" (depending on whether you run serially or in parallel) seems like intermittence. How to spot it: the test is whether the failure is reproducible: serially it passes always, in parallel it fails always. That's deterministic, not flaky. A real flaky fails unpredictably with the same command. How to fix it: treat the failure under -n as an isolation problem (this lesson), not a flakiness one (module 7); the retry wouldn't fix it, only hide it.

Sharing state "to save" and creating order dependencies. What happens: someone uses a module-level object (a Calendar, a list, a connection) reused by several tests "so as not to create it each time", and inadvertently weaves a chain where each test depends on what the previous one left. Why it happens: creating fresh state in each test seems like a waste, and sharing feels efficient. How to spot it: if a test asserts something about a state that another test built (like assert len == 2 when it only added one), you have an order dependency. How to fix it: give each test its own state with a fixture; the cost of creating an empty Calendar per test is negligible, and in exchange you gain isolated tests that run in any order and in parallel.

Exercises

Exercise 1 — Explain the assert 1 == 2. Under -n 3, test_second_booking_is_recorded failed with AssertionError: assert 1 == 2. Explain, in terms of processes and memory, why it found 1 booking instead of 2, and why the same test passed serially.

See solution

The test expected 2 bookings because it assumed test_first_booking_is_recorded ran before and left its booking in shared_cal. Under -n 3, each worker is a separate Python process with its own memory. test_second ran in worker gw1, which imported the module on its own and therefore executed shared_cal = Calendar() in its memory, getting an empty calendar. The first test ran in another worker (gw0), with another copy of shared_cal, and its booking stayed in that memory —unreachable for gw1—. So when test_second added its booking and counted, it found 1 (its own), not 2, and the assert 1 == 2 failed.

Serially it passed because the three tests ran in the same process, in order: a single copy of shared_cal, shared in turns. The first left its booking, the second found it (2), the third found the two (3). Same memory = the state travels from one test to the next. Distinct processes = it doesn't travel. That's the whole difference.

Exercise 2 — Isolate the test. A teammate has this test that passes serially and fails under -n. Rewrite it with a fixture so it passes in both, and explain what you removed.

totals = []

def test_january_total():
    totals.append(30000)
    assert sum(totals) == 30000

def test_year_total():
    totals.append(24000)
    assert sum(totals) == 54000
See solution

The problem is the module-level totals list, shared: test_year_total expects 54000 only if test_january_total ran before and left its 30000 in the list. Under -n, each worker has its own empty totals, so test_year_total sees only its 24000 and fails (assert 24000 == 54000).

Rewritten, each test builds its own state:

import pytest

@pytest.fixture
def totals():
    return []  # a new, empty list for each test

def test_january_total(totals):
    totals.append(30000)
    assert sum(totals) == 30000

def test_year_total(totals):
    totals.append(30000)  # this test builds the data it needs
    totals.append(24000)
    assert sum(totals) == 54000

What I removed: the shared global totals list. What I put in: a totals fixture that returns an empty list per test, and —key— I made test_year_total build itself the two totals it wants to sum, instead of depending on another test leaving the first. Now each test is self-sufficient: it passes serially, in parallel, and in any order, because it shares state with no one.

Exercise 3 — Isolation or flaky? For each situation, say whether it's an isolation problem (this lesson) or a flaky (module 7), and justify with the reproducibility test. (a) "A test passes serially always and fails under -n auto always, with assert 0 == 1." (b) "A test fails one out of every twenty runs of the same command, with no pattern." (c) "I run a test alone to debug it and it fails, but it passes when I run the whole suite."

See solution
  • (a) Isolation. The test: it passes serially always and fails in parallel always. It's deterministic (100% reproducible depending on the mode), so it's a test that shares state with another, and its cure is to isolate it (fixture). It's not flaky.
  • (b) Flaky (module 7). The test: it fails unpredictably with the same command, with no pattern —one out of every twenty—. That's genuine intermittence, the definition of flaky, and its treatment (diagnose the source of non-determinism, maybe retry or quarantine) is module 7's.
  • (c) Isolation. The test: the result depends on which other tests run with it —it fails alone, passes accompanied—. That means it depended on the state another test left; run alone, that state isn't there. It's the same as the failure under -n, seen from another angle, and the cure is the same: isolate. (A well-isolated test passes the same alone as accompanied.)

The rule that separates the two categories: if the failure is reproducible based on a clear condition (serial vs parallel, alone vs accompanied), it's isolation —from here—. If it's unpredictable with the same command, it's flaky —from module 7—.

Summary and next step

In this lesson you demonstrated, by executing for real, parallelism's precondition: isolation. You saw it with the two cooks and the cutting board: in turns, one finds what the other left and everything flows; in parallel, with a board each, the one who depended on the other's onion is left without it. A module-level shared Calendar made three tests pass serially (3 passed) and broke them under -n 3 (2 failed, assert 1 == 2 and assert 1 == 3), because each worker is a process with its own memory and the global state doesn't travel between processes —the [gw1], [gw2] labels gave it away—. And you fixed it: a fixture that gives a fresh Calendar per test made the same cases pass serially and in parallel (3 passed with -n 3), because there was no state to distribute anymore.

You fixed two ideas that hold for your whole testing career. One: isolation isn't a luxury of parallelism, it's its requirement —and a property you want anyway, because it makes tests robust, readable, and debuggable—. Two: a deterministic failure under -n is an isolation problem (from here), not a flaky (module 7); the test is reproducibility. -n auto is, as a bonus, a relentless detector of poorly isolated tests: if it discovers reds for you, thank it for the diagnosis and isolate, don't turn off parallelism.

Before moving on you should be able to: explain why a test with shared state passes serially and fails in parallel, in terms of processes and memory; recognize the anti-pattern of the module-level shared object; fix it with a fixture that gives fresh state per test; and distinguish an isolation failure (deterministic) from a flaky (intermittent).

What's next, in lesson 7, is putting a price on all this speedup. Parallelizing isn't free: each worker consumes CPU, and in CI the bigger runners cost more money. You're going to see the real return curve —-n 2, -n 4, -n 8, -n 12, measured— and how it flattens: there comes a point where adding workers barely speeds up but does make it more expensive. You're going to learn how much to parallelize with a clear head, why the cache almost always pays off while parallelism is weighed, and when -n auto is too much. Isolation gave you permission to parallelize; the trade-off tells you how much.

Resources

← Previous5. Splitting the suite: fast and slowNext →7. The speed/cost trade-off