Module 7: Test Data And Isolation In Integration

2. Why shared real state contaminates

Next →

Description

The previous lesson showed you the problem; this one explains it to the bone, because understanding the cause precisely is what lets you choose the cure well. You're going to see, in slow motion, why a real database shared between tests contaminates them, and why the same pattern with a fake never does. The answer isn't "SQLite is weird" or "fakes are magic": it's a concrete and physical difference in where the state lives and when it dies. In the fake, the state is a dict that's an attribute of a Python instance; it's born when you instantiate the repository and dies when the garbage collector takes the instance away, when the test ends. In the real repository, the state is rows in a SQLite table inside a connection; it's born when you commit and doesn't die until you delete the rows or destroy the database —of course, it doesn't die just because a test ends—. That asymmetry, and only that, is the root of every state-based integration flake.

And you're going to put a name to the problem's signature, because recognizing it in the moment saves you hours of debugging the wrong place. A test contaminated by shared state has an unmistakable signature: it passes when you run it alone and fails when you run it in the suite, and its verdict changes depending on the order. That signature is exactly the one the test-failure-diagnosis module taught you to read: it's not a bug in the test's logic —that's fine—, it's a hidden dependency on what another test left behind. When you see it, don't debug the test that fails; suspect the state. This lesson installs that reflex in you with real output, so that the next time an integration suite goes intermittent, you know exactly what to look at.

Connection to the module: lesson 1 gave you the map and the destination; this one nails the diagnosis —why it contaminates and how it's recognized— before lessons 3 and 4 give you the cures. It's deliberate: a cure you don't understand is applied badly. Here you're going to understand that the problem isn't the real resource itself —the real thing is good, it's what gives integration its value— but the real resource shared without isolation; that the solution isn't "don't use the database" but "give each test a known state"; and that the order-dependent signature is the thread that connects this module with failure diagnosis. With that clarity, lesson 3's rollback and lesson 4's fixture stop being recipes and become two ways of achieving the same thing: cutting the inheritance of state between tests.

Analogy: the whiteboard nobody erases

Imagine a classroom with a single whiteboard, used by teachers who take turns throughout the day, and an unwritten rule: everyone erases their own when done. While everyone complies, it works: each class starts with a clean whiteboard and what appears on it belongs to that class. But one day a teacher leaves in a hurry and leaves their equations written. The next enters, starts explaining geography, and their students see, mixed with the map, some equations that make no sense. If a student asks "where did this come from?", the geography teacher doesn't know: they didn't write it. The state —what's on the whiteboard— came from another class, and contaminates the current one without anyone in the current one having put it there.

Note two things about this analogy, because they're the heart of the lesson. First: the problem isn't the whiteboard —a whiteboard is useful, permanent, and that's why it works—; the problem is sharing it without erasing it between uses. Second: the disaster depends on the order. If the geography teacher had entered before the math one, they wouldn't have seen equations. Their class went badly not because of what they did, but because of who passed through the whiteboard before. A new sheet of paper for each class —the equivalent of the fake's dict— would never have this problem, because each class starts on a blank sheet that's thrown away at the end. The integration database is the shared whiteboard: powerful, permanent, and dangerous if you don't erase it —or don't give each class its own sheet— between one use and the next.

The state's lifecycle, side by side

Let's put the two implementations one next to the other and follow, step by step, what happens to the state in each one over two tests.

With the fake, the state is an instance attribute:

class FakeBookingRepository:
    def __init__(self):
        self._store = {}                  # (1) born here, empty
    def save(self, booking):
        self._store[booking.id] = booking # (2) grows here
    def find_by_room(self, room_id):
        return [b for b in self._store.values() if b.room_id == room_id]

Follow the trail when two tests each do repo = FakeBookingRepository():

  1. Test A calls FakeBookingRepository(). An instance is created, with an empty _store. Test A saves a booking: _store has one entry. Test A ends; the variable repo goes out of scope.
  2. Python collects A's instance —nobody references it—, and with it goes the _store and its single entry. The state died with the test.
  3. Test B calls FakeBookingRepository(). A new instance is created, with a new, empty _store. Test B sees nothing of A, because A's _store no longer exists.

The isolation is a free side effect of the state living in a Python object whose lifecycle coincides with the test's. Now the real repository:

class SqliteBookingRepository:
    def __init__(self, connection):
        self._conn = connection           # (1) the connection comes from OUTSIDE
        self._conn.execute(SCHEMA)
    def save(self, booking):
        self._conn.execute("INSERT ... ON CONFLICT ...", (...))
        self._conn.commit()               # (2) the row stays PERMANENT in the table

The decisive difference is in line (1): the connection comes from outside, you pass it in when constructing the repository. The state doesn't live in the repository; it lives in the database on the other side of that connection. Follow the trail with two tests that share the same connection conn:

  1. Test A uses a repository over conn. It saves a booking; the commit writes it to the bookings table, permanent. Test A ends; its repository goes out of scope and Python collects it.
  2. But the connection conn is not collected —it lives outside, referenced by the module— and the bookings table still has A's row. The state survived the test.
  3. Test B uses another repository, but over the same conn. Its __init__ does CREATE TABLE IF NOT EXISTS —which does nothing, the table already exists— and queries: it sees A's booking. It inherited the state.

The asymmetry is total. In the fake, killing the test kills the state. In the real one, killing the test doesn't touch the database; the state lives while the connection lives, and if you share the connection, the state is shared. There's nothing mystical: it's where the state lives and when it's freed.

Worked example: the contamination in slow motion

Let's set up the smallest case that shows the contamination clearly and diagnose it as the test-failure-diagnosis module would. Two tests share a real repository. The first saves a Focus booking; the second assumes Focus starts without bookings for it.

# tests/test_shared_repo.py — two tests, one shared real DB
import sqlite3
from datetime import datetime
from reservo.models import Booking
from reservo.sqlite_repo import SqliteBookingRepository

# A single connection and a single repo for the WHOLE module. Real state alive.
_conn = sqlite3.connect(":memory:")
REPO = SqliteBookingRepository(_conn)

def make_booking(bid, room_id="focus"):
    return Booking(id=bid, room_id=room_id, member_id="m-ana",
                   start=datetime(2026, 3, 10, 9), end=datetime(2026, 3, 10, 12),
                   status="confirmed", price_cents=6000)

def test_focus_has_one_after_i_save():
    REPO.save(make_booking("bk-1"))
    assert len(REPO.find_by_room("focus")) == 1   # I saved one

def test_focus_starts_empty_for_me():
    # I assume the Focus room starts without bookings for my test.
    assert len(REPO.find_by_room("focus")) == 0   # <- inherits bk-1 from the previous test

Let's run the whole suite and read the failure with a magnifying glass.

What to expect. On my machine (Python 3.14.0, pytest 9.1.1):

python3 -m pytest tests/test_shared_repo.py -v
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 2 items

tests/test_shared_repo.py::test_focus_has_one_after_i_save PASSED          [ 50%]
tests/test_shared_repo.py::test_focus_starts_empty_for_me FAILED           [100%]

=================================== FAILURES ===================================
________________________ test_focus_starts_empty_for_me ________________________

    def test_focus_starts_empty_for_me():
        # I assume the Focus room starts without bookings for my test.
>       assert len(REPO.find_by_room("focus")) == 0   # <- inherits bk-1 from the previous test
E       AssertionError: assert 1 == 0
E        +  where 1 = len([Booking(id='bk-1', room_id='focus', member_id='m-ana', start='2026-03-10T09:00:00', end='2026-03-10T12:00:00', status='confirmed', price_cents=6000)])
E        +    where [Booking(id='bk-1', ...)] = find_by_room('focus')

Read the failure like a detective, not like someone about to fix the test. The assertion that blows up is 1 == 0, and pytest shows you what that 1 is: a list with one booking of id='bk-1'. But look at the body of test_focus_starts_empty_for_me: it never saves a bk-1 booking. The bk-1 was saved by test_focus_has_one_after_i_save, the test above. The failure appears in the second test, but the datum that causes it was put there by the first. That's the mark of contamination: the object that breaks your test carries another's fingerprints.

Now the decisive diagnostic test, the same one you learned in the test-failure-diagnosis module: run the test that fails, alone.

python3 -m pytest tests/test_shared_repo.py::test_focus_starts_empty_for_me -v
============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 1 item

tests/test_shared_repo.py::test_focus_starts_empty_for_me PASSED           [100%]

============================== 1 passed in 0.01s ===============================

Green. The same test, without changing a line, passes when it runs alone and fails when it runs after its neighbor. That difference —passes alone, fails in the suite— is the verdict: the problem isn't in test_focus_starts_empty_for_me, whose logic is flawless, but in the state it inherits from whoever ran before. Debugging the second test would be chasing a ghost; the failure is that the database is shared and nobody cleaned it between one and the other.

The signature: why this is textbook flakiness

It's worth precisely naming why this is the most common integration flake, because the signature is what lets you recognize it fast.

A test is flaky when its result depends not only on its code, but on something external that varies. Shared-state contamination fits perfectly: the second test's result depends on whether the first ran before, and that varies with the execution order, with which tests you select, with whether someone added a new test that now runs first. Three symptoms, all with the same root:

  • Passes alone, fails in the suite. Run in isolation there's nobody to contaminate it; in the suite, there is. It's the diagnostic test we just did.
  • Changes with the order. As you saw in the lesson 1 exercise, reversing the order of two tests reverses which one fails. A result that depends on the order is the operational definition of "not isolated".
  • Breaks when the suite grows. The test passed for months. Someone adds a new test, which runs before and leaves a booking, and suddenly your test —without you touching it— starts failing. The cause is meters away, in code you didn't write.

When you see any of these three in an integration suite, your first hypothesis must be shared state that isn't isolated, not a bug in the test that fails. And your first command, run that test alone. If it passes alone, you stopped looking in the wrong place and gained the hours that go into debugging logic that's fine. This is the hinge with the test-failure-diagnosis module: those symptoms you learned to read there have, in integration, a dominant cause with its own name.

Why the cure isn't "stop using the real thing"

A tempting reflex, on seeing all this, is to conclude that integration is a problem and it's better to go back to the fakes. It's the wrong diagnosis. The value of integration —what you saw in modules 5 and 6— is precisely that it tests against the real thing, with its rules of serialization, types, and persistence; giving that up is giving up catching the class of bugs no fake catches. The problem isn't the real thing; it's the real thing shared without isolation.

The cure, then, isn't removing the database, but giving each test a known state: either a new database per test (lesson 4's fixture), or reverting what the test wrote before moving to the next (lesson 3's rollback). Both achieve the same thing —cutting the inheritance— by different paths, and choosing between them is the topic of the lessons that follow. What this lesson leaves you firm on is the diagnosis: you know what contaminates, where the state that causes it lives, and how to recognize the signature. With that, the cures stop being memorized recipes and become well-founded decisions.

Common mistakes

Debugging the test that fails instead of the one that contaminates. What happens: the failure comes out in test B, so B is debugged, whose logic is perfect, for an hour. Why it happens: the error is reported where the assertion blows up, not where the guilty state was created. How to detect it: run B alone; if it passes, B isn't to blame. Look at what object breaks the assertion and which test it came from (in the example, the bk-1 that B never saved). How to fix it: chase the test that leaves the state, not the one that finds it; and better yet, isolate, so that none leaves anything.

Confusing "in memory" with "ephemeral per test". What happens: :memory: is used and it's assumed each test starts clean by being memory. Why it happens: "memory" sounds like it gets erased. How to detect it: a :memory: database lives as long as its connection; if the connection is module-level (like the example's _conn), the database is a single one for the whole suite and contaminates just like a file. How to fix it: the isolation isn't given by the resource type but by the lifecycle —a new connection per test, or a rollback between tests—.

Believing that if it passes on your machine, it's isolated. What happens: the suite passes locally and it's assumed it's fine. Why it happens: locally the tests usually run in the same order (the definition order), and that order can hide the contamination. How to detect it: run the suite in another order —reverse two tests, or select a subset— and observe whether the verdict changes. How to fix it: lesson 7 makes this systematic; for now, don't trust that "passes" equals "isolated" until it passes in more than one order.

Exercises

Exercise 1 — Read the failure like a detective. You're shown this fragment of a pytest failure in an integration suite:

E       AssertionError: assert 3 == 1
E        +  where 3 = len([Booking(id='bk-seed-1', ...), Booking(id='bk-seed-2', ...), Booking(id='bk-mine', ...)])

The test that fails only saves one booking, bk-mine. Without more context, what's happening and what's your first diagnostic command?

See solution

Shared-state contamination is happening. The test that fails saves a single booking (bk-mine) and expects to see 1, but find returns 3: its own plus bk-seed-1 and bk-seed-2, two bookings with names (bk-seed-...) that suggest another test —or a badly isolated seeding fixture— left them. The object that breaks the assertion carries foreign fingerprints: two ids this test never created.

The first diagnostic command is to run that test alone: pytest path::failing_test -v. If it passes alone, it's confirmed that the cause isn't its logic but the state inherited from its neighbors, and you have to look for who leaves bk-seed-1 and bk-seed-2 without cleaning them —or, better, isolate the suite so nobody inherits anything—. If it also failed alone, then it would indeed be a bug of the test itself, and the diagnosis would change completely. That fork —passes alone versus fails alone— is the first knot to untie in every integration failure.

Exercise 2 — The state that lives outside. Explain why, in the real repository, it's the line self._conn = connection (receiving the connection from outside) —and not the commit itself— that makes contamination between tests possible. What would change if the repository created its own connection in __init__?

See solution

Contamination between tests requires the state to survive the test, and that happens because the connection —where the database lives— is an external object the repository receives and doesn't control. Since the connection lives outside (referenced by the module, by a global variable, by a wide-scope fixture), it isn't collected when the test's repository dies; it survives, with its table and its rows, and the next repository built over that same connection sees them. The commit makes the rows permanent within the database, but it's the shared connection that makes that database persist between tests.

If the repository created its own connection in __init__ —for example, sqlite3.connect(":memory:") inside—, each repository would have its own :memory: database, tied to its own connection, that would die when the repository was collected. In that case, two tests that each create their repository would share nothing, and the isolation would be automatic, just like with the fake. That's exactly why lesson 4's fixture gives each test a new connection: it moves the database's lifecycle to the test's lifecycle. The contamination isn't SQLite's fault; it's the fault of sharing the connection.

Exercise 3 — Design a proof that there's contamination. A colleague swears their 40-test integration suite is well isolated. Without reading the 40 tests, propose two quick experiments that, if the suite were contaminated, would reveal it, and explain what you'd expect to see in each case.

See solution

Two cheap and forceful experiments, both based on the problem's signature:

  1. Run the suite in reversed order. If it's well isolated, the 40 tests pass just as in normal order. If it's contaminated, some test that depended on another running before (or not running before) will change its verdict: one that was passing will fail, or one that was failing will pass. A single change of result on reversing the order proves there's a state dependency. (Tools like pytest-randomly automate this by shuffling the order; by hand, listing a few tests in a different order on the command line is enough.)

  2. Run each test isolated and compare it with the suite. Select a few tests and run them one by one (pytest path::test_x). If any passes alone but was failing in the suite —or vice versa—, you have confirmed contamination in that test. It's the lesson's diagnostic test, applied as an audit.

What neither experiment needs is reading the logic of the 40 tests: contamination is detected by its behavior —sensitivity to order and to isolation—, not by inspecting the code. If the suite survives both experiments without changing a single verdict, there's good evidence it's isolated; if not, you know exactly where to start looking.

Summary and next step

In this lesson you dissected why shared real state contaminates and the fake doesn't. The cause is a concrete asymmetry: in the fake, the state is an instance's dict that dies with the test; in the real one, the state is rows in a table whose connection lives outside and survives the test. You saw the contamination in slow motion with real output —a second test that inherits the bk-1 it never saved— and applied the decisive diagnostic test: running the test alone. And you put a name to the signature —passes alone, fails in the suite, changes with the order—, the thread that connects this module with failure diagnosis, and understood that the cure isn't abandoning the real thing but giving each test a known state.

Before moving on you should be able to: explain where the state lives in the fake and in the real one and why that decides the isolation; read an integration failure and identify whether the object that breaks the assertion carries foreign fingerprints; and use the alone-versus-suite test to separate a bug from contamination.

What comes next is the first cure. In lesson 3 you're going to install the transaction rollback as an isolation technique: each test runs inside a SQLite transaction that's reverted at the end, so that what the test wrote disappears and the database stays clean for the next —without recreating it—. You'll see the mechanism with raw SQL, the fixture that does the rollback in the teardown, and a hard precondition that, if you ignore it, makes the rollback isolate nothing: that the code under test doesn't commit midway. It's the most elegant technique for cutting the inheritance of state, and the first of the two this module gives you.

Resources

← Previous1. Module introduction: the state that doesn't go away on its ownNext →3. The transaction rollback as isolation