7. Independent and repeatable

Description

The whole module was aiming at two properties we're now going to name and demonstrate, because they're the criterion by which an integration suite is judged to be well made. The first is independence: each test starts from a known state and doesn't depend on which other test ran before or after. An independent test gives the same result wherever you put it in the suite, because it brings its own world —it seeds it, it isolates it— and inherits nothing from its neighbors. The second is repeatability: each test gives the same result every time you run it, today and tomorrow, on your machine and the one next door. A repeatable test doesn't depend on the time, the order, data left over from a previous run, or anything that varies between executions. The two together are what makes a green mean something: if your suite is independent and repeatable, a green is a solid statement about the code; if it isn't, a green is a coin toss that came up heads this time.

These two properties aren't abstract ideals: they're the direct consequence of everything you did in the previous lessons. Isolation —rollback or new database per test— is what produces the independence: each test starts clean because nobody left it state. Explicit seeding is what makes that state known instead of just clean. And determinism in everything else —the fixed clock with FixedClock, the doubles instead of external services that vary— is what produces the repeatability. This module, seen from above, was a machine for manufacturing independent and repeatable tests. In this lesson you verify it: you're going to run the isolated suite in normal order, in reversed order, and twice in a row, and see the same green in all three cases. That "same green always" is the signature of a healthy suite, the opposite image of the order-dependent flaky from lesson 2.

Connection to the module: lessons 3 to 6 gave you the tools —isolate, choose resource, seed—; this one names the what for and verifies it. It's the conceptual close before the mini-project: independence and repeatability are the two criteria by which lesson 8 will ask you to judge your own suite. Independence is the exact opposite of the contamination you diagnosed in lesson 2; repeatability is the opposite of the flaky. If you understand that these two properties are achieved with isolation plus determinism, and you know how to demonstrate them by running the suite in various orders, you have the complete criterion for writing integration that doesn't lie. Lesson 8 puts it into practice end to end.

Analogy: the recipe that comes out the same for anyone

Think of the difference between a well-written cooking recipe and the "instructions" of a cook who improvises. The well-written recipe is independent and repeatable: each step starts from a state the recipe itself established —"preheat the oven to 180°", not "use the oven as it was left"—, and doesn't depend on the order in which you made other recipes that day. Anyone who follows it, in any kitchen, with the listed ingredients, gets the same bread. If you give the recipe to ten different people, out come ten identical loaves. That's a recipe you can trust: the result depends on the recipe, not on who or when or in what order.

The "instructions" of the improvising cook are the opposite: "add salt to taste", "cook until it looks good", "make use of the sauté you have left over from before". The result depends on things that vary —whose taste, what was left over, the day's order—, so two people following the same thing get different dishes, and the same person gets different dishes on different days. You can't trust that "it came out well" means anything, because next time it might come out badly without your changing anything. An integration suite is a recipe: if each test establishes its own state (independent) and doesn't depend on anything that varies (repeatable), its green is trustworthy, like the bread that comes out the same for anyone. If it depends on the order, on inherited data, or on the time, its green is the improviser's "it looks good": true today, who knows tomorrow.

Independence: the same green in any order

Let's take the isolated suite —three Reservo integration tests, each with its fresh database by fixture— and test its independence in the most direct way: running it in two different orders and checking that the verdict doesn't change.

# tests/test_suite_isolated.py — ISOLATED integration suite
import sqlite3
import pytest
from datetime import datetime
from reservo.calendar import Calendar
from reservo.doubles import FixedClock, SpyEmailSender, StubPaymentGateway
from reservo.models import Member, Room
from reservo.services import BookingService
from reservo.sqlite_repo import SqliteBookingRepository

FOCUS = Room("focus", "Focus", 4, 2500); ANA = Member("m-ana", "Ana", "pro")

@pytest.fixture
def repo():
    conn = sqlite3.connect(":memory:")     # fresh DB per test
    yield SqliteBookingRepository(conn)
    conn.close()

def service(repo):
    return BookingService(Calendar(), FixedClock(datetime(2026, 3, 1, 9)),
                          StubPaymentGateway(True), SpyEmailSender(), repo)

def test_book_creates_one_focus_booking(repo):
    service(repo).book(FOCUS, ANA, datetime(2026, 3, 10, 9), datetime(2026, 3, 10, 12))
    assert len(repo.find_by_room("focus")) == 1

def test_focus_is_empty_before_my_booking(repo):
    assert len(repo.find_by_room("focus")) == 0     # new DB: it's true
    service(repo).book(FOCUS, ANA, datetime(2026, 3, 11, 9), datetime(2026, 3, 11, 12))
    assert len(repo.find_by_room("focus")) == 1

def test_exactly_one_confirmed_total(repo):
    service(repo).book(FOCUS, ANA, datetime(2026, 3, 12, 9), datetime(2026, 3, 12, 12))
    assert len(repo.find_by_room("focus")) == 1     # its own booking, nothing else

First, definition order.

What to expect. On my machine (Python 3.14.0, pytest 9.1.1):

python3 -m pytest tests/test_suite_isolated.py -v

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 3 items

tests/test_suite_isolated.py::test_book_creates_one_focus_booking PASSED   [ 33%]
tests/test_suite_isolated.py::test_focus_is_empty_before_my_booking PASSED [ 66%]
tests/test_suite_isolated.py::test_exactly_one_confirmed_total PASSED      [100%]

============================== 3 passed in 0.01s ===============================

Now, reversed order, listing the tests backwards on the command line:

python3 -m pytest tests/test_suite_isolated.py::test_exactly_one_confirmed_total \
                  tests/test_suite_isolated.py::test_focus_is_empty_before_my_booking \
                  tests/test_suite_isolated.py::test_book_creates_one_focus_booking -v

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
collected 3 items

tests/test_suite_isolated.py::test_exactly_one_confirmed_total PASSED      [ 33%]
tests/test_suite_isolated.py::test_focus_is_empty_before_my_booking PASSED [ 66%]
tests/test_suite_isolated.py::test_book_creates_one_focus_booking PASSED   [100%]

The same green, order reversed. Compare it with the shared suite from lesson 1, where reversing the order changed which test failed: there the verdict depended on the order, here it doesn't. That insensitivity to order is independence, measured empirically: you don't assert it by reading the code, you verify it by running it in various ways and seeing that the result doesn't move. Each test starts from its own fresh database, so who ran before is irrelevant; the result depends only on what each test does.

Repeatability: the same green every time

Independence answers "does the order matter?"; repeatability answers "does the run matter?". A repeatable test gives the same result if you run it now and again within a second, without changing anything. Let's check it by running the same suite twice in a row.

What to expect. On my machine (Python 3.14.0, pytest 9.1.1):

python3 -m pytest tests/test_suite_isolated.py -v      # first run

collected 3 items

tests/test_suite_isolated.py::test_book_creates_one_focus_booking PASSED   [ 33%]
tests/test_suite_isolated.py::test_focus_is_empty_before_my_booking PASSED [ 66%]
tests/test_suite_isolated.py::test_exactly_one_confirmed_total PASSED      [100%]

============================== 3 passed in 0.01s ===============================

python3 -m pytest tests/test_suite_isolated.py -v      # second run, without changing anything

collected 3 items

tests/test_suite_isolated.py::test_book_creates_one_focus_booking PASSED   [ 33%]
tests/test_suite_isolated.py::test_focus_is_empty_before_my_booking PASSED [ 66%]
tests/test_suite_isolated.py::test_exactly_one_confirmed_total PASSED      [100%]

============================== 3 passed in 0.01s ===============================

Identical. And this, which seems trivial, isn't: it's trivial because we did the work. The suite is repeatable for two reasons worth separating. First, the isolation: each run starts without data inherited from the previous one, because the previous run's :memory: databases were destroyed with their connections —there was no test.db left on disk with yesterday's bookings that would make today's run fail—. Second, the determinism of everything else: the clock is fixed with FixedClock(datetime(2026, 3, 1, 9)), so cancel and the time calculations give the same regardless of when you run; the payment is a StubPaymentGateway that doesn't call any external service that could vary; the email is a SpyEmailSender that doesn't send anything real. Nothing in the suite depends on something that changes between runs.

If either of those two pieces failed, the repeatability would break. A shared test.db on disk would make the second run inherit data from the first. A real datetime.now() instead of the FixedClock would make a refund test give different results depending on the date you run it —the classic test that passes today and fails next month—. Repeatability is the result of having removed, one by one, all the paths by which one run could influence another or the outside world could sneak in.

Why the order should never matter

Let's pause on this lesson's strong claim: in a well-made suite, the execution order should never change a verdict. It's not an aesthetic preference; it's a condition for the tests to mean something. If the verdict depends on the order, then "green" isn't a statement about the code —it's a statement about the code plus the order in which the tests ran today—, and that order can change for a thousand reasons out of your control: pytest reorders, someone adds a test, you run a subset, a tool shuffles the order on purpose to catch exactly these problems.

In fact, that last one is a real practice: there are tools —like pytest-randomly— that shuffle the test order on every run precisely to expose hidden order dependencies. In an independent suite, shuffling changes nothing: green always. In a contaminated suite, shuffling makes it sometimes pass and sometimes fail, revealing the problem a fixed order was hiding. That your suite survives running in any order isn't a luxury: it's the proof that its greens are trustworthy. The manual form of that proof is the one you did above —running in reversed order— and it's enough to detect most dependencies; the automatic one catches them all, run after run.

The connection with the whole module closes here. Lesson 2 showed you the disease: a suite whose verdict depended on the order, the shared-state flaky. Lessons 3 to 6 gave you the cure: isolate so each test starts clean, seed so it starts from a known state, determinism so it doesn't depend on anything that varies. And this lesson gives you the proof the cure worked: run the suite in various orders and various times, and if the green doesn't move, you have independence and repeatability. That's the standard of an integration suite you can trust.

Common mistakes

Trusting that "it passes" equals "it's fine" without testing the order. What happens: the suite passes in the usual order and is taken as good. Why it happens: locally the tests almost always run in the same order, which can hide the dependency. How to detect it: run in reversed order or shuffle with a tool; if any verdict changes, the suite wasn't independent, it just got lucky with the order. How to fix it: make running-in-another-order part of your definition of "trustworthy green"; a green that only holds up in one order isn't trustworthy.

Leaving a real datetime.now() and breaking repeatability. What happens: a cancel test uses the real time instead of the FixedClock, and it passes today but will fail when the date changes the refund window. Why it happens: using the real time seems harmless. How to detect it: if a test depends on "how long until the start" and doesn't fix the clock, its result changes with the calendar. How to fix it: fix the time with a controlled clock (FixedClock), as in the whole suite; temporal determinism is part of repeatability.

Believing that isolating the database already guarantees repeatability. What happens: the database is isolated with a perfect fixture, but a test is still flaky because it depends on the clock or on a non-guaranteed order of results. Why it happens: one thinks the database's state is the only source of variation. How to detect it: if the database is isolated and yet the result varies, look for other sources —time, order of a list without ORDER BY, randomness—. How to fix it: independence and repeatability need isolation plus determinism in everything else; the database is a source of variation, not the only one.

Exercises

Exercise 1 — Independent, repeatable, both, or neither. For each test, say which property it lacks (independence, repeatability, both, or neither) and why: (a) a test that assumes another test seeded a booking before; (b) a test that uses datetime.now() to calculate the refund; (c) a test with a fresh database by fixture and FixedClock; (d) a test that orders results with find_by_room and assumes they come out in a specific order without an ORDER BY.

See solution

• (a) It lacks independence. It depends on another test running before and leaving the booking; if the order changes or that test doesn't run, it fails. It's order dependence, the opposite of independent. (It could be repeatable, if the order were always the same, but it's fragile.)
• (b) It lacks repeatability. datetime.now() changes with the calendar, so the refund window (72 h / 36 h / 12 h before the start) gives different results depending on when you run; it passes today and fails another day without your changing anything. (It can be independent; the problem is the time, not the neighbors.)
• (c) It lacks neither: it's independent and repeatable. Fresh database by fixture → it doesn't inherit from neighbors (independent); fixed clock and no variable external sources → same result always (repeatable). It's the healthy standard.
• (d) It lacks repeatability (and maybe independence). Without ORDER BY, the order in which SQLite returns the rows isn't guaranteed and can vary; a test that assumes a specific order can pass or fail non-deterministically. It's fixed by ordering explicitly in the query or comparing as a set (set), not as an ordered list.

The rule: independence is not depending on other tests; repeatability is not depending on anything that varies between runs (time, non-guaranteed order, randomness, external services). A healthy suite has both.

Exercise 2 — Design the independence test without reading the code. You're handed an integration suite of 50 tests and asked to certify that it's independent, but you don't have time to read the 50. Describe two runs that, compared with the normal run, would give you strong evidence of independence (or reveal that there is none), and say what you'd expect to see in a healthy suite.

See solution

Two runs, both compared against the normal run (which we'll assume green):

1. Reversed or shuffled order. Run the suite with the tests in another order —reversed by hand, or shuffled with a tool like pytest-randomly—. In an independent suite, the result is identical: the same 50 greens. If any test changes verdict (goes from passing to failing, or the reverse), you have proof of order dependence: that test assumes something from a neighbor.

2. Isolated subsets. Run small groups of tests separately —or even each test alone— and compare with its result inside the full suite. In an independent suite, each test gives the same verdict just as it does in a group. If any passes alone but fails in the suite (or the reverse), there's contamination: it depends on what ran alongside it.

In a healthy suite you'd expect neither of the two runs to change a single verdict compared with the normal one: the same greens in any order and in any grouping. The powerful thing about these two tests is that they certify independence by behavior, without reading the 50 tests: if the result is invariant to order and grouping, the tests don't depend on each other; if it varies, you know exactly where to look.

Exercise 3 — The suite that passes today and will fail in June. A cancel test verifies that cancelling with 60 hours' notice returns the full refund (6000), and calculates the notice with now = datetime.now() against a start fixed in the near future. Today it passes. Explain why it will fail at some future moment, which property it violates, and how you fix it without changing what the test wants to prove.

See solution

It will fail because now = datetime.now() advances with the calendar, while the test's start is fixed. Today, the distance between now and that start is 60 hours (more than 48), so it falls in the full-refund window and the test sees 6000: it passes. But as the real days go by, datetime.now() gets closer to the fixed start: there comes a moment when the notice drops below 48 hours (half refund, 3000), then below 24 (no refund, 0), and the test that expected 6000 fails, without anyone touching a line. It's the classic test that passes today and fails in June.

The property it violates is repeatability: its result depends on when it's executed, something that varies between runs. It's not independence —it doesn't depend on other tests—; it's that the real time sneaked in as a hidden input.

The fix, without changing what it tests: fix the clock. Instead of datetime.now(), use an explicit and controlled now —the whole suite's FixedClock(NOW)— with NOW at exactly 60 hours before the start. That way the notice is always 60 hours, whenever the real date falls, and the test verifies the same thing —full refund at more than 48 hours— but deterministically, whenever you run it. The time stops being an input from the world and becomes a fixed datum of the test.

Summary and next step

In this lesson you named and demonstrated the two properties the whole module was pursuing. Independence —each test starts from a known state, without depending on its neighbors— you verified by running the isolated suite in normal order and in reversed order and seeing the same green: the insensitivity to order that lesson 1's shared suite didn't have. Repeatability —the same result every time— you verified by running the suite twice with identical output, and you saw that it rests on two legs: the isolation (nothing is inherited between runs) and the determinism of everything else (the FixedClock, the doubles without external services). And you understood why the order should never change a verdict, and how tools that shuffle the order turn that requirement into an automatic test.

Before moving on you should be able to: distinguish independence from repeatability and say which one a test that violates each is lacking; certify independence by running the suite in various orders and groupings; and recognize the sources of non-repeatability —real time, non-guaranteed order, external services— and how to neutralize them.

What comes next is bringing it all together in a single deliverable. In lesson 8, the mini-project, you're going to take a Reservo integration suite that fails due to shared state —with different verdicts depending on the order, like lesson 1's— and turn it into an isolated and repeatable suite: green in any order and however many times you run it. You're going to deliver the two versions —the contaminated one in red and the isolated one in green—, choose the appropriate isolation technique and justify it. It's the synthesis of the seven lessons and the module's close, with everything you learned put to work on a complete case.

Resources

• pytest documentation — Good practices (independent tests) — the official reference on organizing tests so they're independent and runnable in isolation, the foundation of this lesson's independence.
• pytest-randomly — the plugin that shuffles the test order on every run to expose hidden order dependencies; the automatic version of the independence test you did here by hand by reversing the order.
• Martin Fowler — Eradicating Non-Determinism in Tests — the classic analysis of the sources of non-determinism (shared state, time, order) and why repeatability is non-negotiable, the conceptual framework of this lesson.
• test-failure-diagnosis-guide — the sister diagnosis guide, where the order-dependent flaky is a central case; independence and repeatability are, seen from here, the absence of that flaky.