Module 5: Integration Testing Real Components Together

7. The cost of integration

Next →

Description

Lesson 6 showed you integration's benefit in all its glory: it catches bugs no other tool sees completely. It would be dishonest to close the module there, leaving you with the impression that integration is pure benefit and that, if it catches more bugs, you should integrate everything. It isn't so. Every integration test has a bill, and this lesson puts it on the table with real numbers. You're going to measure, actually running it, how much slower touching SQLite is than a fake —and the difference isn't 10% or double: it's hundreds of times when there's a disk in the middle—. And you're going to see the other cost, the one not measured in milliseconds: the burden of seeding the database before each test and cleaning it afterward, because a real database remembers, and what one test leaves, the next one finds.

Understanding the cost isn't to discourage you from integrating; it's to integrate with criterion. The reason the test pyramid has many unit tests at the base and few integration tests in the middle is exactly this cost: if you invert the proportion and integrate everything, your suite becomes so slow that nobody runs it, and so fragile because of the shared state that nobody trusts it. The criterion that closes the module is a question you can now answer with numbers: for this seam, is it enough for me to assume it (a fast unit test and a cheap contract) or does the risk justify paying for the integration? The answer depends on how much it costs and how much risk there is, and this lesson gives you half the equation —the cost— measured, not intuited.

Connection to the module: this lesson is the flip side of lesson 6 and the module's conceptual close. Lesson 6 put the benefit (it catches the datetime bug); this one puts the cost (slow, with seeding and cleaning), and together they give you the decision criterion. It's also the hinge to module 7: here we name the burden of seeding and cleaning the database and feel it as a cost, but the techniques for doing it well —isolate with rollback, fixtures that create and destroy a temporary database, keep the tests independent when they share real state— are module 7. This lesson leaves you convinced that isolation is a problem that has to be solved; module 7 solves it.

Analogy: the crash test versus the computer simulation

Think of how a carmaker tests a new car's safety. It has two tools. One is the computer simulation: a model of the crash that runs in seconds, can be repeated a thousand times with different parameters, uses no materials, and makes no mess. The other is the real crash test: a real car, a real dummy, smashed against a real wall at real speed. The crash test gives a confidence the simulation can't —it's the physical world, with all its surprises—, but it costs a fortune: it destroys a whole car, requires a track, a team, hours of preparation, and afterward you have to clean up the debris and set everything up again for the next one. Nobody does a thousand crash tests; they do thousands of simulations and a few crash tests, in the scenarios where the real-world confidence justifies the cost.

Integration is the crash test; the unit test with doubles is the simulation. The simulation (in-memory fake) runs in microseconds, repeats without limit, and leaves no trace. The crash test (real SQLite, especially on disk) gives the confidence of the real piece, but costs orders of magnitude more time and demands preparing the track beforehand (seeding the database) and cleaning up the debris afterward (deleting the data). That's why the pyramid has the shape it has: many cheap simulations at the base, few expensive crash tests at the peak. This lesson measures how much the crash test costs, so you know when it's worth crashing a real car and when simulating it is enough.

Measuring the slowness: fake versus SQLite in memory versus SQLite on disk

Enough with intuitions; let's measure it. The following script does the same operation —save followed by get, five thousand times— against three repositories: the FakeBookingRepository (an in-memory dict), a SqliteBookingRepository over :memory: (real SQLite, but in RAM), and a SqliteBookingRepository over a file on disk. It times each one and reports the total and the time per operation.

# bench.py — how much integration costs: fake vs SQLite (memory) vs SQLite (disk)
import os
import sqlite3
import tempfile
import time
from datetime import datetime

from reservo.doubles import FakeBookingRepository
from reservo.models import Booking
from reservo.sqlite_repo import SqliteBookingRepository

START = datetime(2026, 3, 10, 9)
END = datetime(2026, 3, 10, 12)
N = 5000


def a_booking(i):
    return Booking(id=f"bk-{i}", room_id="focus", member_id="m-ana",
                   start=START, end=END, status="confirmed", price_cents=6000)


def bench(repo, label):
    t0 = time.perf_counter()
    for i in range(N):
        repo.save(a_booking(i))
        repo.get(f"bk-{i}")
    dt = time.perf_counter() - t0
    print(f"{label:28s} {N} ops  {dt*1000:8.1f} ms  ({dt/N*1e6:6.1f} us/op)")


bench(FakeBookingRepository(), "fake (dict in memory)")
bench(SqliteBookingRepository(sqlite3.connect(":memory:")), "sqlite :memory:")

fd, path = tempfile.mkstemp(suffix=".db")
os.close(fd)
bench(SqliteBookingRepository(sqlite3.connect(path)), "sqlite on disk")
os.remove(path)

What to expect. On my machine (Python 3.14.0):

python3 bench.py
fake (dict in memory)        5000 ops       2.8 ms  (   0.6 us/op)
sqlite :memory:              5000 ops      39.0 ms  (   7.8 us/op)
sqlite on disk               5000 ops    2002.2 ms  ( 400.4 us/op)

The exact numbers vary between runs and between machines, but the magnitudes hold and are the lesson. Read the ladder. The fake does the five thousand operations in 2.8 ms —half a microsecond per operation—; it's a dict in your process, practically instant. SQLite in memory takes 39 ms, about fourteen times more than the fake: it's still fast in absolute terms, but it already pays the cost of being a real database —parsing SQL, managing the table, applying types— even though the data lives in RAM. And SQLite on disk takes 2002 ms —two whole seconds—, about seven hundred times slower than the fake and fifty times slower than in memory. That brutal jump to disk has a culprit with a name: each save does commit, and each commit forces SQLite to write to the physical disk (an fsync) to guarantee the data survives. That trip to disk, five thousand times, is what eats the two seconds.

The quantitative moral: an integration against the real database isn't "a bit slower"; it's one to three orders of magnitude slower, and the disk is the dominant factor. Multiply that by a suite of thousands of tests and you understand why you can't integrate everything: a suite of a thousand unit tests with a fake runs in a blink; the same suite touching disk would take minutes, and a suite that takes minutes is a suite people stop running.

The other cost: seeding and cleaning

The slowness is the visible cost; there's another, more insidious, that isn't measured in milliseconds: a real database remembers. A fake is born empty in each test —you create it with FakeBookingRepository() and its dict starts clean—. An on-disk database doesn't: what one test wrote is still there for the next, unless you delete it. This creates two tasks the unit test never had:

Seeding (setup): before an integration test, you often need the database to have a certain starting state —a booking already saved so you can cancel it, a schema created, some reference rows—. That "put the database as the test needs it" is the seeding, and it has to be written, maintained, and run before each test.

Cleaning (teardown): after the test, you have to leave the database as it was, or the next test will inherit your garbage. If test A saves a booking bk-1 and doesn't delete it, test B that assumes an empty database —or that also uses bk-1— will fail in confusing ways, and worse still, will fail depending on the order the tests run in. A test that passes alone but fails in the suite, or that passes today and fails tomorrow, is almost always a cleaning problem.

You already saw a preview of this in lesson 3, with the finally: os.remove(path) that deleted the temporary file by hand. That's raw cleaning. The point of this lesson isn't to teach you to do it well —that's module 7—, but for you to feel that it exists: every integration carries the responsibility of seeding its state and cleaning it, a responsibility the unit test with doubles doesn't have because its doubles are born and die with the test. That burden is part of the cost, and sometimes it weighs more than the milliseconds: it's work to write, it's a source of fragility, and it's why poorly maintained integration suites become a minefield of tests that fail because of the state and not because of the code.

The decision criterion, with the cost in hand

Now you can answer the question that closes the module: for a given seam, do you integrate or is it enough to assume? The decision weighs two things —the seam's risk against the cost of integrating it— and now you have both measured.

Integrate when the seam's risk justifies the cost. The repository seam has real risk: it serializes data (the datetime→str), imposes schema constraints, has a round-trip behavior a fake doesn't reproduce. That risk —demonstrated in lesson 6 with the TypeError— justifies paying for a few integration tests, preferably in :memory: (fourteen times slower than the fake, but still fast) and only on disk when you need to test the persistence for real. Few, well chosen, at the seams that can do harm.

Don't integrate when the cost buys no confidence. A seam without a boundary —the price_cents logic, the in-memory Calendar— gains nothing from integration: it doesn't serialize, has no external state, can't diverge. Testing it with a fast unit test is all you need. And to verify that the fake doesn't lie about a seam's behavior, a contract (modules 3-4) is often cheaper than a battery of integrations: it runs the same spec against the fake and the real one once, and protects you without paying disk in each test.

The shape that comes from this criterion is the pyramid: many unit tests (cheap, the logic), one contract per risk seam (cheap, keeps the fake honest), and a few integration tests (expensive, the joints that can really break). It's not "integration versus unit"; it's each tool in the proportion its cost and benefit dictate. The cost you measured in this lesson is exactly what gives the pyramid its shape.

Common mistakes

Integrating everything "to be safe" and ending up with a suite nobody runs. What happens: a team, burned by an integration bug, decides to test everything against SQLite on disk. Why it happens: if integration caught the bug, integration seems always better. How to detect it: if your suite takes minutes and people start skipping it or running only "their" tests, you inverted the pyramid. How to fix it: measure, as in this lesson. Reserve integration for the risk seams, use :memory: when the disk adds nothing, and lean on unit tests and contracts for the rest. A suite that takes seconds is always run; one that takes minutes is abandoned.

Using disk when :memory: was enough. What happens: someone writes all their integrations against a file on disk out of habit, paying the fsync in each test. Why it happens: "disk is more real". How to detect it: if your integration doesn't need to test persistence between connections (surviving a file reopen), the disk is only costing you fifty times more time without buying extra confidence. How to fix it: use sqlite3.connect(":memory:") by default —it's real SQLite, it tests the serialization and the schema, and it's fifty times faster—; reserve the disk for the few tests that specifically test physical persistence. Choose the resource according to what the test needs to demonstrate, not by inertia.

Not cleaning and blaming the code when the suite is flaky. What happens: the integration tests fail intermittently, depending on the order, and the team loses hours looking for a bug in the production code. Why it happens: the shared and uncleaned real state produces failures that seem like code bugs. How to detect it: if a test passes alone but fails in the suite, or passes and fails without the code changing, suspect the state, not the code. How to fix it: every integration must seed its state and clean it, leaving the database as it found it. How to do it well —isolate with rollback, fixtures that create and destroy the database— is module 7; the honest minimum, meanwhile, is not to share state between tests without deleting it.

Exercises

Exercise 1 — Estimate the suite. With the measured numbers —fake ≈ 0.6 µs/op, SQLite :memory: ≈ 7.8 µs/op, SQLite on disk ≈ 400 µs/op—, estimate how long a suite of 2000 tests would take if each did one save+get operation, in each of the three cases. What does the comparison tell you about the pyramid's shape?

See solution

Multiplying the cost per operation by the 2000 tests (one operation each, as a rough approximation; in practice there are more per test, but the proportion holds):

  • Fake: 2000 × 0.6 µs ≈ 1.2 ms. Instant; you don't even notice it.
  • SQLite :memory:: 2000 × 7.8 µs ≈ 15.6 ms. Still a blink.
  • SQLite on disk: 2000 × 400 µs ≈ 800 ms, almost a second —and that's counting a single operation per test; with several operations and the commit of each, it shoots up to several seconds or more—.

The comparison tells you exactly why the pyramid has its shape. With fakes, you can have thousands of tests and the suite runs in the blink of an eye —that's why they go at the wide base—. With SQLite in memory, hundreds are still cheap —the middle band—. With disk, each test really costs, so you want few, only where physical persistence matters —the narrow peak—. The pyramid's shape isn't an aesthetic convention: it's the direct consequence of this cost. Inverting it (many tests on disk) produces a suite that takes minutes, which is a suite people stop running.

Exercise 2 — Why is the disk so slow? The jump from :memory: (7.8 µs/op) to disk (400 µs/op) is about fifty times, much bigger than the jump from fake to :memory:. Explain the technical cause and what you'd change in the repository to measure the effect.

See solution

The cause is the commit with its fsync. Each save of our SqliteBookingRepository ends with self._conn.commit(). In :memory:, a commit doesn't have to go to any disk —the database lives in RAM—, so it's cheap. On disk, each commit forces SQLite to write the changes to the physical file and to ask the operating system to actually sync them to disk (an fsync), to guarantee the data survives even a power outage. That trip to the physical disk is extremely slow compared to an in-memory operation —it's the difference between jotting something on a paper you're holding and walking to the file cabinet, filing it in its folder, and confirming it's in—. Five thousand fsyncs are the two seconds.

To measure the effect, you could save without commit on each operation and commit only once at the end (a transaction that groups the five thousand writes). You'd see the disk time collapse, approaching that of :memory:, because there'd be a single fsync instead of five thousand. That illustrates that the disk's cost isn't "writing the data" but "committing each write to the physical disk one by one". Controlling when you commit —grouping writes in a transaction— is exactly the lever module 6 studies as a tool and module 7 uses to speed up and isolate tests (write without commit, do rollback at the end).

Exercise 3 — Integrate or not, four seams. For each seam, decide whether you'd cover it with integration (and in :memory: or on disk) or whether a unit test or a contract is enough, justifying with cost and risk: (a) the computation of price_cents(room, member, hours); (b) the repository's save/get round-trip; (c) the persistence of a booking between process restarts; (d) that the FakeBookingRepository doesn't diverge from the real one on the repository's clauses.

See solution
  • (a) price_cents → unit test, no integration. It's pure logic: it takes data, returns an integer, no boundary, no state, no serialization. It can't diverge between a fake and the real one because there's no "real one": it's a function. A fast unit test covers it completely; integrating it would buy nothing and only cost.
  • (b) repository's save/get → integration in :memory:. It has real risk (the datetime serialization, schema constraints, round-trip), demonstrated in lesson 6. It justifies paying for the integration, but in :memory: —fourteen times slower than the fake, still fast— because you don't need physical persistence to test the serialization and the schema. A few tests, well chosen.
  • (c) Persistence between restarts → integration on disk. Here the disk does add: what you want to test is precisely that the data survives closing and reopening, which is impossible to verify in :memory:. You pay the disk cost because it buys exactly the confidence you're looking for. One or two tests, no more —it's expensive—.
  • (d) That the fake doesn't diverge → contract (modules 3-4), not a battery of integrations. The question "does the fake behave like the real one?" is best answered with a contract: a single parametrized battery against both, which runs almost as fast as a unit test and protects you from the divergence. Writing separate integrations for this would cost more and guarantee less.

The pattern: unit for logic without a boundary (a), contract for "the fake doesn't lie" (d), integration in :memory: for the serialization/schema risk (b), integration on disk only for physical persistence (c). Each tool where its cost buys the confidence its seam needs.

Summary and next step

In this lesson you put integration's bill on the table, measured. You timed the same operation against three repositories and saw the ladder: the fake at 0.6 µs/op, SQLite in memory fourteen times slower (7.8 µs/op), and SQLite on disk about seven hundred times slower than the fake (400 µs/op), with the commit/fsync as the culprit of the jump to disk. And you named the cost not measured in milliseconds: seeding and cleaning a database that remembers, with the fragility of shared state that produces flaky suites that fail because of the order and not the code. With the crash test versus the simulation you understood why the pyramid has its shape: many cheap simulations, few expensive crash tests, each where its cost buys confidence. And you have the complete decision criterion: integrate where the seam's risk justifies the cost; lean on unit and contract where it doesn't.

Before moving on you should be able to: estimate the cost of a suite according to the resource it touches; explain why the disk is so slow (the fsync of each commit) and when :memory: is enough; and decide, with cost and risk in hand, whether a seam deserves integration, a contract, or just a unit test.

With this module you close the first real integration test and the criterion for writing it well. We named two costs —the difficult boundaries and the isolation of state— and left them pending on purpose. Module 6 takes the first: the specific boundaries in depth —a SQLite transaction with its commit and its rollback, a real file, an HTTP call to a stdlib http.server—, and how to make them fast and deterministic. Module 7 takes the second: the data and isolation —rollback to isolate, fixtures that create and destroy the database, keeping the tests repeatable—. The integration you learned to write here, you're going to learn to do at the difficult boundaries and with the state under control.

Resources

  • time.perf_counter — Python documentation — the high-resolution clock we time the operations with in the benchmark; the tool for measuring the cost of your own seam before deciding whether to integrate it.
  • sqlite3 — Transaction control (Python documentation) — the reference for how SQLite handles transactions and the commit, the key to why the disk is so slow (an fsync per commit) and the lever module 7 will use to isolate and speed up tests.
  • pytest documentation — Fixtures — the tool with which module 7 will solve the seeding-and-cleaning burden this lesson names as a cost; useful to preview how the setup and teardown of a real database is automated.
  • Martin Fowler — TestPyramid — the frame that explains why the suite should have many unit tests and few integration ones; the shape that, as you saw, comes directly from the cost measured in this lesson.
← Previous6. Integration catches what the unit doesn'tNext →8. Mini-project: a `book`→`get` integration against SQLite