Module 1: From Units To Integration

7. The cost and benefit of integration tests

Next →

Description

The whole guide has pushed in one direction: integration catches bugs the unit test doesn't see, so you need integration. It's true, and it's only half the story. The other half, the one this lesson puts on the table without frills, is that an integration test costs more than a unit test —more time per run, more fragility, more setup— and that this cost is the exact reason the pyramid has a narrow middle and not a base of pure integration. If integration were free, we'd test everything against the real thing and there'd be nothing to decide. It's not free, and that's why there's a decision to make at each seam: does this benefit justify this cost?

Until now "costs more" has been a qualitative claim. In this lesson we measure it. You'll see, with real numbers from my machine, how many times slower SqliteBookingRepository over a disk file is than FakeBookingRepository in memory, for the same repeated operation. The figure —hundreds of times— isn't to scare you away from integration; it's to calibrate your intuition: to understand that each integration test you add weighs much more than a unit test on the suite's clock, and that's why you want them few and well chosen, at the seams where the benefit (catching a real divergence) justifies the cost, and not at the seams where a contract or a unit test is enough.

Connection to the module: this lesson closes the module's "why" by putting a price on the tool. Lesson 3 told you what shape the suite should have (the pyramid); this one tells you why that shape is economically obligatory, measuring the cost that holds it up. Lesson 6 gave you the types of integration; here you see that the broadest and most sociable are also the most expensive, which reinforces preferring the narrow. And it prepares the capstone and the following modules: when you write real integration (module 5+), you'll know how to manage its cost —fixtures, isolation, in-memory databases— instead of suffering it.

Analogy: the crash test vs the simulation

Think of how a carmaker verifies a car's safety. It has two tools. One is the computer simulation: a model of the crash runs on a server, thousands of variants per night, for pennies each, and tells you instantly whether the structure holds according to the model. The other is the real crash test: a real car, a real dummy, smashed against a real wall in slow motion. It costs a fortune —a whole car destroyed, a team, a lab—, takes weeks of preparation, and you can only do a few per year. Why not simulate everything, if it's so cheap? Because the simulation is a model, and a model can be wrong exactly where you don't expect it —a material that behaves differently in the real world, a weld the model idealized—. And why not crash real cars all the time, if they're the truth? Because at that cost, the factory would go bankrupt before finishing the first model.

The carmaker's answer is the pyramid: thousands of cheap simulations to explore the design space fast, and a few real crash tests at the critical points, to confirm the simulation didn't lie about what matters. The unit test with a fake is the simulation: dirt cheap, fast, but a model of the real thing that can diverge. The integration test is the crash test: expensive and slow, but the only truth about how the real piece behaves. Nobody crashes a car for every bolt, and nobody writes an integration test for every business rule. Engineering is spending the expensive test where its truth is worth the price, and letting the cheap simulation cover the rest.

Worked example: how much it costs, measured

Let's put a number on "slower". This script does the same thing —five hundred cycles of saving and reading back a booking— against the two providers: the FakeBookingRepository in memory and the SqliteBookingRepository over a real disk file, with a commit for each save (as in production). It measures each one's time.

# timing.py — how much it costs to really touch the disk
import sqlite3, tempfile, os, time
from datetime import datetime, timedelta

from reservo.doubles import FakeBookingRepository
from reservo.models import Booking
from reservo.sqlite_repo import SqliteBookingRepository

N = 500
base = datetime(2026, 3, 10, 9)


def bookings():
    for i in range(N):
        yield Booking(id=f"bk-{i}", room_id="focus", member_id="m-ana",
                      start=base + timedelta(hours=i), end=base + timedelta(hours=i + 1),
                      status="confirmed", price_cents=6000)


# Fake: dict in memory
t0 = time.perf_counter()
fake = FakeBookingRepository()
for b in bookings():
    fake.save(b)
    fake.get(b.id)
t_fake = time.perf_counter() - t0

# SQLite in a real disk file (commit per save)
path = os.path.join(tempfile.mkdtemp(), "reservo.db")
t0 = time.perf_counter()
conn = sqlite3.connect(path)
sql = SqliteBookingRepository(conn)
for b in bookings():
    sql.save(b)
    sql.get(b.id)
t_sqlite = time.perf_counter() - t0
conn.close()

print(f"N = {N} save+get cycles")
print(f"FakeBookingRepository   (dict in memory):   {t_fake*1000:8.2f} ms")
print(f"SqliteBookingRepository (file on disk):     {t_sqlite*1000:8.2f} ms")
print(f"factor: {t_sqlite / t_fake:5.1f}x slower")

What to expect. On my machine (Python 3.14.0):

python3 timing.py
N = 500 save+get cycles
FakeBookingRepository   (dict in memory):       0.86 ms
SqliteBookingRepository (file on disk):       165.20 ms
factor: 186.1x slower

Almost two hundred times slower. The exact figure varies between runs —the disk has its weather, and another run gave me 204 ms and another 159 ms— but the order of magnitude is firm: two, three, almost four orders of magnitude between "in memory" and "on real disk". Where does such a difference come from? From the commit. Each save of the real repository does self._conn.commit(), and a commit forces the database to make sure the data was written to the physical disk before continuing —that durability guarantee is precisely what makes a database useful, and it's extremely slow compared to writing to a Python dict, which only moves a pointer in memory—. The fake has no durability to guarantee; that's why it's free, and why it lies about the cost (and, as you already know, sometimes about the behavior).

Now transfer this figure to your suite. Imagine the twelve unit tests from lesson 3, which ran in 0.01s. If each one, instead of computing in memory, touched a disk file like this, they wouldn't run in hundredths: they'd run in seconds. Multiply by a suite of hundreds of tests and you understand why a base of pure integration is unviable, and why the pyramid has to be wide at the bottom (memory, free) and narrow in the middle (disk, expensive). The cost isn't a minor nuisance: it's the physical force that gives the pyramid its shape.

The other cost: fragility and setup

Slowness is the easiest cost to measure, but not the only one. An integration test brings two more burdens the clock doesn't show.

Fragility: more reasons to fail without your code having a bug. A unit test in memory fails for a single reason: your logic is wrong. An integration test against the disk can fail because the previous file wasn't deleted and the id already existed, because the temp directory filled up, because two tests ran in parallel over the same database and stepped on each other, because a connection was left open. None of those is a bug in your code, and all of them paint your suite red. Each source of fragility is human time spent investigating a failure that wasn't a failure —the most expensive cost of all, because it erodes confidence in the suite (lesson 3)—.

Setup: each integration test asks for preparation and cleanup. The unit test with a fake is built in one line: FakeBookingRepository(). The integration one needs to create the database, apply the schema, and —crucially— leave it clean for the next test, or one's state will contaminate another's. That work of creating-and-destroying real resources is real, and doing it well (with fixtures, transactions that roll back, in-memory databases per test) is a whole topic —module 7's of this guide—. For now keep in mind that an integration test's cost includes the scaffolding of its isolation, not just the time of its execution.

Added up, these three costs —time, fragility, setup— are the "price of the crash test". You don't pay it for fun; you pay it when the benefit warrants it.

The benefit, and the decision rule

Against that cost, what do you buy? A single thing, but it's the one no unit test can give you: the truth about the seam with the real piece. The integration test is the only one that can catch the datetime divergence, the NOT NULL constraint only the database imposes, the id that really collides, the transaction that doesn't roll back. It buys confidence in the joints, which is exactly the hole lesson 1 showed in a suite of pure green unit tests.

With the cost and the benefit on the table, the decision rule becomes concrete. For each seam, ask:

  1. Does the seam have a real risk of divergence? If it's a boundary with serialization, network, or disk (the repository, the payment, the email), yes: the real piece can behave differently from the double. If it's pure logic in memory (Calendar, price_cents), no: there's no "different real piece" that can diverge. Without risk, don't pay for integration.
  2. If there's risk, is a contract enough or do I need real integration? Sometimes the divergence can be pinned down with a contract —a battery of behavior you run against the fake and the real one (modules 3-4)— without paying the cost of integration on every run. Other times the risk lives in the boundary itself (a transaction, a file, HTTP) and you need to cross it for real (modules 5-7).
  3. If I need integration, the narrowest possible? Choose the test that exercises the joint with the minimum of real pieces (lesson 6): you pay less cost for the same truth.

The decision isn't "integration yes or no" in the abstract; it's "for this seam, does the benefit of catching its possible divergence justify its cost, and what's the cheapest way to buy it?". Applied seam by seam, that question produces, on its own, the pyramid: many unit tests where there's no risk or where the logic rules, few narrow integration tests where the boundary's risk demands it.

Common mistakes

Treating the cost as a detail and not as the engine of the shape. What happens: someone accepts "the pyramid" as an arbitrary convention and doesn't understand why they can't have a thousand integration tests. Why it happens: without measuring the cost, the pyramid seems a style rule. How to detect it: if you're surprised that an integration suite takes minutes, you haven't internalized the 200x factor. How to fix it: measure once, as in the worked example, and the pyramid stops being a convention and becomes arithmetic: each integration test costs hundreds of times what a unit test does, so only a few fit. The shape comes from the cost, not from a poster.

Paying the integration cost to test logic with no seam risk. What happens: someone tests the anchors of refund_cents against real SQLite "for more realism", paying 200x for each. Why it happens: the illusion that the real thing always tests better. How to detect it: if an integration test verifies something that doesn't depend on the real piece (the refund arithmetic is identical with any repo), you're paying the price of the crash test to verify something the simulation covers just as well. How to fix it: reserve integration for what only the boundary reveals. Logic goes to the base, free; integration, to the risk seams, where its truth is worth its price.

Suffering the fragility instead of managing it. What happens: someone's integration tests fail at random because of shared state, and their answer is to run them again until they pass. Why it happens: it seems easier to retry than to isolate. How to detect it: if "run it again" is your habitual fix for an integration red, you're paying the fragility cost without managing it. How to fix it: integration fragility is fought with isolation —a clean database per test, transactions that roll back, fixtures that create and destroy the resource—, which is exactly module 7's topic. The cost of setup done well buys the elimination of the fragility cost; they're not two independent evils, one cures the other.

Exercises

Exercise 1 — Explain the 200x factor. In your own words, explain where the enormous difference between the FakeBookingRepository and the on-disk SqliteBookingRepository comes from, and why the fake can't be "fixed" to be as realistic as the real one without ceasing to be fast.

See solution

The difference comes from durability: each save of the real repository does commit(), and a commit forces the database to guarantee that the data was physically written to disk before continuing. Writing to disk and waiting for the confirmation is an extremely slow operation compared to what the fake does, which only stores a reference in a Python dict —moving a pointer in memory—. Hundreds of cycles of "wait for the disk to confirm" against hundreds of "move a pointer" produce the factor of two hundred.

The fake can't become as realistic without losing its speed because its speed is its lack of realism. What makes it fast —not touching the disk, not serializing, not guaranteeing durability— is exactly what makes it a simplified model of the real thing. If you added real durability to it (write to disk, commit), it would stop being an in-memory fake: it would be a real repository, with its real cost. There's no way to have both: the fake buys speed by paying with realism, and the real one buys realism by paying with speed. That's why you want both, each at its level of the pyramid.

Exercise 2 — Apply the decision rule. For each seam, run through the rule's three questions and decide what you do: (a) Calendar.is_available (logic in memory); (b) SqliteBookingRepository saving and reading a booking (disk boundary); (c) the real PaymentGateway charging (network boundary that costs money).

See solution
  • (a) Calendar.is_available — no seam risk, don't pay for integration. Question 1: risk of divergence? No: it's pure logic in memory, there's no "different real piece" that can behave differently (the real Calendar is already cheap and deterministic). Decision: direct unit test, zero integration. Wasting the integration level here buys nothing.
  • (b) SqliteBookingRepository save/get — real risk, narrow integration. Question 1: risk? Yes, a disk boundary with serialization (the datetime trap, the schema constraints). Question 2: contract or integration? Both: a contract to guarantee that the fake doesn't diverge (modules 3-4) and a few narrow integration tests against the real database for serialization and transactions (modules 5-7). Question 3: the narrowest possible —a save+get directly to the repository, without BookingService in between—.
  • (c) real PaymentGateway charging — real risk, but never in your suite. Question 1: risk? Yes, and on top of that it costs real money and lives on the network. Question 2: here integration with the real piece is vetoed by the cost (you'd charge cards on every run); it's covered with a contract against the fake and, at most, a manual test against the provider's sandbox, outside the automatic suite. Decision: always double in the suite, contract to keep the double honest, and the verification against the real thing —if it exists— outside this guide.

The pattern: the rule leads you on its own to double where there's no risk or where the real thing is prohibitive, and to integrate (narrowly) where the boundary's risk justifies it and the cost is bearable. That is the pyramid, decided seam by seam.

Exercise 3 — The suite's budget. Your Reservo suite has 300 unit tests that run in 0.3s total. You want to add integration. A colleague proposes 200 integration tests, each touching the disk. Estimate the impact on the suite's time using the measured factor, and propose an alternative that respects the budget.

See solution

The estimate. A unit test in this suite costs, on average, 0.3s / 300 = 1 ms. An integration test that touches the disk costs, by the measured factor, on the order of 200 times more: about 200 ms each (the same order as the real save+get from the example, dominated by the commit). Two hundred of those are 200 × 200 ms = 40 s —more than a hundred times what the entire current unit suite takes—. The suite would go from running in 0.3s to taking over half a minute, dominated entirely by integration. It would be run less, and you'd lose the fast loop that makes unit tests useful.

The alternative that respects the budget:

  1. Don't duplicate logic in integration. Most of those 200 tests probably re-test business rules (prices, refunds) the unit tests already cover. Those add no seam truth; drop them to the unit level, free.
  2. Keep a handful of narrow integration tests —a dozen, not two hundred— for the real-risk seams: the repository round-trip, get of a missing id, update without duplicating, a transaction, a schema constraint. Narrow and few: maybe 12 × 200 ms = 2.4 s, a bearable cost.
  3. Use a contract (modules 3-4) to guarantee that the fake doesn't diverge from the real one, running it against both: that gives you confidence in all the doubled seams without paying for integration on each one.
  4. Manage the fragility and cost with in-memory databases per test where you can and fixtures that isolate (module 7), so those few integration tests don't become slow or fragile.

Result: a healthy pyramid —300 unit tests in 0.3s, a dozen narrow integrations in a couple of seconds, a contract covering the rest— instead of a 40s ice-cream cone. The budget is respected by spending the expensive test only where its truth is worth the price.

Summary and next step

In this lesson you put a price on integration. You measured, with real numbers, that the on-disk SqliteBookingRepository is on the order of two hundred times slower than the in-memory FakeBookingRepository, and you understood where it comes from —the commit's durability, which is exactly what makes a real database useful and slow—. You saw that on top of the time cost come fragility (more reasons to fail without a bug) and setup (creating and isolating real resources), and that these three costs are the force that gives the pyramid its shape: not a convention, but arithmetic. And you came away with a concrete decision rule —seam risk, contract vs integration, the narrowest possible— that, applied seam by seam, produces the pyramid on its own.

Before moving on you should be able to: explain the 200x factor and why the fake can't be fast and realistic at the same time; run through the decision rule on any seam; and estimate the impact of a test plan on the suite's budget.

With this you close integration's "why": you know what it is, in what proportion, where it lives (the seam), why the green unit test isn't enough, what types there are, and how much they cost. All that's left is to bring it all together with your own hands. Lesson 8, the mini-project, has you demonstrate the gap from start to finish: a unit test with the FakeBookingRepository green, an integration test with the real SqliteBookingRepository red, and the diagnosis of why the green was lying —the whole module, in one deliverable—.

Resources

← Previous6. Types of integrationNext →8. Mini-project: fake green, real red