Module 1: From Loose Scripts To A Framework

7. The return on investing in architecture

Next →

Overview

By the end of this lesson you will know how to decide how much to invest in your suite's architecture and when, which is a skill as important as knowing how to build the fixtures. Because up to here the guide has shown you the pain of not having a framework, and you could leave with the naive conclusion of "then always, the more framework, the better". It is false. Building the framework also costs —time, indirection, a fixture you have to understand before reading a test— and over-building is its own mistake: fixtures nobody uses, factories for objects that appear once, abstractions that solve problems you do not have. The wisdom is not "architecture yes/no", it is where on your suite's curve you are and what investment corresponds to that point.

This lesson traces that curve: why the same fixture that at ten tests seems over-engineering, at a thousand is the only thing that keeps the suite alive. It gives you the operational rule to invest at the right moment —the third duplicate— and the flip side, the signs that you over-invested. And it shows you, executed, how the investment pays out: in a framework, adding test number seven costs one line, and a model change that without architecture would touch hundreds of places is done in a single one, with the whole suite green.

Connection to the module: this is the lesson that turns all the pain of the previous ones into an economic decision. Lessons 2 and 6 measured the cost of not having a framework; this one puts on the other side the cost of having it and teaches you to compare. It is the last conceptual lesson of the module: after it, the mini-project (lesson 8) has you make the smallest and highest-return investment there is —extracting your first shared fixture— and measure its payoff with your own hands. And from here on, every layer you build in modules 2 to 7 you will evaluate with the compass of this lesson: does this framework solve a pain I have, or is it scaffolding I admire but do not need?

Analogy: buying the machine when the volume pays for it

A tailor shop that makes three garments a month sews the buttonholes by hand. It is slow per garment —fifteen minutes each buttonhole— but it works, and buying a forty-thousand-peso industrial buttonholer for three garments a month would be madness: the machine would pay for itself in twenty years. At that volume, "doing it by hand" is the correct decision, not the lazy one.

The same shop, when it starts producing three hundred garments a month, sews buttonholes by hand for entire days. Now the forty-thousand buttonholer pays for itself in two weeks: what used to cost fifteen minutes per garment comes to cost seconds, multiplied by three hundred. At that volume, continuing to sew by hand stops being artisanal humility and becomes an economic mistake —you are burning days of work to save an investment that would pay for itself—.

Notice the two morals, because both matter. The first: the same decision (buying the machine) is wrong at low volume and correct at high volume —there is no universal answer, there is a crossover point—. The second: the mistake exists in both directions —buying the machine for three garments is over-investing; sewing three hundred by hand is under-investing—. Your suite's architecture is the buttonholer. The shared fixture, the marker, the factory: each is a machine that costs to set up and that pays off by volume. Your job is not "set up all the machines always"; it is to recognize when your volume crossed the point where the machine pays for itself.

The cost curve: two lines that cross

Put two lines in your head, on an axis where the horizontal is "size of the suite" (from 10 to 1000 tests) and the vertical is "total cost of maintaining it".

The suite without architecture starts cheap: at ten tests, copying the setup costs almost nothing, and there are no fixtures to understand. But its cost grows fast —remember lesson 6: it is O(N × C), the number of tests times the frequency of changes—. Each new test copies the setup; each model change touches all the places. The line rises with an increasing slope.

The suite with architecture starts more expensive: setting up the first fixture costs five minutes the copy-paste version does not pay, and there is an indirection (the conftest.py) a new reader has to know. But its cost grows slow —O(C), almost flat—: each new test reuses the fixtures without adding setup, and each model change is done in one place. The line rises with an almost horizontal slope.

Two lines: one that starts low and rises steep, another that starts a little higher and rises flat. They cross. To the left of the crossover —small suites—, the without-architecture one is cheaper: that is why setting up a framework for ten tests is genuine over-engineering. To the right of the crossover —suites that grow—, the with-architecture one is dramatically cheaper, and the distance between the lines opens without limit. All the skill of this lesson is to estimate where the crossover is for your case and not stay on the wrong side: neither setting up the machine before the crossover (over-investment), nor continuing by hand long after (under-investment, the debt of lesson 6).

The operational rule: the third duplicate

"Estimate the crossover" is correct but hard to apply in the moment. Fortunately there is a practical rule, inherited from software engineering and tuned for tests, that gets it right almost always:

The first time you write a setup block, write it. The second time you need it, copy it —yes, copy it— and endure the discomfort. The third time, extract it to a shared fixture.

Why the third and not the second? Because with two occurrences you still do not know which is the stable part of the pattern and which varies. Extracting at the second makes you guess the abstraction, and guessing early produces fixtures with too many parameters or with assumptions the third case breaks. By the third use you already see the pattern: what repeats identically (goes to the fixture) and what changes between cases (stays as a parameter or in the test). Extracting at the third is extracting with evidence, not with a hunch. And the cost of having copied twice is trivial —two blocks— compared to the cost of a wrong abstraction that has to be undone.

The rule also protects you from the opposite mistake. If a setup block appears only once, the rule tells you explicitly: do not extract it. A fixture used by a single test shares nothing; it only adds an indirection that forces the reader to jump files to understand a test they would have read straight through. That is over-investment, and the third-duplicate rule prevents it as well as it prevents under-investment.

Worked example: how the investment pays out

Let us see the return concretely, with a Reservo pricing suite already built on fixtures. The conftest.py has the shared world:

# conftest.py
import pytest

from reservo.models import Room, Member


@pytest.fixture
def focus_room():
    return Room(id="r1", name="Focus", capacity=1, hourly_cents=2500)


@pytest.fixture
def basic_member():
    return Member(id="m1", name="Ana", tier="basic")


@pytest.fixture
def pro_member():
    return Member(id="m2", name="Bruno", tier="pro")

With that investment made, look at what each new test costs. There is no setup: each test is one line that requests its world and asserts its anchor.

# test_pricing.py
from reservo.pricing import price_cents


def test_basic_1h(focus_room, basic_member):
    assert price_cents(focus_room, basic_member, 1) == 2500


def test_basic_3h(focus_room, basic_member):
    assert price_cents(focus_room, basic_member, 3) == 7500


def test_pro_1h(focus_room, pro_member):
    assert price_cents(focus_room, pro_member, 1) == 2000


def test_pro_3h(focus_room, pro_member):
    assert price_cents(focus_room, pro_member, 3) == 6000


def test_basic_5h(focus_room, basic_member):
    assert price_cents(focus_room, basic_member, 5) == 12500


def test_pro_5h(focus_room, pro_member):
    assert price_cents(focus_room, pro_member, 5) == 10000

What to expect. With pytest test_pricing.py -q:

......                                                                   [100%]
6 passed in 0.01s

Return 1: the marginal cost of a new test is one line. I want to cover two more cases —basic 2 h (5000) and pro 2 h (4000)—. In the copy-paste suite, each would be eight lines (build room, member, calendar, assert). Here they are two lines each:

def test_basic_2h(focus_room, basic_member):
    assert price_cents(focus_room, basic_member, 2) == 5000


def test_pro_2h(focus_room, pro_member):
    assert price_cents(focus_room, pro_member, 2) == 4000

What to expect. With the two tests added, pytest test_pricing.py -q:

........                                                                 [100%]
8 passed in 0.01s

Eight tests, and the two new ones cost one setup line each —zero, really: they requested fixtures that already existed—. That is the first dividend of the investment: once the framework is set up, the suite grows almost for free. Each test you add inherits the world, instead of rebuilding it.

Return 2: the model change is done in one place. Now the blow that in lesson 2 broke six tests and in lesson 6 broke two hundred: Room gains owner_id. With the copy-paste suite, it would be eight edits (one per test). With the framework, the Focus room is built in a single place —the focus_room fixture—, so the fix is one edit there:

@pytest.fixture
def focus_room():
    return Room(id="r1", name="Focus", capacity=1, hourly_cents=2500, owner_id="o1")

What to expect. With the model changed and the fixture fixed in that single place, pytest test_pricing.py -q:

........                                                                 [100%]
8 passed in 0.01s

Eight tests green, and we did not touch any of the eight: just the fixture. There is the investment paying out. The same Room move that without architecture was eight (or two hundred) fragile edits, with architecture is one edit in the obvious place. Multiply those two returns —tests that grow for free, changes done at a single point— by the life of a project and you understand why, past the crossover, the with-architecture line is not only cheaper: it makes possible a suite that without it would have become ungovernable.

The other side: when NOT to invest

For the lesson to be honest, we have to say with the same force when architecture is a mistake. Over-investing has its own symptoms, and recognizing them keeps you from becoming the person who sets up the buttonholer for three garments:

  • Fixtures that a single test uses. If a fixture has a single consumer, it shares nothing: it is a helper function disguised as architecture that only adds the cost of jumping files. Leave the setup in the test.
  • Factories for objects that appear once. A factory is justified when many tests manufacture variations of the same object. For a single object, it is construction with extra steps.
  • Markers that never filter. A marker that no -m selects orders nothing; it is a dead label that clutters the config.
  • Abstraction before the third use. Extracting a fixture at the first or second use is guessing the shape of the pattern before having evidence; it usually produces the wrong abstraction, which costs more to undo than the copy-paste it avoided.
  • Framework for a suite that is not going to grow. A single-use test script, an exploratory test you will delete tomorrow: it does not deserve infrastructure. Architecture is a bet that the suite will live and grow; if it will not, do not bet.

The mental test: before building any piece of the framework, ask yourself "what concrete and present pain does this cure?". If you can name the pain —"I have the Focus setup copied in twenty tests and Room changes often"—, invest. If the answer is "it is good practice" or "just in case it grows", do not invest yet: you are admiring a machine your volume does not yet pay for.

Common mistakes

Over-architecting out of fear of the pain of lesson 2. What happens: someone leaves scared from seeing 200 failed and sets up fixtures, factories and markers for their fifteen-test suite, "to avoid suffering that". They end up with more framework than suite, and with abstractions nobody needs. Why it happens: fear confuses "this may hurt someday" with "it hurts me now". How to detect it: count the consumers of each piece; if you have fixtures with one user or markers that do not filter, you over-invested. How to fix it: apply the third duplicate. Architecture is a response to a present pain, not insurance against a hypothetical one.

Under-investing waiting for "the right moment" that never comes. What happens: "when it scales, we'll architect". The suite scales, but migrating a thousand copy-paste tests to fixtures is so expensive that it is never prioritized, and the debt becomes permanent. Why it happens: the retrofit is visible and expensive; the debt is invisible and diffuse. How to detect it: if your architecture plan is always in the future, you are already under-investing. How to fix it: invest incrementally, at the third duplicate, every time —that way you never accumulate a thousand places to migrate at once—. Cheap architecture is the one built little by little, when each pattern matures, not the one postponed for a big redesign.

Measuring the return only in lines saved. What happens: someone evaluates whether a fixture "is worth it" by counting how many setup lines it deletes, and concludes that a fixture saving two lines per test does not pay off. Why it happens: the writing saving is the most visible return, but it is the smallest. How to detect it: if your return calculation ignores the cost of future changes and confidence, you are measuring wrong. How to fix it: the big return of a fixture is not writing less today; it is changing in one place when the model evolves (lesson 2) and protecting the signal of the suite (lesson 6). A fixture that saves two lines per test but turns two hundred future edits into one is worth a great deal, even if the writing saving seems small.

Exercises

Exercise 1 — Apply the third duplicate. For each situation, say whether you would extract a fixture/helper now, wait, or never do it, and why. (a) You just wrote the first test that builds a Calendar with three bookings. (b) It is the second time you copy that block of three bookings. (c) It is the fourth time. (d) A single, exploratory test that you will delete once you understand a bug. (e) A setup block that appears in 40 tests from day one of a new suite.

See solution
  • (a) First time: do not extract. Write it in the test. You still do not know whether the pattern will repeat or which is its stable part.
  • (b) Second time: copy it and endure. Two occurrences are not enough to see the shape of the pattern; extracting now is guessing the abstraction. Copy and move on.
  • (c) Fourth time: extract now. You passed the third duplicate comfortably; the pattern is clear and the copy-paste already hurts. It goes to a fixture (or factory if each use varies).
  • (d) Single, disposable test: never. It does not deserve infrastructure; you will delete it. Architecting something ephemeral is pure cost with no return.
  • (e) 40 uses from day one: extract immediately. Here the "third duplicate" is already met forty times before starting. You do not have to wait to copy it forty times to know it repeats; the pattern is obvious. The rule of the third is a floor, not a ceiling: if you see forty uses up front, you invest up front.

Exercise 2 — Locate the crossover. Two Reservo suites: suite A has 8 tests and its model almost never changes (it is a stable library); suite B has 600 tests and its model changes every two weeks (it is a product in evolution). For each one: (a) which side of the crossover is it on? (b) what architecture investment corresponds? (c) what mistake would be more likely to make in each one?

See solution
  • Suite A (8 tests, stable model). (a) To the left of the crossover: few tests, low change frequency, that is, N and C low. (b) Minimal investment: perhaps one or two fixtures if there is setup repeated three times, but no elaborate factories or markers. Moderate copy-paste is acceptable here. (c) The likely mistake is over-investing: setting up a full framework for eight tests that barely change is the buttonholer for three garments.
  • Suite B (600 tests, volatile model). (a) To the right of the crossover, very far: N high and C high, that is, the multiplier N × C is enormous. (b) Strong and early investment: fixtures for all the shared setup, factories for the variations, markers to run subsets, per-environment config. (c) The likely mistake is under-investing: leaving copied setup in a suite that changes every two weeks is accumulating the debt of lesson 6, with routine massive reds and erosion of confidence.

The lesson: the same amount of architecture is over-investment in A and under-investment in B. There is no correct answer in the abstract; there is a correct answer for the point on the curve where your suite is, and that point is set by the product N × C.

Exercise 3 — Count the full return, not just the lines. A focus_room fixture saves 1 setup line per test. The suite has 300 tests that use it, and in the next year the Room model will change 5 times. (a) How many writing lines does the fixture save (the visible return)? (b) How many future maintenance edits does it save (the big return)? (c) If someone argues "it only saves one line per test, not worth it", what are they ignoring?

See solution
  • (a) 299 writing lines, approximately. The room is defined once in the fixture instead of 300 times in the tests: you save the 300 constructions minus the single definition. It is real, but it is the smaller return.
  • (b) 1495 maintenance edits (5 changes × 299 places you do not have to touch). Without the fixture, each of the 5 Room changes would force editing 300 lines: 1500 fragile edits in the year. With the fixture, each change is 1 edit: 5 in total. The saving is ~1495 edits, and each one avoided is also an error of omission avoided.
  • (c) It ignores the big return and the invisible return. "One line per test" measures only the initial writing (smaller return, (a)). It does not count the ~1495 maintenance edits the fixture saves (b), nor the protection of confidence —those 5 changes, without the fixture, would produce massive reds that erode the suite's signal (lesson 6)—. The return of a fixture is almost never in writing less today; it is in changing in one place tomorrow and in keeping the suite credible. Measuring only today's lines is like valuing the buttonholer by what it saves on the first garment.

Summary and next step

In this lesson you did the math of architecture. You learned that the framework also costs and that over-investing is a mistake as real as under-investing —the buttonholer for three garments is as wrong as sewing three hundred by hand—. You saw the cost curve: two lines that cross, the without-architecture one cheap at first but with an increasing slope (O(N × C)), the with-architecture one a little more expensive at first but almost flat (O(C)); all the skill is not staying on the wrong side of the crossover. You have the operational rule —the third duplicate: write, copy, extract— that gets it right almost always and protects you in both directions. And you saw the return executed: once the framework is set up, each new test costs one line (8 passed with two tests added for free) and a model change is done in a single place (8 passed touching only the fixture). The big return is not writing less today; it is changing at one point tomorrow and protecting confidence in the suite.

Before moving on you should be able to: draw the cost curve with its two lines and explain the crossover; apply the third-duplicate rule in concrete cases; recognize the signs of over-investment (single-use fixtures, markers that do not filter, abstraction before the third use); and calculate the full return of a fixture —writing, future maintenance and confidence—, not just the lines saved.

What comes next is to leave the theory and make the smallest and most profitable investment of all with your own hands. The mini-project of lesson 8 hands you the Reservo suite with setup duplicated in six tests —the same one that in lesson 2 broke in six places— and asks you to extract its first shared fixture into a conftest.py, without changing what each test asserts. You will run it until you see 6 passed, apply the model change that used to break six places to fix it in one, and deliver the two outputs side by side: the "before" and the "after" of the whole module, measured with pytest.

Resources

← Previous6. The suite that doesn't scaleNext →8. Mini-project: extract your first shared fixture