7. The speed/cost trade-off

Description

The previous lessons gave you two levers to speed up CI and a condition to use them. You might leave here thinking "more parallelism is always better: I bump -n to the max and I'm done". This lesson defuses that idea, because parallelism has a price and its benefit doesn't grow forever. Each worker you open consumes a CPU core; on your laptop that translates into the machine getting slow for everything else, and in CI it translates, very literally, into money: runners with more cores cost more per minute. And —this is what surprises people— past a certain point, adding workers barely speeds things up. Ignoring this leads to paying double for a 10% improvement.

By the end you'll see the real return curve of parallelism, measured on the Reservo suite: how the time drops fast at first (-n 2, -n 4) and then flattens (-n 8, -n 12), so the last workers contribute less and less. You'll understand why module 4's matrix multiplies the cost when you add parallelism, how to decide how much to parallelize weighing speed against resources, and why the module's two levers aren't equally cheap: the cache almost always pays off (it saves without costing you anything real), while parallelism is weighed. The rule you take away is the whole module's, now with numbers: measure where it hurts, parallelize what blocks, don't do everything by reflex.

Connection to the module: this lesson closes the technical arc. Lesson 3 (cache) and lesson 4 (parallelism) gave you the tools; lesson 5 (splitting) and lesson 6 (isolating) gave you how to apply them well; this one gives you the economic criterion for deciding how much. It leans directly on lesson 4 —it takes up its three reasons why the speedup isn't infinite and turns them into a cost curve— and on module 4 —the matrix, which multiplies everything—. It introduces no new tools: it's the decision lesson, as module 4's lesson 7 was for the matrix. The boundary holds: here we decide how much to spend on speed; the strategy of which tests to have is another guide's.

The moving crew: more movers, always faster?

Imagine you hire movers to move a house. With one mover, the move takes eight hours. You hire a second: now two work at once, and they finish in a little over four —almost half—. A third and a fourth: they drop to about two and a half hours. You're doing well. But you keep adding: with eight movers, they finish in two hours; with twelve, in one hour and fifty minutes. The last four barely moved the needle, and you pay all twelve by the hour.

What happened? At first, each extra mover made an enormous difference, because there was a lot of work waiting and few hands. But there comes a point where the movers start to get in each other's way —they cross in the hallway, wait their turn at the door, there's a single staircase— and where, simply, there isn't that much work left to distribute. Mover number twelve charges full and contributes a minute. Hiring twelve when eight finished almost the same is overpaying for almost nothing.

Your tests' parallelism is identical. Each worker is a mover: the first ones speed things up enormously, the last ones get in the way and contribute little. And in CI, each worker costs —CPU, and on big runners, money—. The question isn't "how many workers can I open?" (as many as you want), but "how many are worth it before the next one charges full and contributes a minute?". To answer it, you have to look at the curve.

Parallelism has diminishing returns: the first workers speed things up a lot, the last ones almost nothing, and they all cost. The decision isn't "the maximum of workers", but "the point where the next worker no longer pays its cost".

The real curve, measured

Here's the return curve, measured for real on Reservo's twelve slow tests (each half a second), on the guide's machine (12 cores), with Python 3.14.0. I ran the same -m slow selection varying the number of workers:

Command	Workers	Real time	Speedup vs serial
pytest -m slow (serial)	1	6.11 s	1.0×
pytest -m slow -n 2	2	3.28 s	1.9×
pytest -m slow -n 4	4	1.83 s	3.3×
pytest -m slow -n 8	8	1.49 s	4.1×
pytest -m slow -n auto	12	1.17 s	5.2×

Read the time column top to bottom and look at where it drops and where it flattens:

• From 1 to 2 workers: 6.11 → 3.28 s. Almost half. The second mover paid off enormously.
• From 2 to 4: 3.28 → 1.83 s. Another big drop, almost half again. Doubling workers almost doubled the speed.
• From 4 to 8: 1.83 → 1.49 s. You doubled the workers (from 4 to 8) and only gained ~0.34 s. The return is already flattening.
• From 8 to 12: 1.49 → 1.17 s. You added four more workers and gained ~0.32 s. The last four movers contributed a sigh.

The shape is unmistakable: the time drops fast at first and flattens at the end. Going from 1 to 4 workers gave you most of the benefit (from 6.11 to 1.83, a 3.3×); going from 4 to 12 —three times the workers— only cut from 1.83 to 1.17. If each worker cost money, the first four would be a bargain and the last eight, an expensive luxury. That curve —a lot of return at first, little at the end— is the law of parallelism, and it's the reason "the maximum of workers" is almost never the right answer.

Each row of that table is a real run. This is how, for example, the -n 8 one looks —eight workers for the twelve slow tests—, with its header in view:

python -m pytest -m slow -n 8

What to expect (real output):

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /private/tmp/reservo-m5
configfile: pyproject.toml
testpaths: tests
plugins: xdist-3.8.0
created: 8/8 workers
8 workers [12 items]

............                                                             [100%]
============================== 12 passed in 1.51s ==============================

8 workers [12 items]: eight workers splitting twelve tests, so two of them get a second test —there's the imperfect distribution that prevents the ideal speedup—. An honesty note about these figures: the wall-clock time oscillates a smidge between runs (here 1.51 s; in the table's measurement, 1.49 s), because it depends on the process startup and how the operating system distributes the CPU at that instant. That variation of hundredths doesn't change the curve's shape —what matters is the trend: it drops sharply at first, flattens at the end—, but it's good to know a time benchmark never gives the identical number twice.

There are two reasons behind the plateau, the same as lesson 4's. One: starting workers costs, and that fixed cost weighs more the less work there is to distribute. Two: with twelve tests, more than twelve workers wouldn't help at all (there's no thirteenth test to give the thirteenth worker), and even with twelve, the imperfect distribution leaves some worker with two tests while others finish and wait. The speedup hits against the amount of available work and against the cost of coordinating.

The matrix multiplies the cost

Now connect this with module 4, because together they reveal the real cost. A matrix runs your suite in several combinations —three Python versions, say—. Parallelism opens several workers within each run. The two multiply.

Think about it: a matrix of 3 versions × 3 operating systems is 9 cells, each a complete run of the suite. If each cell runs with -n auto on a runner of, say, 4 cores, you're using 9 × 4 = 36 CPU "worker-runs" per push. In billed minutes, that can be the order of magnitude between "CI costs little" and "CI is a serious line on the cloud bill". Parallelism within each cell speeds up that cell, yes —which is fine, because a slow matrix is doubly annoying—, but the total resources consumed is the product of the two dimensions.

The practical consequence isn't "don't use matrix or parallelism" —both are valuable—, but be aware of the product. Module 4's lesson 7 already taught you to trim the matrix to what really matters (test what you ship plus what you promise to support, and nothing more); this lesson adds the other half: within each cell, parallelize enough not to block, not the maximum possible. A minimal matrix with sensible parallelism costs a fraction of an inflated matrix with -n auto on everything, and protects almost the same.

The two levers don't cost the same

Here's an asymmetry worth recording, because it changes the order in which you turn on the levers. Cache and parallelism aren't equally expensive.

The cache almost always pays off, and its cost is nearly nil. Saving and restoring a few megabytes of dependencies is cheap, and the saving —not re-downloading or reinstalling on every run— is pure. It doesn't consume extra CPU during the tests, doesn't force you into a bigger runner, doesn't have diminishing returns: either there's a cache hit (you save) or a miss (you don't save this time, but you don't lose either). Except in rare cases (enormous caches that take longer to restore than to reinstall), turning on the cache is an easy decision: do it almost always. It's the first lever, the lowest-risk one.

Parallelism is weighed. It really speeds things up, but it consumes CPU proportional to the workers, has diminishing returns (the curve above), and in CI it translates into cost. It's not "turn it on to the max and forget"; it's "measure how much you need and stop where the return flattens". -n auto is a great default —it squeezes what's there without you nailing down a number—, but on a shared or per-core-paid runner, a deliberate -n 4 that captures most of the speedup at a third of the cost can be the best engineering decision.

That's why the recommended order is: first the cache (cheap, almost always pays off), then parallelism (powerful, but measured). If your CI is slow, cache the dependencies without much thought, and then look at the curve to decide how many workers are worth it.

When -n auto is too much

-n auto isn't always the answer. Three cases where something else is best:

A suite that's already fast. If your suite runs in 0.2 s serially, -n auto will make it slower, because starting twelve workers costs more than the 0.2 s you save. Here the answer is not to parallelize: serial is correct. Parallelism pays off when there's enough work; the instant suite doesn't have it.

A shared or budget-limited runner. If CI runs on runners you pay for per core, or shared with other teams, -n auto hogs the whole machine. An -n 4 that captures, per the curve, a 3.3× at a third of the resources can be the sweet spot: almost all the speed, much less cost. You measure your own curve and pick the elbow —where the line stops dropping sharply—.

Your laptop while you work. Locally, -n auto with all the cores can freeze your machine for everything else while the tests run. An -n 4 leaves cores free for you to keep coding. The fixed number is control over how much machine you cede.

The rule that unites the three: -n auto is the default for squeezing dedicated hardware, but when the resource is shared, paid, or scarce, a fixed number chosen with the curve in hand is wiser. Measuring your curve —running -n 2, -n 4, -n 8 and seeing where it flattens— takes five minutes and tells you your exact elbow.

Common mistakes

Bumping -n to the max believing it always speeds up proportionally. What happens: someone goes from -n 4 to -n 16 expecting to quadruple the speed, and the time barely drops —or goes up, if the machine doesn't have 16 cores and the workers fight over the ones it has—. Why it happens: the intuition "twice the workers, twice as fast" ignores diminishing returns and the core ceiling. How to spot it: measure with -n 2/4/8; if the time stops dropping, you found your plateau. How to fix it: pick the number at the curve's elbow, not the maximum; beyond the elbow you pay for workers that contribute a sigh.

Combining a big matrix with -n auto without seeing the product. What happens: someone has a 9-cell matrix and puts -n auto on each, and the CI minute bill skyrockets without anyone understanding why. Why it happens: you think about each lever separately, not their multiplication. How to spot it: multiply cells × workers; if the product is big, that's where your budget goes. How to fix it: trim the matrix to the essential (module 4) and use a measured parallelism per cell; the total cost is the product, so lower either of the two factors.

Parallelizing before caching. What happens: someone invests effort in fine-tuning -n while their CI keeps spending two minutes per run reinstalling dependencies that didn't change. Why it happens: parallelism is more "exciting" and visible than the cache. How to spot it: look at how much goes into installing dependencies; if it's a big slice, the cache gives you more saving with less effort than fine-tuning workers. How to fix it: turn on the cache first (cheap, almost always pays off) and then adjust the parallelism. The order matters: the cheap lever goes before the one that's weighed.

Exercises

Exercise 1 — Find the elbow. With Reservo's measured curve (serial 6.11 s; -n 2 3.28 s; -n 4 1.83 s; -n 8 1.49 s; -n 12 1.17 s), a teammate asks how many workers you'd recommend if CI charges per core. Answer with a number and justify it with the curve.

See solution

I'd recommend -n 4. Justification with the curve: from 1 to 4 workers, the time drops from 6.11 to 1.83 s —a 3.3×, most of the possible benefit—. From 4 to 8 workers (twice the resources) you only gain from 1.83 to 1.49 s (~0.34 s), and from 4 to 12 (three times the resources) only to 1.17 s (~0.66 s). That is: the first four workers capture most of the speedup, and the next eight contribute less and less for a growing cost.

If CI charges per core, -n 4 is the sweet spot: almost all the speed (3.3× of a possible 5.2×) at a third of -n 12's resources. Paying for 12 workers to gain 0.66 s over 4 is "hiring four extra movers to finish a minute earlier". If the hardware were dedicated and free, -n auto (12) would be fine —it squeezes what's there—; but with per-core cost, the curve's elbow is around 4, and that's where it's best to stop.

Exercise 2 — Calculate the matrix × parallelism product. A team has a matrix of 3 Python versions × 3 operating systems, and runs each cell with -n auto on 4-core runners. (a) How many cells are there? (b) How many CPU "worker-runs" does a push consume? (c) What two things could they trim to lower the cost, and which would you trim first?

See solution

• (a) 3 versions × 3 operating systems = 9 cells.
• (b) Each cell uses -n auto = 4 workers (on a 4-core runner), so 9 × 4 = 36 CPU worker-runs per push. That's the product of the two dimensions —matrix and parallelism— that multiplies the cost.
• (c) They could trim the matrix (fewer cells: maybe they don't need all 9, only the versions/OSes they really use or promise to support —module 4—) or the parallelism per cell (drop from -n auto to a lower -n if the curve shows 4 workers don't contribute that much). I'd trim the matrix first, because each cell you remove saves a complete run of the suite (the heaviest factor), and a 9-cell matrix for an app that runs in a single environment is usually pure noise (module 4's lesson 7). Then I'd fine-tune the parallelism within the remaining cells.

The idea: the cost is the product of cells × workers, so lowering either of the two factors helps —but trimming the matrix usually gives the biggest saving with the least risk, because it eliminates whole runs.

Exercise 3 — Order the levers. A CI takes 10 minutes: ~4 min reinstalling dependencies on every run and ~6 min running 2000 independent tests in a line. With what you know about the trade-off, in what order would you apply the levers and why?

See solution

I'd apply the cache first, then parallelism. Reasons:

1. The cache first, because it's cheap and almost always pays off. The 4 minutes of reinstalling identical dependencies on every run are pure sink-1 waste: actions/cache with the hash key drops them to almost zero (the packages are restored in seconds) with no real cost —it consumes no extra CPU or forces a bigger runner—. It's the biggest improvement with the least effort and risk. From 10 min we'd drop to ~6.
2. Parallelism after, measured. The 6 minutes of 2000 tests in a line are sink 2, and the tests are independent (lesson 6's requirement is met), so -n distributes them. But here I measure the curve before fixing the number: I try -n 4, -n 8, and pick the elbow, instead of putting -n auto blindly —especially if the runner is paid per core—. With sensible parallelism, those 6 min can drop to one or two.

The order matters: the cache is the low-risk, high-immediate-return lever, so it goes first; parallelism is powerful but weighed (consumes CPU, has diminishing returns), so it goes after and with the curve in hand. Result: from 10 minutes to ~3, without deleting a single test.

Summary and next step

In this lesson you put a price on speed. You saw it with the moving crew: the first movers speed things up enormously, the last ones get in the way and contribute a minute, and you pay them all. You measured the real parallelism curve on Reservo —serial 6.11 s, -n 2 3.28 s, -n 4 1.83 s, -n 8 1.49 s, -n 12 1.17 s— and saw its law-like shape: it drops fast at first and flattens at the end, so going from 1 to 4 workers gives most of the benefit and from 4 to 12 cuts barely. You understood that the matrix multiplies the cost (cells × workers), that the two levers don't cost the same —the cache almost always pays off, parallelism is weighed—, and that -n auto is a good default but not always the answer: on shared, paid runners or on your laptop, a fixed number chosen with the curve is wiser.

The rule you take away, now with numbers behind it: turn on the cache almost always (cheap), parallelize what blocks with a measured number (not the max), and trim the matrix to the essential —because the total cost is the product of everything—. Speeding up with a clear head is as important as speeding up.

Before moving on you should be able to: describe the shape of the parallelism return curve and why it flattens; find the elbow of a measured curve and recommend a number of workers based on cost; calculate the matrix × parallelism product; and justify why the cache is turned on before parallelism.

What's next, in lesson 8, is the mini-project that pulls the whole module together: you speed up Reservo's CI end to end. You're going to write the workflow with actions/cache and pytest -n auto, measure the real speedup with local parity (serial vs -n auto, with times), diagnose and fix the test that broke in parallel —the shared Calendar to a fixture, verified by running before and after—, and write a trade-off and scope note reflecting this lesson's decisions. It's the practical exam of everything you learned about making CI fast without breaking it.

Resources

• How to run tests in parallel — pytest-xdist documentation — the official guide to how to choose the number of workers and the distribution modes, the basis for measuring your own curve. The reference for deciding your elbow.
• About billing for GitHub Actions — how CI minutes are counted and billed, including the bigger runners that cost more per minute. This lesson's "cost in money", concretely.
• Using larger runners — GitHub Actions — the documentation of the runners with more cores, which speed up parallelism but cost more. Useful for understanding the other side of the resource trade-off.
• Caching dependencies to speed up workflows — GitHub Actions — the cheap lever this lesson recommends turning on first. Review it to confirm why the cache almost always pays off.