5. Cache and parallelism for speed

Description

The matrix of the previous lesson gave you version coverage, but it charged you in time: now the pipeline runs the complete work three times per push. If each cell downloads the dependencies from the internet from scratch and runs the tests single file, the feedback cycle —that loop of "I push, I wait, I see the result" that makes the CI useful— lengthens right when you use it most. The layer we stack here attacks that cost on two fronts: caching the dependencies so as not to download them again in each run, and parallelizing the suite with pytest-xdist to run the tests in several processes at once.

It is the layer of module 5, and it brings with it the most important —and most uncomfortable— lesson of performance: not every optimization pays off. The cache almost always helps. The parallelism helps when the suite is slow, and gets in the way when it is already fast, because it has a fixed overhead that has to be amortized. You are going to see pytest-xdist really executed on a slow suite —serial 4.07s dropping to 1.19s with -n auto— and also the honesty of Reservo's case: its real suite is 0.02 seconds, and there -n auto only adds overhead. Measure before optimizing is not a hollow phrase; it is the difference between accelerating and braking.

By the end you will know how to cache dependencies in the YAML, run the suite in parallel with xdist, and —what separates whoever copies -n auto from a tutorial from whoever decides with data— judge when the parallelism pays off in your project and when it does not.

Connection with the module: this is the fourth layer, stacked on the matrix (lesson 4) that made it necessary. It modifies two steps you already have: the dependency-install one (which gains the cache) and the pytest-run one (which gains -n auto). And it lays an uncomfortable bridge to lesson 7: the parallelism changes the order in which the tests run, and that change of order is one of the classic triggers of the flaky —a test that assumed running after another, or with a resource all to itself, is uncovered when xdist distributes them—. You accelerate here; in lesson 7 you manage the side effect.

The moving crew and the boxes they brought again

Think of a moving crew emptying a big house. The first version, naive, does this: a single carrier takes a box up to the truck, comes down, takes the next one up, comes down, one by one, alone. They take the whole day. Two inefficiencies jump out. The first: a single carrier when there could be four working in parallel, each with their pile of boxes. The second, sillier: every time they need tape or bubble wrap, someone goes to the store to buy it again, even though they bought it yesterday and the roll is in the warehouse.

The efficient crew fixes both things. Against the first, they parallelize: four carriers at once, and the move that took eight hours takes two. Against the second, they cache: they keep the tape and the wrap in the warehouse and reuse them, instead of running to the store on each move. The two optimizations attack different times —the parallelism, the repeatable work that can be divided; the cache, the material that does not change between moves— and together they turn a day of work into a morning.

Your pipeline has the two inefficiencies. Downloading the dependencies from the internet on each run is running to the store for tape you already have: Reservo's dependencies do not change from push to push, so downloading them each time is waste —the cache keeps them and reuses them—. Running the tests in series is the solitary carrier: pytest-xdist puts several processes to run different tests at once. With a nuance the crew also knows: hiring four carriers to move one box is absurd —the time to organize them exceeds the time to carry the box—. The parallelism pays off when there are many boxes, not when there is one.

Two optimizations, two wastes. The cache avoids re-downloading what does not change (running to the store for tape you already have). The parallelism divides the work among processes (several carriers). The cache almost always pays off; the parallelism pays off when there is enough work to divide.

Caching the dependencies

Every time the pipeline runs, pip install -r requirements-dev.txt downloads pytest, cov, xdist, rerunfailures, and their transitive ones from the internet. Those packages do not change between runs —they are pinned (lesson 3)—, so downloading them each time is pure waste of time and network. The cache saves them after the first run and restores them in the following ones.

The simplest way is the cache: pip option that setup-python brings, which wraps the caching mechanism for you:

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip                            # caches the pip download
          cache-dependency-path: requirements-dev.txt

Two new lines. cache: pip tells setup-python to save and restore pip's download directory. cache-dependency-path points to the file whose content defines the key of the cache: as long as requirements-dev.txt does not change, the key is the same and the cache is reused; the day you edit a version, the key changes and the cache is rebuilt. That is the magic and the trap at the same time: the cache is valid because it is tied to the content of the dependency file. If the cache were tied to something that does not reflect the dependencies, you could restore old packages —the classic "the cache is poisoned" bug—. Tying it to the hash of requirements-dev.txt keeps it honest.

For fine control —caching specific directories, composing the key by hand— there is actions/cache, more explicit:

      - name: Cache pip downloads
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: pip-${{ hashFiles('requirements-dev.txt') }}

hashFiles('requirements-dev.txt') computes a hash of the file: if the file changes, the hash changes, the key changes, and the cache is rebuilt. For Reservo, cache: pip of setup-python suffices and is cleaner; actions/cache earns its place when you cache something more than pip downloads (build artifacts, models, datasets). Both are the same mechanism underneath.

Parallelizing with pytest-xdist

pytest-xdist runs your suite in several worker processes at once. The star flag is -n:

• -n auto — xdist detects how many CPUs the machine has and starts a worker for each one. It is the idiomatic choice for CI, because it adapts to the runner without your fixing a number.
• -n 4 — a fixed number of workers, when you want control (for example, to not saturate a shared machine).

The pytest step gains the flag:

      - name: Run the test suite
        run: python -m pytest -n auto

xdist distributes the tests among the workers, each one runs its own in its own process, and at the end xdist joins the results. On a slow suite, the effect is dramatic. But "slow" is the keyword, and here is where the honesty of the guide matters.

Worked example 1: the parallelism on a slow suite (it pays off)

Reservo's real suite is 0.02s —too fast for the parallelism to mean anything, as we will see—. To see xdist pay off, I need a slow suite. I built one of eight tests that simulate slow work (each one waits half a second, as a test that hits the network or an external resource would). The sleep stands in for a real call; the point is the time, not what it does. First, in series:

python -m pytest perf_demo/

What to expect (real, eight tests of 0.5s each, in series):

collected 8 items

perf_demo/test_slow_integration.py ........                              [100%]

============================== 8 passed in 4.07s ===============================

Eight tests × 0.5s ≈ 4.07 seconds, one after another. Now with -n auto:

python -m pytest -n auto perf_demo/

What to expect (real, same suite in parallel):

created: 12/12 workers
12 workers [8 items]

........                                                                 [100%]
============================== 8 passed in 1.19s ===============================

From 4.07s to 1.19s —more than three times faster—. Look at the line created: 12/12 workers: the machine has twelve logical CPUs, so -n auto started twelve workers, and with eight tests each one fit in its own worker, running the eight half-seconds almost at once. The total time stopped being "the sum of all" and became "the slowest one plus the overhead of starting and coordinating workers." With a fixed number:

python -m pytest -n 4 perf_demo/

created: 4/4 workers
4 workers [8 items]

============================== 8 passed in 1.32s ===============================

With four workers, the eight tests are distributed in two rounds (four and four), so the theoretical time is ~1.0s plus the overhead: 1.32s measured. A bit slower than the twelve workers, but still an enormous leap over the 4.07s in series. That is the shape of the curve: more workers help until you run out of CPU or of tests to distribute.

Worked example 2: the parallelism on Reservo's suite (it does not pay off)

Now the honesty. Reservo's real suite runs in 0.02 seconds. What happens if I throw -n auto at it? Measure it mentally with what you already know: xdist has a fixed overhead —starting twelve processes, distributing tests to them, collecting results— on the order of a second. That overhead is fifty times greater than the 0.02s the suite takes in series. The result is that -n auto would make Reservo's suite slower, not faster: it would go from 0.02s to over a second, almost all of it in organizing workers that have nothing heavy to do.

It is the crew hiring twelve carriers to move one box: the time to organize them far exceeds the time to carry it. The conclusion is not "xdist is bad"; it is "xdist is the right tool for a slow suite, and Reservo does not have one today." The parallelism is justified when the suite takes long enough to amortize the overhead —seconds, minutes—, not when it is already instantaneous. That is why, in Reservo's pipeline, -n auto is a future investment: you leave it in place knowing that today it does not pay off, so that the day the suite grows to hundreds of slow tests, the acceleration is ready without touching the YAML. Or, with equal judgment, you omit it until the suite asks for it. Both are defensible decisions; the indefensible one is turning it on by reflex believing that "parallelize = faster" always.

The speed/cost trade-off

Accelerating is not free in two currencies. The first is the compute cost: more workers use more CPU and memory at once; on a shared or resource-billed runner, -n auto on a many-core machine can raise the instantaneous consumption. The second, more subtle, is the complexity: the parallelism introduces non-determinism in the execution order, and that non-determinism is a breeding ground for the flaky (lesson 7). A test that in series always ran after another —and unintentionally depended on that order— can fail intermittently when xdist distributes them into different workers.

The practical rule: always cache (it almost never has a downside), and parallelize when the suite is slow enough for the acceleration to exceed the overhead and the risk of uncovering flaky. Measure before deciding: run the suite in series, time it, and only add -n auto if the number hurts. Optimizing without measuring is how you arrive at a Reservo suite fifty times slower "to speed it up."

Common mistakes

Turning on -n auto by reflex on a fast suite. What happens: someone reads "xdist speeds up the CI," puts -n auto on a 0.02s suite like Reservo's, and makes it slower —from instantaneous to over a second— without understanding why. Why it happens: "parallelize = faster" is a mental shortcut that ignores the fixed overhead of starting workers. How to detect it: time the suite in series and with -n auto; if the parallel version is slower or equal, the overhead is costing you. How to fix it: parallelize only when the serial suite takes long enough to amortize the worker startup (seconds and up). For an instantaneous suite, series wins.

Tying the cache to something that does not reflect the dependencies. What happens: someone puts a fixed cache key (key: pip-cache, without a hash) or one tied to something that does not change when the dependencies change. They edit requirements-dev.txt, bump a version, but the cache keeps restoring the old packages because the key did not change, and the pipeline runs with outdated dependencies —a red or, worse, a deceptive green—. Why it happens: the cache "works" (it restores something), so the bug is silent. How to detect it: if you changed a version and the CI keeps using the old one, suspect the cache key. How to fix it: tie the key to the hash of the dependency file (hashFiles('requirements-dev.txt') or cache-dependency-path), so that any change in the dependencies invalidates the cache and rebuilds it.

Optimizing without measuring. What happens: someone stacks cache, -n auto, and other "performance" flags without having timed anything, guided by the feeling that "faster is better." Some help, others get in the way, and without measurement they do not know which or why. Why it happens: optimization feels productive even when blind. How to detect it: if you cannot say in seconds how long it took before and how long after each change, you are optimizing on faith. How to fix it: measure first (time the suite in series), identify the real bottleneck (is it the installation?, is it the tests?), and apply the optimization that attacks that bottleneck. For Reservo today, the bottleneck is not the suite (0.02s); if anything, it would be the repeated installation in the matrix —which the cache attacks—, not the parallelism.

Exercises

Exercise 1 — Predict the effect of -n auto. You have two suites: A takes 90 seconds in series with 600 tests; B takes 0.05 seconds with 20 tests. For each one, predict whether -n auto (on an 8-core machine) would make it faster or slower, and why.

Exercise 2 — The cache key that poisons. A colleague writes key: pip-deps (a fixed key, without a hash) in their actions/cache. Explain what bug this hides and how you fix it, using what you know about how the cache decides to reuse.

key: pip-${{ hashFiles('requirements-dev.txt') }}

Exercise 3 — Cache yes, parallelize not yet. For Reservo's pipeline as it is, decide (and justify) what you do with each of the two optimizations of this lesson: do you cache the dependencies? do you put -n auto? Write the decision as you would defend it in a review.

Summary and next step

In this lesson you stacked the fourth layer: cache and parallelism for speed, the answer to the cost the matrix introduced. You learned to cache the dependencies —cache: pip in setup-python, or actions/cache with the key tied to the hash of the dependency file— so as not to re-download in each run what does not change, the optimization that almost always pays off. And to parallelize with pytest-xdist —-n auto or -n N— which distributes the tests among workers.

Above all, you saw measured the nuance that separates optimizing from braking: on a slow suite, -n auto dropped from 4.07s to 1.19s (twelve workers); on Reservo's real suite, at 0.02s, the parallelism would only add overhead and make it slower. The underlying lesson —measure before optimizing; the parallelism pays off when the suite is slow, not always— is what distinguishes a performance decision with method from an -n auto copied from a tutorial. And the bridge to lesson 7 was seeded: the parallelism changes the execution order, and that change uncovers flaky.

Before moving on you should be able to: cache dependencies with the key tied to the hash of the file; run the suite with -n auto and -n N; predict whether the parallelism will help or get in the way according to how long the suite takes; and justify in a review why you always cache but only parallelize when it pays off.

What follows, in lesson 6, is the layer that changes the nature of the pipeline from "runs the tests" to "demands quality": the coverage gate. So far your pipeline goes red if a test fails; the gate makes it go red if the coverage drops below a threshold —if new code arrives without tests that exercise it—. You are going to see --cov-fail-under break the build for real (exit 1 at 95%, exit 0 at 85%), read which lines are missing, and face the honest decision of where to put the threshold —including why 100% is a fetish and how a Reservo coverage gap is, in fact, code covered by another matrix cell—.

Resources

• pytest-xdist — the documentation of the parallelism plugin: -n auto, -n N, the distribution modes, and the warnings about tests that depend on the order. The reference for what you executed in this lesson.
• Caching dependencies to speed up workflows — GitHub Actions — how actions/cache works, the key, the file hash, and the invalidation. The page that explains why hashFiles keeps the cache honest.
• actions/setup-python: caching packages — the cache: pip option that we used as the simple way, with its cache-dependency-path. Dependency caching in two lines.
• Speeding up your tests — pytest documentation — good practices about suite performance, including when the parallelism helps and when the overhead is not worth it. The backing for the "measure before optimizing" rule.