2. Why a slow CI gets ignored

Description

A slow pipeline doesn't fail loudly. It doesn't crash, doesn't error out, doesn't paint anything red. It simply takes time, and that delay, which seems like the most benign problem in the world, is the one that ends up emptying of meaning all the work you did in the previous modules. Because a CI exists to give feedback: it warns you, on every change, whether you broke something. And feedback has a property people underestimate: its value plummets over time. A warning in thirty seconds changes what you do next; the same warning in twenty minutes arrives when you've already switched tasks, and you ignore it.

By the end of this lesson you'll understand why speed isn't a luxury but a condition for CI to be used, and you'll be able to point out where the time goes in a run. You'll see that slowness breaks the feedback loop in concrete and human ways —people merge without waiting for the green, silence the check that "always takes forever", lose track of what they were doing—, and that a run's time is split, almost always, between two big sinks: installing the dependencies (over and over, identically) and running the tests (in a line, one after another). Those two sinks are, exactly, the ones the module's two levers are going to attack. This lesson runs no new tools; it frames the problem precisely so the solutions that follow land on the real waste.

Connection to the module: lesson 1 gave you the map and a speedup demo. This lesson answers the question left open: why that speedup is worth it, beyond comfort. It's the "why" that justifies the three technical lessons coming —cache (3), parallelism (4), splitting the suite (5)— and also the trade-off lesson (7), because once you understand what slowness costs you, you can decide with a clear head how much to invest in removing it. Here we touch no new code; we measure the problem. The boundary holds: the intermittent failure that erodes trust in another way —the flaky— belongs to module 7; here we talk about the silent failure of waiting, which doesn't give red but does just as much damage.

The traffic light that takes five minutes to change

Imagine an intersection with a broken traffic light: it takes a full five minutes to go from red to green. At first, drivers respect the law and wait. But five minutes is an eternity stopped in front of an empty street, and soon the inevitable starts to happen: someone looks both ways, no one's coming, and runs it. Then another. In a week, the light still "works" —it changes color, completes its cycle—, but no one obeys it. It became an ornament. And the dangerous part is that the day a car really does come down the cross street, the driver who got used to running the red no longer brakes.

The light did its technical job perfectly; its problem was time. Five minutes is more than human patience tolerates for a benefit it doesn't see, so people rationalized skipping it. A light that changes in twenty seconds is obeyed without thinking; one that takes five minutes is ignored without guilt.

Your CI is that traffic light. Its red says "don't merge this, something broke"; its green says "go ahead, the suite passes". If the green takes twenty seconds, people wait for it: it's faster to wait than to risk it. If it takes twenty minutes, people start to "run the red" —to merge without seeing the result, to trust that "it surely passes", to silence the notification that always arrives late—. The pipeline keeps running, keeps changing color, but it stopped governing the team's behavior. And the day you really do break something, the warning will arrive when the change is already merged and someone has built on top of it.

CI doesn't protect by existing, but because people wait for its verdict before acting. When the wait becomes intolerable, the team stops waiting —and a verdict no one waits for protects nothing—. Speed is what keeps the traffic light obeyed.

How the feedback loop breaks, concretely

"The slow CI gets ignored" sounds like a motivational phrase. It's worth bringing down to the exact mechanisms, because each has a different remedy and they all point to speed.

The merge without green. The healthy flow is: you open a pull request, CI runs, you wait for the green, you merge. When CI takes fifteen minutes, that "wait for the green" becomes a dead hole in your day. So people do one of two things: they merge as soon as their local tests pass (without waiting for CI, which "surely matches"), or they configure the branch not to require the check (so as not to be blocked). In both cases, CI stopped being a gate and became a late comment. The failure it caught —a version incompatibility, a test that only breaks on Linux— is now discovered after the merge, when it costs ten times more to fix.

The context switch. When CI gives feedback in thirty seconds, you're still looking at the same screen, with the problem fresh in your head; if something fails, you fix it right there. When it takes fifteen minutes, you're not going to sit staring at a progress bar: you open another branch, start another task, answer a message. And when the red finally arrives, you have to go back to a context that's already cooled: remember what you were doing, reload the problem into memory, reconstruct the thread. That re-entry cost is real and expensive, and it grows with the wait time. A fast CI lets you fix while it's hot; a slow one forces you to reheat.

The normalization of slowness. The most insidious: people get used to it. "CI takes forever, it always takes forever, that's just how it is." The fifteen-minute wait stops being perceived as a problem to solve and becomes a fact of nature, like rain. And once normalized, no one attacks it —why, if "it's always been this way"?—. The pipeline swells year after year, everyone adds their slow test, and the number goes from twelve to twenty to thirty minutes without anyone pulling the brake, because the increase is gradual and slowness is already the air you breathe.

The three mechanisms share a root: waiting costs, and above a certain threshold the team stops paying it and starts evading CI instead of using it. The good news is that the wait is, almost always, avoidable waste. To see it, you have to look at where it goes.

Where the time really goes

When CI takes long, the instinctive reaction —"let's buy it a more powerful machine"— is usually the wrong one, because it doesn't attack the source. Before speeding up, you have to measure where the time goes. In a typical Python test pipeline, almost all the clock is split between two sinks, and it's worth seeing them separately because each has its own lever.

Sink 1: installing the dependencies. Each run starts on a clean machine (that's CI's whole point, you saw it in module 1: a fresh environment that doesn't drag your local garbage). Clean means without your libraries, so each run runs pip install -r requirements.txt from scratch: it downloads the packages from the internet, unpacks them, installs them. For Reservo it's fast —one dependency—, but a real project with twenty or fifty dependencies can spend one or two minutes just on this, on every run, even though requirements.txt hasn't changed in weeks. It's the buffet cook cutting the same vegetables over and over. And if you have a matrix of three versions, that's three identical reinstalls per push. This sink is repeated work, and its lever is the cache (lesson 3).

Sink 2: running the tests. Once everything's installed, pytest goes through the suite and runs the tests one after another, in a single process. For 11 instant tests, imperceptible. For a big suite, or one with expensive tests —the ones that wait for the network, render something, spin up a subprocess—, this dominates the clock. Reservo's report suite exaggerates it on purpose: twelve half-second tests, run in a line, are six seconds of pure waiting in sequence. This sink is queued work, and its lever is parallelism (lesson 4): distributing the tests among several processes that run at once.

Here's the split, measured on the Reservo suite with the slow part included. Serially, the complete suite takes this:

============================== 23 passed in 6.10s ==============================

Of those 6.10 s, practically 6 s are the twelve slow tests running in a line (sink 2), and a tiny fraction is everything else. In a real project there would also be a good chunk of "installing dependencies" before even starting to run tests (sink 1), which you don't see locally because your packages are already installed, but which the clean runner pays every time.

The operational conclusion is the one I want you to take away: don't speed up blindly. Look at the run and ask yourself what fraction goes into installing (→ cache) and what fraction into running the tests in a line (→ parallelism). Buying a faster CPU doesn't help the first sink (downloading from the internet doesn't go faster with more GHz) and helps little with the second if you don't distribute the work. The module's two levers are tailored to these two sinks; that's why they work.

A nuance: fast isn't the same as less coverage

There's a dangerous temptation worth defusing now, because it's the bad way to speed up a CI: deleting tests. "CI takes too long, let's remove the slow tests." That does speed it up, but at the cost of the only thing CI contributes —the coverage, the certainty that something works—. It's like removing the traffic light so no one has to wait for it: fast, yes, and dangerous.

This whole module is about speeding up without losing coverage. The cache deletes no test: it installs the same, only reusing instead of re-downloading. Parallelism deletes no test: it runs the 23 tests, exactly the same ones, only at once instead of in a line —that's why the result is still 23 passed, not 18 passed with five deleted—. Splitting the suite (lesson 5) also deletes nothing: it reorders what runs first. The goal is to lower the wall-clock time —the seconds a human waits— while keeping intact the amount of truth the pipeline verifies. If you ever "speed up" your CI by removing tests, you didn't speed it up: you weakened it.

Common mistakes

Measuring CI health only by color, never by the clock. What happens: a team checks that the pipeline is green and calls its health good, without ever looking at how long it takes. The time climbs month over month until CI is a bottleneck everyone evades, but since "it's green", no one counts it as a problem. Why it happens: color is visible and binary; time is a gray number you have to go look for. How to spot it: note the time of your last ten runs and look at the trend. If it climbs and climbs, you have a problem that color wasn't going to tell you about. How to fix it: treat run time as a first-class metric, with a budget ("CI shouldn't exceed N minutes"), and act when it exceeds it.

Speeding up by buying hardware without looking at where the time goes. What happens: someone moves to a bigger, more expensive runner expecting CI to fly, and it barely improves, because half the time went into downloading dependencies from the internet —which doesn't depend on the CPU— and the other half into tests in a line —which one faster process barely speeds up—. Why it happens: "more powerful = faster" is a reasonable intuition but blind to the problem's structure. How to spot it: if you doubled the power and the time dropped 10%, it wasn't a power problem. How to fix it: measure the two sinks and apply the correct lever —cache for the repeated, parallelism for what goes in a line—; hardware is the last card, not the first.

"Speeding up" by deleting or skipping tests. What happens: under pressure from a slow CI, someone deletes the slow tests or marks them to be skipped permanently, and CI drops from fifteen minutes to three. It feels like a victory until a bug those tests caught reaches production. Why it happens: deleting tests is the fastest and most tempting way to lower the number, and the cost (less coverage) isn't seen until later. How to spot it: if your CI sped up and the test count dropped, you didn't optimize: you cut coverage. How to fix it: speed up with cache and parallelism, which keep the 23 tests running; if a test is slow and low-value, that's a strategy decision (another guide), not an excuse to delete blindly.

Exercises

Exercise 1 — Classify the sink. For each line of a CI log, say whether the time belongs to sink 1 (install dependencies → cache) or sink 2 (run tests in a line → parallelism). (a) Collecting numpy==2.1.0 ... Downloading numpy-2.1.0 (18 MB). (b) test_reports.py ............ [100%] that took 6 seconds. (c) Successfully installed pytest-9.1.1 iniconfig-2.3.0 .... (d) collected 800 items followed by three minutes of dots.

See solution

• (a) Sink 1 (install → cache). Downloading numpy (18 MB) is downloading a package from the internet, part of pip install. It repeats identically on every run if you don't cache. Lever: actions/cache.
• (b) Sink 2 (run tests → parallelism). The six seconds of test_reports.py are the tests running in a line. Lever: pytest-xdist -n auto.
• (c) Sink 1 (install → cache). Successfully installed ... is the end of pip install: the time to install the packages. Cacheable.
• (d) Sink 2 (run tests → parallelism). 800 tests dropping dots for three minutes is queued execution. Lever: parallelism (and maybe splitting the suite, lesson 5).

The rule: if the line talks about downloading or installing packages, it's sink 1; if it talks about tests running, it's sink 2. Each sink, its lever.

Exercise 2 — Diagnose the team that evades CI. A teammate tells you: "No one on the team waits for CI's green; we merge as soon as the local tests pass. CI takes eighteen minutes." Explain, using this lesson's mechanisms, why that's dangerous and which module lever would attack the root.

See solution

The danger is that CI stopped being a gate and became a late comment. When they merge without waiting for the green, any failure CI catches —a version incompatibility their local doesn't have, a test that only breaks in the runner's environment— enters the main branch and is discovered after, when someone has already built on top and fixing it costs much more. It's the lesson's "merge without green": eighteen minutes is more than patience tolerates, so the team rationalized skipping the wait, and with it the protection.

The root isn't that the team is careless; it's that CI is too slow to be worth waiting for. The solution isn't to scold people ("wait for the green"), but to remove the reason they don't wait: bring those eighteen minutes down to two or three. Which lever depends on where the eighteen minutes go —if it's in reinstalling, cache; if it's in running tests in a line, parallelism—, and that's why the first step is to measure the two sinks. A two-minute CI is waited for effortlessly, and the gate becomes a gate again.

Exercise 3 — The speedup that isn't worth it. A teammate "sped up" CI from fifteen to four minutes and is proud. When you ask how, they answer: "I deleted the twelve monthly-report tests, which were the slow ones." Why is this speedup a bad deal, and what should they have done instead?

See solution

It's a bad deal because they traded speed for coverage, and coverage is exactly what CI contributes. The twelve monthly-report tests verified that Reservo's revenue arithmetic is correct —in cents, with no rounding errors—. Deleting them makes CI faster, yes, but now a bug in the report calculation passes the pipeline without anyone noticing and reaches production. It's removing the traffic light so as not to wait for it: the intersection flows better until the day of the crash.

What they should have done is speed up without losing coverage, which is the whole point of the module. The twelve slow tests are a textbook case for parallelism: run with -n auto they drop from six seconds to a little over one, and they're still twelve green tests, not zero. If CI was also reinstalling dependencies on every run, the cache would cut another chunk. The goal is to lower the wall-clock time while keeping the 23 tests; deleting tests lowers the time by lowering the truth, which isn't optimizing but weakening.

Summary and next step

In this lesson you measured the problem before solving it. A slow CI doesn't fail loudly: it gets ignored, and an ignored verdict protects nothing. You saw it with the five-minute traffic light: technically it works, but people run it, and on the day of the crash no one brakes anymore. Slowness breaks the feedback loop by three concrete routes —the merge without green, the cost of switching context and coming back, and the normalization of the wait— and they all share a root: above a certain threshold, waiting costs more than the team is willing to pay.

You also located where the time goes: two distinct sinks. Installing the dependencies, over and over identically —repeated work, cache lever—; and running the tests in a line, one after another —queued work, parallelism lever—. And you defused the dangerous temptation: speeding up by deleting tests isn't optimizing, it's trading speed for coverage. The module's goal is to lower the wall-clock time without touching the amount of truth the pipeline verifies.

Before moving on you should be able to: explain why a slow CI ends up ignored, with at least two of the three mechanisms; name the two time sinks and the lever that attacks each; and recognize "deleting tests" as a false optimization.

What's next, in lesson 3, is the first complete lever: caching dependencies with actions/cache. You're going to see how you tell CI "don't reinstall the same thing if it didn't change", how the cache key is derived from the requirements.txt hash so it reuses when the list is the same and reinstalls when it changes, and how to read a cache hit versus a cache miss in the log. It's the exact remedy for sink 1.

Resources

• Caching dependencies to speed up workflows — GitHub Actions — the official guide to why and how to cache in CI, which motivates sink 1 and its remedy. The ideal prior reading for lesson 3.
• About continuous integration with GitHub Actions — the official overview of what CI is and what fast feedback is for, this lesson's conceptual frame. Useful for connecting speed with the pipeline's purpose.
• How to invoke pytest (pytest documentation) — the reference for the ways to run the suite, including --durations, which in lesson 5 we'll use to measure sink 2 precisely. The first step to "measure before speeding up".