7. When the matrix pays off and when it's noise

Description

This module has taught you to build a matrix, sculpt it, and read it. All that is the how. This lesson is the when, and it's the most important, because a poorly sized matrix isn't a technical error —the YAML works the same— but an error of judgment that's paid every day in billed minutes and in noise that trains your team to ignore reds. The question isn't "can I make a nine-cell matrix?" —yes you can, it's trivial—, but "does my project deserve nine cells, or does it have six too many?".

By the end you'll be able to decide, with judgment and a cost model, which matrix corresponds to a project depending on what it is. You'll distinguish the case where the matrix is mandatory —a library that many install, whose job is to work on every version and system it promises— from the case where it's noise and cost —an internal app you deploy to a single version on a single system, where eight of nine cells test environments where your code will never run—. You'll see the cost model (N cells = N× minutes per push), the rule that solves almost every case —"test what you ship plus what you promise to support, and nothing more"— and how to trim an inflated matrix without losing real coverage.

Connection to the module: lessons 3 to 6 gave you the tools; this one gives you the judgment to use them. It leans on everything before: lesson 2's catalog of differences (to know if your code touches one), lesson 4's multiplication (to calculate the cost), and lesson 5's exclude (to trim). It's the second-to-last lesson, and it prepares the mini-project: when in lesson 8 you configure Reservo's matrix, the question "which matrix does Reservo deserve?" will be the one that decides the design, not a copied tutorial.

The insurance you buy based on what you risk

No one buys the same insurance for a bicycle as for a fleet of cargo trucks. Not because the truck insurance is "better" —it's more expensive and more complete—, but because what you risk is different. For the bicycle, basic theft insurance is enough; paying for fleet insurance would be throwing money away on coverage you'll never use. For the trucks, skimping on coverage is reckless: an accident without insurance ruins you. The correct insurance isn't the biggest or the smallest: it's the one that corresponds to what you really risk.

The matrix is insurance against "it breaks in an environment I didn't test". Its correct size depends on how many environments you really risk. A library published for the world is the truck fleet: thousands of people install it on versions and systems you don't control, and if it breaks on Python 3.11 on Windows, it's a real problem for a real user. There the big matrix isn't luxury, it's responsibility. An internal app that only you deploy, to a Linux server with Python 3.12 nailed down, is the bicycle: there's only one environment that really matters —production's—, and testing on eight more is paying fleet insurance for a bike. Your matrix's size should follow your risk, not your anxiety.

The correct matrix is the one that covers the environments you really risk: the ones your code will step on in others' hands or in production. Not more (noise and cost), not less (a promised environment untested).

The cost model: why each cell costs

Before deciding, you have to see the price, because a matrix isn't free and its cost is easy to underestimate. Each matrix cell is a complete run of your suite on a runner, and that costs in three currencies:

• Compute minutes. GitHub runners are billed per minute (with a free quota that runs out). A nine-cell matrix consumes, on each push, nine times the minutes of a single job. If your suite takes two minutes, a single job costs two minutes per push; the nine-cell matrix costs eighteen. Multiplied by each push of each person on the team, every day, it becomes real. And note: macOS and Windows runners usually cost more per minute than Linux ones (often several times more), so a macOS column doesn't cost the same as a Linux one.
• Wait time. Even though the cells run in parallel, waiting for all nine to finish —and for runners to free up if there's a queue— lengthens the feedback cycle. A huge matrix can turn a two-minute feedback into a ten-minute one.
• Noise and attention. This is the hidden cost and the most expensive in the long run. If you have cells that fail for reasons you don't care about —a version you don't support, an environment where your app never runs—, your team learns to see reds and shrug. And once the team ignores reds, the matrix stopped protecting anything: it's a traffic light no one looks at.

The mental math is simple and you should always do it: number of cells = product of the lists, and cost per push ≈ cells × suite duration (weighted, because not all runners cost the same). Nine cells aren't "a bit more" than one; they're nine times. That factor is what justifies asking of each cell whether it contributes.

Worked example: the same suite, one cell against nine

Let's run the Reservo suite once locally to have the base number, and from there reason about the matrix cost:

python -m pytest tests/

What to expect. On Python 3.14.0, measured for real:

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /private/tmp/reservo-m4
collected 9 items

tests/test_pricing.py ...                                                [ 33%]
tests/test_refunds.py ...                                                [ 66%]
tests/test_version_features.py ..s                                       [100%]

========================= 8 passed, 1 skipped in 0.01s =========================

One run: 8 passed, 1 skipped in 0.01s. That's one cell. Now think about matrices over that same work:

• 1 cell (one job, Linux, one version): you run the suite once per push. It's what Reservo, being pure logic, almost needs.
• 3 cells (Linux, three versions): you run the suite three times. Justified if you promise to support three versions and use features that vary between them —like lesson 1's report_pages—.
• 9 cells (three systems × three versions): you run the suite nine times. Justified only if your code really behaves differently on all three systems and all three versions —which, for Reservo's pure logic, mostly doesn't happen—.

The point isn't that nine is bad and one good. It's that each jump —from 1 to 3 to 9— multiplies the cost, and that cost is only justified if the new cells catch something the old ones wouldn't. For Reservo, jumping to nine because "it looks more complete" would be paying fleet insurance for a bike: nine runs of 8 passed, 1 skipped where eight discover nothing the first didn't already discover.

The rule that solves almost every case

There's a rule that decides the vast majority of matrices well:

Test what you SHIP plus what you PROMISE TO SUPPORT, and nothing more.

Broken down:

• What you ship — the environment where your code will really run. For an app you deploy, it's the exact version and system of production. If production is Python 3.12 on Linux, that cell has to be there, because it's the only environment that really matters to you: if it fails there, your app is really broken.
• What you promise to support — for a library, every version and system your README, your pyproject.toml, or your documentation claim to support. If you promise "Python 3.11+", each of those versions is a promise the matrix must verify. Promising and not testing is lying with confidence.
• And nothing more — the part people forget. Don't test versions you don't promise or systems where your code will never run. Each extra cell is cost without coverage, and noise that erodes trust in the matrix.

Apply the rule to the two canonical cases:

A library (reservo published on PyPI for the world). It's installed by people you don't control, on the versions and systems they have. Your pyproject.toml says requires-python = ">=3.11" and your README promises Linux, macOS, and Windows. So "what you promise to support" is big: 3.11, 3.12, 3.13 × Linux, macOS, Windows = the nine cells, and all pay off, because each is a real user with that environment. Here the complete matrix isn't excess, it's exactly the promise. Trimming it would be promising support you don't verify.

An internal app (Reservo running as a service, deployed by your team). Only you deploy it, to an environment you choose: Python 3.12 in a Linux container, and nothing more. No one installs it on Windows; no one runs it on 3.11. "What you ship" is one cell —(ubuntu, 3.12)—, and "what you promise to support" is empty (it's not a library, you don't promise versions to anyone). So the correct matrix is one cell, maybe two if you're about to migrate versions and want to test the new one first. The other seven cells of a 3×3 matrix would test environments where your app will never run: pure cost and noise.

The same codebase —Reservo— deserves opposite matrices depending on how it's distributed. It's not a property of the code; it's a property of where it will live.

Intermediate cases and how to think about them

Few projects are pure "library for the world" or "single-cell app". For the intermediate ones, some guidelines:

• An app that runs in several deployment environments. If you deploy the same app to Linux and Windows (rare, but it happens), then "what you ship" is those two, and the matrix must cover both —but only those two, not macOS, which you don't deploy—.
• A library with a recent version floor. If your library uses itertools.batched without a fallback and declares requires-python = ">=3.12", then 3.11 is not a promise: it doesn't go in the matrix. Raising the version floor is a legitimate way to trim the matrix —fewer promises, fewer cells—.
• Focus on the boundaries. When a feature changes behavior in one version (lesson 2), make sure to have cells on both sides of that boundary, and don't obsess over the middle ones. If the change is between 3.11 and 3.12, those two matter more than adding 3.13 and 3.14 "for completeness".
• The representative one, not the exhaustive one. For the system dimension, if your code touches paths but you don't care about each system equally, sometimes "one POSIX (Linux) and Windows" is enough, skipping macOS —which shares almost all POSIX behavior with Linux—. Two cells cover the real difference (POSIX vs Windows) without the nearly redundant third.

How to trim an inflated matrix

If you inherit or detect a matrix that's pure reflex —nine cells copied from a tutorial for a single-cell app—, this is how you trim it without losing real coverage:

1. List the environments you really risk (ship + promise). For the internal app: (ubuntu, 3.12). Period.
2. Compare with the current cells. Everything in the matrix but not in your risk list is a candidate to go.
3. Remove whole dimensions if they don't contribute. If your code is pure logic (doesn't touch the system), remove the entire os dimension —leave it at [ubuntu-latest]—. If you only deploy to one version, reduce python-version to that one.
4. Use exclude for the specific corners that are excess but whose rest of the row/column you do want (lesson 5).
5. Document the why in a YAML comment: # only Linux 3.12: it's the only deployment environment. That way the next person who reads it doesn't inflate it again "just in case".

Trimming isn't being careless; it's being honest about what you risk. A well-justified one-cell matrix protects more than a nine-cell one the team learned to ignore.

Common mistakes

Copying a nine-cell matrix from a tutorial without asking whether it applies. What happens: a tutorial shows a "professional" 3×3 matrix, someone pastes it into an internal single-version app, and now each push spends nine runs to test eight environments where the code will never run. Why it happens: the big matrix looks more serious and "complete". How to spot it: for each cell ask yourself "is someone really going to run my code here?". If the honest answer is no for most, the matrix is inflated. How to fix it: apply the rule —ship + promise, nothing more— and trim with the procedure above.

Promising support the matrix doesn't verify. What happens: the README says "supports Python 3.9+", but the matrix only tests 3.12 and 3.13. A 3.9 user installs and it blows up; you promised something you never tested. Why it happens: the promise and the matrix are edited at different moments and get out of sync. How to spot it: compare the version list of your pyproject.toml/README with the matrix list; they must match. How to fix it: either you raise the version floor to what you really test (and adjust the README), or you add the promised versions to the matrix. The matrix is the promise made verifiable; make them say the same.

Confusing "more cells" with "more quality". What happens: someone adds versions and systems "to be safer", and the matrix grows to fifteen cells that take time and cost, without any of the new ones catching a real bug (the code doesn't touch them). Why it happens: quantity of coverage gets equated with quality of coverage. How to spot it: ask yourself for each new cell "what bug does this catch that the others wouldn't?". If you have no answer, it doesn't contribute. How to fix it: measure coverage by what you risk, not by the size of the grid. A cell that catches something real is worth more than five that give the same green.

Exercises

Exercise 1 — Size the matrix by the case. For each project, say which matrix corresponds (how many cells and of what) and justify it with the "ship + promise" rule: (a) a utility library published on PyPI, with requires-python = ">=3.11" and a README that promises Linux/macOS/Windows; (b) an internal web app deployed only to a Linux container with Python 3.12; (c) a command-line script your team runs on their laptops, some Macs and some Windows, always on Python 3.12.

Exercise 2 — Calculate the cost and decide. An internal app has a suite that takes 3 minutes per run. The team makes on average 20 pushes a day. Compare the monthly cost (30 days) in runner minutes between a 1-cell matrix and a 9-cell one, and say whether the jump is justified knowing the app only deploys to (ubuntu, 3.12).

Exercise 3 — Trim the inflated matrix. You inherit this workflow of an internal app that only deploys to Linux with Python 3.12. Explain what's wrong and rewrite the strategy.matrix section to what it really deserves, with a comment justifying the trim.

strategy:
  matrix:
    os: [ubuntu-latest, macos-latest, windows-latest]
    python-version: ["3.10", "3.11", "3.12", "3.13"]

strategy:
  matrix:
    # The app only deploys to Linux with Python 3.12 (production).
    # It's not a library: it promises no other versions or systems to anyone.
    # That's why: a single cell, the deployment one.
    os: [ubuntu-latest]
    python-version: ["3.12"]

Summary and next step

In this lesson you learned the judgment that governs every matrix: its correct size follows what you really risk, like insurance follows what you insure. A library for the world deserves the complete matrix —each version and system it promises is a real user—; an internal app with a single deployment deserves one cell —the others test environments that don't exist for it—. The rule that solves almost everything: test what you ship plus what you promise to support, and nothing more.

You saw the cost model —each cell is a complete run, N cells = N× minutes per push, with macOS and Windows more expensive— and applied it with numbers: nine cells of 8 passed, 1 skipped for a one-cell app are eight runs that catch nothing, pure cost and noise that trains the team to ignore reds. And you learned to trim an inflated matrix: list what you risk, remove dimensions that don't contribute, use exclude for the corners, and document the why.

Before moving on you should be able to: size a project's matrix based on how it's distributed; calculate its cost per push; apply the "ship + promise, nothing more" rule; and trim a reflex matrix to the one it really deserves, justifying each cell.

What's next, in lesson 8, is the mini-project that pulls the whole module together: you configure the three-Python-version matrix for the Reservo suite —the YAML with strategy.matrix, the feature with skipif that behaves differently by version, the local run that shows 8 passed, 1 skipped, reading the three expected results, and —with what you learned here— a note that justifies which matrix Reservo really deserves—.

Resources

• About billing for GitHub Actions — GitHub Docs — how runner minutes are billed, including the macOS and Windows multipliers relative to Linux. The real number behind this lesson's "cost model".
• requires-python in pyproject.toml — Python Packaging User Guide — where a library declares which versions it supports, the source of truth of "what you promise". The matrix should reflect exactly this value.
• Using a matrix for your jobs: example matrices — GitHub Actions — the reference for expressing whatever matrix you decide, big or one-cell. You choose the size with this lesson's criteria; this page gives the syntax.
• Supported Python versions and their lifecycle — Python Developer's Guide — which Python versions are still officially supported and which aren't. Useful for deciding your matrix's version floor: testing a version that's already reached end of life rarely pays off.