Module 4: The Matrix Versions And Environments

1. Module introduction: the matrix

Next →

Description

Up to now, your pipeline had a virtue and a blind spot. The virtue: it ran the Reservo suite automatically on every push, without anyone remembering to do it. The blind spot: it ran it in a single environment. One Python version —the one the runner shipped by default—, one operating system —usually Linux—, and that's it. When that single job came up green, you read "the suite passes". But what it actually said was more humble: "the suite passes in that environment". And there's the problem this module solves, because your code almost never lives only in that environment.

By the end of this lesson you'll understand what the CI matrix strategy is and why it exists: a table of environments —several Python versions, several operating systems— that GitHub Actions expands into one job per combination, all running in parallel, each with its own green or red verdict. Instead of one check that says "passes", you have N checks that say "passes on 3.11", "passes on 3.12", "passes on 3.13", and if one turns red, you know exactly which version broke without guessing. You're going to see the skeleton of the YAML that defines it, how to read the result, and —this is what separates someone who copies a matrix from the internet from someone who designs one— when the matrix pays off and when it's pure noise and cost.

Connection to the module: this lesson is the map. Here you install the idea —why a single green isn't enough and what a matrix is— and you watch it beat once with a real local demo. Lesson 2 opens up what changes between environments: standard-library features that appear in one version and not the previous one, path separators that differ by operating system. Lesson 3 writes the Python-version matrix in the YAML, piece by piece. Lesson 4 adds the operating-system dimension and multiplies the grid. Lesson 5 fine-tunes it with include, exclude, and fail-fast. Lesson 6 teaches you to read the N results and locate which cell a bug lives in. Lesson 7 is the business decision: when this machine pays off and when it gets in the way. And lesson 8, the mini-project, has you configure the three-version matrix for Reservo from start to finish.

A note about the boundary, because this module leans on the three previous ones and doesn't repeat them. Reproducing a CI failure locally was module 3: here, when a matrix cell turns red, we're going to know which version to match to reproduce it, but the mechanics of reproduction and in-depth diagnosis you already have. Speeding up CI —caching dependencies, parallelizing— is module 5, which comes right after, and it's no coincidence: a matrix multiplies the work, so speed matters more than ever. Here the focus is a single one: the matrix. What it is, how it's written, how it's read, and when it's worth it.

The restaurant that only tested its dish on one stove

Think of a chef who invents a new dish. They cook it in their kitchen, on their gas stove, with their pots, and it comes out perfect. They add it to the menu. The first week, odd complaints arrive: at the downtown branch the dish comes out raw inside, at the north one it burns. The chef doesn't understand: "it works in my kitchen".

The problem isn't the recipe. It's that each branch cooks in a different environment. One has an electric stove, which heats slower; another has induction, which heats on a different curve; the north one is a thousand meters higher in altitude, where water boils at a lower temperature. The recipe the chef tested on one stove assumed, without noticing, things about that stove. They never tested it on the others. And "it works in my kitchen" turned out to be a much narrower claim than they believed.

A serious chef's solution isn't to cross their fingers. It's to test the recipe on each type of stove before sending it to the menu: gas, electric, induction, at different altitudes. If it comes out well on all five, they publish it with confidence. If it comes out badly on induction, they discover it in their test kitchen —not in a customer's face— and adjust the recipe or note "this one isn't for induction".

A CI matrix is exactly that test kitchen with five stoves. Your code is the recipe. Each Python version and each operating system is a type of stove. The matrix cooks your Reservo suite on all the stoves you care about, at once, and delivers a verdict for each. "It works on my machine" stops being a hope and becomes a table of results: it works on gas, electric, induction; it fails at high altitude, and you already know why.

A CI matrix runs your same suite in several environments —Python versions, operating systems— and gives you a verdict per combination. It turns "it works on my machine" into "it works on these six, fails on this one, and I know which".

Reservo, as we left it — and a new feature that gives away the version

We continue with Reservo, the coworking meeting-room booking system we've been testing since the first module. Pure Python logic: no database, no network, no hidden clocks. Its pieces, in case you need a refresher:

  • Room (id, name, capacity, hourly_cents), Member (id, name, tier: "basic" or "pro"), Booking (with its price_cents field, the start, the end as a half-open range [start, end), and its status).
  • The core functions: price_cents(room, member, hours), refund_cents(booking, price_paid_cents, now), overlaps, is_available, book, and the Calendar that stores the bookings in memory.
  • The anchor numbers, the whole guide's checksum: basic 3 h → 7500, pro 3 h → 6000 (20% discount), and the refund on 6000 paid: 6000 if you cancel 72 h ahead (≥ 48 h, 100%), 3000 at 36 h (24–48 h, 50%), 0 at 12 h (< 24 h).

You already have all that tested. For this module we add to Reservo a new function that, on purpose, depends on the Python version: a daily report that groups the bookings into pages. We use it because we need something that behaves differently depending on the version, and this one is genuine, not a lab trick.

# reservo/reports.py
import sys

# `itertools.batched` is part of the stdlib ONLY since Python 3.12.
# On 3.11 and earlier you have to bring it by hand. Reservo chooses the path
# based on the running version — and that difference is exactly what a
# CI matrix exists to watch over.
if sys.version_info >= (3, 12):
    from itertools import batched

    def report_pages(bookings, size):
        """Group the bookings into pages of `size` for the daily report."""
        return [list(page) for page in batched(bookings, size)]
else:
    def report_pages(bookings, size):
        """Manual fallback for Python < 3.12, where itertools.batched doesn't exist."""
        return [bookings[i:i + size] for i in range(0, len(bookings), size)]

Stop at the if sys.version_info >= (3, 12). sys.version_info is a tuple Python gives you with the version running at this precise moment: on the machine where I wrote this it's (3, 14, 0). The comparison >= (3, 12) is True on 3.12, 3.13, and 3.14, and False on 3.11 and earlier. So report_pages uses itertools.batched —a standard-library function that appeared in Python 3.12— when it's available, and a hand-written list comprehension when not. The visible behavior is identical in both branches; what changes is the internal path. And that "path that changes depending on the version" is exactly the kind of thing a single green job doesn't see and a matrix does.

First contact with skipif: the same suite, a different verdict per version

To test a feature that depends on the version, pytest gives you the exact tool: @pytest.mark.skipif. It's a marker that puts a condition on a test: "if this condition is true, skip this test and don't count it as passed or failed". We use it so a test that only makes sense on 3.12+ doesn't run —doesn't pretend to pass, doesn't pretend to fail— on versions where it doesn't apply.

Here's the test file for the version feature. Read it slowly: there are three tests, and two of them carry a skipif with opposite conditions.

# tests/test_version_features.py
import sys

import pytest

from reservo.reports import report_pages


def test_report_pages_groups_bookings():
    # This rule holds in EVERY version: 5 bookings in pages of 2 -> [2, 2, 1].
    pages = report_pages(["b1", "b2", "b3", "b4", "b5"], 2)
    assert [len(p) for p in pages] == [2, 2, 1]


@pytest.mark.skipif(
    sys.version_info < (3, 12),
    reason="itertools.batched is part of the stdlib only since Python 3.12",
)
def test_report_pages_uses_stdlib_batched():
    # On 3.12+ report_pages uses itertools.batched internally; here we confirm
    # that the stdlib import is available in the running version.
    from itertools import batched
    assert list(batched("abcde", 2)) == [("a", "b"), ("c", "d"), ("e",)]


@pytest.mark.skipif(
    sys.version_info >= (3, 12),
    reason="the manual fallback is only exercised on Python < 3.12",
)
def test_report_pages_manual_fallback_on_old_python():
    # This test only makes sense on 3.11 and earlier, where report_pages
    # uses the manual comprehension instead of itertools.batched.
    assert "batched" not in dir(__import__("itertools"))

Notice the two conditions. test_report_pages_uses_stdlib_batched skips when sys.version_info < (3, 12) —that is, it skips on 3.11, and runs on 3.12, 3.13, 3.14—. test_report_pages_manual_fallback_on_old_python does the opposite: it skips when sys.version_info >= (3, 12) —runs on 3.11, and skips on 3.12 and above—. They're mirrors. On any version, one of the two runs and the other skips. That means the same suite produces a result with a different nuance in each matrix cell, and that's the idea I want you to feel before writing a single line of YAML.

Worked example: run the suite on your version and look at what skips

We're going to run Reservo's complete suite on the machine where I'm writing, which has Python 3.14.0 and pytest 9.1.1. This run is the one a pipeline would do in one matrix cell; here we do it locally to see it for real. The -v flag (--verbose) lists each test with its verdict, instead of summarizing them with dots.

python -m pytest -v tests/

What to expect. On Python 3.14.0 this comes out, measured for real (not invented):

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- /private/tmp/reservo-m4/.venv/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/reservo-m4
collected 9 items

tests/test_pricing.py::test_basic_three_hours PASSED                     [ 11%]
tests/test_pricing.py::test_pro_three_hours PASSED                       [ 22%]
tests/test_pricing.py::test_basic_one_hour PASSED                        [ 33%]
tests/test_refunds.py::test_full_refund_72h_before PASSED                [ 44%]
tests/test_refunds.py::test_half_refund_36h_before PASSED                [ 55%]
tests/test_refunds.py::test_no_refund_12h_before PASSED                  [ 66%]
tests/test_version_features.py::test_report_pages_groups_bookings PASSED [ 77%]
tests/test_version_features.py::test_report_pages_uses_stdlib_batched PASSED [ 88%]
tests/test_version_features.py::test_report_pages_manual_fallback_on_old_python SKIPPED [100%]

========================= 8 passed, 1 skipped in 0.01s =========================

Read the final summary slowly: 8 passed, 1 skipped. Eight tests passed and one skipped. Look at which one skipped: test_report_pages_manual_fallback_on_old_python, the one that only makes sense on Python < 3.12. Since this machine runs 3.14, its condition sys.version_info >= (3, 12) is true, and pytest skipped it cleanly. It didn't fail —the manual fallback doesn't apply here, so there'd be nothing to test—, but it also didn't disappear silently: pytest counts it as skipped and even marks it with SKIPPED in the list.

Now think about what would happen in another matrix cell. If this same file ran on Python 3.11, the conditions invert: test_report_pages_manual_fallback_on_old_python would run (and pass, because on 3.11 itertools doesn't have batched), while test_report_pages_uses_stdlib_batched would skip (because itertools.batched doesn't exist there). The summary would still say 8 passed, 1 skipped, but the test that skips would be a different one. That's the matrix in miniature: the same suite, run on different versions, exercising different paths, and each cell telling you what it tested and what it didn't.

If you want to see why it skipped, pytest has a flag for that, -rs (report skipped), which prints the reason we wrote in the skipif:

=========================== short test summary info ============================
SKIPPED [1] tests/test_version_features.py:25: the manual fallback is only exercised on Python < 3.12
========================= 8 passed, 1 skipped in 0.01s =========================

There's the reason we put in the marker, exactly. A skip without a reason is a mystery; a skip with a reason is documentation. In lesson 2 you're going to squeeze this idea, and in the mini-project you're going to deliver it as part of the report.

What a matrix is and what it isn't

It's worth making the anatomy clear before diving in, so the next lessons fall into place.

A matrix is, literally, one or more lists of values that CI multiplies to generate jobs. A list of three Python versions generates three jobs. Two lists —three versions × three operating systems— generate nine, one for each pair. Each job is a copy of the same work (checkout, install Python, install dependencies, run pytest) parameterized by its combination. You don't write nine jobs by hand; you write the lists and CI does the multiplication for you. That automatic multiplication is the whole value of the tool: you express "I want to test these versions on these systems" in four lines, and you get the complete grid.

What a matrix is not: it's not magic that fixes incompatibilities. If your code breaks on 3.11, the matrix doesn't repair it; it shows it to you, which is different and much more useful. Nor is it free: each cell is a runner that consumes minutes, and those minutes are billed and accumulate. A matrix of three versions by three systems is nine runs of your suite every time someone pushes. That's why lesson 7 exists: the question isn't "can I make a huge matrix?" —yes you can—, but "which matrix does this project deserve?".

And an honesty of the guide, the same as always: the CI workflow is executed for real on a GitHub runner, which we don't have here. So the matrix YAML you're going to write and read as content —I show you how it looks and how its log would read—, while the pytest runs are real, done locally with Python 3.14.0, because your machine plays one of the matrix cells. When you cite "8 passed, 1 skipped", I measured that number by executing; when I show a CI log with three jobs, that's the honest format of how it would look, not a screenshot of a phantom runner.

Common mistakes

Reading "CI is green" as "my code works everywhere". What happens: a single-job pipeline comes up green on Linux with Python 3.12, and the team concludes "done, it works". A user with Python 3.11 installs it and it blows up on the first import. Why it happens: a green job only asserts the environment it ran in, but it's easy to forget that fine print and read it as a universal verdict. How to spot it: ask yourself "in what exactly is this green?". If the answer is "in one environment" and your code is used in five, you have a blind spot. How to fix it: a matrix that covers the environments that really matter. The rest of the module is how.

Turning the whole matrix on by reflex, without asking whether it pays off. What happens: someone copies a 3×3 matrix from a tutorial for an internal app that only runs on Linux with Python 3.12 in production. Now each push spends nine runs, eight of which test environments where the code will never run. Why it happens: the matrix is easy to turn on and feels "more complete". How to spot it: look at your cells and ask for each one "is someone really going to run my code here?". If the answer is no, that cell is noise and cost. How to fix it: test what you ship plus what you promise to support, and nothing more. Lesson 7 gives you the complete rule.

Confusing skip with pass or with fail. What happens: someone sees 1 skipped in the summary and gets scared —"did something skip? is it broken?"—, or the reverse, ignores it believing that "skipped" is the same as "passed". Why it happens: skipped is a third state that isn't always explained: it's not green or red, it's "this test doesn't apply under these conditions and we said so on purpose". How to spot it: run with -rs and read the reason. A skip with a clear reason ("the fallback only applies on < 3.12") is healthy and expected. A skip with no reason, or one that skipped by accident (an import that failed), is a signal to investigate. How to fix it: always put a reason in your skipif, and treat the skip count as information, not an alarm.

Exercises

Exercise 1 — Predict the skip on another version. On Python 3.14 you ran the suite and saw 8 passed, 1 skipped, and the one that skipped was test_report_pages_manual_fallback_on_old_python. Without running anything, predict: if the same suite ran on Python 3.11, how many pass and how many skip, and which test skips? Explain why in one sentence.

See solution

It would still be 8 passed, 1 skipped, but the test that skips would be test_report_pages_uses_stdlib_batched, not the other. Reason: on 3.11, that test's condition, sys.version_info < (3, 12), is true (3.11 is less than 3.12), so it skips; and the other's condition, sys.version_info >= (3, 12), is false, so test_report_pages_manual_fallback_on_old_python runs and passes (on 3.11 itertools really doesn't have batched). The two skipif are mirrors: on every version, exactly one of the two skips. What changes between matrix cells isn't the total count, but which code path was exercised —and that's why it's worth running both versions.

Exercise 2 — Translate the analogy to the matrix. The restaurant's chef tested their recipe on five types of stove (gas, electric, induction, and two altitudes). Map each element of that analogy to its equivalent in a CI matrix: (a) the recipe, (b) a type of stove, (c) testing on all five stoves before publishing, (d) the dish that comes out raw only on induction.

See solution
  • (a) The recipe → your code (the Reservo suite and the code it tests). It's the same on all stoves; the only thing that changes is where it's cooked.
  • (b) A type of stove → an environment, that is, a combination of Python version and operating system. Gas ≈ "Python 3.12 on Linux"; induction ≈ "Python 3.11 on Windows".
  • (c) Testing on all five before publishing → the CI matrix, which runs the suite on all combinations at once before approving the change, in the test kitchen (the runner) and not in the customer's face (production).
  • (d) The raw dish only on induction → a red matrix cell: a test that passes in almost all environments but fails in a specific one. The matrix tells you which, just as the chef knew it was induction and not the others.

The moral on both sides: "it works on my machine/kitchen" is a claim about one environment, not all. The matrix turns it into an honest table.

Exercise 3 — Which module solves this? For each situation, say whether this module (the matrix) solves it or whether it belongs to another module of the guide, and name it in one sentence. (a) "The 3.11 cell came out red and I want to reproduce that failure on my laptop." (b) "My 3×3 matrix takes 12 minutes and I want it to be faster." (c) "I want to run the suite on Python 3.11, 3.12, and 3.13 at once." (d) "A test passes sometimes and fails sometimes, without changing the code."

See solution
  • (a) Reproducing the 3.11 cell's failure locally → module 3 (reproducing a CI failure locally). This module tells you which version it failed in, which is the first datum; matching that version on your machine and reproducing the failure is module 3's technique. Here we only make the bridge.
  • (b) Making the matrix faster → module 5 (fast CI: cache and parallelism). A matrix multiplies the work, and speeding it up —caching dependencies, parallelizing— is exactly the topic that follows. This module builds the matrix; module 5 speeds it up.
  • (c) Running the suite on three versions at once → this module, and specifically lesson 3. It's the very definition of a Python-version matrix.
  • (d) A test that passes sometimes and fails sometimes → module 7 (flaky tests in CI). A non-deterministic test is a flaky, and its treatment —retry, quarantine, the failure that only happens in CI— belongs to module 7. A consistently red matrix cell is not flaky; it's a real version or OS incompatibility, which does belong here.

The mechanical rule: if the question is "in which environments do I run my suite?", it's this module. "How do I reproduce what failed?" is module 3, "how do I make it fast?" is 5, "why is it inconsistent?" is 7.

Summary and next step

In this lesson you installed the idea that holds up the module: a single green job asserts a single environment, and your code lives in many. The matrix strategy is the five-stove kitchen: it runs your same suite on several Python versions and operating systems, in parallel, and gives you a verdict per combination, turning "it works on my machine" into an honest table of where it works and where it doesn't.

You watched it beat with a real local demo: Reservo debuted report_pages, a feature that uses itertools.batched on Python 3.12+ and a manual fallback before, and two mirror tests with @pytest.mark.skipif that skip on opposite versions. You ran the suite on Python 3.14.0 and read 8 passed, 1 skipped, understanding that the skip isn't an error but a third state —"this case doesn't apply here, and we said so on purpose"— and that on another version the one that skips would be the other test. The guide's honesty also became clear: the pytest runs are real, the matrix YAML is content you learn to read and write.

Before moving on you should be able to: explain in your own words what a CI matrix is and why it exists; say what a single green job asserts —and what it doesn't—; predict which test skips on 3.11 versus 3.14 and why; and distinguish skipped from passed and from failed.

What's next, in lesson 2, is opening up the underlying question: what really changes between one environment and another? You're going to see the concrete types of difference —stdlib features that appear in one version, new syntax, dependencies that don't compile, path separators that differ by operating system— to understand not only that the matrix is useful, but against which exact dangers it protects you.

Resources

← Previous8. Mini-project: reproduce and fix a failure from an unpinned dependencyNext →2. Why test in several environments