Module 8: Project A Ci Pipeline For Reservo

4. The version matrix

Next →

Description

Your pipeline already runs the suite on every push (lesson 2), on a reproducible environment (lesson 3). But it runs on one Python: the machine's 3.14. Green there means "it passes on 3.14," nothing more. And Reservo, as a library that other teams install, promises to support 3.11, 3.12, and 3.13 —three promises to users you do not control—. The layer we stack here, the version matrix, turns those promises into tests: it wraps the base workflow in a strategy.matrix that runs it on the three versions at once, each one in its own parallel job, each one with its verdict.

It is the layer of module 4, now woven into the whole. You are going to see how a list of three versions multiplies into three jobs, how ${{ matrix.python-version }} injects each cell's version into setup-python, and why fail-fast: false gives you the complete map instead of canceling at the first red. And you are going to reencounter Reservo's version-dependent feature —report_pages, which uses itertools.batched on 3.12+ and a manual fallback before— with its two mirror skipif tests, running your machine as a witness cell. The run is real: 13 passed, 1 skipped on 3.14, with the fine detail of which test gets skipped according to the version.

By the end you will have the matrix in place inside the pipeline and know how to justify its size: why three Python versions and why —today— a single operating-system row. Because the capstone is evaluated by the method, and turning on cells by reflex is the opposite of the method.

Connection with the module: this is the third layer, stacked on the floor (lesson 2) and the reproducibility (lesson 3). The matrix wraps exactly the steps you already have: each cell does checkout, setup-python, install, and pytest, just like the floor, only parameterized by its version. And it is the layer that makes the next one urgent: running the suite three times per push multiplies the time, so lesson 5 —cache and parallelism— comes right after, not by chance. The reproducibility of lesson 3 is what makes each cell a clean experiment: the pinned snapshot guarantees that "the 3.11 cell failed" means "the bug is from 3.11," not "maybe that cell installed something weird."

The seatbelt they tested on a single crash dummy

Think of a seatbelt factory. They design a new one, test it with a crash dummy —an average adult, 1.75 m, 78 kg— and it comes out perfect: it holds, it does not break, it saves the simulated life. They approve it. Months later reports arrive: in real accidents, the seatbelt hurts small people and does not hold the big ones well. "But it passed the test," they say. Yes —the test with one dummy—.

The problem is not the seatbelt; it is that they tested it against a single body. A seatbelt is used by bodies of all sizes: children, tall adults, heavy people. Testing it only with the average dummy asserts that it works for that body, and stays silent about all the others. The serious factory tests the seatbelt against a battery of dummies —a child, a small woman, a big man, the average— before approving it. If it holds all four well, it comes out with confidence; if it fails with the child, they discover it in the lab, not in an accident.

The version matrix is that battery of dummies. Your code is the seatbelt. Each Python version is a different body that is going to use it. Testing only on 3.14 —your average dummy— asserts that Reservo works on 3.14 and stays silent about 3.11, 3.12, 3.13, which are the versions your users really have. The matrix runs the suite against the four bodies at once, and gives you a verdict for each: it holds on 3.11, holds on 3.12, holds on 3.13, holds on 3.14. "It works in my version" stops being a hope and becomes a table.

The matrix tests your code against each version you promise to support, like a seatbelt against a battery of dummies. A single green environment asserts a single body; the matrix asserts —or refutes— all the ones that are going to use it.

The matrix, woven over the base workflow

Here is the base pipeline of lesson 2, now wrapped in a matrix of three versions. Notice how little changes and how much it gains:

# .github/workflows/tests.yml
name: tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false      # we want to see the three cells, not cancel at the first red
      matrix:
        python-version: ["3.11", "3.12", "3.13"]   # exactly what the README promises
    steps:
      - name: Check out the code
        uses: actions/checkout@v5

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements-dev.txt

      - name: Run the test suite
        run: python -m pytest

The three new pieces, and why each one:

  • strategy.matrix.python-version: ["3.11", "3.12", "3.13"] — the list of versions, in quotes (lesson 2). This list is the README's promise made executable: not one version too many, not one too few. A list of three generates three jobs, identical except for the version.
  • ${{ matrix.python-version }} in setup-python — the connection that makes each cell install its version. Without this line, the three cells would run the same version and the matrix would be theater. With it, the test (3.11) cell installs 3.11, the test (3.12) cell installs 3.12, and so on.
  • fail-fast: false — the switch that, Reservo being a library, I want off: if one version fails, I want to know whether the others do too, not for GitHub to cancel the sibling cells at the first red. The complete map, not the first shot.

Everything else —the four steps— is the floor of lesson 2, unchanged. That is the beauty of the matrix: it does not rewrite the work, it wraps it and multiplies it. You write the steps once and the list of versions once; GitHub does the product for you.

The feature that betrays the version, and its mirror tests

For the matrix to have something to see, Reservo includes a feature that behaves differently according to the version: report_pages, which groups bookings into pages for the daily report. It uses itertools.batched —a standard-library function that appeared in Python 3.12— when available, and a manual fallback when not:

# reservo/reports.py
import sys

if sys.version_info >= (3, 12):
    from itertools import batched

    def report_pages(bookings, size):
        """Groups the bookings into pages of `size` for the daily report."""
        return [list(page) for page in batched(bookings, size)]
else:
    def report_pages(bookings, size):
        """Manual fallback for Python < 3.12, where itertools.batched does not exist."""
        return [bookings[i:i + size] for i in range(0, len(bookings), size)]

The visible behavior is identical in both branches; what changes is the internal path according to the version. To test both branches —each one in the version where it lives— the suite uses two tests with @pytest.mark.skipif of opposite conditions, plus a universal test:

# tests/test_version_features.py
import sys

import pytest

from reservo.reports import report_pages


def test_report_pages_groups_bookings():
    # Rule that holds in EVERY version: 5 bookings in pages of 2 -> [2, 2, 1].
    pages = report_pages(["b1", "b2", "b3", "b4", "b5"], 2)
    assert [len(p) for p in pages] == [2, 2, 1]


@pytest.mark.skipif(
    sys.version_info < (3, 12),
    reason="itertools.batched is part of the stdlib only since Python 3.12",
)
def test_report_pages_uses_stdlib_batched():
    from itertools import batched
    assert list(batched("abcde", 2)) == [("a", "b"), ("c", "d"), ("e",)]


@pytest.mark.skipif(
    sys.version_info >= (3, 12),
    reason="the manual fallback is only exercised on Python < 3.12",
)
def test_report_pages_manual_fallback_on_old_python():
    assert "batched" not in dir(__import__("itertools"))

The two skipif are mirrors: test_report_pages_uses_stdlib_batched is skipped on 3.11 (where batched does not exist) and runs on 3.12+; test_report_pages_manual_fallback_on_old_python does the opposite. In any version, exactly one of the two runs and the other is skipped. Between the 3.11 and 3.12+ cells, both branches are exercised, each one where it lives.

Worked example: your machine as a witness cell

Let us run the full suite on the guide's machine —Python 3.14.0— which plays the part of a matrix cell (3.14 is ≥ 3.12, so it exercises the same branch as 3.12 and 3.13):

python -m pytest -v -rs

What to expect (real output on Python 3.14.0, with -rs to see the reason for the skip):

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- /private/tmp/reservo-m8/.venv/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/reservo-m8
configfile: pyproject.toml
testpaths: tests
plugins: xdist-3.8.0, rerunfailures-16.4, cov-7.1.0
collected 14 items

tests/test_pricing.py::test_basic_three_hours PASSED                       [ 35%]
tests/test_pricing.py::test_pro_three_hours PASSED                         [ 42%]
tests/test_pricing.py::test_basic_one_hour PASSED                          [ 50%]
tests/test_version_features.py::test_report_pages_groups_bookings PASSED   [ 85%]
tests/test_version_features.py::test_report_pages_uses_stdlib_batched PASSED [ 92%]
tests/test_version_features.py::test_report_pages_manual_fallback_on_old_python SKIPPED [100%]

=========================== short test summary info ============================
SKIPPED [1] tests/test_version_features.py:24: the manual fallback is only exercised on Python < 3.12
========================= 13 passed, 1 skipped in 0.02s =========================

(I trimmed the pricing and refund rows so as not to repeat; the final count is from the complete run.) The summary: 13 passed, 1 skipped. The test that is skipped is test_report_pages_manual_fallback_on_old_python, because on 3.14 (≥ 3.12) the fallback branch does not apply, and its reason is printed thanks to -rs. A skip is not a failure: it is "this case does not apply here, and we said so on purpose," a third state that does not alter the exit code 0.

How the three cells would read

Without a runner, this is how the matrix's list of jobs would look, all green (honest format of the log; the per-cell count is the one the cell would report):

tests · push to main   (fail-fast: false)
  ✓ test (3.11)   —  13 passed, 1 skipped
  ✓ test (3.12)   —  13 passed, 1 skipped
  ✓ test (3.13)   —  13 passed, 1 skipped

The three say 13 passed, 1 skipped, but —the fine detail— the test that is skipped is different on 3.11 than on 3.12/3.13:

CellBranch it exercisesTest that gets skipped
test (3.11)manual fallbacktest_report_pages_uses_stdlib_batched
test (3.12)itertools.batchedtest_report_pages_manual_fallback_on_old_python
test (3.13)itertools.batchedtest_report_pages_manual_fallback_on_old_python

On 3.11 the fallback is exercised (and the batched branch, which does not exist there, is skipped). On 3.12 and 3.13, batched is exercised. Between the three cells, both branches were really tested. A single job could not give you that: it would test one branch and leave the other untouched. And if a cell went red —say test (3.11) with ImportError: cannot import name 'batched'—, the name would tell you at once that the bug is from 3.11 (someone used batched without the fallback), and reproducing it would be installing 3.11 and running the suite (the technique of module 3, the reproducibility of the previous lesson).

What matrix Reservo deserves

The capstone is evaluated by the method, so the matrix is not chosen by reflex but by what the project really risks. The rule: test what you ship plus what you promise to support, and nothing more.

  • Three Python versions: yes, they pay off. Reservo is a library that others install, and its pyproject.toml/README promise 3.11, 3.12, and 3.13. Each one is a promise to users you do not control. Besides, Reservo uses a feature that varies by version (report_pages), so without the three cells one branch would go unexercised where it lives. The version dimension is not reflex: it is the promise made verifiable.
  • The operating-system dimension: it does not pay off (today). Reservo's core logic —price_cents, refund_cents, overlaps, book— is integer arithmetic and date comparison: it gives identical results on Linux, macOS, and Windows. A 3×3 matrix would run nine cells to get the same green six times over. It is cost (nine runs per push, with macOS and Windows more expensive) and noise, without catching a bug that the Linux row does not already catch.
  • The trigger for adding OS. The day Reservo writes reports to disk —paths, encoding, line breaks, which differ by system—, there os: [ubuntu-latest, windows-latest] would enter (POSIX vs. Windows, the real difference). Until then, adding it would be advancing a cost without coverage.

Conclusion: three cells (3.11, 3.12, 3.13 on Linux), with fail-fast: false. The matrix that corresponds to what Reservo really risks as a library, not the nine of a 3×3 by reflex nor the single cell that would leave two promised versions untested.

Common mistakes

Forgetting ${{ matrix.python-version }} in setup-python. What happens: someone writes the list ["3.11", "3.12", "3.13"] but leaves setup-python with python-version: "3.14" fixed. GitHub opens three jobs —the log looks "correct," with three cells—, but all three install 3.14, so the matrix is theater: it tests the same version three times. Why it happens: the matrix list and the step that consumes it are separated in the YAML, and it is easy to update one and forget the other. How to detect it: look at the header of each cell in the log; if all three say the same Python version, the connection is missing. How to fix it: the line python-version: ${{ matrix.python-version }} is what makes each cell install its version. Without it, the matrix spins in a vacuum.

Covering only one branch of the version feature. What happens: the itertools.batched test is written but not the fallback one, so in the 3.11 cell the fallback branch is never asserted —it goes green by omission—. Why it happens: on your machine (3.12+) you only see the batched branch, and it is easy to forget the other. How to detect it: for each if sys.version_info in the code, ask yourself "do I have a test for each branch, with a skipif that runs it where it applies?". How to fix it: the two mirror tests, with opposite conditions, plus at least one version on each side of the border (3.12) in the matrix.

Inflating the matrix to 3×3 "to be safe." What happens: os: [ubuntu, macos, windows] is added, going up to nine cells, even though Reservo's code touches nothing system-specific. Why it happens: more cells feels more robust. How to detect it: for the OS dimension, ask yourself "what bug does it catch that the Linux row would not catch?"; for pure logic, none. How to fix it: a single system row while the code is pure logic, documented in a comment; add the OS dimension only when the code starts to touch the disk.

Exercises

Exercise 1 — Predict the skip on 3.11. On 3.14 the run gave 13 passed, 1 skipped, and the one skipped was test_report_pages_manual_fallback_on_old_python. Without running anything, predict: in the 3.11 cell, how many pass and how many are skipped, and which test is skipped? One sentence of why.

See solution

It would still be 13 passed, 1 skipped, but the test that is skipped would be test_report_pages_uses_stdlib_batched. Reason: on 3.11, the condition of that test, sys.version_info < (3, 12), is true (3.11 < 3.12), so it is skipped; and that of the other, sys.version_info >= (3, 12), is false, so test_report_pages_manual_fallback_on_old_python runs and passes (on 3.11 itertools really does not have batched). The two skipif are mirrors: in every version, exactly one is skipped. What changes between cells is not the total count, but which code path was exercised —and that is why it is worth running both versions—.

Exercise 2 — The matrix makes lesson 5 urgent. Explain, in two or three sentences, why adding the matrix of three versions makes the next layer —cache and parallelism— go from "luxury" to "necessity."

See solution

The matrix multiplies the work by three: each push now runs the complete pipeline three times —three checkouts, three dependency installations, three suite runs—, one per version. Any slowness that was tolerable in a single job is paid triple: if installing dependencies takes a minute, now it is three minutes just installing, repeated in each cell even though the dependencies are almost the same. That is why the cache (not downloading the dependencies again in each cell) and the parallelism (running each cell's tests in several processes) stop being optional optimizations and become what keeps the feedback cycle short when there is a matrix. Lesson 5 comes right after this one not by chance: the matrix creates the speed problem that lesson 5 solves.

Exercise 3 — Justify (or reject) a fourth cell. The team proposes adding macos-latest to the matrix "because several developers use Mac." Is it defensible for Reservo as it is? What would you ask before deciding?

See solution

For Reservo as it is —pure logic of integer arithmetic and dates—, adding macos-latest is not defensible by reflex, because the code gives identical results on macOS as on Linux: it does not touch paths, or files, or line breaks, or anything that differs by operating system. The macOS cell would run the same suite and give the same green as the Linux one, without catching a single extra bug, and macOS is among the most expensive runners in billed minutes. It would be pure cost.

The key question before deciding is: "what behavior of Reservo could differ on macOS that Linux does not already cover?". If the honest answer is "none" (the current case), the cell does not pay off. The reason "several developers use Mac" confuses where it is developed with where the code behaves differently: that the team uses Mac to write does not mean Reservo's code runs differently there. The macOS cell would be justified the day Reservo started doing something system-dependent —writing files, invoking OS commands, handling paths— and the team had users on macOS reporting bugs. Until then, the correct matrix is the one of versions on a single Linux row. The capstone's discipline: each cell is justified by a real risk, not by a team habit.

Summary and next step

In this lesson you stacked the third layer: the version matrix. You wrapped the base workflow in a strategy.matrix of ["3.11", "3.12", "3.13"], connected to setup-python with ${{ matrix.python-version }} and with fail-fast: false to see the complete map, understanding that the matrix does not rewrite the work but wraps it and multiplies it. You reencountered report_pages with its two mirror skipif tests, ran your machine as a witness cell —13 passed, 1 skipped on 3.14— and saw that between the 3.11 and 3.12+ cells both branches of the feature are tested, each one where it lives.

You learned to read the three cells —same count, different test skipped by version— and to locate a bug by the name of the red cell. And you justified which matrix Reservo deserves: three versions (the README's promise made verifiable) on a single system row (because its pure logic gives identical results on every OS), neither the nine of a 3×3 by reflex nor the single cell that would leave promises untested. The method, not the reflex.

Before moving on you should be able to: write a strategy.matrix of versions and connect it to setup-python; explain what fail-fast: false does; predict which test is skipped in each version and why; and justify the size of a matrix by the real risk of the project.

What follows, in lesson 5, is the direct consequence of having a matrix: speed. Running the suite three times per push multiplies the time, and if each cell downloads the dependencies from the internet and runs the tests in series, the feedback cycle lengthens right when you use it most. You are going to stack the layer of cache (saving the dependencies between runs) and parallelism (pytest-xdist, -n auto), with a real speed demo —and an important honesty about when the parallelism pays off and when, in a suite like Reservo's, it only adds overhead—.

Resources

← Previous3. Pinned dependencies and reproducibilityNext →5. Cache and parallelism for speed