Module 4: The Matrix Versions And Environments

6. Reading the matrix's N jobs

Next →

Description

A matrix doesn't produce one result, it produces N. Where before you had one check —"the suite passes" or "the suite fails"—, now you have a list of nine, or six, or four, each with its name and its traffic light. And that multiplication isn't just more to look at: it's more information. A single red job tells you "something broke". Nine cells, eight green and one red, tell you "something broke, and it was exactly on Python 3.11 on Windows, nowhere else". This lesson teaches you to read that grid of results: to turn N traffic lights into a precise diagnosis of where a bug lives.

By the end you'll be able to read the job list of a matrix run —understand how GitHub names each cell, test (windows-latest, 3.11)—, distinguish an all-green grid from one with a red cell or a pattern of reds, and read the log of that red cell to know what failed: the line that tells you the version and the system, and pytest's failure report that tells you the test and the assertion. You're going to see, with really executed output, how a real red Reservo log reads, and how that reading gives you exactly the datum —"it failed in this environment"— you need to reproduce it locally (module 3) by matching that environment.

Connection to the module: lessons 3, 4, and 5 built and sculpted the matrix; this one teaches you to read its output. It's the counterpart of reading a single job's log (module 2), but multiplied: now the first datum isn't "did it pass?" but "which passed and which didn't?". Lesson 7 will use this skill to justify the matrix's size —a cell that never turns red informatively is a cell that might be excess—. And the bridge back is explicit: when you read "it failed on 3.11", the next step —reproducing it on your machine with 3.11— is module 3's technique, which here we only hook into.

The airport arrivals board

At an airport, the arrivals board doesn't just say "there are problems". It shows a row per flight, each with its status: on time, landed, delayed, cancelled. If your flight says "delayed" and all the others say "on time", you know the problem is with your flight, not the airport —maybe the aircraft is coming late from another city—. But if suddenly all of one airline's flights say "cancelled" and the others stay on time, the pattern tells another story: the problem is that airline's, not a single flight's. The board doesn't just report; its pattern diagnoses.

A matrix's job list is that arrivals board. Each cell is a row with its status. A lone red cell, surrounded by greens, points to that specific combination: "the problem is Python 3.11 on Windows". A whole column of reds —the three Windows versions— points to the system: "the problem is Windows, regardless of version". A whole row of reds —the three systems on 3.11— points to the version: "the problem is 3.11, everywhere". Reading the matrix is reading the pattern, not just counting reds.

A single red job says "something failed". A red matrix says "something failed, and the pattern of which cells fail tells you whether it's a version problem, a system problem, or an exact combination". The diagnosis starts in the shape of the red.

How each cell is named

To read the grid you need to understand the names, because the name is the cell's coordinates. GitHub builds each job's name like this: the base job name, plus, in parentheses, the values of the matrix dimensions, separated by commas.

test (ubuntu-latest, 3.11)     <- job "test", os=ubuntu-latest, python-version=3.11
test (windows-latest, 3.12)    <- job "test", os=windows-latest, python-version=3.12
test (3.13)                    <- if there's only a version dimension, only that appears

The order of the values within the parentheses follows the order in which you declared the dimensions in the YAML. This matters because the name is what you see in a pull request's check list, in the Actions tab, and in branch protection. When someone says "test (windows-latest, 3.11) failed", they're giving you the exact coordinates: Windows system, version 3.11. Without reading a single log yet, you already know where to look.

This is how an all-green run of the 3×3 matrix looks:

tests · push to main
  ✓ test (ubuntu-latest, 3.11)    ✓ test (ubuntu-latest, 3.12)    ✓ test (ubuntu-latest, 3.13)
  ✓ test (macos-latest, 3.11)     ✓ test (macos-latest, 3.12)     ✓ test (macos-latest, 3.13)
  ✓ test (windows-latest, 3.11)   ✓ test (windows-latest, 3.12)   ✓ test (windows-latest, 3.13)

Nine greens. Your code works in the nine combinations you promised to support. That's what the matrix buys you: not "it works on my machine" but "it works on these nine, verified".

Reading the pattern of the reds

Now the shapes of red, each with its diagnosis. Suppose a change introduced a bug. The matrix could look several ways, and each tells you something different.

A lone red cell — the bug depends on that exact combination:

  ✓ test (ubuntu-latest, 3.11)    ✓ test (ubuntu-latest, 3.12)    ✓ test (ubuntu-latest, 3.13)
  ✓ test (macos-latest, 3.11)     ✓ test (macos-latest, 3.12)     ✓ test (macos-latest, 3.13)
  ✗ test (windows-latest, 3.11)   ✓ test (windows-latest, 3.12)   ✓ test (windows-latest, 3.13)

Only (windows, 3.11) red. Diagnosis: the bug needs both conditions at once —Windows and 3.11—. Maybe a dependency with no wheel for that exact pair, or a feature missing only in that corner. It's the most specific red and, sometimes, the rarest.

A column of reds — the bug depends on the system:

  ✓ test (ubuntu-latest, 3.11)    ✓ test (ubuntu-latest, 3.12)    ✓ test (ubuntu-latest, 3.13)
  ✓ test (macos-latest, 3.11)     ✓ test (macos-latest, 3.12)     ✓ test (macos-latest, 3.13)
  ✗ test (windows-latest, 3.11)   ✗ test (windows-latest, 3.12)   ✗ test (windows-latest, 3.13)

The three Windows cells red, the Linux and macOS ones green. Diagnosis: the bug is Windows's, on all versions —typically the path separator \ or the line ending \r\n from lesson 4—. The version doesn't matter; the system does.

A row of reds — the bug depends on the version:

  ✗ test (ubuntu-latest, 3.11)    ✓ test (ubuntu-latest, 3.12)    ✓ test (ubuntu-latest, 3.13)
  ✗ test (macos-latest, 3.11)     ✓ test (macos-latest, 3.12)     ✓ test (macos-latest, 3.13)
  ✗ test (windows-latest, 3.11)   ✓ test (windows-latest, 3.12)   ✓ test (windows-latest, 3.13)

The three 3.11 cells red, the 3.12 and 3.13 ones green. Diagnosis: the bug is Python 3.11's, on all systems —typically a stdlib function you used that doesn't exist before 3.12, like lesson 2's itertools.batched without its fallback—. The system doesn't matter; the version does.

Notice the power of this: without opening a single log, the shape of the red already gave you the hypothesis. Windows column → look at path or file handling. 3.11 row → look at which new feature you used. Lone cell → look at what that exact combination needs. The log confirms the hypothesis; the pattern generates it.

Reading the log of a red cell

When the pattern points you to a cell, you open its log to confirm. A matrix cell's log is identical to a normal job's —you saw it in module 2—, with a crucial difference: the platform line tells you which cell you're looking at. Let's see a real red Reservo log, executed for real. It's the cell that catches lesson 4's path-separator bug —a test that compares against a Windows separator and therefore fails on macOS—:

Worked example

python -m pytest -v tests_red/test_report_path_bug.py

What to expect. On Python 3.14.0 on macOS, measured by really executing:

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- /private/tmp/reservo-m4/.venv/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/reservo-m4
collecting ... collected 1 item

tests_red/test_report_path_bug.py::test_report_path_hardcoded_separator FAILED [100%]

=================================== FAILURES ===================================
_____________________ test_report_path_hardcoded_separator _____________________

    def test_report_path_hardcoded_separator():
        # BUG: compares against a '/' separator written by hand. Passes on POSIX,
        # breaks on Windows. Here we force it to fail to see a RED cell.
        path = os.path.join("reports", "2026-08", "daily.txt")
>       assert path == "reports\\2026-08\\daily.txt"  # Windows separator, fails on macOS
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       AssertionError: assert 'reports/2026-08/daily.txt' == 'reports\\2026-08\\daily.txt'
E
E         - reports\2026-08\daily.txt
E         ?        ^       ^
E         + reports/2026-08/daily.txt
E         ?        ^       ^

tests_red/test_report_path_bug.py:8: AssertionError
=========================== short test summary info ============================
FAILED tests_red/test_report_path_bug.py::test_report_path_hardcoded_separator - AssertionError: assert 'reports/2026-08/daily.txt' == 'reports\\2026-08\\da...
========================= 1 failed in 0.02s ===============================

Read it in the order a matrix log is read, top to bottom:

  1. platform darwin -- Python 3.14.0 — the first thing. This line confirms which cell you're in: system darwin (macOS), version 3.14. In CI, if the job name said test (macos-latest, 3.14), this line must match. If they didn't match, you'd be looking at the wrong log —a classic error when debugging a matrix—.
  2. test_report_path_hardcoded_separator FAILED — which test failed, name and all. In a matrix with many tests, this narrows it down at once.
  3. The FAILURES block — the detailed report: the test's code, the exact line marked with >, and the AssertionError with the two values. Here it reads crystal-clear: the test expected reports\2026-08\daily.txt (with \, Windows's) but os.path.join produced reports/2026-08/daily.txt (with /, macOS's). The diff with -/+ and the ^ pointing to the positions that differ nails the problem: the separators.
  4. The short test summary info — the one-line summary, perfect for when there are many failures and you want the list without scrolling.

Now connect with the pattern. In this example we forced the failure on macOS to see it, but in the real world this bug would produce a red Windows column (the well-written test compares against the real system separator, and the Windows separator \ is the one that doesn't match the / the careless code assumed). The log confirms what the pattern anticipated: it's a path-separator problem, which is an operating-system problem.

From the red to the fix: the bridge to reproducing locally

Reading the matrix leaves you with a precise sentence: "it failed in this environment, in this test, over this assertion". That sentence is exactly module 3's starting point. If the matrix says "it failed in test (ubuntu-latest, 3.11)", your next step is to reproduce it on your machine by matching that environment: install Python 3.11 locally (with pyenv or similar), run the suite there, and see the same red in your terminal, where you can use the debugger and experiment. The matrix gave you the what and the where; reproducing and fixing is module 3.

It's worth saying why the matrix makes this so easy compared to a single job. With a single job on 3.12, if something broke only on 3.11, you didn't even find out until a user complained, and then you had to guess the environment. With the matrix, the red already brings the environment coordinates: you don't guess, you read. "It failed on 3.11" isn't a hypothesis you build; it's a datum the cell handed you. That cuts the diagnosis work in half —module 3 starts with the environment already identified—.

A note on required checks and branch protection

To close, the most common use of these N results: branch protection. GitHub lets you mark certain checks as "required" to be able to merge to the main branch. With a matrix, each cell is a check with its own name, so you can require, for example, that test (ubuntu-latest, 3.12) and test (windows-latest, 3.12) be green before allowing the merge. If any of those cells is red, the merge button is blocked.

This turns the matrix from "useful information" into "a real gate": it's not that you see the red and decide to ignore it; it's that the red prevents the broken code from entering main. We won't go deep into the configuration —it's a repository setting, not a YAML one—, but keep the idea: naming the cells well matters because those names are the ones you choose as required checks. A matrix whose results no one requires is an arrivals board no one looks at; connected to branch protection, it's access control.

Common mistakes

Counting reds instead of reading the pattern. What happens: someone sees "three red cells" and concludes "there are three bugs", when in reality it's one bug affecting three cells (a Windows column, a single separator problem). They start fixing three things when it was one. Why it happens: you count the number of reds instead of looking at their shape. How to spot it: before touching anything, ask yourself "do these reds form a row, a column, or are they scattered?". How to fix it: read the pattern first. A column is a system problem; a row, a version problem; scattered, several problems. The pattern tells you how many bugs there really are and of what type.

Looking at the wrong cell's log. What happens: the matrix says test (windows-latest, 3.11) failed, but you open the log of test (ubuntu-latest, 3.12) (which is green), see nothing odd, and get confused. Why it happens: with nine nearly identical logs, it's easy to open the wrong one. How to spot it: look at the platform ... Python X.Y line at the start of the log; if it doesn't match the cell you think you're looking at, it's the wrong log. How to fix it: always use the job name to open its log, and verify the platform line as confirmation before drawing conclusions.

Ignoring a red cell because "the others pass". What happens: eight green cells and one red on 3.11, and someone merges anyway thinking "almost everything passes". The 3.11 user —whom you promised support— gets the bug. Why it happens: the green majority gives a false sense of "it's fine". How to spot it: if a cell of a version or system you promise to support is red, your code is broken for those users, no matter how many others pass. How to fix it: treat each matrix cell as a promise; a red one is a broken promise. If you really no longer support that environment, remove it from the matrix (lesson 5/7), don't ignore it in red.

Exercises

Exercise 1 — Diagnose by the pattern. For each results grid (matrix of 3 systems × 3 versions), say whether the bug seems version-related, system-related, or from an exact combination, and what you'd look at first: (a) only the three cells of the 3.11 row red; (b) only (macos-latest, 3.13) red; (c) the three cells of the Windows column red.

See solution
  • (a) 3.11 row red → version bug. The three 3.11 cells fail on all systems, and 3.12/3.13 pass. The system doesn't matter, the version does. I'd look first at which new feature I used that doesn't exist in 3.11 —a stdlib function, a syntax— without its fallback (lesson 2's pattern).
  • (b) (macos-latest, 3.13) lone red → exact-combination bug. Only that corner fails. It needs both conditions: macOS and 3.13. I'd look first at something specific to that pair —a dependency with no wheel for macOS+3.13, or a behavior that changed only in that combination—. It's the rarest red and sometimes the hardest.
  • (c) Windows column red → system bug. The three Windows cells fail, the Linux/macOS ones pass. The version doesn't matter, the system does. I'd look first at path and file handling —\ vs / separator, \r\n line endings— which is the classic source of Windows-only reds (lesson 4).

The rule: row = version, column = system, lone cell = combination. The shape of the red generates the hypothesis before opening a log.

Exercise 2 — Read the log and locate the cell. You're given this header of a CI log and the failure's first line. Which matrix cell are you in, and what kind of problem does it suggest?

platform win32 -- Python 3.11.9, pytest-9.1.1
...
E   ImportError: cannot import name 'batched' from 'itertools'
See solution

The cell is test (windows-latest, 3.11) —or any 3.11 cell; what the platform win32 -- Python 3.11.9 line fixes is the system (Windows, win32) and the version (3.11)—. The kind of problem is screamed by the failure: ImportError: cannot import name 'batched' from 'itertools'. itertools.batched doesn't exist before Python 3.12, so the code used it without a fallback and blows up on 3.11.

Complete diagnosis: even though you see this red in the Windows cell, it's not a Windows problem —it's a version one—. If you opened the (ubuntu, 3.11) and (macos, 3.11) cells, they'd also be red with the same ImportError: it's a 3.11 row, not a Windows column. The platform line gave you the coordinates, and the nature of the error (batched doesn't exist in 3.11) tells you the culprit dimension is the version. The fix: lesson 2's fallback, or raising the minimum supported version. Reproduce locally (module 3) by matching Python 3.11.

Exercise 3 — One bug or several? A 3×3 matrix run shows these four cells red: (windows-latest, 3.11), (windows-latest, 3.12), (windows-latest, 3.13), and (ubuntu-latest, 3.11). The rest, green. How many distinct bugs does this pattern suggest, and of what type is each?

See solution

The pattern suggests two distinct bugs, because the reds don't form a single clean shape but two overlapping ones:

  1. A complete Windows column (3.11, 3.12, 3.13 of windows-latest) → an operating-system bug, present on all Windows versions. Probable cause: path handling (\ vs /) or line endings. The version doesn't affect it.
  2. An extra Linux cell on 3.11 (ubuntu-latest, 3.11), which doesn't fit in the Windows column → a second problem, version-related, that affects 3.11 but on Windows is "covered" by the system bug. Probable cause: a stdlib feature missing in 3.11.

How to verify it: the (macos-latest, 3.11) cell would be the proof —if it were also red, it would confirm that the 3.11 bug is version-related and affects all systems (a 3.11 row), and that it only shows up "clean" on Linux and macOS because Windows is already red for another cause—. In the statement (macos, 3.11) is green, which complicates the reading: it could be that the 3.11 bug additionally depends on something of Linux. The real lesson: when the reds don't form a clean shape, suspect more than one bug, and use the surrounding green/red cells to separate the causes. Counting "four reds = four bugs" would be the error; reading the pattern reveals they're probably two causes, one system-related and one version-related.

Summary and next step

In this lesson you learned to read a matrix's N results like an arrivals board: you don't count reds, you read their pattern. A lone red cell points to an exact combination; a column, to a system; a row, to a version. The shape of the red generates the hypothesis before opening a single log. You saw how GitHub names each cell with its coordinates —test (windows-latest, 3.11)— and how that name is your map, both to locate and to require checks in branch protection.

You read a real red Reservo log top to bottom: the platform darwin -- Python 3.14.0 line that confirms which cell you're looking at, the FAILED that says which test, the FAILURES block with the assertion and the diff that nails the cause (the path separators), and the short test summary. And the bridge became clear: reading the matrix leaves you with "it failed in this environment, in this test, over this assertion" —exactly the starting point for reproducing it locally by matching that environment, which is module 3—.

Before moving on you should be able to: read a results grid and diagnose by its pattern (row/column/cell); name a cell's coordinates from its name; read a red log and confirm which cell you're in by the platform line; and distinguish a bug that affects several cells from several distinct bugs.

What's next, in lesson 7, is the question that's been hovering over the whole module: when does this matrix pay off and when is it pure noise and cost? You already know how to build it, sculpt it, and read it; now you're going to decide, with judgment and a cost model, which matrix a project really deserves —a library used by thousands versus an internal single-version app— so you don't turn on nine cells by reflex when one tells the whole story.

Resources

← Previous5. `include`, `exclude`, and `fail-fast`Next →7. When the matrix pays off and when it's noise