Module 4: The Matrix Versions And Environments

2. Why test in several environments

Next →

Description

In the previous lesson we installed the matrix idea with an analogy —the five-stove kitchen— and a demo. But an analogy isn't enough to design well: to know which matrix your project deserves, you first have to know against which exact dangers it protects you. This lesson opens up the underlying question: what really changes between one environment and another? Because if nothing changed, the matrix would be a waste; you'd run the same suite in nine identical environments to get the same green nine times. The matrix is worth it precisely because the environments are not identical, and the differences, though small, are real and they bite.

By the end you'll be able to name the concrete categories of difference between environments —standard-library features that appear in one version and not the previous one, new language syntax, dependency wheels that compile on one operating system and not another, path separators and line endings that differ by OS, default encodings— and recognize which of those differences your code touches without noticing. You're going to see with Reservo, running for real, how itertools.batched exists on your Python 3.14 but wouldn't exist on 3.11: a test that passes for you would break for the 3.11 user, and you'd never know it with a single job. The skipif reappears here not as a trick, but as the exact scalpel to isolate that difference and test each branch where it belongs.

Connection to the module: lesson 1 gave you the what (the matrix) and the what for (it works on my machine isn't enough). This one gives you the against what: the catalog of differences that justifies turning on more than one cell. It's the conceptual base of the four lessons that follow. Lesson 3 will express the "Python version" dimension in the YAML; this lesson tells you why that dimension matters. Lesson 4 will do the same with the "operating system" dimension; here you'll see in advance what changes between systems. And lesson 7, when you decide whether the matrix pays off, will lean on this catalog: you only turn on a cell if your code touches one of these differences in an environment you care about.

The same outlet, different voltage

When you travel from Mexico to Europe with your charger, you take an adapter. The wall outlet looks almost the same —two or three holes—, but inside runs something different: 127 volts on one side of the Atlantic, 230 on the other. If you plug a 127-volt device straight into a 230 outlet without an adapter, it's not that it "almost works": it burns out. And the cruel part is that the device worked perfectly for years in your home. "It works in my outlet" was true. It was simply a truth about your outlet, not about all the outlets in the world.

Execution environments are outlets. Python 3.11 and Python 3.13 look almost the same —the same language, the same syntax 99% of the way—, but inside they have voltage differences: a stdlib function that exists in one and not the other, a syntax the new one accepts and the old one rejects with a syntax error. Linux and Windows look almost the same to your Python code —you open files, you join paths—, but the path separator is / on one and \ on the other, and the line ending is one character on one and two on the other. Your code, like the device, can work for years in your outlet and burn out the first time it's plugged into another.

The matrix is the cautious traveler's drawer of adapters: before sending your device to another country, you test it at that country's voltage, on your own table, with a transformer. If it holds, you send it with peace of mind. If it heats up, you discover it at home. This is the map of the "voltages" that change between Python environments —the ones your code touches without noticing.

What really changes between Python versions

Not everything changes between 3.11 and 3.13. Reservo's integer arithmetic —2500 * 3 == 7500— gives the same in all versions; if it depended only on that, you wouldn't need a version matrix. What changes, and bites, falls into a few concrete categories.

Standard-library features that appear in one version. This is the most common one, and the one we use in Reservo. itertools.batched didn't exist before Python 3.12; it was added in 3.12. If your code does from itertools import batched, it works on 3.12+ and blows up with an ImportError on 3.11. Same with tomllib (reading TOML files, added in 3.11), datetime.UTC (a short alias, added in 3.11), typing.Self (added in 3.11), and dozens more. Each Python version brings new functions, and using one of them silently ties your code to that version or higher.

New language syntax. Sometimes it's not a function but the grammar itself. The generic type-parameter syntax —def first[T](items: list[T]) -> T— arrived in Python 3.12. On 3.11, that file doesn't even compile: it bursts with a SyntaxError on import, before running a single test. This kind of difference is more brutal than a missing function's, because there's no way to dodge it with an if: the old interpreter can't even read the file.

Behavior that changed without changing name. The most treacherous case: a function that exists in both versions but behaves differently. A real example: in Python 3.12, certain error messages and reprs changed format; a test that did assert "foo" in str(error) could pass on one version and fail on another because the message's exact text changed. Here there's no ImportError or SyntaxError to warn you: the test simply gives red in one cell and green in another, and you have to read the diff to understand that the behavior moved.

Dependencies that have no wheel for your combination. Your dependencies (the ones you install with pip) are sometimes distributed as precompiled wheels for specific combinations of Python version and operating system. If a library didn't publish a wheel for "Python 3.13 on Windows", pip tries to compile it from source, and that can fail if a compiler is missing. Reservo is pure stdlib and doesn't suffer this, but it's one of the most common reasons a green pip install on your machine turns red in a matrix cell.

What really changes between operating systems

The other dimension. Here the differences are more subtle because the same Python code runs on all three systems —there's no ImportError—, but the result differs. I'll measure them for real below; for now, the catalog.

The path separator. On Linux and macOS, paths use / (reservo/reports.py). On Windows, they use \ (reservo\reports.py). If your code builds paths by gluing strings with "/", it works on your Mac and breaks on Windows. The correct way —os.path.join or pathlib.Path— uses the correct separator per system, but a lot of code doesn't use it.

The line ending. When writing a text file, Linux and macOS end each line with a single character, \n. Windows uses two, \r\n. A test that does assert content == "line1\nline2\n" can pass on Linux and fail on Windows if the file was written with the system's line endings.

The default encoding. When opening a file without saying the encoding, Python uses the "system default", which was historically UTF-8 on Linux/macOS and something different (like cp1252) on Windows. A file with an accent —an ó in a room name— could be read fine on one system and come out as garbage on another. (Python 3.15 makes UTF-8 the default everywhere, but for years this was a classic source of Windows-only reds.)

Case sensitivity. On Linux, Reservo.py and reservo.py are two different files. On macOS and Windows, by default, they're the same. An import that works on your Mac by accident —you wrote from Reservo import x and the system didn't distinguish— bursts on the runner's Linux.

The demo: itertools.batched on your version and on the one that doesn't have it

Let's bring this down to Reservo, running for real. Remember that in lesson 1 we added report_pages, which uses itertools.batched on 3.12+ with a manual fallback before. The reason we did that if sys.version_info >= (3, 12) wasn't a whim: it was to dodge exactly the first category of difference. Let's look at it.

First, let's confirm on the real machine —Python 3.14.0— that itertools.batched really exists here:

python -c "print('batched' in dir(__import__('itertools')))"

What to expect. On Python 3.14.0:

True

batched is in itertools, because 3.14 is ≥ 3.12. Now imagine the same command on Python 3.11: it would print False, because there the function doesn't exist. That single difference —True here, False there— is what makes or breaks a test.

Look at the test that uses it directly, without protection:

def test_report_pages_uses_stdlib_batched():
    from itertools import batched
    assert list(batched("abcde", 2)) == [("a", "b"), ("c", "d"), ("e",)]

On your Python 3.14, this test passes: the import works and batched("abcde", 2) groups by twos. But if the same test ran on Python 3.11, the line from itertools import batched would fail before reaching the assert, with an error like this:

ImportError: cannot import name 'batched' from 'itertools'

And here's the exact danger the matrix prevents: on your machine, that test is green and you're happy. There's no signal that anything is wrong. If your pipeline runs a single job on 3.14, you'll see green and publish. The user who installs your code on 3.11 will be the one who discovers the ImportError, in production, in their face —the raw dish on the stove you never tested—. A single green job lied to you by omission: it was true about 3.14 and you read it as true about everything.

Worked example: the skipif as a scalpel for the difference

What do you do when a test only makes sense on certain versions? You don't delete it —you want to test it where it applies— and you don't let it blow up where it doesn't apply. You mark it with @pytest.mark.skipif, which is precisely the scalpel to isolate a version difference. Let's review Reservo's two mirror tests, now with an eye on which difference each one is isolating:

# tests/test_version_features.py (fragment)
import sys
import pytest

@pytest.mark.skipif(
    sys.version_info < (3, 12),
    reason="itertools.batched is part of the stdlib only since Python 3.12",
)
def test_report_pages_uses_stdlib_batched():
    from itertools import batched
    assert list(batched("abcde", 2)) == [("a", "b"), ("c", "d"), ("e",)]


@pytest.mark.skipif(
    sys.version_info >= (3, 12),
    reason="the manual fallback is only exercised on Python < 3.12",
)
def test_report_pages_manual_fallback_on_old_python():
    assert "batched" not in dir(__import__("itertools"))

The first skipif says: "skip this test if the version is lower than 3.12". So, on 3.11 —where from itertools import batched would blow up— the test doesn't even attempt the import: it skips cleanly, with its reason noted. On 3.12+ it runs and really tests the branch that uses the stdlib. The second skipif is the mirror: it tests the manual-fallback branch, and only makes sense on 3.11, so it skips on 3.12+.

Let's run only this file on the real machine to see the scalpel in action, with -rs so it prints the skip reasons:

python -m pytest -v -rs tests/test_version_features.py

What to expect. On Python 3.14.0, measured for real:

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0 -- /private/tmp/reservo-m4/.venv/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/reservo-m4
collecting ... collected 3 items

tests/test_version_features.py::test_report_pages_groups_bookings PASSED  [ 33%]
tests/test_version_features.py::test_report_pages_uses_stdlib_batched PASSED [ 66%]
tests/test_version_features.py::test_report_pages_manual_fallback_on_old_python SKIPPED [100%]

=========================== short test summary info ============================
SKIPPED [1] tests/test_version_features.py:25: the manual fallback is only exercised on Python < 3.12
========================= 2 passed, 1 skipped in 0.00s =========================

Read the story this result tells. test_report_pages_groups_bookings —the rule that holds in every version— passed. test_report_pages_uses_stdlib_batched —the stdlib branch— passed, because on 3.14 itertools.batched exists. And test_report_pages_manual_fallback_on_old_python —the fallback branch— skipped, with its crystal-clear reason: "the manual fallback is only exercised on Python < 3.12". Nothing blew up. The skipif cut exactly where it had to cut: it tested what applies, skipped what doesn't, and left a written record of the decision.

Now tie the loop with the matrix. This same file, in the 3.11 cell, would invert the skips: it would test the fallback (which does apply there) and skip the batched branch (which doesn't exist there). With the two cells —3.11 and 3.12+— both branches of report_pages end up really exercised, each in the version where it lives. That's what a single job can't give you: a single job tests one branch and leaves the other untouched. The matrix covers both.

An important nuance: skipif isn't the solution, it's the honesty

Beware an easy reading: "ah, so I put skipif on everything and done, solved". No. The skipif doesn't fix the incompatibility; it recognizes and locates it. If your code uses itertools.batched without the fallback, a skipif on the test doesn't change that your code breaks on 3.11 —it only keeps the test from faking a verdict where it can't have an opinion—. The real incompatibility you solve in the code (with the if sys.version_info that chooses the branch, or by raising your minimum supported version), not in the test.

So, what is skipif for? For two honest things. First: testing a branch that only exists in certain versions, without the test blowing up in the others (our case). Second: documenting, in an executable way, that "this behavior only applies here". A skipif with a clear reason is a comment the runner respects. What it should not be is a rug under which you sweep an incompatibility so CI shuts up: if you catch yourself putting skipif so a test stops failing on a version you do promise to support, you're not testing —you're hiding the problem.

Common mistakes

Assuming "it passes on my version" is "it passes on all". What happens: you write a test that uses a new stdlib function, it passes on your 3.14, and you publish. On 3.11 it bursts with ImportError and a user discovers it. Why it happens: your machine has a single version, and that version gives you a single answer; you don't see the others. How to spot it: every time you use a stdlib function, ask yourself "since which version does this exist?" —the docs say it with an "Added in version X"—. How to fix it: either you raise your minimum supported version to that one, or you add a fallback, or —to know before publishing— you run a matrix that includes your minimum version.

Gluing paths with strings and testing only on your OS. What happens: you build a path with folder + "/" + file, it works on your Mac, and on a teammate's Windows the \ separator breaks it. Why it happens: your OS uses /, so you never see the problem; the hand-written / is comfortable and silently incorrect. How to spot it: search your code for path concatenations with "/" or "\\"; each one is a candidate to break on another OS. How to fix it: use os.path.join or pathlib.Path, which put the correct separator per system, and —if the code really touches the file system— include Windows in your matrix to verify it.

Using skipif to silence instead of to locate. What happens: a test fails on 3.11, and instead of fixing the incompatibility, someone puts @pytest.mark.skipif(sys.version_info < (3, 12), ...) on it so CI goes green. But the project does promise to support 3.11. Now CI lies: it says green, but the code is broken on a version you promised. Why it happens: it's faster to silence a test than to fix the code. How to spot it: for each skipif, ask yourself "does the code really not apply here, or am I just covering that it breaks?". How to fix it: if you promise to support the version, fix the code (fallback or minimum version); reserve skipif for cases where the code branch genuinely doesn't exist in that version.

Exercises

Exercise 1 — Classify the difference. For each situation, say which category of difference between environments it belongs to —(i) stdlib function that appears in one version, (ii) new language syntax, (iii) operating-system difference— and whether it would manifest as ImportError, SyntaxError, or a test that gives red with no import error: (a) using from tomllib import load (added in 3.11) on Python 3.10; (b) writing def wrap[T](x: T) (3.12 syntax) and running it on 3.11; (c) a test that does assert path == "data/out.txt" running on Windows.

See solution
  • (a) Category (i), stdlib function. tomllib was added in 3.11; on 3.10 it doesn't exist. It manifests as an ImportError on the line from tomllib import load, before running anything. It's dodged with a fallback (the external package tomli) or by raising the minimum version to 3.11.
  • (b) Category (ii), new syntax. The type parameters [T] in the signature arrived in 3.12. On 3.11 the file doesn't compile: SyntaxError on import. It's the harshest, because there's no if that dodges it: the old interpreter can't even read the file. The only way out is to not use that syntax if you support 3.11.
  • (c) Category (iii), OS difference. On Windows the separator is \, so the real path would be data\out.txt, not data/out.txt. There's no import error: the test simply gives red because the expected string doesn't match. It's fixed by building the path with os.path.join/pathlib and comparing in a separator-independent way.

The usefulness of classifying: (i) and (ii) you see by including them in the version matrix; (iii) you see by including them in the operating-system matrix. Knowing the category tells you which matrix dimension to turn on.

Exercise 2 — Honest green or lying green? Your project promises in its README "supports Python 3.11+". A test uses itertools.batched directly. A teammate, so CI stops failing in the 3.11 cell, puts @pytest.mark.skipif(sys.version_info < (3, 12), reason="batched isn't in 3.11") on it. CI goes green. Is it an honest green? Explain what's wrong and what the correct fix would be.

See solution

It's a lying green. The README promises to support 3.11, but the code uses itertools.batched, which doesn't exist on 3.11: the code is broken on a version you promised to support. The skipif doesn't fix that; it only makes the test stop reporting it, so CI says "green" while a 3.11 user receives an ImportError. The problem was swept under the rug.

The correct fixes, in order of preference:

  1. Fix the code, not the test. Put the fallback like in Reservo: if sys.version_info >= (3, 12): from itertools import batched … else: manual comprehension. That way the code really works on 3.11, and the test can run (with the two mirror skipif testing each branch where it applies).
  2. Raise the minimum version. If you really want to use batched without a fallback, change the README and the config to "supports 3.12+" and remove 3.11 from the matrix. It's honest: you no longer promise something you don't deliver.

What doesn't count is leaving the skipif silencing the failure while the support promise still says 3.11. Rule: skipif documents branches that genuinely don't apply; it doesn't hide broken code on versions you promise.

Exercise 3 — Design the minimal matrix for a difference. Reservo debuts a function that reads its pricing configuration from a TOML file with tomllib (added in Python 3.11), with a fallback to the external package tomli for earlier versions. The team promises to support Python 3.10, 3.11, and 3.12. Which versions should the matrix include at a minimum to really test both code branches, and why those?

See solution

At a minimum, the matrix must include a version < 3.11 and a version ≥ 3.11, because that's where the behavior's boundary is. Specifically, since 3.10, 3.11, and 3.12 are promised:

  • 3.10 exercises the fallback branch (uses the external package tomli, because tomllib doesn't exist yet). Without a cell < 3.11, that branch would never really be tested.
  • 3.11 exercises the tomllib branch (the stdlib), and is also exactly the boundary version where the behavior changes: the best place to catch an edge error.
  • 3.12 confirms that the tomllib branch stays fine on the newest supported version.

The principle: for a difference that occurs at a version boundary, the matrix must have at least one cell on each side of that boundary, plus the other versions you promise to support. Testing only 3.12 would leave the fallback branch (the 3.10 one) completely unexercised —green by omission, silently broken—. This is exactly the reasoning lesson 7 formalizes: test what you promise to support, with focus on the boundaries where the behavior changes.

Summary and next step

In this lesson you opened the box of "what changes between environments", which is what justifies turning on more than one matrix cell. Between Python versions: stdlib functions that appear in one version (itertools.batched since 3.12), new syntax that doesn't even compile on the old version, behavior that changed without changing name, and dependencies without a wheel for your combination. Between operating systems: the path separator (/ vs \), the line ending (\n vs \r\n), the default encoding, and case sensitivity. Each one is a different "voltage" your code can touch without noticing.

You saw it running: itertools.batched exists on your Python 3.14 (True) and not on 3.11; a test that uses it directly passes for you and would break with ImportError for the 3.11 user —the exact danger a single green job doesn't see—. And you saw the skipif as the honest scalpel: it isolates the difference, tests each branch where it applies (2 passed, 1 skipped), and leaves a record with its reason, without faking a verdict where it can't have an opinion. The nuance became clear: skipif locates the difference, it doesn't fix it; the real fix lives in the code.

Before moving on you should be able to: name at least three categories of difference between environments and say how each one manifests (ImportError, SyntaxError, red with no error); explain why a single green job can lie by omission; and distinguish a skipif that documents a legitimate branch from one that silences broken code.

What's next, in lesson 3, is writing the first matrix dimension in the YAML: strategy.matrix with several Python versions. You're going to see how a list of three versions becomes three jobs, how the version is injected into setup-python, and how the CI log would look —with the local run on 3.14 as the witness cell you execute for real.

Resources

← Previous1. Module introduction: the matrixNext →3. `strategy.matrix` of Python versions