3. Matrix Testing — Multiple Python Versions

Overview

Your workflow from the previous capsule runs pytest on Python 3.12 and it passes. But your users — or your production servers — may be on Python 3.10 or 3.11. A dependency that works on 3.12 can fail on 3.10. A Python feature you use without realizing it (like match/case) doesn't exist in 3.10. And you don't find out until someone reports an error in production.

Matrix testing solves this: it runs the same tests on multiple Python versions simultaneously. One push generates 3 parallel jobs — one per version. If any of them fails, you know immediately. You don't have to install 3 versions of Python on your machine or remember to test manually on each one.

Think of matrix testing as a confidence multiplier. Without a matrix, your tests validate one specific combination (Python 3.12 + Ubuntu). With a matrix, they validate 3, 6, or however many you need. The computational cost is low (GitHub runs the jobs in parallel) and the benefit is high.

Why does it matter? — The real problem

Let's look at a concrete example. This code works perfectly on Python 3.10+:

# src/ai_app/router.py
def route_request(request_type: str) -> str:
    match request_type:
        case "completion":
            return "openai_completion_handler"
        case "embedding":
            return "openai_embedding_handler"
        case _:
            return "default_handler"

The problem? match/case was introduced in Python 3.10. If someone tries to use your library with Python 3.9, they get:

SyntaxError: invalid syntax

Another example — list[dict] as a type hint (builtin generics) only works from Python 3.9+. In Python 3.8 you need from typing import List, Dict.

Without matrix testing: You only discover these problems when someone reports a bug. With matrix testing: You discover it in the PR, before merging.

strategy.matrix — The basic syntax

The matrix configuration lives inside the job's strategy key:

# .github/workflows/test.yml
name: Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      # Each version spawns an independent parallel job — no manual duplication needed
      matrix:
        # Quotes required: without them, YAML reads 3.10 as float 3.1
        python-version: ["3.10", "3.11", "3.12"]

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        run: pytest tests/ -v --tb=short

What happens when you push

Push to main
    │
    ├── Job: test (Python 3.10) ──► Runner 1 ──► pytest
    ├── Job: test (Python 3.11) ──► Runner 2 ──► pytest
    └── Job: test (Python 3.12) ──► Runner 3 ──► pytest

3 jobs in parallel, each with its own VM

GitHub creates 3 identical jobs, each with a different Python version. All 3 run at the same time. The workflow only passes if all 3 pass.

How it looks in GitHub Actions' UI

Tests
├── test (3.10) ✅ Passed (42s)
├── test (3.11) ✅ Passed (38s)
└── test (3.12) ✅ Passed (35s)

Important detail: the quotes on the versions

# CORRECT
python-version: ["3.10", "3.11", "3.12"]

# INCORRECT — 3.10 is read as 3.1
python-version: [3.10, 3.11, 3.12]

YAML reads 3.10 as the number 3.1 (it drops the trailing zero). With quotes, "3.10" is the string "3.10". This is a classic error.

The ${{ matrix.python-version }} variable

Inside the steps, you access the matrix's current value with ${{ matrix.python-version }}. In each parallel job, this variable has a different value:

You can use this variable in any step — in the name, in commands, in conditions.

fail-fast: Stop at the first failure or see them all?

By default, fail-fast is true: if one job fails, GitHub cancels the others immediately.

fail-fast: true (default)

test (3.10) ❌ Failed (15s)
test (3.11) ⚪ Cancelled
test (3.12) ⚪ Cancelled

Advantage: Fast feedback. Disadvantage: You don't know whether the error affects all the versions.

fail-fast: false

    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]

test (3.10) ❌ Failed (15s)
test (3.11) ✅ Passed (38s)
test (3.12) ✅ Passed (35s)

Advantage: You see the full picture. Disadvantage: You use more CI minutes.

Which one to choose?

For most projects, fail-fast: false in matrix testing is better. The extra time is minimal (the jobs run in parallel) and the extra information is worth a lot.

Practical example: Code that fails on one version

Let's look at a case where matrix testing saves you:

# src/ai_app/parser.py
def parse_model_response(response: dict) -> dict:
    """Parse the model's response and extract metadata."""
    content = response.get("choices", [{}])[0].get("message", {}).get("content", "")
    model = response.get("model", "unknown")
    usage = response.get("usage", {})

    match model:
        case str(m) if "gpt-4" in m:
            tier = "premium"
        case str(m) if "gpt-3.5" in m:
            tier = "standard"
        case _:
            tier = "basic"

    return {
        "content": content,
        "model": model,
        "tier": tier,
        "tokens": usage.get("total_tokens", 0),
    }

# tests/test_parser.py
from src.ai_app.parser import parse_model_response

def test_parse_gpt4_response():
    response = {
        "choices": [{"message": {"content": "Hello!"}}],
        "model": "gpt-4o-mini",
        "usage": {"total_tokens": 42},
    }
    result = parse_model_response(response)
    assert result["tier"] == "premium"
    assert result["tokens"] == 42

If your README says "Python 3.9+" but you use match/case, someone with Python 3.9 gets a SyntaxError. To catch it, you would add "3.9" to your matrix.

# src/ai_app/parser.py (compatible with Python 3.9+)
def parse_model_response(response: dict) -> dict:
    content = response.get("choices", [{}])[0].get("message", {}).get("content", "")
    model = response.get("model", "unknown")
    usage = response.get("usage", {})

    if "gpt-4" in model:
        tier = "premium"
    elif "gpt-3.5" in model:
        tier = "standard"
    else:
        tier = "basic"

    return {
        "content": content,
        "model": model,
        "tier": tier,
        "tokens": usage.get("total_tokens", 0),
    }

Matrix with include and exclude

Sometimes you need specific combinations that don't follow the simple pattern.

include: Add extra combinations

    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        include:
          - python-version: "3.13"
            experimental: true

This generates 4 jobs. You can use the experimental variable to allow failures:

    continue-on-error: ${{ matrix.experimental || false }}

test (3.10) ✅ Passed
test (3.11) ✅ Passed
test (3.12) ✅ Passed
test (3.13) ❌ Failed (but the workflow stays green)

exclude: Remove specific combinations

When you combine multiple dimensions, exclude removes combinations that don't make sense:

    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        os: [ubuntu-latest, macos-latest]
        exclude:
          - python-version: "3.10"
            os: macos-latest

Without exclude, you'd have 6 combinations (3 × 2). With the exclude, you have 5.

Matrix with multiple dimensions

You can combine several variables in the matrix. Each combination generates a job:

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        os: [ubuntu-latest, macos-latest]

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - run: pytest tests/ -v --tb=short

This generates 6 jobs (3 versions × 2 OS).

When to add OS to the matrix?

For most AI projects (APIs, pipelines), ubuntu-latest is enough.

Cost of the matrix

Each combination uses CI minutes. The jobs run in parallel, so the real time is that of the slowest job. But the billed minutes add up. For private repos (2,000 min/month free), a matrix of 9 jobs on every push can consume your quota fast.

Recommendation: Start with a matrix of Python versions only. Add OS only if your project needs it.

Complete recommended workflow

# .github/workflows/test.yml
name: Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      # false ensures ALL versions report results, not just the first failure
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        run: pytest tests/ -v --tb=short
        env:
          # Needed because the runner doesn't auto-configure import paths like your IDE does
          PYTHONPATH: ${{ github.workspace }}

Expected output

Run pytest tests/ -v --tb=short
============================= test session starts ==============================
platform linux -- Python 3.10.16, pytest-8.3.4, pluggy-1.5.0
collected 8 items

tests/test_parser.py::test_parse_gpt4_response PASSED                  [ 12%]
tests/test_parser.py::test_parse_gpt35_response PASSED                 [ 25%]
tests/test_parser.py::test_parse_unknown_model PASSED                   [ 37%]
tests/test_utils.py::test_format_prompt PASSED                          [ 50%]
tests/test_utils.py::test_validate_input PASSED                         [ 62%]
tests/test_utils.py::test_count_tokens PASSED                           [ 75%]
tests/test_utils.py::test_sanitize_output PASSED                        [ 87%]
tests/test_utils.py::test_truncate_context PASSED                       [100%]

============================== 8 passed in 0.22s ===============================

Same tests, same results, different Python version. That's the confidence matrix testing gives you.

Troubleshooting

"Error: Version 3.1 was not found"

Cause: You forgot the quotes around the Python version.

# WRONG — YAML reads 3.10 as 3.1
python-version: [3.10, 3.11, 3.12]

# RIGHT
python-version: ["3.10", "3.11", "3.12"]

"All the jobs fail with the same error"

Cause: The error isn't about compatibility — it's a bug in your code or configuration that affects all the versions.

Solution: If all the jobs fail with the same error, the problem isn't the Python version. Check: missing dependency in requirements.txt, incorrect path, broken test.

"One single job fails, the others pass"

Cause: A real incompatibility with that specific version:

1. Syntax feature not available (match/case, walrus operator)
2. Type hint syntax incompatible (list[str] vs List[str] for Python < 3.9)
3. Dependency with no support for that version

Solution: Check the log of the job that failed. The error tells you whether it's a SyntaxError, an ImportError, or an AssertionError.

Exercises

Exercise 1: Your first matrix

Create a workflow that runs tests on Python 3.10, 3.11, and 3.12. Use fail-fast: false.

# .github/workflows/test.yml
name: Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        run: pytest tests/ -v --tb=short

Exercise 2: Find the compatibility bug

This code has a problem that only shows up on Python 3.9. Identify the error without running it:

# src/ai_app/config.py
def load_config(overrides: dict | None = None) -> dict:
    """Load configuration with optional overrides."""
    defaults = {
        "model": "gpt-4o-mini",
        "temperature": 0.7,
        "max_tokens": 1000,
    }
    if overrides:
        defaults |= overrides
    return defaults

Which Python version(s) does this code support? How would you make it compatible with 3.9?

from typing import Optional

def load_config(overrides: Optional[dict] = None) -> dict:
    defaults = {
        "model": "gpt-4o-mini",
        "temperature": 0.7,
        "max_tokens": 1000,
    }
    if overrides:
        defaults.update(overrides)
    return defaults

test (3.9)  ❌ Failed — TypeError: unsupported operand type for |
test (3.10) ✅ Passed
test (3.11) ✅ Passed
test (3.12) ✅ Passed

Exercise 3: Matrix with include for an experimental version

Configure a matrix that tests Python 3.10, 3.11, and 3.12 normally, but that also tests Python 3.13 as experimental (it can fail without blocking the merge).

# .github/workflows/test.yml
name: Tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        experimental: [false]
        include:
          - python-version: "3.13"
            experimental: true
    continue-on-error: ${{ matrix.experimental }}

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run tests
        run: pytest tests/ -v --tb=short

Tests
├── test (3.10) ✅ Passed
├── test (3.11) ✅ Passed
├── test (3.12) ✅ Passed
└── test (3.13) ❌ Failed (continue-on-error)

Workflow: ✅ (3.13 doesn't block)

Exercise 4: Multi-dimension matrix

Your project needs to run on Ubuntu and macOS, but only with Python 3.11 and 3.12 (not 3.10 on macOS). Create the matrix with exclude.

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        os: [ubuntu-latest, macos-latest]
        exclude:
          - python-version: "3.10"
            os: macos-latest

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - run: pytest tests/ -v --tb=short

Tests
├── test (3.10, ubuntu-latest) ✅
├── test (3.11, ubuntu-latest) ✅
├── test (3.11, macos-latest)  ✅
├── test (3.12, ubuntu-latest) ✅
└── test (3.12, macos-latest)  ✅

Summary

• ✅ Matrix testing runs your tests on multiple Python versions simultaneously
• ✅ strategy.matrix defines the combinations; each combination generates a parallel job
• ✅ Quotes are mandatory on versions: "3.10", not 3.10 (YAML reads 3.10 as 3.1)
• ✅ fail-fast: false shows the full picture; true gives fast feedback
• ✅ include adds extra combinations (like experimental versions)
• ✅ exclude removes combinations you don't need
• ✅ continue-on-error allows experimental jobs to fail without affecting the result
• ✅ Multi-dimension matrix (Python × OS) is possible but increases the jobs multiplicatively
• ✅ Start with Python versions and add OS only if your project needs it

Additional resources

1. Using a matrix for your jobs - GitHub Docs - Complete official reference
2. actions/setup-python - Supported versions - Available Python versions
3. Python Release Schedule - Which versions are active and when they expire
4. What's New in Python 3.10 - Relevant changes (match/case, union types)
5. GitHub Actions billing - How CI minutes are billed