7. Advanced Matrix Strategies

Overview

In Module 2 you learned basic matrix testing: running tests on multiple Python versions. But GitHub Actions' matrix strategies go far beyond that. You can use include to add specific combinations with extra variables, exclude to skip combinations that don't make sense, and fail-fast to control whether one failure cancels the rest of the matrix.

The most powerful use case for AI pipelines: testing your code on Python 3.10, 3.11, and 3.12, but building the Docker image only for 3.12. Or running prompt regression on gpt-4o-mini by default, but adding an extra combination with gpt-4o for smoke testing. These are combinations you can't express with a simple matrix — you need include and exclude.

Connection with the final pipeline: In the capstone pipeline (Module 8), matrix strategies let you test against multiple Python versions and multiple AI models in a single pipeline run.

Recap: The basic matrix

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: pytest tests/ -v

This generates 3 parallel jobs: one per Python version. Simple, direct, useful. But the basic matrix generates the Cartesian product of every value. When you need more control, include, exclude, and fail-fast come in.

include: Adding specific combinations

Adding extra variables to existing combinations

strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    include:
      - python-version: "3.12"
        build-docker: true
      - python-version: "3.10"
        build-docker: false
      - python-version: "3.11"
        build-docker: false

Now every combination has a build-docker variable you can use in the steps:

- name: Build Docker image
  if: matrix.build-docker
  run: docker build -t my-app .

Adding entirely new combinations

include can also add combinations that didn't exist in the original matrix:

strategy:
  matrix:
    python-version: ["3.11", "3.12"]
    os: [ubuntu-latest]
    include:
      - python-version: "3.12"
        os: macos-latest
        experimental: true

This generates:

1. Python 3.11 + Ubuntu (from the matrix)
2. Python 3.12 + Ubuntu (from the matrix)
3. Python 3.12 + macOS (from the include — a new combination)

Combination 3 has the extra variable experimental: true that the others don't.

An AI use case: Testing with multiple models

strategy:
  matrix:
    python-version: ["3.12"]
    model: ["gpt-4o-mini"]
    include:
      - python-version: "3.12"
        model: "gpt-4o"
        cost-threshold: "1.00"
        smoke-test-only: true
      - python-version: "3.12"
        model: "gpt-4o-mini"
        cost-threshold: "0.10"
        smoke-test-only: false

- name: Run prompt regression
  run: |
    python scripts/prompt_regression.py \
      --model ${{ matrix.model }} \
      --cost-threshold ${{ matrix.cost-threshold || '0.10' }}

- name: Run full test suite
  if: ${{ !matrix.smoke-test-only }}
  run: pytest tests/ -v

This tests with gpt-4o-mini (the full suite) and gpt-4o (only a smoke test, to verify compatibility without spending on the full suite).

exclude: Skipping combinations

Removing combinations that don't make sense

strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    os: [ubuntu-latest, macos-latest, windows-latest]
    exclude:
      - python-version: "3.10"
        os: windows-latest
      - python-version: "3.10"
        os: macos-latest

The complete matrix would be 3×3 = 9 combinations. With exclude, we remove 2, leaving 7.

Why exclude?

BEFORE (no exclude):
  3.10 + Ubuntu     ← useful (legacy support)
  3.10 + macOS      ← unnecessary (nobody uses 3.10 on macOS)
  3.10 + Windows    ← unnecessary (nobody uses 3.10 on Windows)
  3.11 + Ubuntu     ← useful
  3.11 + macOS      ← useful
  3.11 + Windows    ← useful
  3.12 + Ubuntu     ← useful (production)
  3.12 + macOS      ← useful (development)
  3.12 + Windows    ← useful

  9 jobs × 3 minutes each = 27 minutes of compute
  Cost: ~$0.027

AFTER (with exclude):
  3.10 + Ubuntu     ← keep (legacy)
  3.11 + everything ← keep
  3.12 + everything ← keep

  7 jobs × 3 minutes = 21 minutes
  Savings: ~22%

For a pipeline that runs 20 times a day, that 22% adds up.

fail-fast: Failure control

The default behavior: fail-fast: true

strategy:
  fail-fast: true  # The default
  matrix:
    python-version: ["3.10", "3.11", "3.12"]

With fail-fast: true, if any combination fails, the others get cancelled immediately. It's the default because usually, if it fails on one version, you want to know quickly.

When to disable it: fail-fast: false

strategy:
  fail-fast: false
  matrix:
    python-version: ["3.10", "3.11", "3.12"]

With fail-fast: false, every combination runs to the end, even if one fails. Useful when you want to know which versions it fails on and which it doesn't.

The comparison

The recommendation for AI pipelines

strategy:
  fail-fast: false  # Recommended for AI pipelines
  matrix:
    python-version: ["3.10", "3.11", "3.12"]

In AI pipelines, a failure can be from rate limiting (transient) or from a version incompatibility (permanent). With fail-fast: false you can distinguish: if it only failed on one version, it's probably an incompatibility; if it failed on all of them, it's probably rate limiting or a real bug.

A complete example: Test all versions, build Docker only for 3.12

This is the most useful pattern for AI pipelines. You want to guarantee compatibility with multiple Python versions, but you only need one Docker image for production:

name: CI Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    name: "Test Python ${{ matrix.python-version }}"
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        include:
          - python-version: "3.12"
            build-docker: true
            run-ai-checks: true
          - python-version: "3.11"
            build-docker: false
            run-ai-checks: false
          - python-version: "3.10"
            build-docker: false
            run-ai-checks: false

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run linter
        run: ruff check src/

      - name: Run tests
        run: pytest tests/ -v --tb=short

      - name: Run AI checks
        if: matrix.run-ai-checks
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python scripts/prompt_regression.py --mode check
          python scripts/cost_estimation.py --threshold 0.10

      - name: Build Docker image
        if: matrix.build-docker
        run: |
          docker build -t ghcr.io/${{ github.repository }}:test .
          echo "Docker build successful"

      - name: Summary
        run: |
          echo "## Python ${{ matrix.python-version }}" >> $GITHUB_STEP_SUMMARY
          echo "- Tests: ✅" >> $GITHUB_STEP_SUMMARY
          echo "- AI Checks: ${{ matrix.run-ai-checks && '✅' || '⏭️ Skipped' }}" >> $GITHUB_STEP_SUMMARY
          echo "- Docker Build: ${{ matrix.build-docker && '✅' || '⏭️ Skipped' }}" >> $GITHUB_STEP_SUMMARY

Why Docker only for 3.12?

Your Dockerfile uses python:3.12-slim as the base image.
The Docker image ALWAYS uses Python 3.12.
There's no point building Docker images for 3.10 and 3.11
  — they would never get deployed.

But it does make sense to TEST on 3.10 and 3.11:
  - It verifies that your code doesn't use 3.12-exclusive features
  - It guarantees that libraries work on earlier versions
  - It's useful for open source, where users may have 3.10

A dynamic matrix

For advanced cases, you can generate the matrix dynamically:

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - id: set-matrix
        run: |
          if [ "${{ github.event_name }}" = "schedule" ]; then
            echo 'matrix={"python-version":["3.10","3.11","3.12"],"model":["gpt-4o-mini","gpt-4o"]}' >> $GITHUB_OUTPUT
          else
            echo 'matrix={"python-version":["3.12"],"model":["gpt-4o-mini"]}' >> $GITHUB_OUTPUT
          fi

  test:
    needs: setup
    strategy:
      matrix: ${{ fromJson(needs.setup.outputs.matrix) }}
    runs-on: ubuntu-latest
    steps:
      - run: echo "Testing Python ${{ matrix.python-version }} with ${{ matrix.model }}"

In this example, normal pushes only test Python 3.12 with gpt-4o-mini (fast and cheap). The nightly scheduled runs test every combination (complete but more expensive).

Comparisons

A simple matrix vs one with include/exclude

A static vs a dynamic matrix

Troubleshooting

"The matrix generates combinations I don't want"

Cause: The matrix generates the Cartesian product of every value, and you need to exclude some combinations.

Solution: Use exclude to remove specific combinations:

strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    os: [ubuntu-latest, macos-latest]
    exclude:
      - python-version: "3.10"
        os: macos-latest

"The include variable doesn't exist in every combination"

Cause: include only adds variables to specific combinations. The other combinations don't have that variable.

Solution: Use the || operator for a default value:

- if: ${{ matrix.build-docker || false }}

Or add the variable to every combination via include.

"fail-fast cancels my jobs before I can see the results"

Cause: fail-fast: true (the default) cancels every combination when one fails.

Solution: Disable it:

strategy:
  fail-fast: false

"The dynamic matrix fails with 'unexpected type'"

Cause: The JSON you generate for the matrix has the wrong format.

Solution: Verify the JSON with jq:

- id: set-matrix
  run: |
    MATRIX='{"python-version":["3.12"]}'
    echo "$MATRIX" | jq .  # It validates the JSON
    echo "matrix=$MATRIX" >> $GITHUB_OUTPUT

Exercises

Exercise 1: A matrix with include for a conditional Docker build

Create a matrix that tests on Python 3.10, 3.11, and 3.12, but only builds Docker on 3.12.

strategy:
  fail-fast: false
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    include:
      - python-version: "3.12"
        build-docker: true
      - python-version: "3.11"
        build-docker: false
      - python-version: "3.10"
        build-docker: false

steps:
  - uses: actions/setup-python@v5
    with:
      python-version: ${{ matrix.python-version }}

  - run: pytest tests/ -v

  - name: Build Docker
    if: matrix.build-docker
    run: docker build -t test-image .

Exercise 2: Exclude to optimize the matrix

You have a matrix of 3 Python versions × 3 OSes. You want to exclude Python 3.10 on Windows and macOS (Ubuntu only).

strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    os: [ubuntu-latest, macos-latest, windows-latest]
    exclude:
      - python-version: "3.10"
        os: macos-latest
      - python-version: "3.10"
        os: windows-latest

Exercise 3: A dynamic matrix per trigger

Create a matrix that's small for pushes (only Python 3.12) and complete for scheduled runs (3.10, 3.11, 3.12).

jobs:
  determine-matrix:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set.outputs.matrix }}
    steps:
      - id: set
        run: |
          if [ "${{ github.event_name }}" = "schedule" ]; then
            echo 'matrix={"python-version":["3.10","3.11","3.12"]}' >> $GITHUB_OUTPUT
          else
            echo 'matrix={"python-version":["3.12"]}' >> $GITHUB_OUTPUT
          fi

  test:
    needs: determine-matrix
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix: ${{ fromJson(needs.determine-matrix.outputs.matrix) }}
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - run: echo "Testing on Python ${{ matrix.python-version }}"

Exercise 4: Compute the number of combinations

How many combinations does each matrix generate?

1. python-version: [3.10, 3.11, 3.12] × os: [ubuntu, macos]
2. The previous matrix + exclude: [{python: 3.10, os: macos}]
3. python-version: [3.12] + include: [{python: 3.12, os: macos, experimental: true}]

Best practices for matrices in AI pipelines

1. Separate test from build

# ✅ Good practice: test on multiple versions, build only for production
strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
    include:
      - python-version: "3.12"
        build-docker: true
      - python-version: "3.11"
        build-docker: false
      - python-version: "3.10"
        build-docker: false

2. Use fail-fast: false for AI checks

AI checks depend on external APIs. A failure from rate limiting in one combination doesn't mean the others will fail. With fail-fast: false, you get the complete picture.

3. Minimize API calls in the matrix

# ❌ Expensive: AI checks on every Python version
strategy:
  matrix:
    python-version: ["3.10", "3.11", "3.12"]
# → 3 × the AI API calls

# ✅ Efficient: AI checks only on the production version
include:
  - python-version: "3.12"
    run-ai-checks: true
  - python-version: "3.11"
    run-ai-checks: false
  - python-version: "3.10"
    run-ai-checks: false
# → 1 × the AI API calls

4. Name your jobs with the matrix variable

name: "Test Python ${{ matrix.python-version }}"

This makes the Actions UI show "Test Python 3.10", "Test Python 3.11", etc. — much more readable than "Test (1 of 3)".

5. Limit the matrix on PRs, expand it nightly

# PRs: only the production version (fast)
# Nightly: every version (complete)

Use a dynamic matrix with fromJson to change the combinations depending on the trigger.

Summary

• ✅ include adds extra variables to existing combinations or creates new ones
• ✅ exclude removes combinations from the Cartesian product that don't make sense
• ✅ fail-fast: false lets every combination run even if one fails — recommended for AI pipelines
• ✅ The key pattern: Test all Python versions, build Docker only for the production version
• ✅ A dynamic matrix with fromJson lets you change the combinations depending on the trigger
• ✅ The AI use case: Testing with gpt-4o-mini (complete) + gpt-4o (a smoke test) in the same matrix
• ✅ Conditional variables: ${{ matrix.build-docker || false }} for safe defaults
• ✅ Cost: Every combination is a separate job — optimize with exclude to reduce compute

Additional resources

1. GitHub Actions — Matrix Strategy - The complete official documentation
2. GitHub Actions — include/exclude - The include and exclude reference
3. GitHub Actions — fromJson - The function for a dynamic matrix
4. GitHub Actions — fail-fast - Failure control in a matrix
5. GitHub Actions Pricing - The per-minute cost of runners
6. GitHub Actions — Strategy Context - The strategy context's variables