7. Strategies for Large Test Suites

Overview

When your AI project has 10 tests, everything is simple: you run pytest tests/ and in 5 seconds you have results. But projects grow. You get to 50 tests, then 200. Some are unit tests that take milliseconds, others are integration tests that call APIs and take seconds. Some need API keys, others don't.

Without a strategy, your CI pipeline turns into a slow monolith: 200 tests in a single job, all sequential, all on every push. One integration test fails from rate limiting and cancels everything — including the 180 unit tests that were passing.

This capsule gives you the tools to avoid that scenario: separating tests by type, running subsets depending on the trigger, parallelizing execution, and configuring failure strategies that balance speed with information.

fail-fast: true vs false

fail-fast is a strategy option in GitHub Actions that controls what happens when a job in a matrix fails:

• fail-fast: true (default) — If one job fails, it cancels all the other jobs in the matrix
• fail-fast: false — All the jobs run to the end

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v

With fail-fast: true: If 3.11 fails, 3.12 gets cancelled. You don't know whether it would have failed too.

With fail-fast: false: Every version runs. If only 3.11 fails, you know it's version-specific.

Recommendation for AI projects: fail-fast: false. The complete information is worth more than the minutes saved.

continue-on-error on steps

continue-on-error: true lets the job continue even if a step fails. The step is marked with a warning in the UI.

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt

      - name: Run unit tests (must pass)
        run: pytest tests/unit/ -v

      - name: Run LLM integration tests (informational)
        continue-on-error: true
        id: llm_tests
        run: pytest tests/integration/ -v --timeout=60
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Report LLM test status
        if: always()
        run: |
          if [ "${{ steps.llm_tests.outcome }}" == "failure" ]; then
            echo "LLM integration tests failed — review recommended" >> $GITHUB_STEP_SUMMARY
          fi

fail-fast controls matrix jobs; continue-on-error controls individual steps. They're complementary.

Separating unit tests from integration tests

The fundamental reason: unit tests and integration tests have completely different requirements.

The split into jobs

name: CI Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run unit tests
        # Coverage gate here ensures no feature ships without tests — integration tests don't need it
        run: pytest tests/unit/ -v --timeout=10 --cov=src --cov-fail-under=80

  integration-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    # Don't waste API credits if unit tests already failed — fast feedback, lower cost
    needs: unit-tests
    # Only on PRs: push to main already ran unit tests, integration validates before merge
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run integration tests
        run: pytest tests/integration/ -v --timeout=60
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

needs: unit-tests creates a dependency: the integration tests only run if the unit tests pass. There's no point spending API calls if the unit tests already failed.

Pytest markers: Organizing by test type

Defining markers

# pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
markers = [
    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
    "integration: marks tests that need external APIs",
    "llm: marks tests that call LLM APIs (subset of integration)",
    "unit: marks pure unit tests",
]

Tagging tests

# tests/unit/test_prompts.py
import pytest

@pytest.mark.unit
def test_format_system_prompt():
    result = format_system_prompt("You are a helpful assistant")
    assert result == {"role": "system", "content": "You are a helpful assistant"}


# tests/integration/test_openai_service.py
import pytest

@pytest.mark.integration
@pytest.mark.llm
@pytest.mark.timeout(30)
def test_generate_summary_real():
    result = generate_summary("CI/CD automates building, testing, and deploying software.")
    assert result is not None
    assert len(result) > 10


@pytest.mark.slow
@pytest.mark.integration
@pytest.mark.llm
@pytest.mark.timeout(120)
def test_batch_processing_10_items():
    items = [f"Summarize topic #{i}" for i in range(10)]
    results = process_batch(items, model="gpt-4o-mini")
    assert len(results) == 10

Running subsets with markers

pytest -m "unit"                              # Unit tests only
pytest -m "integration"                       # Integration tests only
pytest -m "not slow"                          # Everything EXCEPT the slow ones
pytest -m "not llm"                           # Everything EXCEPT the ones that call LLMs
pytest -m "unit or (integration and not llm)" # Unit + integration without LLM

Running subsets in CI depending on the trigger

The implementation uses if: conditions with github.event_name to control which jobs run on each trigger. You can combine it with workflow_dispatch inputs to allow manual selection of the scope:

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch:
    inputs:
      test_scope:
        type: choice
        options: [unit, integration, all]

Each job uses conditions like:

integration-tests:
  needs: unit-tests
  if: |
    github.event_name == 'pull_request' ||
    github.event_name == 'schedule'

full-suite:
  needs: [unit-tests, integration-tests]
  if: github.event_name == 'schedule'

The flow: push → [unit] | PR → [unit] → [integration] | cron → [unit] → [integration] → [full-suite]

Parallelizing tests with pytest-xdist

pytest-xdist runs tests in parallel using multiple workers. On a runner with 2 CPUs, you can run tests ~2x faster.

pip install pytest-xdist

# Auto-detect the number of CPUs
pytest tests/ -n auto -v

# A specific number of workers
pytest tests/ -n 4 -v

When NOT to use pytest-xdist

In GitHub Actions

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run unit tests in parallel
        # -n auto detects CPU count — unit tests are independent so parallelism is safe
        run: pytest tests/unit/ -n auto -v --timeout=10

  integration-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    needs: unit-tests
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run integration tests sequentially
        # No -n here: parallel API calls would hit rate limits and cause flaky failures
        run: pytest tests/integration/ -v --timeout=60
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Unit tests get parallelized (they're independent). Integration tests run sequentially (to avoid rate limits).

Organizing test directories

Recommended structure

project/
├── src/
│   ├── main.py
│   ├── api.py
│   ├── ai_service.py
│   └── prompts.py
├── tests/
│   ├── conftest.py              # Shared fixtures
│   ├── unit/
│   │   ├── conftest.py          # Unit test fixtures (mocks)
│   │   ├── test_main.py
│   │   └── test_prompts.py
│   ├── integration/
│   │   ├── conftest.py          # Integration fixtures (API clients)
│   │   └── test_openai_service.py
│   └── e2e/
│       └── test_full_pipeline.py
├── pyproject.toml
└── requirements.txt

The advantages: running subsets is trivial (pytest tests/unit/), fixtures are scoped by type, and CI can run different directories in different jobs.

# tests/conftest.py — Shared by every test
import pytest

@pytest.fixture
def sample_text():
    return "Artificial intelligence is transforming how we build software."

# tests/unit/conftest.py — Unit tests only
import pytest
from unittest.mock import MagicMock

@pytest.fixture
def mock_openai_client():
    client = MagicMock()
    client.chat.completions.create.return_value = MagicMock(
        choices=[MagicMock(message=MagicMock(content="Mocked response"))]
    )
    return client

# tests/integration/conftest.py — Integration tests only
import pytest
import os

@pytest.fixture(scope="session")
def openai_client():
    import openai
    api_key = os.environ.get("OPENAI_API_KEY")
    if not api_key:
        pytest.skip("OPENAI_API_KEY not set")
    return openai.OpenAI(api_key=api_key)

@pytest.fixture(autouse=True)
def slow_down_between_tests():
    """Pause between tests to respect rate limits."""
    yield
    import time
    time.sleep(0.5)

AI-specific: Separating tests that need API keys

In CI, secrets are not available in every context: PRs from forks don't have access to the repo's secrets. If a test needs an API key and doesn't have it, it fails and blocks other tests.

The solution: Automatic skip

# tests/integration/conftest.py
import pytest
import os

skip_without_api_key = pytest.mark.skipif(
    not os.environ.get("OPENAI_API_KEY"),
    reason="OPENAI_API_KEY not set"
)

# tests/integration/test_openai_service.py
from tests.integration.conftest import skip_without_api_key

@skip_without_api_key
@pytest.mark.integration
@pytest.mark.llm
@pytest.mark.timeout(30)
def test_generate_summary_real():
    result = generate_summary("CI/CD automates the software delivery process.")
    assert result is not None

@pytest.mark.integration
def test_format_prompt_no_api_needed():
    """This integration test does NOT need an API key."""
    result = format_system_prompt("You are a CI/CD expert.")
    assert result["role"] == "system"

Tests that need an API key get skipped gracefully — they don't fail, they don't block other tests.

In the workflow, separate the jobs with and without API keys. Use the condition github.event.pull_request.head.repo.full_name == github.repository to verify that the PR doesn't come from a fork (which wouldn't have access to secrets).

Comparisons

Execution strategies

pytest-xdist vs parallel jobs in Actions

Troubleshooting

1. Tests pass locally but fail in CI with "OPENAI_API_KEY not set"

Cause: The secret isn't configured, or the job has no access (PR from a fork).

Solution:

@pytest.mark.skipif(
    not os.environ.get("OPENAI_API_KEY"),
    reason="OPENAI_API_KEY not set"
)
def test_generate_summary():
    ...

2. pytest-xdist causes intermittent failures

Cause: The tests share state (files, global variables).

Solution:

# Use tmp_path for temporary files (isolated per test)
def test_write_output(tmp_path):
    output_file = tmp_path / "result.json"
    output_file.write_text('{"status": "ok"}')
    assert output_file.exists()

3. Integration tests exceed the rate limit when they run together

Solution:

# tests/integration/conftest.py
import time, pytest

@pytest.fixture(autouse=True)
def rate_limit_pause():
    yield
    time.sleep(1)

And don't use pytest-xdist for integration tests:

- run: pytest tests/integration/ -v --timeout=60   # No -n auto

Exercises

Exercise 1: Design the testing strategy

Your AI project has:

• 40 unit tests (all < 1s, they don't need API keys)
• 15 integration tests (5 with an LLM, 10 without, between 2-10s each)
• 5 e2e tests (full pipeline, 30-60s each)

Your GitHub Actions plan has 2000 minutes/month. Design a strategy that maximizes coverage while minimizing consumption.

name: Smart CI Strategy
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: "0 3 * * 1-5"

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 8
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: pytest tests/unit/ -n auto -v --timeout=5 --cov=src --cov-fail-under=80

  integration-no-llm:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    needs: unit-tests
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: pytest tests/integration/ -v -m "not llm" --timeout=30

  integration-llm:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    needs: unit-tests
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: pytest tests/integration/ -v -m "llm" --timeout=60
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

  e2e-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    needs: [unit-tests, integration-no-llm]
    if: github.event_name == 'schedule'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: pytest tests/e2e/ -v --timeout=120
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Exercise 2: Implement markers and conftest.py

Given this test file without markers, reorganize it with the right markers and write the conftest.py:

def test_validate_input():
    assert validate_input("Hello") == True

def test_generate_with_openai():
    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Say hi"}],
    )
    assert response.choices[0].message.content

def test_batch_10_items():
    results = process_batch(["item"] * 10)
    assert len(results) == 10

# tests/unit/test_validation.py — Marked as unit
@pytest.mark.unit
def test_validate_input():
    assert validate_input("Hello") == True

# tests/integration/test_openai_real.py — Marked with skip + markers
skip_without_api = pytest.mark.skipif(not os.environ.get("OPENAI_API_KEY"), reason="OPENAI_API_KEY not set")

@skip_without_api
@pytest.mark.integration
@pytest.mark.llm
@pytest.mark.timeout(15)
def test_generate_with_openai():
    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Say hi"}],
    )
    assert response.choices[0].message.content

@skip_without_api
@pytest.mark.slow
@pytest.mark.llm
@pytest.mark.timeout(120)
def test_batch_10_items():
    results = process_batch(["item"] * 10)
    assert len(results) == 10

# tests/conftest.py — Automatic timeouts per directory
TIMEOUT_MAP = {"/unit/": 5, "/integration/": 30, "/e2e/": 90}

def pytest_collection_modifyitems(config, items):
    for item in items:
        if any(m.name == "timeout" for m in item.iter_markers()):
            continue
        test_path = str(item.fspath)
        for pattern, timeout_val in TIMEOUT_MAP.items():
            if pattern in test_path:
                item.add_marker(pytest.mark.timeout(timeout_val))
                break

Exercise 3: fail-fast decision matrix

For each scenario, decide fail-fast: true or false:

1. A matrix of Python 3.10, 3.11, 3.12 on a PR
2. A matrix of 3 OSes (ubuntu, macos, windows) with tests that take 20 min each
3. A developer is debugging a test that only fails on 3.10

Exercise 4: Selective parallelization

You have 150 unit tests (independent, < 0.1s), 30 integration tests without an LLM (independent, 1-3s), and 20 integration tests with an LLM (they share a rate limit, 3-10s). Write the pytest command for each group.

# Unit tests: parallelize as much as possible
pytest tests/unit/ -n auto -v --timeout=5

# Integration without LLM: parallelize
pytest tests/integration/ -m "not llm" -n auto -v --timeout=15

# Integration with LLM: sequential (rate limits)
pytest tests/integration/ -m "llm" -v --timeout=60

jobs:
  unit:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: pytest tests/unit/ -n auto -v --timeout=5

  integration-fast:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    needs: unit
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: pytest tests/integration/ -m "not llm" -n auto -v --timeout=15

  integration-llm:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    needs: unit
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: pytest tests/integration/ -m "llm" -v --timeout=60
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Summary

• ✅ fail-fast: false gives you the full picture — use it on PRs and nightly runs
• ✅ continue-on-error: true on steps that shouldn't block (like informational LLM tests)
• ✅ Separate unit from integration into different jobs — different requirements and timeouts
• ✅ Pytest markers (@pytest.mark.slow, @pytest.mark.llm) let you run subsets with -m
• ✅ pytest -m "not slow" on every push, the full suite on a schedule
• ✅ pytest-xdist (-n auto) parallelizes unit tests; do NOT parallelize tests with rate limits
• ✅ Directory structure: tests/unit/, tests/integration/, tests/e2e/
• ✅ Tests without an API key must skip, not fail — use pytest.mark.skipif
• ✅ needs: unit-tests creates dependencies between jobs — don't spend API calls if the unit tests fail

Additional resources

1. pytest markers documentation — Complete reference for markers and -m expressions
2. pytest-xdist documentation — Parallelizing tests with multiple workers
3. GitHub Actions strategy — fail-fast and matrix strategies
4. GitHub Actions job dependencies — needs and dependencies between jobs
5. Scheduled workflows — Cron syntax for nightly runs