6. Timeouts for LLM Tests

Overview

LLM APIs are external services with unpredictable behavior. Sometimes they respond in 2 seconds, sometimes in 30, and sometimes they simply don't respond. An integration test that calls OpenAI can hang waiting for a response that never arrives. Without a timeout, your CI workflow keeps running until GitHub Actions cancels it automatically — after 6 hours.

Six hours of a runner running while doing nothing. Burning minutes from your plan. Blocking your pipeline. All because a test hung.

Timeouts operate at three levels — job, step, and individual test — and they form a hierarchy of protection. This capsule teaches you to configure all three levels and to choose the right values for each type of test.

Why timeouts are critical for AI

Imagine this test, which runs locally and passes in 3 seconds:

# tests/integration/test_openai.py
import openai

def test_generate_response():
    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user", "content": "Explain CI/CD in one sentence."}
        ],
    )
    assert response.choices[0].message.content
    assert len(response.choices[0].message.content) > 10

One day, the API has high latency. The test waits... and waits. With no timeout configured, GitHub Actions uses its default of 360 minutes (6 hours). The consequences: minutes burned, a blocked pipeline, and if there's a matrix (3 Python versions), that's 18 hours of runner time.

The most common causes of hung tests in AI:

Level 1: Job timeout (timeout-minutes)

timeout-minutes at the job level defines the maximum time a whole job can run. It's the final guardrail.

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v

When the timeout triggers: GitHub Actions sends SIGTERM, then SIGKILL. The job is marked as cancelled. Steps with if: always() do run.

Recommended values:

If you don't define timeout-minutes, GitHub Actions uses 360 minutes (6 hours). Always define an explicit timeout.

Level 2: Step timeout (timeout-minutes on steps)

Each step can have its own timeout, more granular than the job timeout:

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 20

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - name: Install dependencies
        timeout-minutes: 5
        run: pip install -r requirements.txt

      - name: Run unit tests
        timeout-minutes: 5
        run: pytest tests/unit/ -v

      - name: Run integration tests
        timeout-minutes: 10
        run: pytest tests/integration/ -v --timeout=60
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

When a step timeout triggers: the step is cancelled and marked as failed. The following steps are skipped (except if: always()). The job is marked as failed.

Level 3: Test timeout with pytest-timeout

pytest-timeout adds timeouts at the individual test level. If a test exceeds its time, it's marked as failed with a stack trace that shows exactly where it hung.

pip install pytest-timeout

Using it with a flag

pytest tests/ --timeout=30 -v

Output when a test exceeds the timeout:

tests/integration/test_openai.py::test_batch_processing

+++ Timeout +++

tests/integration/test_openai.py::test_batch_processing - Timeout >30.0s

... (stack trace showing where the test was stuck) ...

========================= 1 failed, 6 passed in 37.42s ====

Using it with a decorator

For finer control, use @pytest.mark.timeout on individual tests:

# tests/integration/test_openai.py
import pytest
import openai


@pytest.mark.timeout(10)
def test_quick_completion():
    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Say hello"}],
        max_tokens=10,
    )
    assert response.choices[0].message.content


@pytest.mark.timeout(60)
def test_batch_processing():
    client = openai.OpenAI()
    prompts = ["Summarize CI/CD", "Explain Docker in one sentence", "What is pytest?"]

    results = []
    for prompt in prompts:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            max_tokens=50,
        )
        results.append(response.choices[0].message.content)

    assert len(results) == 3
    assert all(r is not None for r in results)

Configuration in pyproject.toml

[tool.pytest.ini_options]
timeout = 30
timeout_method = "signal"

Combining a global timeout with decorators

# pyproject.toml
[tool.pytest.ini_options]
timeout = 10    # Default: 10 seconds for every test

# tests/unit/ — they use the 10s default (enough)
def test_greet():
    assert greet("World") == "Hello, World!"

# tests/integration/ — override with a decorator
@pytest.mark.timeout(30)
def test_generate_summary():
    ...

@pytest.mark.timeout(60)
def test_batch_processing():
    ...

The timeout hierarchy

┌─────────────────────────────────────────────┐
│  Job timeout (timeout-minutes: 20)          │  ← Final guardrail
│  ┌─────────────────────────────────────┐    │
│  │  Step timeout (timeout-minutes: 10) │    │  ← Per-step protection
│  │  ┌─────────────────────────────┐    │    │
│  │  │  Test timeout (30 seconds)  │    │    │  ← Per-test protection
│  │  └─────────────────────────────┘    │    │
│  └─────────────────────────────────────┘    │
└─────────────────────────────────────────────┘

The most specific timeout always acts first:

1. Test timeout (pytest-timeout): marks the test as FAILED, pytest continues with the next one
2. Step timeout (Actions step): cancels the whole step
3. Job timeout (Actions job): cancels the whole job

Complete example with all three levels

name: CI with Timeouts

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    # LEVEL 1: Job — final guardrail, prevents 6-hour default from burning your free tier
    timeout-minutes: 25

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - name: Install dependencies
        # LEVEL 2: Step — catches pip hanging on slow mirrors or network issues
        timeout-minutes: 5
        run: pip install -r requirements.txt

      - name: Run unit tests
        timeout-minutes: 5
        # LEVEL 3: Test — 10s is 10x margin for <1s tests; catches infinite loops early
        run: pytest tests/unit/ -v --timeout=10

      - name: Run integration tests
        timeout-minutes: 15
        # 60s per test: LLM APIs can spike to 30s under load, so 2x margin is minimum
        run: pytest tests/integration/ -v --timeout=60
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Trigger scenarios:

• One test hangs: pytest-timeout acts (60s) → test FAILED → pytest continues normally
• Several tests hang: each one fails at 60s → if they add up to more than 15 min → the step timeout cancels
• pip install hangs: the step timeout (5 min) cancels → the job is marked as failed

Differentiated timeouts by test type

conftest.py with automatic timeouts

# tests/conftest.py
import pytest


def pytest_collection_modifyitems(config, items):
    """Apply automatic timeouts based on the test's directory."""
    for item in items:
        test_path = str(item.fspath)

        if any(m.name == "timeout" for m in item.iter_markers()):
            continue

        if "/integration/" in test_path:
            item.add_marker(pytest.mark.timeout(60))
        elif "/unit/" in test_path:
            item.add_marker(pytest.mark.timeout(10))

Workflow with differentiated timeouts

name: CI with Differentiated Timeouts

on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run unit tests
        timeout-minutes: 5
        run: pytest tests/unit/ -v --timeout=10

  integration-tests:
    runs-on: ubuntu-latest
    # Higher limit than unit-tests because LLM API latency is unpredictable
    timeout-minutes: 25
    # Only run on PRs — no reason to spend API credits on every push to main
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run integration tests
        timeout-minutes: 20
        run: pytest tests/integration/ -v --timeout=60
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Best practices: Timeout values

Reference table

How to determine the values

Step 1: Measure the local baseline

pytest tests/ -v --durations=0

============================= slowest durations ================================
3.21s call     tests/integration/test_openai.py::test_batch_processing
1.45s call     tests/integration/test_openai.py::test_generate_summary
0.05s call     tests/unit/test_main.py::test_greet

Step 2: Multiply by 5-10x for CI. CI runners are slower, and APIs have variable latency. If your slowest test takes 3.21s locally, set a timeout of 15-30 seconds.

Step 3: Adjust with experience. Monitor the real times in CI for a week with --durations=0.

The principle: Tight but not hostile

• Too tight: A 5s timeout for a test that takes 3s → intermittent failures
• Too loose: A 300s timeout for a test that takes 3s → a hang isn't detected for 5 min
• Correct: A 30s timeout for a test that takes 3s → 10x margin, detects hangs quickly

The practical rule: timeout = expected_time × 5 to 10.

Comparisons

pytest-timeout vs a manual timeout in code

Job timeout vs step timeout

Troubleshooting

1. The test timeout triggers intermittently

Cause: Variability in the API's latency.

Solution:

@pytest.mark.timeout(60)  # Increase the margin
def test_generate_summary():
    ...

Or add retry logic:

@pytest.mark.timeout(45)
def test_with_retry():
    for attempt in range(3):
        try:
            result = call_openai()
            assert result
            return
        except Exception:
            if attempt < 2:
                import time
                time.sleep(2)
            else:
                raise

2. The step timeout cancels pytest before it generates the report

Solution: Use test-level timeouts so pytest handles the hangs internally:

- name: Prepare
  run: mkdir -p test-results

- name: Run tests
  timeout-minutes: 15
  run: |
    pytest tests/ -v \
      --timeout=60 \
      --junitxml=test-results/report.xml \
      || true

- name: Upload results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: test-results
    path: test-results/
    if-no-files-found: warn

3. The job was cancelled but I don't know which test caused it

Solution: Add pytest-timeout and --durations=0 for diagnosis:

- name: Run tests
  run: pytest tests/ -v --durations=0 --timeout=30 --junitxml=results.xml

- name: Upload results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: test-results
    path: results.xml

With --timeout=30, pytest detects and reports the hung test before the job timeout kicks in.

Exercises

Exercise 1: Configure the timeout hierarchy

Your project has this structure:

tests/
├── unit/          (20 tests, all < 1 second)
├── integration/   (8 tests, they call APIs, 2-5 seconds each)
└── e2e/           (3 tests, full pipeline, 10-30 seconds each)

Write a workflow with the right timeouts for each type.

name: CI with Full Timeout Hierarchy
on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 8
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run unit tests
        timeout-minutes: 3
        run: pytest tests/unit/ -v --timeout=5

  integration-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run integration tests
        timeout-minutes: 10
        run: pytest tests/integration/ -v --timeout=30
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

  e2e-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt
      - name: Run E2E tests
        timeout-minutes: 20
        run: pytest tests/e2e/ -v --timeout=120
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Exercise 2: pytest-timeout in pyproject.toml

Configure pyproject.toml so every test has a default timeout of 15 seconds with the signal method. Then write a test that uses @pytest.mark.timeout(45) to override the default.

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --tb=short"
timeout = 15
timeout_method = "signal"
markers = [
    "slow: marks tests as slow",
    "integration: marks tests that need API keys",
    "llm: marks tests that call LLM APIs directly",
]

# tests/integration/test_llm_batch.py
import pytest
import openai


@pytest.mark.timeout(45)
@pytest.mark.integration
@pytest.mark.llm
def test_batch_summarization():
    """Expected total: 15-25s, a 45s timeout gives enough margin."""
    client = openai.OpenAI()
    texts = [
        "CI/CD is a software development practice...",
        "Docker containers package applications...",
        "Pytest is a testing framework for Python...",
        "GitHub Actions automates workflows...",
        "Coverage measures how much code tests execute...",
    ]

    summaries = []
    for text in texts:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"Summarize in 10 words: {text}"}],
            max_tokens=30,
        )
        summaries.append(response.choices[0].message.content)

    assert len(summaries) == 5
    assert all(s is not None for s in summaries)

Exercise 3: Diagnose a timeout

This workflow fails with "job cancelled due to timeout". The log doesn't say which test hung. How do you fix it?

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - run: pip install -r requirements.txt pytest-timeout

      - name: Run tests with per-test timeouts
        timeout-minutes: 10
        run: |
          pytest tests/ \
            -v \
            --timeout=30 \
            --durations=0 \
            --junitxml=test-results/report.xml

      - name: Upload results for diagnosis
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: test-results/
          if-no-files-found: warn

Exercise 4: conftest.py with automatic timeouts

Write a conftest.py that applies timeouts automatically: 5 seconds for unit/, 30 seconds for integration/, and 90 seconds for e2e/.

# tests/conftest.py
import pytest

TIMEOUT_MAP = {
    "/unit/": 5,
    "/integration/": 30,
    "/e2e/": 90,
}


def pytest_collection_modifyitems(config, items):
    """Assign automatic timeouts based on the test's location."""
    for item in items:
        has_explicit_timeout = any(
            m.name == "timeout" for m in item.iter_markers()
        )
        if has_explicit_timeout:
            continue

        test_path = str(item.fspath)
        for path_pattern, timeout_value in TIMEOUT_MAP.items():
            if path_pattern in test_path:
                item.add_marker(pytest.mark.timeout(timeout_value))
                break

Summary

• ✅ LLM APIs are unpredictable — they can hang from latency, rate limits, or outages
• ✅ Without a timeout, a hung test can run for 6 hours (GitHub Actions' default)
• ✅ Three levels of timeout: Job (final guardrail) > Step (per step) > Test (per individual test)
• ✅ timeout-minutes on the job is mandatory — always define it
• ✅ pytest-timeout adds per-test timeouts with --timeout=N or @pytest.mark.timeout(N)
• ✅ The test timeout acts first: pytest reports the failure and continues with the next test
• ✅ timeout_method = "signal" gives a better stack trace on Linux/macOS (GitHub's runners)
• ✅ The timeout rule: expected_time × 5-10 = a reasonable timeout
• ✅ conftest.py can assign automatic timeouts per directory

Additional resources

1. pytest-timeout documentation — Official documentation of the timeout plugin
2. GitHub Actions timeout-minutes — Reference for job-level timeouts
3. GitHub Actions step timeout — Reference for step-level timeouts
4. pytest --durations — How to profile test duration
5. OpenAI API rate limits — Understanding the rate limits that cause timeouts