3. pytest Configuration for AI Apps

Description

Configuring pytest correctly for an AI app isn't just installing the library. You need: conftest.py with shared fixtures specific to LLMs, markers to separate fast tests from those that call the API (and cost money), parametrize to test multiple variations of prompts and parsers, and pytest.ini or pyproject.toml configuration adapted to AI projects.

This session is 80% practical. By the end you'll have all the configuration files ready to copy into your project, with an explanation of each design decision. It's not a generic pytest tutorial — it's the specific configuration that works for projects with LLMs.

By the end you'll be able to configure a testing project from scratch, organize your suite with markers that separate tests by speed and cost, create reusable fixtures for LLM mocking, and run specific subsets of tests depending on the context (local development, PR, nightly CI).

Complete directory structure

Before looking at the configuration files, here's the directory structure we're going to build:

my-ai-app/
│
├── src/
│   └── app/
│       ├── __init__.py
│       ├── llm.py              # LLM client wrapper
│       ├── parsers.py          # Output parsers
│       ├── validators.py       # Input/output validators
│       ├── prompts.py          # Prompt templates
│       └── main.py             # Main function
│
├── tests/
│   ├── conftest.py             # Shared fixtures (root-level)
│   ├── unit/
│   │   ├── conftest.py         # Fixtures specific to unit tests
│   │   ├── test_parsers.py
│   │   ├── test_validators.py
│   │   └── test_llm_chain.py
│   ├── integration/
│   │   ├── conftest.py         # Fixtures specific to integration
│   │   └── test_e2e.py
│   └── regression/
│       ├── conftest.py
│       └── test_model_regression.py
│
├── pytest.ini                  # Main pytest configuration
├── .env                        # Environment variables (do not commit)
├── .env.test                   # Environment variables for tests
└── requirements-test.txt       # Testing dependencies

Why this structure:

• tests/ separated from src/ → standard convention, makes coverage configuration easier
• Subdirectories by test type → you can run pytest tests/unit/ for only unit tests
• conftest.py at multiple levels → more specific fixtures override the general ones
• Separate .env.test → different API keys or config for tests

pytest.ini — Base configuration

# pytest.ini
[pytest]
# Where to look for tests
testpaths = tests

# Naming conventions
python_files = test_*.py
python_classes = Test*
python_functions = test_*

# Markers (each one must be documented here)
markers =
    unit: Tests with mocks. No cost, <1 second. Run on every commit.
    integration: Tests with a real LLM. They have a cost. Run on PR.
    regression: Regression tests vs golden set. Expensive. Run weekly.
    smoke: Basic smoke tests. Verify that the system boots.
    slow: Tests that take >5 seconds.
    contract: Tests that verify the structural contracts of prompts.

# Default options: verbose + short traceback
addopts = -v --tb=short

# Async mode (critical for apps that use asyncio)
asyncio_mode = auto

Run subsets:

# Only fast tests (local development)
pytest -m "unit or smoke"

# Only unit tests (even faster)
pytest -m unit

# Exclude expensive ones (CI by default)
pytest -m "not integration and not regression"

# Only the ones that failed (fast debug)
pytest --lf

# Stop at the first failure
pytest -x

# Show print() output in tests (useful for debug)
pytest -s

# Run a specific test
pytest tests/unit/test_parsers.py::TestParseJson::test_valid_json -v

conftest.py — Root level (global fixtures)

The conftest.py at the root of tests/ is available to ALL tests in all subdirectories:

# tests/conftest.py
"""
Shared fixtures for the whole testing suite.
Automatically available in all tests without importing.
"""
import os
import pytest
from unittest.mock import MagicMock


# ─────────────────────────────────────────────────────────────
# Fixtures for environment variables
# ─────────────────────────────────────────────────────────────

@pytest.fixture(autouse=True)
def set_test_environment(monkeypatch):
    """
    Ensures we always run with a test configuration.
    autouse=True: applies to ALL tests without needing to declare it.
    """
    monkeypatch.setenv("ENVIRONMENT", "test")
    monkeypatch.setenv("LOG_LEVEL", "WARNING")  # Silences logs in tests


@pytest.fixture
def api_key():
    """
    Reads the API key from the environment. Skips if it's not defined.
    Use in integration tests that need a real API key.
    """
    key = os.getenv("OPENAI_API_KEY")
    if not key:
        pytest.skip("OPENAI_API_KEY not defined — integration test skipped")
    return key


# ─────────────────────────────────────────────────────────────
# Mock response factory (reusable across the whole suite)
# ─────────────────────────────────────────────────────────────

def create_openai_response(content: str, tokens_used: int = 150) -> MagicMock:
    """
    Factory that creates a mock object with the exact structure
    of a real OpenAI chat completions response.

    Important: it mirrors the REAL structure of the API so that
    the code that accesses resp.choices[0].message.content works
    the same with a mock as with the real API.
    """
    response = MagicMock()

    # Structure of choices (list with at least one element)
    choice = MagicMock()
    choice.message.content = content
    choice.message.role = "assistant"
    choice.finish_reason = "stop"
    response.choices = [choice]

    # Structure of usage (tokens)
    response.usage.prompt_tokens = tokens_used // 3
    response.usage.completion_tokens = tokens_used * 2 // 3
    response.usage.total_tokens = tokens_used

    # Metadata
    response.model = "gpt-4o-mini"
    response.id = "chatcmpl-test-mock-id"

    return response


@pytest.fixture
def mock_openai_response():
    """
    Fixture that returns the response factory.
    Usage: mock_openai_response('{"key": "value"}')

    Returns the factory function, not a fixed response.
    This lets you create multiple responses with different contents.
    """
    return create_openai_response


@pytest.fixture
def mock_openai_client(mock_openai_response):
    """
    Fully mocked OpenAI client.

    Usage in tests:
        def test_something(mock_openai_client):
            mock_openai_client.chat.completions.create.return_value = (
                mock_openai_response('{"result": "ok"}')
            )
            ...
    """
    client = MagicMock()
    # Default response (can be overridden in each test)
    client.chat.completions.create.return_value = mock_openai_response(
        '{"default": "mock response"}'
    )
    return client


# ─────────────────────────────────────────────────────────────
# Test data fixtures
# ─────────────────────────────────────────────────────────────

@pytest.fixture
def sample_texts():
    """Test texts for text-analysis tests."""
    return {
        "positive": "I absolutely love this product! Best purchase ever.",
        "negative": "Terrible experience. Never buying again.",
        "neutral": "The package arrived on Tuesday as expected.",
        "empty": "",
        "long": "a" * 5000,
        "with_json": 'The result is {"key": "value"} embedded.',
        "multilingual": "Esta es una oración en español.",
    }


@pytest.fixture
def sample_json_responses():
    """Typical JSON responses from LLMs for parser tests."""
    return {
        "clean": '{"sentiment": "positive", "confidence": 0.9}',
        "in_markdown": '```json\n{"sentiment": "positive", "confidence": 0.9}\n```',
        "with_prefix": 'Here is the result: {"sentiment": "positive", "confidence": 0.9}',
        "with_suffix": '{"sentiment": "positive", "confidence": 0.9}\nNote: confidence is high.',
        "malformed": '{"sentiment": "positive", "confidence": ',  # Incomplete JSON
        "empty": '',
        "no_json": 'The sentiment is positive with high confidence.',
    }

conftest.py — Unit tests level

More specific fixtures for unit tests (available only inside tests/unit/):

# tests/unit/conftest.py
"""
Fixtures specific to unit tests.
Available only in tests/unit/ and subdirectories.
"""
import pytest
from unittest.mock import patch, MagicMock


@pytest.fixture
def patched_llm_client():
    """
    Patches the LLM client directly in the app module.
    More convenient than @patch in each test when many tests
    need the same patch.
    """
    with patch("app.llm.client") as mock_client:
        yield mock_client


@pytest.fixture
def sentiment_mock_responses():
    """
    Predefined mock responses for sentiment analysis tests.
    Uses a fixture factory for flexibility.
    """
    def _make_response(sentiment: str, confidence: float) -> MagicMock:
        from tests.conftest import create_openai_response
        return create_openai_response(
            f'{{"sentiment": "{sentiment}", "confidence": {confidence}}}'
        )
    return _make_response


@pytest.fixture
def summary_mock_responses():
    """Mock responses for summarization tests."""
    def _make_response(points: list) -> MagicMock:
        import json
        from tests.conftest import create_openai_response
        return create_openai_response(json.dumps({"points": points}))
    return _make_response

conftest.py — Integration tests level

Fixtures for integration tests (require a real API key):

# tests/integration/conftest.py
"""
Fixtures for integration tests.
They require a real API key. They have a cost per run.
"""
import os
import pytest
from openai import OpenAI


@pytest.fixture(scope="module")
def real_openai_client():
    """
    Real OpenAI client for integration tests.
    scope="module": a single client for all the module's tests.
    Avoids the overhead of creating multiple clients.
    """
    api_key = os.getenv("OPENAI_API_KEY")
    if not api_key:
        pytest.skip("OPENAI_API_KEY required for integration tests")
    return OpenAI(api_key=api_key)


@pytest.fixture(scope="module")
def budget_tracker():
    """
    Simple token cost tracker for integration tests.
    Helps monitor that tests don't exceed the budget.
    """
    class BudgetTracker:
        def __init__(self, max_tokens: int = 10_000):
            self.total_tokens = 0
            self.max_tokens = max_tokens
            self.calls = 0

        def track(self, response) -> None:
            """Records the token usage of a response."""
            if hasattr(response, 'usage') and response.usage:
                self.total_tokens += response.usage.total_tokens
                self.calls += 1

        def check_budget(self) -> None:
            """Fails the test if the budget was exceeded."""
            if self.total_tokens > self.max_tokens:
                pytest.fail(
                    f"Budget exceeded: {self.total_tokens} tokens "
                    f"(max: {self.max_tokens}). "
                    f"Calls: {self.calls}"
                )

        def __repr__(self) -> str:
            return f"BudgetTracker(tokens={self.total_tokens}, calls={self.calls})"

    return BudgetTracker()

Markers in action

Markers are the most important tool for organizing your suite:

# tests/unit/test_parsers.py
import pytest
from app.parsers import parse_json_from_llm_output


# Mark individual tests
@pytest.mark.unit
def test_parse_valid_json():
    raw = '{"key": "value"}'
    result = parse_json_from_llm_output(raw)
    assert result == {"key": "value"}


# Mark a whole class (applies to all methods)
@pytest.mark.unit
class TestParseJsonFromLLMOutput:

    def test_pure_json(self):
        result = parse_json_from_llm_output('{"x": 1}')
        assert result == {"x": 1}

    def test_json_in_markdown_block(self):
        raw = '```json\n{"x": 1}\n```'
        result = parse_json_from_llm_output(raw)
        assert result == {"x": 1}

    def test_json_with_surrounding_text(self):
        raw = "Here is the result: {\"x\": 1} end."
        result = parse_json_from_llm_output(raw)
        assert result == {"x": 1}

    def test_no_json_raises_value_error(self):
        with pytest.raises(ValueError, match="No JSON found"):
            parse_json_from_llm_output("No JSON here")

    def test_malformed_json_raises_json_decode_error(self):
        import json
        with pytest.raises(json.JSONDecodeError):
            parse_json_from_llm_output('{"incomplete":')


# Combine multiple markers
@pytest.mark.integration
@pytest.mark.slow
def test_full_pipeline_with_real_llm():
    """This test calls the real API — slow and with a cost."""
    ...


# Conditional marker — skip if configuration is missing
@pytest.mark.skipif(
    not os.getenv("OPENAI_API_KEY"),
    reason="OPENAI_API_KEY not defined"
)
def test_requires_api_key():
    ...

parametrize — Test multiple variations

parametrize is especially useful for testing parsers with multiple input formats:

Basic use case: parser with multiple formats

# tests/unit/test_parsers.py
import pytest
import json
from app.parsers import parse_json_from_llm_output

# Table of cases: (input, expected_output)
JSON_PARSE_CASES = [
    # Description, raw input, expected result
    ("pure JSON", '{"key": "value"}', {"key": "value"}),
    ("JSON in markdown", '```json\n{"key": "value"}\n```', {"key": "value"}),
    ("JSON in markdown no lang", '```\n{"key": "value"}\n```', {"key": "value"}),
    ("JSON with prefix text", 'Result: {"key": "value"}', {"key": "value"}),
    ("JSON with suffix text", '{"key": "value"}\nNotes: etc.', {"key": "value"}),
    ("JSON with leading whitespace", '  \n{"key": "value"}', {"key": "value"}),
    ("nested JSON", '{"outer": {"inner": 1}}', {"outer": {"inner": 1}}),
    ("JSON with list", '{"items": [1, 2, 3]}', {"items": [1, 2, 3]}),
]

@pytest.mark.unit
@pytest.mark.parametrize("description,raw_input,expected", JSON_PARSE_CASES)
def test_parse_json_variations(description, raw_input, expected):
    """
    Tests the parser with multiple LLM output formats.
    Each row of JSON_PARSE_CASES is a separate test.
    The names will be: test_parse_json_variations[pure JSON-...]
    """
    result = parse_json_from_llm_output(raw_input)
    assert result == expected, f"Failed for case: {description}"


# For error cases
ERROR_CASES = [
    ("empty string", "", ValueError),
    ("no JSON", "No JSON here at all", ValueError),
    ("malformed JSON", '{"key":}', json.JSONDecodeError),
    ("None input", None, (TypeError, ValueError)),  # Multiple valid exceptions
]

@pytest.mark.unit
@pytest.mark.parametrize("description,bad_input,expected_exception", ERROR_CASES)
def test_parse_json_errors(description, bad_input, expected_exception):
    with pytest.raises(expected_exception):
        parse_json_from_llm_output(bad_input)

Advanced use case: parametrize with fixtures

# Parametrize that generates dynamic mock responses
@pytest.mark.unit
@pytest.mark.parametrize("sentiment,confidence", [
    ("positive", 0.9),
    ("negative", 0.1),
    ("neutral", 0.5),
    ("positive", 1.0),  # Edge case
    ("negative", 0.0),  # Edge case
])
@patch("app.sentiment.client.chat.completions.create")
def test_analyze_sentiment_valid_outputs(mock_create, sentiment, confidence, mock_openai_response):
    """
    For each sentiment/confidence combination, verify that
    the function returns the correct structure.
    """
    mock_create.return_value = mock_openai_response(
        f'{{"sentiment": "{sentiment}", "confidence": {confidence}}}'
    )
    result = analyze_sentiment("Some text")
    assert result["sentiment"] == sentiment
    assert result["confidence"] == confidence

Fixture scopes — When to use each one

The scope determines how long a fixture "lives":

# Comparison of scopes with usage examples

# SCOPE: function (default)
# The fixture is created and destroyed for each test
@pytest.fixture  # scope="function" by default
def fresh_mock_client():
    """Each test receives a new, clean mock."""
    return MagicMock()
# Use when: the test modifies the mock's state and
# you don't want it to affect other tests

# SCOPE: class
# One instance per test class
@pytest.fixture(scope="class")
def class_mock_client():
    """Shared within a TestX class."""
    return MagicMock()
# Use when: multiple tests in the same class
# use the mock without modifying it

# SCOPE: module
# One instance per test file
@pytest.fixture(scope="module")
def module_real_client():
    """A real client for the whole integration tests module."""
    from openai import OpenAI
    return OpenAI()
# Use when: the object is expensive to create (real connections)
# and the module's tests share it without modifying it

# SCOPE: session
# One instance for the whole test session
@pytest.fixture(scope="session")
def session_config():
    """Global configuration loaded once for the whole suite."""
    from dotenv import load_dotenv
    load_dotenv(".env.test")
    return {
        "model": os.getenv("TEST_MODEL", "gpt-4o-mini"),
        "max_tokens": int(os.getenv("TEST_MAX_TOKENS", "500")),
    }
# Use when: loading expensive configuration a single time

Rule of thumb:

pyproject.toml as an alternative to pytest.ini

If your project uses pyproject.toml, you can put the configuration there:

# pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
asyncio_mode = "auto"
addopts = "-v --tb=short"
markers = [
    "unit: Tests with mocks. No cost, fast.",
    "integration: Tests with a real LLM. They have a cost.",
    "regression: Regression tests. Expensive, weekly.",
    "smoke: Basic smoke tests.",
    "slow: Slow tests (>5s).",
    "contract: Prompt contract tests.",
]

Async testing with pytest-asyncio

If your app uses async/await for LLM calls (recommended for production), you need pytest-asyncio:

# For the AsyncOpenAI client
import asyncio
import pytest
from unittest.mock import AsyncMock, patch
from app.async_sentiment import async_analyze_sentiment


@pytest.mark.asyncio  # Only needed if asyncio_mode != "auto" in pytest.ini
async def test_async_sentiment_analysis():
    """Test of an async function."""
    with patch("app.async_sentiment.async_client.chat.completions.create") as mock_create:
        # AsyncMock to simulate an async response
        mock_response = MagicMock()
        mock_response.choices[0].message.content = '{"sentiment": "positive", "confidence": 0.9}'
        mock_create.return_value = mock_response
        # If the function is async, the mock must be too:
        mock_create = AsyncMock(return_value=mock_response)

        result = await async_analyze_sentiment("Great product!")

        assert result["sentiment"] == "positive"
        assert mock_create.await_count == 1


# Async fixture
@pytest.fixture
async def async_app_client():
    """Async fixture for FastAPI tests with HTTPX."""
    from httpx import AsyncClient
    from app.main import app
    async with AsyncClient(app=app, base_url="http://test") as client:
        yield client


async def test_api_endpoint(async_app_client):
    response = await async_app_client.post(
        "/analyze",
        json={"text": "Great product!"}
    )
    assert response.status_code == 200

Coverage — Measure what's tested

# Install
pip install pytest-cov

# Run with coverage
pytest --cov=app --cov-report=term-missing

# Typical output:
# Name                    Stmts   Miss  Cover   Missing
# -------------------------------------------------------
# app/__init__.py             0      0   100%
# app/llm.py                 25      5    80%   45-50, 67
# app/parsers.py             18      0   100%
# app/validators.py          22      2    91%   38, 42
# -------------------------------------------------------
# TOTAL                      65      7    89%

# Generate HTML report (more detailed)
pytest --cov=app --cov-report=html
# Open htmlcov/index.html in the browser

# Fail if coverage < threshold
pytest --cov=app --cov-fail-under=80

Configure coverage in .coveragerc:

# .coveragerc
[run]
source = app
omit =
    app/__init__.py
    app/config.py  # If it's only configuration
    tests/*

[report]
exclude_lines =
    pragma: no cover
    def __repr__
    raise NotImplementedError
    if TYPE_CHECKING:

Comparison of configuration options

Recommendation: Use pyproject.toml for pytest configuration if your project already has it. If the project is only for testing, pytest.ini is simpler. Fixtures always go in conftest.py.

Connection with the module's project

This configuration is exactly what you'll use in Project 07: Test Suite Setup. By the time you reach that project you'll have:

• pytest.ini with the correct markers for your app
• conftest.py at root with fixtures for LLM mocking
• conftest.py in unit/ with fixtures specific to your components
• The unit/ + integration/ directory structure ready

The project consists of applying this structure to a real LLM app, not learning it from scratch.

Troubleshooting

Problem: pytest: error: unrecognized arguments: -m unit Cause: The markers aren't registered in pytest.ini. Solution: Add the marker to the markers = section in pytest.ini. If you register it, pytest won't give the PytestUnknownMarkWarning warning.

Problem: fixture 'mock_openai_response' not found Cause: The conftest.py with the fixture isn't in the correct directory or the file has a syntax error. Solution: The conftest.py must be in the tests/ directory (or a parent subdirectory). Verify that the file has no errors: python -c "import tests.conftest".

Problem: Integration tests run in CI and fail due to a missing API key. Cause: The API key isn't configured in the CI environment. Solution 1: pytest -m "not integration" in CI by default. Solution 2: In GitHub Actions, add OPENAI_API_KEY as a secret and use it with -m integration only in PR workflows.

Problem: ModuleNotFoundError: No module named 'app' Cause: The src/app path isn't in PYTHONPATH. Solution A: pip install -e . with a setup.py or pyproject.toml with [tool.setuptools.packages]. Solution B: Add to the root conftest.py:

import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))

Solution C (recommended): Use pytest-pythonpath or configure in pyproject.toml:

[tool.setuptools.packages.find]
where = ["src"]

Problem: fixture 'event_loop' not found with pytest-asyncio. Cause: Incompatible version of pytest-asyncio. Solution: Add asyncio_mode = "auto" in pytest.ini (pytest-asyncio >= 0.21).

Problem: Coverage shows 0% for a module that does have tests. Cause: The module isn't in coverage's source. Solution: pytest --cov=app where app is the package name (directory with __init__.py).

Exercises

Exercise 1: Create pytest.ini

Create a pytest.ini for a project with 3 types of tests: unit (mocks), integration (real LLM), and smoke (basic). Include at least 4 documented markers and the command to run only fast tests.

[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*

markers =
    unit: Tests with mocks. Fast, no cost. Run on every commit.
    integration: Tests with a real LLM. They have a cost. Run on PR.
    smoke: Basic smoke tests. Verify that the system boots.
    contract: Structural contract tests for prompts.

addopts = -v --tb=short
asyncio_mode = auto

# Command for fast tests:
# pytest -m "unit or smoke"

Exercise 2: conftest.py with a fixture for the API key

Create an api_key fixture that reads OPENAI_API_KEY from the environment and skips if it's not defined. Also add a model_name fixture that returns the model to use in tests (with a default configurable via env).

# tests/conftest.py
import os
import pytest


@pytest.fixture
def api_key():
    """API key for integration tests. Skip if it's not defined."""
    key = os.getenv("OPENAI_API_KEY")
    if not key:
        pytest.skip("OPENAI_API_KEY not defined — this test requires a real API")
    return key


@pytest.fixture
def model_name():
    """
    Name of the model to use in tests.
    Configurable via environment variable for flexibility in CI.
    Default: gpt-4o-mini (cheaper for tests)
    """
    return os.getenv("TEST_MODEL", "gpt-4o-mini")


# Usage in an integration test:
# def test_with_real_api(api_key, model_name):
#     from openai import OpenAI
#     client = OpenAI(api_key=api_key)
#     response = client.chat.completions.create(
#         model=model_name,
#         messages=[{"role": "user", "content": "Say hi"}]
#     )
#     assert response.choices[0].message.content

Exercise 3: parametrize for a parser

You have this function:

def extract_number_from_response(raw: str) -> float:
    """Extracts the first floating-point number from a string."""
    import re
    match = re.search(r'\d+\.?\d*', raw)
    if not match:
        raise ValueError(f"No number found in: {raw!r}")
    return float(match.group())

Write a parametrized test that covers: integer, float, number in text, no number (must fail).

import pytest
from app.parsers import extract_number_from_response


@pytest.mark.unit
@pytest.mark.parametrize("raw_input,expected", [
    ("0.95", 0.95),
    ("42", 42.0),
    ("The confidence is 0.87 for this response.", 0.87),
    ("Score: 100 out of 100", 100.0),
    ("3.14159", 3.14159),
])
def test_extract_number_valid(raw_input, expected):
    result = extract_number_from_response(raw_input)
    assert result == expected


@pytest.mark.unit
@pytest.mark.parametrize("bad_input", [
    "",
    "No numbers here at all.",
    "abc def ghi",
])
def test_extract_number_raises_on_no_number(bad_input):
    with pytest.raises(ValueError, match="No number found"):
        extract_number_from_response(bad_input)

Exercise 4: Fixture scope

You have a db_connection fixture that takes 2 seconds to initialize. You have 50 tests that use it. How much time do you save using scope="module" vs scope="function" if all the tests are in the same file?

# scope="function" (default):
# The fixture is created and destroyed 50 times
# Time: 50 × 2s = 100 seconds

# scope="module":
# The fixture is created 1 time for the whole file
# Time: 1 × 2s = 2 seconds

# Saving: 98 seconds (50x faster)

# If you had 5 test files with 50 tests each:
# scope="function": 250 × 2s = 500 seconds
# scope="module": 5 × 2s = 10 seconds
# scope="session": 1 × 2s = 2 seconds

# IMPORTANT: Only use wider scopes if the fixture
# is SAFE to share (it doesn't have state that gets modified between tests).
# A database connection in read-only mode is safe.
# A mock that's configured differently in each test is NOT safe with scope="module".

Exercise 5: Configure CI for cost-free runs

Write the pytest command you'd use in GitHub Actions to:

1. Run only fast, cost-free tests (by default on every push)
2. Include integration tests only when the API key is available

# .github/workflows/tests.yml

name: Tests

on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with: { python-version: "3.11" }
      - run: pip install -r requirements-test.txt

      # Always: only unit + smoke (no cost)
      - name: Run fast tests
        run: pytest -m "unit or smoke" --cov=app --cov-fail-under=80

  integration-tests:
    runs-on: ubuntu-latest
    # Only on PRs (not on every push to feature branches)
    if: github.event_name == 'pull_request'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v4
        with: { python-version: "3.11" }
      - run: pip install -r requirements-test.txt

      # Integration tests: only if the secret is available
      - name: Run integration tests
        if: secrets.OPENAI_API_KEY != ''
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: pytest -m integration -v

Summary

• pytest.ini with markers documents the testing strategy and lets you run subsets
• conftest.py at multiple levels organizes fixtures from general to specific
• The create_openai_response() factory must replicate the real API structure so the code works the same with a mock as with the real API
• parametrize is essential for testing parsers with the multiple formats the LLM can produce
• Fixture scope: use the widest possible for fixtures that don't have mutable state
• pytest-asyncio with asyncio_mode = "auto" simplifies testing async code
• Coverage: --cov-fail-under=80 in CI ensures the tests cover the new code

Additional resources

1. pytest Documentation — Complete official documentation
2. pytest conftest.py — Official guide to conftest and fixture scopes
3. pytest markers — Marking and selecting tests
4. pytest parametrize — Advanced parametrization
5. pytest-asyncio — Testing async code (critical for async LLM calls)
6. pytest-cov — Coverage reports
7. pytest-mock — mocker fixture for cleaner mocking