5. Fixtures for LLM Apps

Description

Fixtures for testing AI apps aren't simple test data — they're exact representations of the behavior of an expensive external API. A poorly designed fixture can make all your tests pass while the code fails in production because the mock doesn't reflect the real API structure. This session covers how to design robust fixtures: static ones for typical cases, factories for variations, and fixtures with the appropriate scope to optimize speed.

The goal of this session is to give you a complete set of reusable fixtures you can copy directly into your conftest.py. Each fixture has its design rationale explained — not just the code, but why it's built that way.

By the end you'll have fixtures for: OpenAI responses with the full structure, LLM client mocks, factories for output variations (including edge cases), and fixtures for async. All ready to use in the following modules.

Foundation: Why are LLM fixtures different?

The problem with simple mocks

# ❌ DANGEROUS MOCK: too simple
@pytest.fixture
def mock_llm_response():
    return "This is the LLM response"  # A simple string

# The test passes:
def test_process_response(mock_llm_response):
    result = process(mock_llm_response)
    assert "response" in result  # ✅ Passes

# But the REAL app does this:
def process(api_response):
    content = api_response.choices[0].message.content  # AttributeError!
    # The mock doesn't have .choices, the string doesn't have .choices[0]

The simple mock doesn't reflect the real API structure. When the code does response.choices[0].message.content, the string doesn't have that attribute.

The real OpenAI API structure

# Real structure of client.chat.completions.create()
# (Documentation: platform.openai.com/docs/api-reference/chat)

response = ChatCompletion(
    id="chatcmpl-abc123",
    object="chat.completion",
    model="gpt-4o-mini",
    choices=[
        Choice(
            index=0,
            message=ChatCompletionMessage(
                role="assistant",
                content='{"sentiment": "positive", "confidence": 0.9}'
            ),
            finish_reason="stop",
            logprobs=None
        )
    ],
    usage=CompletionUsage(
        prompt_tokens=100,
        completion_tokens=50,
        total_tokens=150
    ),
    created=1699000000,
    system_fingerprint="fp_abc"
)

# Your code accesses it like this:
content = response.choices[0].message.content
tokens = response.usage.total_tokens
finish = response.choices[0].finish_reason

The mock MUST replicate this access structure exactly.

Static fixture: the base

Factory function (without pytest)

# tests/helpers.py
"""
Reusable testing helpers.
They aren't pytest fixtures — they're functions that create mock objects.
They're imported in conftest.py to create the fixtures.
"""
from unittest.mock import MagicMock


def create_openai_chat_response(
    content: str,
    prompt_tokens: int = 100,
    completion_tokens: int = 50,
    finish_reason: str = "stop",
    model: str = "gpt-4o-mini",
) -> MagicMock:
    """
    Creates a mock that exactly replicates the ChatCompletion structure.

    Args:
        content: The assistant message content (what response.choices[0].message.content returns)
        prompt_tokens: Prompt tokens (for cost tracking)
        completion_tokens: Response tokens
        finish_reason: "stop" (normal) | "length" (truncated) | "content_filter" (filtered)
        model: Model name

    Returns:
        MagicMock that behaves like a real ChatCompletion
    """
    response = MagicMock()

    # Metadata
    response.id = "chatcmpl-test-fixture"
    response.object = "chat.completion"
    response.model = model
    response.created = 1699000000

    # Choice (the main result)
    choice = MagicMock()
    choice.index = 0
    choice.finish_reason = finish_reason
    choice.message.role = "assistant"
    choice.message.content = content
    choice.logprobs = None
    response.choices = [choice]

    # Usage (tokens — important for cost tracking in module 5)
    response.usage.prompt_tokens = prompt_tokens
    response.usage.completion_tokens = completion_tokens
    response.usage.total_tokens = prompt_tokens + completion_tokens

    return response

Fixtures in conftest.py

# tests/conftest.py
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
from tests.helpers import create_openai_chat_response


# ─────────────────────────────────────────────────────────────────────
# Base fixture: response factory
# ─────────────────────────────────────────────────────────────────────

@pytest.fixture
def make_llm_response():
    """
    Factory fixture: returns the create_openai_chat_response function.

    Usage in tests:
        def test_something(make_llm_response):
            response = make_llm_response('{"sentiment": "positive", "confidence": 0.9}')
            # Configure the client mock with this response

    Why a factory instead of a direct fixture:
    - Different tests need different contents
    - Lets you customize tokens, finish_reason, etc.
    - Avoids repeating the MagicMock creation in each test
    """
    return create_openai_chat_response


# ─────────────────────────────────────────────────────────────────────
# Predefined fixtures for common cases
# ─────────────────────────────────────────────────────────────────────

@pytest.fixture
def llm_response_valid_json():
    """Typical response: clean, valid JSON."""
    return create_openai_chat_response(
        content='{"summary": "Test summary", "confidence": 0.9, "points": ["a", "b", "c"]}',
        prompt_tokens=100,
        completion_tokens=60,
    )


@pytest.fixture
def llm_response_json_in_markdown():
    """Response with JSON inside a markdown block (common format)."""
    return create_openai_chat_response(
        content='```json\n{"summary": "Test summary", "confidence": 0.9}\n```'
    )


@pytest.fixture
def llm_response_empty_content():
    """Edge case: the LLM returns an empty string."""
    return create_openai_chat_response(content="")


@pytest.fixture
def llm_response_malformed_json():
    """Edge case: incomplete/malformed JSON."""
    return create_openai_chat_response(
        content='{"summary": "Incomplete summary"',  # Missing closing brace
    )


@pytest.fixture
def llm_response_truncated():
    """Edge case: truncated response (finish_reason="length")."""
    return create_openai_chat_response(
        content='{"summary": "This summary was truncated by',  # Truncated
        finish_reason="length",  # Indicates it was truncated by the token limit
    )


@pytest.fixture
def llm_response_with_extra_text():
    """Edge case: LLM adds text before/after the JSON."""
    return create_openai_chat_response(
        content='Here is the analysis:\n{"sentiment": "positive", "confidence": 0.8}\nHope it is useful.'
    )

Fixtures for the LLM client

# tests/conftest.py (continued)

@pytest.fixture
def mock_llm_client():
    """
    Fully mocked OpenAI client.

    Basic usage:
        def test_something(mock_llm_client, make_llm_response):
            mock_llm_client.chat.completions.create.return_value = make_llm_response("content")
            # ... test code ...

    Usage with @patch (more common for isolated unit tests):
        @patch("app.sentiment.client")
        def test_something(mock_client, make_llm_response):
            mock_client.chat.completions.create.return_value = make_llm_response("content")

    This fixture is useful when you need to inject the client
    as a dependency into the function (dependency injection).
    """
    client = MagicMock()
    # Default response (override in each test)
    client.chat.completions.create.return_value = create_openai_chat_response(
        '{"default": "mock response — override this in your test"}'
    )
    return client


@pytest.fixture
def mock_llm_client_with_side_effect(make_llm_response):
    """
    Mock client that returns different responses on successive calls.
    Useful for testing chains or pipelines with multiple LLM calls.
    """
    client = MagicMock()
    client.chat.completions.create.side_effect = [
        make_llm_response('{"step": "extraction", "data": "extracted"}'),
        make_llm_response('{"step": "analysis", "result": "positive"}'),
        make_llm_response('{"step": "summary", "text": "Final summary"}'),
    ]
    return client

Advanced fixture factories

Factory with params for parametrize

# You can use params in fixtures to run a test multiple times

SENTIMENT_TEST_CASES = [
    pytest.param("positive", 0.9, id="positive_high_confidence"),
    pytest.param("negative", 0.8, id="negative_high_confidence"),
    pytest.param("neutral", 0.5, id="neutral_medium_confidence"),
    pytest.param("positive", 0.1, id="positive_low_confidence"),
]

@pytest.fixture(params=SENTIMENT_TEST_CASES)
def sentiment_response(request, make_llm_response):
    """
    Parametrized fixture: generates a response for each combination
    of sentiment and confidence in SENTIMENT_TEST_CASES.

    A test that uses this fixture runs 4 times automatically.
    """
    sentiment, confidence = request.param
    return make_llm_response(
        f'{{"sentiment": "{sentiment}", "confidence": {confidence}}}'
    )


# Usage:
def test_parser_handles_all_sentiments(sentiment_response):
    """This test runs 4 times — one per param."""
    content = sentiment_response.choices[0].message.content
    result = parse_sentiment_response(content)
    assert result["sentiment"] in ["positive", "negative", "neutral"]
    assert 0.0 <= result["confidence"] <= 1.0

Factory with error cases

PARSER_ERROR_CASES = [
    pytest.param("", id="empty_content"),
    pytest.param('{"summary": "incomplete', id="malformed_json"),
    pytest.param("No JSON here", id="no_json"),
    pytest.param('{"wrong_key": "value"}', id="missing_required_key"),
    pytest.param("null", id="null_response"),
]

@pytest.fixture(params=PARSER_ERROR_CASES)
def problematic_llm_content(request):
    """
    Fixture that generates problematic contents to test error handling.
    """
    return request.param


# Usage in a robust parser test:
@pytest.mark.unit
def test_parser_handles_problematic_content_gracefully(problematic_llm_content, make_llm_response):
    """
    The parser must not crash with problematic content.
    It must return None or raise a controlled exception.
    """
    response = make_llm_response(problematic_llm_content)
    content = response.choices[0].message.content

    try:
        result = parse_llm_response(content)
        # If it doesn't raise, verify that the result is empty or None
        assert result is None or result == {}
    except (ValueError, KeyError, json.JSONDecodeError):
        # Controlled exception — acceptable
        pass
    except AttributeError as e:
        pytest.fail(f"Parser should not raise AttributeError: {e}")
    except Exception as e:
        pytest.fail(f"Parser raised an unexpected exception: {type(e).__name__}: {e}")

Fixtures for async apps

If your app uses async/await for LLM calls (recommended for production with FastAPI):

# tests/conftest.py (for async apps)
import pytest
from unittest.mock import AsyncMock, MagicMock


def create_async_openai_response(content: str, tokens: int = 150) -> MagicMock:
    """
    Creates a mock for the AsyncOpenAI client.
    The difference: create() must be awaitable.
    """
    response = MagicMock()
    response.choices[0].message.content = content
    response.usage.total_tokens = tokens
    return response


@pytest.fixture
def mock_async_llm_client():
    """
    AsyncOpenAI mock for testing async code.
    Uses AsyncMock so that 'await client.chat.completions.create(...)' works.
    """
    client = MagicMock()
    # AsyncMock makes 'await create(...)' work in tests
    client.chat.completions.create = AsyncMock(
        return_value=create_async_openai_response(
            '{"default": "async mock response"}'
        )
    )
    return client


# Usage in an async test:
@pytest.mark.asyncio
async def test_async_sentiment_analysis(mock_async_llm_client, make_llm_response):
    """Test of an async function that calls the LLM."""
    # Configure the response
    mock_async_llm_client.chat.completions.create = AsyncMock(
        return_value=make_llm_response('{"sentiment": "positive", "confidence": 0.9}')
    )

    # Run the async function
    with patch("app.async_sentiment.async_client", mock_async_llm_client):
        result = await async_analyze_sentiment("I love this!")

    assert result["sentiment"] == "positive"

Fixtures for testing FastAPI

# tests/conftest.py (for projects with FastAPI)
import pytest
from fastapi.testclient import TestClient
from httpx import AsyncClient


@pytest.fixture(scope="module")
def test_client():
    """
    HTTP client for FastAPI tests.
    scope="module": a single client for all the module's tests.

    Usage:
        def test_health_check(test_client):
            response = test_client.get("/health")
            assert response.status_code == 200
    """
    from app.main import app
    with TestClient(app) as client:
        yield client


@pytest.fixture(scope="module")
async def async_test_client():
    """Async client for async FastAPI tests."""
    from app.main import app
    async with AsyncClient(app=app, base_url="http://test") as client:
        yield client

Organizing fixtures by module

For large projects, organize the fixtures across multiple files:

tests/
├── conftest.py              # Global fixtures (make_llm_response, mock_llm_client)
├── helpers.py               # Helper functions (create_openai_chat_response)
├── fixtures/                # Specialized fixtures
│   ├── sentiment.py         # Fixtures for the sentiment module
│   ├── summarizer.py        # Fixtures for the summarization module
│   └── rag.py               # Fixtures for the RAG pipeline
├── unit/
│   ├── conftest.py          # Fixtures specific to unit tests
│   └── test_*.py
└── integration/
    ├── conftest.py          # Fixtures for integration tests
    └── test_*.py

# tests/fixtures/sentiment.py
"""Specialized fixtures for the sentiment module tests."""
import pytest
from tests.helpers import create_openai_chat_response

@pytest.fixture
def positive_sentiment_response():
    return create_openai_chat_response(
        '{"sentiment": "positive", "confidence": 0.95, "aspects": ["quality", "value"]}'
    )

@pytest.fixture
def negative_sentiment_response():
    return create_openai_chat_response(
        '{"sentiment": "negative", "confidence": 0.88, "aspects": ["shipping", "price"]}'
    )

# In tests/unit/conftest.py — import the specialized fixtures:
# pytest lets you re-export fixtures from external files

Comparison of fixture approaches

Fixture anti-patterns

# ❌ ANTI-PATTERN 1: Fixture that returns a dict instead of a MagicMock
@pytest.fixture
def bad_llm_response():
    return {"choices": [{"message": {"content": "..."}}]}
# The real code does response.choices[0].message.content (dot notation)
# The dict requires response["choices"][0]["message"]["content"] (bracket notation)
# They're incompatible → the test passes but the real code fails

# ✅ CORRECT:
@pytest.fixture
def good_llm_response(make_llm_response):
    return make_llm_response('{"content": "..."}')
# MagicMock supports dot notation: response.choices[0].message.content ✅


# ❌ ANTI-PATTERN 2: Overly specific fixture
@pytest.fixture
def response_for_test_42():
    return create_openai_chat_response('{"very": "specific"}')
# Creates one fixture per test → conftest.py becomes garbage

# ✅ CORRECT: Factory that any test can customize
@pytest.fixture
def make_llm_response():
    return create_openai_chat_response  # Returns the function


# ❌ ANTI-PATTERN 3: Global fixture with scope="session" for mutable mocks
@pytest.fixture(scope="session")
def shared_mock_client():
    client = MagicMock()
    client.chat.completions.create.return_value = ...
    return client
# If a test modifies client.chat.completions.create.return_value,
# ALL the following tests in the session use the modified value

# ✅ CORRECT: scope="function" for mocks (or "module" if they're read-only)
@pytest.fixture  # scope="function" by default
def mock_client():
    return MagicMock()  # Each test receives a clean mock

Connection with the module's project

The fixtures you define here are exactly the ones you'll use in Project 07: Test Suite Setup. The project consists of applying these fixtures to a real LLM app, configuring the complete conftest.py, and writing the first tests using these fixtures.

In the following modules:

• Module 2: You'll use make_llm_response to create prompt contract tests
• Module 3: You'll extend the fixtures for integration tests with a real LLM
• Module 4: You'll add fixtures for testing guardrails
• Module 5: You'll add fixtures that capture logs to verify logging

Fixtures are an investment: the time you spend now designing them well multiplies into productivity throughout the whole guide.

Troubleshooting

Problem: AttributeError: 'dict' object has no attribute 'choices' Cause: The fixture returns a dict instead of a MagicMock. The code does response.choices[0] (dot notation), not response["choices"][0] (bracket notation). Solution: Always use MagicMock(): response = MagicMock(); response.choices[0].message.content = "...".

Problem: TypeError: 'MagicMock' object is not subscriptable Cause: The code accesses with bracket notation response["choices"] but the fixture uses MagicMock with dot notation. Solution: Identify how your code accesses the response. If it does response["choices"], use a dict. If it does response.choices, use MagicMock. The safest thing is to match the real API (OpenAI uses dot notation → use MagicMock).

Problem: The fixture runs many times and the test is slow (>2s per test). Cause: scope="function" on an expensive fixture (real connection, model loading). Solution: Change to scope="module" if the fixture is read-only and shareable between tests.

Problem: The side_effect doesn't work as expected. Cause: side_effect with a list consumes one element per call. If there are more calls than elements, it raises StopIteration. Solution:

• For exceptions: mock.side_effect = Exception("error")
• For a list of responses: mock.side_effect = [resp1, resp2, resp3] (one per call)
• For a dynamic function: mock.side_effect = lambda *args, **kwargs: compute_response(args)

Problem: The async fixture doesn't work (TypeError: object MagicMock can't be used in 'await' expression) Cause: MagicMock() isn't awaitable. For async functions you need AsyncMock. Solution: from unittest.mock import AsyncMock; mock.create = AsyncMock(return_value=response)

Exercises

Exercise 1: Create empty and malformed response fixtures

Create two fixtures in conftest.py: one that simulates content="" (empty) and another that simulates truncated JSON. Write tests that verify how your parser handles each case.

# tests/conftest.py
@pytest.fixture
def llm_response_empty():
    """Edge case: the LLM returns an empty string."""
    return create_openai_chat_response(content="")


@pytest.fixture
def llm_response_truncated_json():
    """Edge case: truncated JSON (finish_reason='length')."""
    return create_openai_chat_response(
        content='{"summary": "This summary was',  # Incomplete JSON
        finish_reason="length"
    )


# tests/unit/test_parser_robustness.py
def test_parser_handles_empty_content(llm_response_empty):
    content = llm_response_empty.choices[0].message.content
    with pytest.raises((ValueError, json.JSONDecodeError)):
        parse_llm_response(content)


def test_parser_handles_truncated_json(llm_response_truncated_json):
    content = llm_response_truncated_json.choices[0].message.content
    finish = llm_response_truncated_json.choices[0].finish_reason

    # If finish_reason == "length", it should raise an error or return None
    assert finish == "length"
    with pytest.raises((ValueError, json.JSONDecodeError)):
        parse_llm_response(content)

Exercise 2: Factory with parameters

Modify the create_openai_chat_response function to accept a multiple_choices: bool parameter that, when True, generates a response with 2 choices (some APIs return multiple options with n=2).

def create_openai_chat_response_multi(
    contents: list[str],
    tokens: int = 200,
) -> MagicMock:
    """
    Creates a mock with multiple choices (when n>1 is used in the API).
    """
    response = MagicMock()
    response.usage.total_tokens = tokens

    choices = []
    for i, content in enumerate(contents):
        choice = MagicMock()
        choice.index = i
        choice.finish_reason = "stop"
        choice.message.content = content
        choices.append(choice)

    response.choices = choices
    return response


# Usage:
@pytest.fixture
def multi_choice_response():
    return create_openai_chat_response_multi([
        '{"sentiment": "positive", "confidence": 0.9}',
        '{"sentiment": "positive", "confidence": 0.85}',
    ])


def test_function_uses_first_choice(multi_choice_response):
    content = multi_choice_response.choices[0].message.content
    result = parse_sentiment_response(content)
    assert result["sentiment"] == "positive"

Exercise 3: Fixture that depends on another

Create a mock_sentiment_client fixture that:

1. Uses the make_llm_response factory (conftest fixture)
2. Configures the client to always return a positive sentiment
3. Returns an already-configured MagicMock of the client

@pytest.fixture
def mock_sentiment_client(make_llm_response):
    """
    Pre-configured mock client for sentiment tests.
    Already configured to return a positive sentiment.
    """
    client = MagicMock()
    client.chat.completions.create.return_value = make_llm_response(
        '{"sentiment": "positive", "confidence": 0.9}'
    )
    return client


# Usage in a test:
def test_positive_text_is_analyzed_correctly(mock_sentiment_client):
    with patch("app.sentiment.client", mock_sentiment_client):
        result = analyze_sentiment("I love this!")

    assert result["sentiment"] == "positive"
    mock_sentiment_client.chat.completions.create.assert_called_once()

Exercise 4: AsyncMock fixture

Create a mock_async_openai_client fixture using AsyncMock that's compatible with code that does await client.chat.completions.create(...).

# tests/conftest.py
from unittest.mock import AsyncMock

@pytest.fixture
def mock_async_openai_client(make_llm_response):
    """
    AsyncOpenAI mock for testing async code.
    The create() method is an AsyncMock to support 'await'.
    """
    client = MagicMock()
    # AsyncMock makes 'await client.chat.completions.create(...)' work
    client.chat.completions.create = AsyncMock(
        return_value=make_llm_response('{"result": "async mock default"}')
    )
    return client


# Usage:
@pytest.mark.asyncio
async def test_async_function(mock_async_openai_client, make_llm_response):
    # Configure a specific response
    mock_async_openai_client.chat.completions.create = AsyncMock(
        return_value=make_llm_response('{"sentiment": "positive", "confidence": 0.9}')
    )

    with patch("app.async_module.async_client", mock_async_openai_client):
        result = await my_async_function("test input")

    assert result["sentiment"] == "positive"
    mock_async_openai_client.chat.completions.create.assert_awaited_once()

Exercise 5: Parametrized fixture for automatic coverage

Create a parametrized fixture all_finish_reasons that generates responses with finish_reason in ["stop", "length", "content_filter"]. Write a test that uses this fixture to verify that your code correctly handles each case.

@pytest.fixture(params=[
    pytest.param("stop", id="normal_completion"),
    pytest.param("length", id="truncated_by_token_limit"),
    pytest.param("content_filter", id="filtered_by_safety"),
])
def response_with_finish_reason(request, make_llm_response):
    """Fixture parametrized by finish_reason."""
    finish = request.param
    content = '{"result": "some content"}' if finish == "stop" else '{"result": "partial'
    return (finish, create_openai_chat_response(content=content, finish_reason=finish))


def test_handles_all_finish_reasons(response_with_finish_reason):
    """Verifies that the code correctly handles each finish_reason."""
    finish_reason, response = response_with_finish_reason

    if finish_reason == "stop":
        # Normal: must process correctly
        result = process_llm_response(response)
        assert result is not None
    elif finish_reason == "length":
        # Truncated: must raise an error or return partial with a flag
        with pytest.raises(ValueError, match="truncated"):
            process_llm_response(response)
    elif finish_reason == "content_filter":
        # Filtered: must raise ContentFilterError
        with pytest.raises((ValueError, ContentFilterError)):
            process_llm_response(response)

Summary

• LLM fixtures must use MagicMock() with a structure that replicates the real API (dot notation, not dict)
• The create_openai_chat_response() factory is the base of all fixtures — copy it into tests/helpers.py
• Use factory fixtures (return create_openai_chat_response) for tests that need different contents
• Parametrized fixtures (params=[...]) run the test multiple times and maximize coverage
• For async code: use AsyncMock instead of MagicMock so await works
• Design fixtures for edge cases from the start: empty, malformed, truncated, with extra text

Additional resources

1. pytest fixtures — official documentation — Scopes, factory fixtures, fixture parametrization
2. unittest.mock — MagicMock — How it configures attributes automatically
3. unittest.mock — AsyncMock — For async code
4. pytest-mock — Plugin that simplifies using mocks in pytest
5. OpenAI API Reference — Chat — Real structure of the ChatCompletion object
6. Factories as Fixtures — Pattern documented in pytest