Module 2: Unit Testing LLM Applications

6. Fixture Factories

Description

Fixture factories are functions that return fixture-generating functions. Instead of a static fixture that always returns the same thing, you have a factory that can create parametrized variations. This technique is fundamental for testing LLM apps because it lets you generate dozens of response variations without duplicating code. The result: tests that are more expressive, DRY, and that cover more edge cases.

The problem with static fixtures

Static fixtures have a problem: they only define one case:

# Static fixture — only one case
@pytest.fixture
def mock_client_positive():
    client = MagicMock()
    client.chat.completions.create.return_value = create_openai_chat_response(
        '{"sentiment": "positive", "score": 0.9, "keywords": ["good"]}'
    )
    return client

@pytest.fixture
def mock_client_negative():
    client = MagicMock()
    client.chat.completions.create.return_value = create_openai_chat_response(
        '{"sentiment": "negative", "score": 0.1, "keywords": ["bad"]}'
    )
    return client

@pytest.fixture
def mock_client_neutral():
    client = MagicMock()
    client.chat.completions.create.return_value = create_openai_chat_response(
        '{"sentiment": "neutral", "score": 0.5, "keywords": ["normal"]}'
    )
    return client

# To test 3 sentiments, you need 3 fixtures and 3 separate tests
def test_positive(mock_client_positive): ...
def test_negative(mock_client_negative): ...
def test_neutral(mock_client_neutral): ...

# If you add another sentiment, you create another fixture...
# If you need 10 variations, you write 10 fixtures...
# ❌ This pattern doesn't scale

The solution: factory fixture

A factory fixture returns a function that generates the desired object:

# Factory fixture — a single place to create variations
@pytest.fixture
def mock_client_factory():
    """
    Factory that creates mock clients with different responses.
    
    Usage:
        def test_x(mock_client_factory):
            client = mock_client_factory(sentiment="positive", score=0.9)
            result = analyze_sentiment("text", client=client)
            assert result["sentiment"] == "positive"
    """
    def _create(
        sentiment: str = "neutral",
        score: float = 0.5,
        explanation: str = "Test analysis",
        keywords: list = None
    ) -> MagicMock:
        if keywords is None:
            keywords = []
        
        content = json.dumps({
            "sentiment": sentiment,
            "score": score,
            "explanation": explanation,
            "keywords": keywords
        })
        
        client = MagicMock()
        client.chat.completions.create.return_value = create_openai_chat_response(content)
        return client
    
    return _create

# Now a single fixture creates ANY variation:
def test_positive_sentiment(mock_client_factory):
    client = mock_client_factory(sentiment="positive", score=0.9)
    result = analyze_sentiment("positive text", client=client)
    assert result["sentiment"] == "positive"

def test_negative_sentiment(mock_client_factory):
    client = mock_client_factory(sentiment="negative", score=0.1)
    result = analyze_sentiment("negative text", client=client)
    assert result["sentiment"] == "negative"

def test_boundary_score(mock_client_factory):
    client = mock_client_factory(sentiment="positive", score=1.0)
    result = analyze_sentiment("extreme text", client=client)
    assert result["score"] == 1.0

Factory with smart defaults

Well-chosen defaults make the factory more expressive:

# tests/conftest.py

from tests.helpers import create_openai_chat_response
import json
from unittest.mock import MagicMock
import pytest

# Realistic default values
DEFAULT_SENTIMENT_RESPONSE = {
    "sentiment": "neutral",
    "score": 0.5,
    "explanation": "The text doesn't express a clear sentiment.",
    "keywords": []
}

@pytest.fixture
def make_sentiment_client():
    """
    Factory to create sentiment mock clients.
    
    Defaults: neutral, score 0.5 — the "emptiest" case.
    Override only what you need to test.
    """
    def _create(**kwargs) -> MagicMock:
        # Merge defaults with the provided kwargs
        response_data = {**DEFAULT_SENTIMENT_RESPONSE, **kwargs}
        
        client = MagicMock()
        client.chat.completions.create.return_value = create_openai_chat_response(
            json.dumps(response_data)
        )
        return client
    
    return _create

# Expressive usage — you only specify what changes:
def test_sentiment(make_sentiment_client):
    # I only need to change the sentiment
    client = make_sentiment_client(sentiment="positive", score=0.95)
    result = analyze_sentiment("I love it", client=client)
    assert result["sentiment"] == "positive"

def test_low_confidence(make_sentiment_client):
    # I only care about the low score
    client = make_sentiment_client(score=0.1)
    result = analyze_sentiment("Ambiguous text", client=client)
    assert result["score"] == 0.1

Factory for edge cases

Factories shine when you need to cover many edge cases:

@pytest.fixture
def make_error_client():
    """
    Factory to create clients that simulate LLM errors.
    
    Usage:
        client = make_error_client("rate_limit")
        client = make_error_client("timeout")
        client = make_error_client("empty_response")
    """
    def _create(error_type: str = "generic") -> MagicMock:
        client = MagicMock()
        
        error_map = {
            "rate_limit": openai.RateLimitError(
                message="Rate limit exceeded",
                response=MagicMock(status_code=429),
                body={}
            ),
            "timeout": openai.APITimeoutError(request=MagicMock()),
            "connection": openai.APIConnectionError(request=MagicMock()),
            "auth": openai.AuthenticationError(
                message="Invalid API key",
                response=MagicMock(status_code=401),
                body={}
            ),
            "service_unavailable": openai.APIStatusError(
                message="Service unavailable",
                response=MagicMock(status_code=503),
                body={}
            ),
            "empty_response": None,  # Special case: returns None
            "generic": Exception("Unexpected LLM error")
        }
        
        if error_type == "empty_response":
            client.chat.completions.create.return_value = create_openai_chat_response("")
        else:
            error = error_map.get(error_type, error_map["generic"])
            client.chat.completions.create.side_effect = error
        
        return client
    
    return _create

# Error handling tests:
@pytest.mark.parametrize("error_type,expected_result_key", [
    ("rate_limit", "error"),
    ("timeout", "error"),
    ("empty_response", "sentiment"),  # Must use a fallback, not crash
])
def test_error_handling(make_error_client, error_type, expected_result_key):
    """The app handles all error types gracefully."""
    client = make_error_client(error_type)
    result = analyze_sentiment("text", client=client)
    
    assert result is not None, "The app must not return None"
    assert expected_result_key in result, \
        f"Expected '{expected_result_key}' in the result, got: {result}"

Factory + parametrize: the most powerful combination

When you combine factories with parametrize, you get massive coverage with little code:

# Define the test cases as data
SENTIMENT_TEST_CASES = [
    pytest.param(
        {"sentiment": "positive", "score": 0.95, "keywords": ["excellent", "incredible"]},
        "positive",
        id="very_positive"
    ),
    pytest.param(
        {"sentiment": "positive", "score": 0.6},
        "positive",
        id="slightly_positive"
    ),
    pytest.param(
        {"sentiment": "neutral", "score": 0.5},
        "neutral",
        id="exactly_neutral"
    ),
    pytest.param(
        {"sentiment": "negative", "score": 0.3},
        "negative",
        id="slightly_negative"
    ),
    pytest.param(
        {"sentiment": "negative", "score": 0.05, "keywords": ["terrible", "horrible"]},
        "negative",
        id="very_negative"
    ),
]

@pytest.mark.parametrize("response_data,expected_sentiment", SENTIMENT_TEST_CASES)
def test_sentiment_variations(make_sentiment_client, response_data, expected_sentiment):
    """
    Tests multiple sentiment variations with a single function.
    The factory creates the client with the appropriate data for each case.
    """
    client = make_sentiment_client(**response_data)
    result = analyze_sentiment("test text", client=client)
    
    assert result["sentiment"] == expected_sentiment
    assert 0 <= result["score"] <= 1
    assert isinstance(result.get("keywords", []), list)

Factory for mocking multiple calls (chains)

Factories are especially useful for chains that make multiple LLM calls:

@pytest.fixture
def make_chain_client():
    """
    Factory to create clients that respond to multiple LLM calls.
    
    Usage for chains:
        client = make_chain_client([
            '{"summary": "Text summary"}',                # First call
            '{"category": "technology", "tags": ["AI"]}'   # Second call
        ])
    """
    def _create(responses: list[str]) -> MagicMock:
        """
        responses: list of JSON strings, one per LLM call.
        """
        client = MagicMock()
        client.chat.completions.create.side_effect = [
            create_openai_chat_response(resp)
            for resp in responses
        ]
        return client
    
    return _create

# Chain test with multiple calls:
def test_summarize_and_classify_chain(make_chain_client):
    """Chain that summarizes and then classifies."""
    client = make_chain_client([
        '{"summary": "Python is popular for AI"}',              # First call: summary
        '{"category": "technology", "confidence": 0.95}'         # Second call: classification
    ])
    
    result = summarize_and_classify("Long article about Python and AI...", client=client)
    
    assert result["summary"] == "Python is popular for AI"
    assert result["category"] == "technology"
    assert client.chat.completions.create.call_count == 2

# Test that verifies the behavior when the second call fails:
def test_chain_second_call_fails(make_chain_client, make_error_client):
    """Chain handles a failure on the second call."""
    client = MagicMock()
    client.chat.completions.create.side_effect = [
        create_openai_chat_response('{"summary": "Summary"}'),  # First: success
        openai.APITimeoutError(request=MagicMock())               # Second: timeout
    ]
    
    result = summarize_and_classify("text", client=client)
    
    # Must return the summary even if the classification fails
    assert result["summary"] == "Summary"
    assert result.get("category") is None or "error" in result

Factory with async support

For async apps, the factory must return an AsyncMock:

@pytest.fixture
def make_async_client():
    """Factory for async mock clients."""
    def _create(
        sentiment: str = "neutral",
        score: float = 0.5,
        **kwargs
    ) -> MagicMock:
        content = json.dumps({"sentiment": sentiment, "score": score, **kwargs})
        
        client = MagicMock()
        # AsyncMock for the async function
        client.chat.completions.create = AsyncMock(
            return_value=create_openai_chat_response(content)
        )
        return client
    
    return _create

@pytest.mark.asyncio
async def test_async_analyze(make_async_client):
    client = make_async_client(sentiment="positive", score=0.9)
    result = await analyze_sentiment_async("text", client=client)
    assert result["sentiment"] == "positive"

Factory that combines client + patch

For cases where you need to patch the app's global client:

@pytest.fixture
def patched_openai_factory(mocker):
    """
    Factory that patches the global OpenAI client and lets you configure the response.
    
    Usage:
        def test_x(patched_openai_factory):
            mock_create = patched_openai_factory(sentiment="positive")
            result = analyze_sentiment("text")  # Without passing a client
            assert result["sentiment"] == "positive"
            mock_create.assert_called_once()  # Verifies the LLM was called
    """
    def _create(**response_kwargs) -> MagicMock:
        mock_create = mocker.patch("app.sentiment.client.chat.completions.create")
        
        content = json.dumps({
            "sentiment": response_kwargs.get("sentiment", "neutral"),
            "score": response_kwargs.get("score", 0.5),
            "explanation": response_kwargs.get("explanation", "Test analysis"),
            "keywords": response_kwargs.get("keywords", [])
        })
        
        mock_create.return_value = create_openai_chat_response(content)
        return mock_create  # Returns the mock so you can make assertions
    
    return _create

# Usage:
def test_with_patched_factory(patched_openai_factory):
    mock_create = patched_openai_factory(sentiment="positive", score=0.92)
    
    # You don't need to pass a client — the patch acts on the global client
    result = analyze_sentiment("I love it")
    
    assert result["sentiment"] == "positive"
    mock_create.assert_called_once()
    
    # Verify the call parameters
    call_kwargs = mock_create.call_args.kwargs
    assert "I love it" in str(call_kwargs.get("messages", []))

Organizing factories in conftest.py

For a project with multiple modules, organize the factories:

# tests/conftest.py — factories shared by all tests

@pytest.fixture
def make_sentiment_client():
    """Factory for sentiment analysis tests."""
    def _create(sentiment="neutral", score=0.5, **extra):
        content = json.dumps({"sentiment": sentiment, "score": score, **extra})
        client = MagicMock()
        client.chat.completions.create.return_value = create_openai_chat_response(content)
        return client
    return _create

@pytest.fixture
def make_summary_client():
    """Factory for text summary tests."""
    def _create(summary="Test summary", confidence=0.8, sources=None):
        if sources is None:
            sources = []
        content = json.dumps({"summary": summary, "confidence": confidence, "sources": sources})
        client = MagicMock()
        client.chat.completions.create.return_value = create_openai_chat_response(content)
        return client
    return _create

@pytest.fixture
def make_classification_client():
    """Factory for classification tests."""
    def _create(category="general", confidence=0.8, tags=None):
        if tags is None:
            tags = []
        content = json.dumps({"category": category, "confidence": confidence, "tags": tags})
        client = MagicMock()
        client.chat.completions.create.return_value = create_openai_chat_response(content)
        return client
    return _create

# tests/unit/conftest.py — factories specific to unit tests
@pytest.fixture
def make_chain_client():
    """Factory for chain tests with multiple calls."""
    def _create(responses: list[str]) -> MagicMock:
        client = MagicMock()
        client.chat.completions.create.side_effect = [
            create_openai_chat_response(r) for r in responses
        ]
        return client
    return _create

Anti-patterns in fixture factories

# ❌ Anti-pattern 1: Factory that does too much
@pytest.fixture
def uber_factory():
    def _create(sentiment=None, summary=None, classification=None, error=None, ...):
        # 50 lines of complex logic
        # Hard to understand exactly what it creates
        ...
    return _create
# ✅ Better: small, specific factories per prompt type

# ❌ Anti-pattern 2: Factory with shared mutable state
SHARED_STATE = {"call_count": 0}

@pytest.fixture
def factory_with_shared_state():
    def _create():
        SHARED_STATE["call_count"] += 1  # Global mutable state
        ...
    return _create
# ✅ Better: each factory call creates a fresh object with no shared state

# ❌ Anti-pattern 3: Factory that mixes sync and async without declaring it
@pytest.fixture
def ambiguous_factory():
    def _create(async_mode=False):
        if async_mode:
            return AsyncMock(...)
        else:
            return MagicMock(...)
    return _create
# ✅ Better: separate factories for sync and async

# ❌ Anti-pattern 4: Not using a factory when there are 5+ identical fixtures with variations
@pytest.fixture
def client_v1():
    client = MagicMock()
    client.chat.completions.create.return_value = create_openai_chat_response('{"x": 1}')
    return client

@pytest.fixture
def client_v2():
    client = MagicMock()
    client.chat.completions.create.return_value = create_openai_chat_response('{"x": 2}')
    return client

# ... client_v3, client_v4, client_v5 ...
# ✅ Better: make_client(x=1), make_client(x=2)

When to use each pattern

Situation	Recommended pattern
A single standard case	Static fixture
2-3 known predefined cases	`@pytest.fixture(params=[...])`
Dynamic variations in each test	Factory fixture
Many combinations	Factory + `@pytest.mark.parametrize`
Error handling tests	Factory specialized for errors
Chains with multiple LLM calls	Factory with `side_effect` list

Exercises

Exercise 1: Create a basic factory

Create a factory fixture make_classification_client that generates clients with classification responses. The response has: category (string), confidence (float), tags (list).

See solution

@pytest.fixture
def make_classification_client():
    """Factory for document classification tests."""
    def _create(
        category: str = "general",
        confidence: float = 0.8,
        tags: list = None
    ) -> MagicMock:
        if tags is None:
            tags = []
        
        content = json.dumps({
            "category": category,
            "confidence": confidence,
            "tags": tags
        })
        
        client = MagicMock()
        client.chat.completions.create.return_value = create_openai_chat_response(content)
        return client
    
    return _create

# Tests that use the factory:
def test_tech_classification(make_classification_client):
    client = make_classification_client(category="technology", confidence=0.95, tags=["AI", "Python"])
    result = classify_document("Article about Python and AI", client=client)
    assert result["category"] == "technology"
    assert result["confidence"] == 0.95

def test_default_classification(make_classification_client):
    client = make_classification_client()  # No arguments, uses defaults
    result = classify_document("Generic text", client=client)
    assert result["category"] == "general"

Exercise 2: Factory with parametrize

Use the factory from Exercise 1 with @pytest.mark.parametrize to test 5 different categories:

See solution

CLASSIFICATION_CASES = [
    pytest.param("technology", 0.95, ["AI", "Python"], id="technology"),
    pytest.param("science", 0.88, ["physics", "research"], id="science"),
    pytest.param("sports", 0.92, ["soccer", "competition"], id="sports"),
    pytest.param("politics", 0.75, ["elections"], id="politics"),
    pytest.param("general", 0.60, [], id="no_clear_category"),
]

@pytest.mark.parametrize("category,confidence,tags", CLASSIFICATION_CASES)
def test_all_categories(make_classification_client, category, confidence, tags):
    """Verifies that all categories are processed correctly."""
    client = make_classification_client(
        category=category,
        confidence=confidence,
        tags=tags
    )
    
    result = classify_document("Test text", client=client)
    
    assert result["category"] == category
    assert result["confidence"] == confidence
    assert result["tags"] == tags

Exercise 3: Factory for error handling

Create a factory make_error_client that lets you simulate different error types. Write tests for:

Rate limit → the app returns {"error": "rate_limit"}
Timeout → the app returns {"error": "timeout"}
Empty response → the app uses category="unknown"

See solution

@pytest.fixture
def make_error_client():
    def _create(error_type: str) -> MagicMock:
        client = MagicMock()
        
        if error_type == "rate_limit":
            client.chat.completions.create.side_effect = openai.RateLimitError(
                message="Rate limit exceeded",
                response=MagicMock(status_code=429),
                body={}
            )
        elif error_type == "timeout":
            client.chat.completions.create.side_effect = openai.APITimeoutError(
                request=MagicMock()
            )
        elif error_type == "empty_response":
            client.chat.completions.create.return_value = create_openai_chat_response("")
        
        return client
    
    return _create

def test_rate_limit_handling(make_error_client):
    client = make_error_client("rate_limit")
    result = classify_document("text", client=client)
    assert result == {"error": "rate_limit"}

def test_timeout_handling(make_error_client):
    client = make_error_client("timeout")
    result = classify_document("text", client=client)
    assert result == {"error": "timeout"}

def test_empty_response_fallback(make_error_client):
    client = make_error_client("empty_response")
    result = classify_document("text", client=client)
    assert result.get("category") == "unknown"

Exercise 4: Factory for a chain

Create a factory make_pipeline_client for a three-step pipeline:

First LLM call: extracts entities
Second LLM call: classifies the entities
Third LLM call: generates a summary

See solution

@pytest.fixture
def make_pipeline_client():
    """Factory for 3-step pipelines."""
    def _create(
        entities: list = None,
        classifications: list = None,
        summary: str = "Generated summary"
    ) -> MagicMock:
        if entities is None:
            entities = [{"entity": "Python", "type": "TECHNOLOGY"}]
        if classifications is None:
            classifications = [{"entity": "Python", "category": "language"}]
        
        client = MagicMock()
        client.chat.completions.create.side_effect = [
            create_openai_chat_response(json.dumps(entities)),
            create_openai_chat_response(json.dumps(classifications)),
            create_openai_chat_response(json.dumps({"summary": summary}))
        ]
        return client
    
    return _create

def test_pipeline_three_steps(make_pipeline_client):
    client = make_pipeline_client(
        entities=[{"entity": "FastAPI", "type": "FRAMEWORK"}],
        classifications=[{"entity": "FastAPI", "category": "web"}],
        summary="FastAPI is a modern web framework"
    )
    
    result = run_pipeline("Article about FastAPI", client=client)
    
    assert result["summary"] == "FastAPI is a modern web framework"
    assert client.chat.completions.create.call_count == 3

Exercise 5: When NOT to use a factory

Describe 3 situations where a static fixture is better than a factory:

See guide

A single standard case for all the module's tests: If all the module's tests use the same mock response (the generic "happy path"), a static fixture is clearer than a factory with defaults that nobody overrides.
```
# Clearer:
@pytest.fixture
def standard_client():
    client = MagicMock()
    client.chat.completions.create.return_value = create_openai_chat_response(STANDARD_RESPONSE)
    return client
```
An expensive fixture that must be reused (scope="session"): Factories create new objects on every call. If the creation is expensive (e.g., initializing a real connection), a fixture with scope="module" or scope="session" is more efficient.
```
@pytest.fixture(scope="module")
def db_connection():
    conn = create_real_db_connection()  # Expensive
    yield conn
    conn.close()
```
The case is very specific and has no variations: If you only have one test that needs a response with a specific API error, it's clearer to write it directly in the test with with patch(...) than to create a factory for that single case.

Summary

Factory fixture = a fixture that returns a function that generates configurable objects
Eliminates the duplication of static fixtures with similar variations
Smart defaults: specify only what changes in each test
Combine with parametrize for massive coverage with little code
Specialized factories: for errors, chains, async — one per responsibility
When to use a static one: a single case, elevated scope, no variations needed

Additional resources

Factories as Fixtures — pytest docs — The official reference
pytest fixtures — scope — When to use function vs session scope
pytest.mark.parametrize — To combine with factories
Python functools.partial — A functional alternative to factories
Dependency Injection in pytest — How fixtures are DI
conftest.py — pytest docs — Sharing factories across modules
pytest-lazy-fixture — For parametrize with fixtures

← Previous5. Testing Parsers and Output Processors Next →7. Project: Prompt Contract Tests