1. Introduction: Unit Testing LLM Applications

Description

Module 1 configured pytest and established the fundamental mindset. Now we apply the core technique: making the non-deterministic deterministic through mocks, and treating prompts as contracts that can be validated. This module gives you the superpower of testing AI apps without calling the real LLM.

Context: Where are we?

Before getting into the technical detail, place yourself on the map of what you learned and what's coming:

Module 2 is the bridge between "I know how to configure pytest" and "I have real tests that protect my app". Here you build the foundation of your test suite: fast, deterministic, with no API cost.

The module's mental transformation

Before this module

Developer → "My app uses LLMs, it can't be tested well"
Developer → "The tests would be slow and expensive"
Developer → "The output varies, how do I make assertions?"
Developer → "I can only test manually"

After this module

Developer → "I mock the LLM → tests in milliseconds, no API calls"
Developer → "My prompt is a contract → I can validate it automatically"
Developer → "Parsers are deterministic → 100% coverage is easy"
Developer → "Snapshot testing detects regressions effortlessly"

This change in mindset is the module's most important outcome.

Central idea: Prompts as contracts

A prompt isn't just text — it's a behavioral contract:

"Given this input, the output will have this structure, meet these constraints, and respect this format."

If the contract is clear, it's testable. If you change the prompt and the contract breaks, the test fails. This idea transforms prompts from "text I modify and pray it works" to "a specification with automatic validation".

Implicit vs explicit contract

Most developers have implicit contracts in their heads. The goal is to make them explicit and testable:

Implicit contract (in your head):

"The LLM summarizes texts and returns something useful"

Explicit contract (testable):

"The output is JSON with:

• summary: a string of 10 to 500 characters
• confidence: a float between 0.0 and 1.0
• sources: a list of strings (can be empty)
• Valid format always, even if the input is short"

# The contract as code:
def test_summary_prompt_contract(mock_client):
    """The summary prompt meets the structure and constraints contract."""
    # Arrange
    text = "Python is an interpreted programming language."

    # Act
    result = summarize(text, client=mock_client)

    # Assert: structure
    assert isinstance(result, dict), "The result must be a dictionary"
    assert "summary" in result, "The 'summary' key must exist"
    assert "confidence" in result, "The 'confidence' key must exist"
    assert "sources" in result, "The 'sources' key must exist"

    # Assert: types
    assert isinstance(result["summary"], str)
    assert isinstance(result["confidence"], (int, float))
    assert isinstance(result["sources"], list)

    # Assert: constraints
    assert 10 <= len(result["summary"]) <= 500, \
        f"summary must be 10-500 chars, it has {len(result['summary'])}"
    assert 0.0 <= result["confidence"] <= 1.0, \
        f"confidence must be 0-1, it is {result['confidence']}"

Mocking as a superpower

The LLM is non-deterministic: same input → slightly different outputs. That makes testing hard. The solution: mock the LLM so it always returns what you define.

Why mocking wins

# Without a mock: slow, expensive, variable
def test_slow_and_costly():
    client = openai.OpenAI()                    # Real API
    result = summarize("text", client=client)   # ~2 seconds, ~$0.001
    assert "summary" in result                  # can fail due to variation
    # ❌ Slow (seconds), expensive, non-deterministic

# With a mock: fast, free, deterministic
def test_fast_and_free(mock_client):
    result = summarize("text", client=mock_client)  # <1ms, $0
    assert "summary" in result                       # always the same
    # ✅ Fast (<1ms), free, deterministic

Anatomy of a basic LLM mock

from unittest.mock import MagicMock
import pytest

@pytest.fixture
def mock_client():
    """OpenAI client mock with a realistic structure."""
    client = MagicMock()

    # Real structure of the OpenAI response
    mock_choice = MagicMock()
    mock_choice.message.content = '{"summary": "Test summary", "confidence": 0.9, "sources": []}'
    mock_choice.finish_reason = "stop"

    mock_response = MagicMock()
    mock_response.choices = [mock_choice]
    mock_response.usage.total_tokens = 150
    mock_response.model = "gpt-4o-mini"

    client.chat.completions.create.return_value = mock_response
    return client

def test_with_mock(mock_client):
    result = summarize("Input text", client=mock_client)
    assert result["summary"] == "Test summary"
    assert result["confidence"] == 0.9
    # The mock guarantees this always passes — no variation

The three measurable benefits

Speed:         Mock = 0.001s   |  Real LLM = 2-5s   → 2000-5000x faster
Cost:          Mock = $0       |  Real LLM = $0.001+ → saves thousands in CI
Determinism:   Mock = always the same | Real LLM = variable → 0 flaky tests

The 70/30 of testing in AI apps

Module 1 introduced this concept. In Module 2 you apply it:

70% of your code is deterministic:
  ├── Parsers (convert raw string → dict)
  ├── Validators (verify the output structure)
  ├── Formatters (prepare the input for the LLM)
  ├── Chains (orchestrate multiple calls)
  └── Error handlers (handle incorrect responses)

30% is non-deterministic:
  └── The LLM (black box, variable outputs)

Module 2 tests the 70% with mocks + deterministic tests. Module 3 tests the 30% with integration tests and semantic assertions.

What you'll achieve in this module

By the end of Module 2 you'll have:

Technical skills

• ✅ Mock LLM responses with unittest.mock and pytest-mock
• ✅ Create mocks with a realistic structure (choices, message, usage)
• ✅ Define specific, testable prompt contracts
• ✅ Write tests that validate contracts with precise assertions
• ✅ Implement snapshot testing to detect regressions
• ✅ Test parsers and output processors in isolation
• ✅ Create fixture factories for parametrized variations

The mini-project

A suite of Prompt Contract Tests for the Module 1 app that:

• Mocks all the LLM calls (0 real calls)
• Defines contracts for each of the app's prompts
• Validates structure, types, and constraints automatically
• Detects regressions with snapshots
• Runs the complete suite in < 10 seconds

Comparison: Module 1 vs Module 2

Mental architecture: what you mock and what you test

User input
       ↓
[Formatter] ← test directly (deterministic)
       ↓
[Prompt builder] ← test directly (deterministic)
       ↓
[LLM call] ← MOCK HERE
       ↓
[Raw response] ← mocked
       ↓
[Parser] ← test directly (deterministic)
       ↓
[Validator] ← test directly (deterministic)
       ↓
[Output processor] ← test directly (deterministic)
       ↓
Final output

Rule of thumb: You mock the LLM. You test everything else directly. Most bugs are in the "everything else" — and that's where it's easiest to write tests.

The difference between a Contract Test and Evaluation

This confusion comes up frequently. Definitive clarification:

Summary: A contract test verifies "does it have the correct form?". Evaluation verifies "is it good?". Both are necessary. This module covers contract tests; evaluation is another topic (Guide #12).

Module roadmap

Setup: what do you need from Module 1?

Before starting, verify that you have from Module 1:

# 1. Project structure
your-project/
├── src/
│   └── app/
│       ├── __init__.py
│       ├── config.py
│       ├── parsers.py
│       ├── sentiment.py
│       └── main.py
├── tests/
│   ├── conftest.py
│   ├── smoke/
│   └── unit/
├── pytest.ini
└── requirements.txt

# 2. pytest works
pytest --collect-only  # must show tests without error

# 3. Markers configured
pytest -m smoke  # must run smoke tests
pytest -m unit   # must run unit tests

If you have that, you're ready for Module 2. If not, go back to Module 1 and set up the base.

A note about the reference app

In Module 1 you built (or received) a sentiment analysis app. This is the base we'll use in Module 2 to build the prompt contract tests.

The app does:

1. Receives a text
2. Calls an LLM with a sentiment analysis prompt
3. Parses the LLM's JSON output
4. Returns {sentiment: str, score: float, explanation: str}

In Module 2 you will:

1. Mock the LLM call
2. Define the sentiment prompt's contract
3. Test the parser in isolation
4. Create a snapshot of the expected output
5. Use fixture factories to test edge cases

Exercises

Exercise 1: Identify your prompt's contract

Take a prompt you use at work or in a project. Write the contract in prose (not in code yet):

• What keys must the output have?
• What types are each value?
• What are the constraints (ranges, lengths, allowed values)?
• What should happen if the input is unusual (empty, very long, a different language)?

Exercise 2: Identify the deterministic 70%

For the following app, identify which parts are deterministic (mockable with a mock) and which are the LLM:

def classify_ticket(ticket_text: str) -> dict:
    # 1. Normalize the text
    clean_text = ticket_text.strip().lower()[:2000]

    # 2. Build the prompt
    prompt = f"Classify this ticket: {clean_text}\nCategory: urgent/normal/low"

    # 3. Call the LLM
    raw_response = call_llm(prompt)

    # 4. Parse the output
    category = parse_category(raw_response)

    # 5. Validate and structure
    return {
        "category": category,
        "original_length": len(ticket_text),
        "truncated": len(ticket_text) > 2000
    }

def test_classify_ticket(mock_llm):
    # mock_llm always returns "urgent"
    result = classify_ticket("System down in production")

    assert result["category"] in ["urgent", "normal", "low"]
    assert isinstance(result["original_length"], int)
    assert isinstance(result["truncated"], bool)

Exercise 3: Contract vs Evaluation

For each assertion, indicate whether it's a contract test or an evaluation metric:

1. assert "sentiment" in result
2. assert result["score"] >= 0 and result["score"] <= 1
3. assert result["explanation"].startswith("The text")
4. Verify that the summary captures the key points of the original
5. assert isinstance(result["tags"], list)
6. Verify that the sentiment classification is correct for negative texts

Exercise 4: Design the mock

For this function, design the mock you'd use for unit testing:

def analyze_sentiment(text: str, client: openai.OpenAI) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Analyze the sentiment."},
            {"role": "user", "content": text}
        ],
        response_format={"type": "json_object"}
    )
    raw = response.choices[0].message.content
    return parse_sentiment(raw)

What structure must the mock have? What should choices[0].message.content return?

from unittest.mock import MagicMock
import pytest

@pytest.fixture
def mock_openai_client():
    client = MagicMock()

    # Structure that exactly replicates the OpenAI API
    mock_choice = MagicMock()
    mock_choice.message.content = '{"sentiment": "positive", "score": 0.85, "explanation": "The text expresses satisfaction."}'
    mock_choice.finish_reason = "stop"

    mock_response = MagicMock()
    mock_response.choices = [mock_choice]
    mock_response.usage.prompt_tokens = 50
    mock_response.usage.completion_tokens = 30
    mock_response.usage.total_tokens = 80
    mock_response.model = "gpt-4o-mini"
    mock_response.id = "chatcmpl-test-id"

    client.chat.completions.create.return_value = mock_response
    return client

def test_analyze_sentiment(mock_openai_client):
    result = analyze_sentiment("The product is excellent", mock_openai_client)
    assert result["sentiment"] == "positive"
    assert result["score"] == 0.85
    assert isinstance(result["explanation"], str)

Exercise 5: When to use mocks vs a real LLM?

Classify each situation: mock or real LLM?

Summary

• Prompts as contracts: structure, constraints, format — all testable
• Mocking eliminates non-determinism in unit tests: fast, free, deterministic
• 70% of your code is deterministic — test it directly, without a mock
• Contract test ≠ Evaluation: one verifies form, the other verifies quality
• Most of your suite should use mocks; integration tests with a real LLM are a strategic complement
• Parsers and processors are the easiest, highest-ROI logic to test

Additional resources

1. unittest.mock — Official Python documentation — The foundation of mocking in Python
2. pytest-mock — pytest plugin for more ergonomic mocking
3. Pact — Contract Testing — Contract concepts (more oriented to microservices, but useful for understanding the philosophy)
4. OpenAI API Reference — Real structure of the responses for creating realistic mocks
5. Property-based testing with Hypothesis — Complementary technique (covered in M3)
6. Module 1: Testing Fundamentals — Prerequisite
7. Software Testing Anti-patterns — Common mistakes to avoid