4. Anatomy of the Test: Arrange-Act-Assert for LLM

Description

A well-structured test follows the Arrange-Act-Assert (AAA) pattern. For LLM apps, the "Arrange" phase has additional complexities: setting up LLM mocks, preparing fixtures with realistic responses, and making sure the mock is injected in the right place. If Arrange is wrong, the test can look like it passes but it isn't testing what you think.

This session goes deep into each phase of the AAA pattern adapted for LLM apps: how to do a robust Arrange with mocks, how to structure the Act to test exactly one thing, and how to write assertions specific to outputs that can vary. It also covers naming conventions for tests, which is more important than it seems when you have 200 tests and a failure at 3am.

By the end you'll know how to structure any test for an LLM app so it's readable, maintainable, and diagnoses exactly where the problem is when it fails.

The Arrange-Act-Assert pattern

Basic structure

def test_descriptive_name():
    # ─────────── ARRANGE ───────────
    # Prepare everything needed for the test
    # - Inputs
    # - Mocks and stubs
    # - Initial state

    # ─────────── ACT ───────────────
    # Execute EXACTLY one thing
    # - The function under test

    # ─────────── ASSERT ────────────
    # Verify the result
    # - One or several assertions about the same behavior

Why this separation matters:

• When a test fails, you immediately know which phase it failed in
• Easy to read and maintain (others can understand the test without comments)
• Prevents tests that do too much (hard to debug)

Basic example: deterministic function

# tests/unit/test_parsers.py
import pytest
from app.parsers import parse_json_from_llm_output


def test_parse_json_from_markdown_block():
    # ─── ARRANGE ───
    # Input: LLM response with JSON inside a markdown block
    raw_output = '```json\n{"sentiment": "positive", "confidence": 0.9}\n```'
    expected = {"sentiment": "positive", "confidence": 0.9}

    # ─── ACT ───
    # Run the parser
    result = parse_json_from_llm_output(raw_output)

    # ─── ASSERT ───
    # Verify that the output is exactly what's expected
    assert result == expected

This is the simplest case. The parser is 100% deterministic — you can use assert result == expected.

Arrange for LLM: setting up mocks

When the function under test calls the LLM, the Arrange includes setting up the mock. There are three ways to do it:

Option 1: @patch as a decorator

# tests/unit/test_sentiment.py
import pytest
from unittest.mock import patch, MagicMock
from app.sentiment import analyze_sentiment


@patch("app.sentiment.client.chat.completions.create")
def test_analyze_sentiment_positive(mock_create):
    """
    @patch intercepts the API call and returns whatever we define.
    mock_create is the argument the function receives — it's the object
    that replaces client.chat.completions.create during the test.
    """
    # ─── ARRANGE ───
    # Configure what the mock returns when it's called
    mock_response = MagicMock()
    mock_response.choices[0].message.content = (
        '{"sentiment": "positive", "confidence": 0.9}'
    )
    mock_create.return_value = mock_response

    # Test input
    text = "I absolutely love this product!"

    # ─── ACT ───
    result = analyze_sentiment(text)

    # ─── ASSERT ───
    assert result["sentiment"] == "positive"
    assert result["confidence"] == 0.9
    # We can also verify that the LLM was called
    mock_create.assert_called_once()

Option 2: with patch as a context manager

def test_analyze_sentiment_with_context_manager():
    """
    Useful when you need multiple patches or the test is very short.
    """
    # ─── ARRANGE ───
    mock_response = MagicMock()
    mock_response.choices[0].message.content = (
        '{"sentiment": "negative", "confidence": 0.8}'
    )

    with patch("app.sentiment.client.chat.completions.create") as mock_create:
        mock_create.return_value = mock_response

        # ─── ACT ───
        result = analyze_sentiment("This is terrible.")

    # ─── ASSERT ─── (outside the with — the patch no longer applies)
    assert result["sentiment"] == "negative"
    assert 0.0 <= result["confidence"] <= 1.0

Option 3: conftest fixture

# tests/conftest.py (already configured in session 3)
# The mock_openai_client fixture is already available

def test_analyze_sentiment_with_fixture(mock_openai_client, mock_openai_response):
    """
    Uses the conftest fixture — cleaner when many tests
    need the same type of mock.
    """
    # ─── ARRANGE ───
    mock_openai_client.chat.completions.create.return_value = mock_openai_response(
        '{"sentiment": "neutral", "confidence": 0.5}'
    )
    # Inject the mocked client into the function
    # (requires analyze_sentiment to accept client as an argument or
    #  that you use patch to replace the global client)

    # ─── ACT ───
    with patch("app.sentiment.client", mock_openai_client):
        result = analyze_sentiment("The package arrived.")

    # ─── ASSERT ───
    assert result["sentiment"] == "neutral"

Comparison of the three options

Rule of thumb: Use a fixture when >3 tests use the same mock. Use @patch for mocks unique to a specific test.

Act: test exactly one thing

The principle

# ❌ BAD: multiple Acts in one test
def test_pipeline_bad():
    # Act 1
    validated = validate_input("text")
    assert validated is not None

    # Act 2 (should be a separate test)
    result = summarize(validated)
    assert "summary" in result

    # Act 3 (should be another separate test)
    formatted = format_for_user(result)
    assert len(formatted) > 0
    # Which part failed? validate_input? summarize? format_for_user?

# ✅ GOOD: one Act per test
def test_validate_input_returns_stripped_text():
    result = validate_input("  text with spaces  ")
    assert result == "text with spaces"

def test_summarize_returns_summary_key():
    with patch("app.llm.client.chat.completions.create") as mock:
        mock.return_value = create_mock_response('{"summary": "summary", "points": []}')
        result = summarize("text")
    assert "summary" in result

def test_format_for_user_includes_header():
    result = format_for_user({"summary": "summary", "points": []})
    assert result.startswith("## Summary")

When test_summarize_returns_summary_key fails, you know exactly where the problem is: in the summarize function, in the integration between the mock and the response parser.

Valid exceptions: multiple assertions in one Act

A test can (and should) have multiple assertions if they all verify the same behavior:

# ✅ ACCEPTABLE: multiple assertions about the same result
def test_sentiment_response_structure():
    with patch("app.sentiment.client.chat.completions.create") as mock:
        mock.return_value = create_mock_response(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("I love this!")

    # All these assertions verify "the structure of the result is correct"
    # They're aspects of the same behavior, not different behaviors
    assert isinstance(result, dict), f"Expected dict, got {type(result)}"
    assert "sentiment" in result, "Missing 'sentiment' key"
    assert "confidence" in result, "Missing 'confidence' key"
    assert result["sentiment"] in ["positive", "negative", "neutral"]
    assert isinstance(result["confidence"], float)
    assert 0.0 <= result["confidence"] <= 1.0

Key difference: Multiple assertions about the same result → ✅. Multiple calls to different functions → ❌.

Assert: specific and useful when they fail

The problem with vague assertions

# ❌ BAD: vague, when it fails you don't know what you expected
assert result
assert result["confidence"]
assert result["items"]

# ✅ GOOD: specific, informative error messages
assert result is not None, "analyze_sentiment returned None"
assert result.get("confidence") is not None, "Missing 'confidence' key"
assert len(result.get("items", [])) > 0, "'items' list is empty"

Assertions with custom messages

@patch("app.sentiment.client.chat.completions.create")
def test_sentiment_confidence_range(mock_create):
    mock_create.return_value = create_mock_response(
        '{"sentiment": "positive", "confidence": 1.5}'  # invalid confidence
    )

    result = analyze_sentiment("text")

    # The message is shown when the assert fails
    assert 0.0 <= result["confidence"] <= 1.0, (
        f"Confidence out of range [0,1]: {result['confidence']}\n"
        f"Full result: {result}"
    )

Assertions for types

def test_response_types():
    with patch(...) as mock:
        mock.return_value = create_mock_response(
            '{"sentiment": "positive", "confidence": 0.9, "aspects": ["price", "quality"]}'
        )
        result = analyze_sentiment("text")

    # Verify types explicitly
    assert isinstance(result, dict), f"Expected dict, got: {type(result).__name__}"
    assert isinstance(result["sentiment"], str), f"sentiment must be str"
    assert isinstance(result["confidence"], (int, float)), "confidence must be numeric"
    assert isinstance(result["aspects"], list), "aspects must be a list"
    assert all(isinstance(a, str) for a in result["aspects"]), "All aspects must be str"

Assertions for non-deterministic outputs

When the test uses the real LLM (not mocked), the assertions must be about invariant properties:

@pytest.mark.integration
def test_sentiment_real_llm_properties():
    """
    Integration test that calls the real API.
    Assert about invariant properties (not the exact value).
    """
    result = analyze_sentiment("I love this product!")

    # Properties that must ALWAYS hold, regardless of the output:
    assert isinstance(result, dict), "Result must be a dict"
    assert "sentiment" in result, "Must have 'sentiment' key"
    assert result["sentiment"] in ["positive", "negative", "neutral"], (
        f"Unexpected value: {result['sentiment']}"
    )
    assert 0.0 <= result.get("confidence", -1) <= 1.0, (
        f"Confidence out of range: {result.get('confidence')}"
    )
    # For clearly positive text, we can be more specific:
    assert result["sentiment"] == "positive", (
        f"For unambiguously positive text we expected 'positive', "
        f"got: {result['sentiment']}"
    )

Naming conventions: names that diagnose

The test name is the first piece of information you see when it fails. A good name makes the failure self-explanatory.

Recommended pattern: test_[function]_[condition]_[expected_result]

# ✅ GOOD names — the failure is self-explanatory:
def test_parse_json_with_markdown_block_returns_dict():
    pass

def test_analyze_sentiment_with_empty_text_raises_value_error():
    pass

def test_summarize_with_valid_input_returns_three_points():
    pass

def test_validate_input_with_text_over_limit_raises_value_error():
    pass

# ❌ BAD names — when they fail, you don't know what happened:
def test_parse():
    pass

def test_it_works():
    pass

def test_error():
    pass

def test_sentiment_1():
    pass

Names for parametrized tests

@pytest.mark.parametrize(
    "raw_input,expected",
    [
        pytest.param('{"x": 1}', {"x": 1}, id="pure_json"),
        pytest.param('```json\n{"x": 1}\n```', {"x": 1}, id="markdown_block"),
        pytest.param('Result: {"x": 1}', {"x": 1}, id="with_prefix_text"),
    ]
)
def test_parse_json_from_llm_output(raw_input, expected):
    """
    Using pytest.param with id= makes the test name descriptive:
    test_parse_json_from_llm_output[pure_json]
    test_parse_json_from_llm_output[markdown_block]
    test_parse_json_from_llm_output[with_prefix_text]
    """
    assert parse_json_from_llm_output(raw_input) == expected

Names for test classes

class TestAnalyzeSentiment:
    """Groups all the tests for the analyze_sentiment function."""

    class TestHappyPath:
        """Tests of the normal flow (valid inputs)."""

        def test_positive_text_returns_positive_sentiment(self):
            ...

        def test_negative_text_returns_negative_sentiment(self):
            ...

    class TestEdgeCases:
        """Tests of edge cases."""

        def test_empty_text_raises_value_error(self):
            ...

        def test_very_long_text_is_truncated(self):
            ...

    class TestMocking:
        """Tests that verify the integration with the LLM mock."""

        def test_calls_llm_exactly_once(self):
            ...

        def test_passes_text_in_prompt(self):
            ...

Complete test: integrated example

Here's a complete, well-structured test for a sentiment analysis function:

# tests/unit/test_sentiment_analysis.py
"""
Unit tests for app.sentiment.analyze_sentiment.
All tests use mocks — no real API calls.
"""
import pytest
from unittest.mock import patch, MagicMock, call
from app.sentiment import analyze_sentiment, SentimentResult


def create_sentiment_mock(sentiment: str, confidence: float) -> MagicMock:
    """Helper to create mocks with the real OpenAI API structure."""
    response = MagicMock()
    response.choices[0].message.content = (
        f'{{"sentiment": "{sentiment}", "confidence": {confidence}}}'
    )
    response.usage.total_tokens = 150
    return response


@pytest.mark.unit
class TestAnalyzeSentiment:
    """Tests for the main sentiment analysis function."""

    @patch("app.sentiment.client.chat.completions.create")
    def test_positive_text_returns_positive_sentiment(self, mock_create):
        """Analysis of clearly positive text returns sentiment=positive."""
        # ─── ARRANGE ───
        mock_create.return_value = create_sentiment_mock("positive", 0.9)
        text = "I absolutely love this product!"

        # ─── ACT ───
        result = analyze_sentiment(text)

        # ─── ASSERT ───
        assert result["sentiment"] == "positive", (
            f"Positive text should return 'positive', got: {result['sentiment']}"
        )

    @patch("app.sentiment.client.chat.completions.create")
    def test_returns_required_keys(self, mock_create):
        """The result always has the keys 'sentiment' and 'confidence'."""
        # ─── ARRANGE ───
        mock_create.return_value = create_sentiment_mock("neutral", 0.5)

        # ─── ACT ───
        result = analyze_sentiment("Some text")

        # ─── ASSERT ───
        assert "sentiment" in result, "Missing 'sentiment' key"
        assert "confidence" in result, "Missing 'confidence' key"

    @patch("app.sentiment.client.chat.completions.create")
    def test_confidence_is_float_in_valid_range(self, mock_create):
        """Confidence is a float in the range [0.0, 1.0]."""
        # ─── ARRANGE ───
        mock_create.return_value = create_sentiment_mock("positive", 0.85)

        # ─── ACT ───
        result = analyze_sentiment("Great!")

        # ─── ASSERT ───
        assert isinstance(result["confidence"], (int, float)), (
            f"confidence must be numeric, got: {type(result['confidence'])}"
        )
        assert 0.0 <= result["confidence"] <= 1.0, (
            f"confidence out of range: {result['confidence']}"
        )

    @patch("app.sentiment.client.chat.completions.create")
    def test_calls_llm_exactly_once_per_analysis(self, mock_create):
        """The function calls the LLM exactly once per analysis."""
        # ─── ARRANGE ───
        mock_create.return_value = create_sentiment_mock("positive", 0.9)

        # ─── ACT ───
        analyze_sentiment("Some text")

        # ─── ASSERT ───
        mock_create.assert_called_once()

    @patch("app.sentiment.client.chat.completions.create")
    def test_passes_text_in_the_prompt(self, mock_create):
        """The analysis text is included in the prompt sent to the LLM."""
        # ─── ARRANGE ───
        mock_create.return_value = create_sentiment_mock("positive", 0.9)
        text = "unique_test_text_12345"

        # ─── ACT ───
        analyze_sentiment(text)

        # ─── ASSERT ───
        # Verify that the text was passed to the LLM
        call_args = mock_create.call_args
        messages = call_args.kwargs.get("messages") or call_args.args[0]
        # The text must appear in one of the messages
        all_content = " ".join(
            m.get("content", "") for m in messages
            if isinstance(m, dict)
        )
        assert text in all_content, (
            f"The text '{text}' doesn't appear in the prompt sent to the LLM"
        )


@pytest.mark.unit
class TestAnalyzeSentimentEdgeCases:
    """Edge case tests for analyze_sentiment."""

    def test_empty_text_raises_value_error(self):
        """Empty text raises ValueError before calling the LLM."""
        # ─── ARRANGE / ACT / ASSERT combined (exception test)
        with pytest.raises(ValueError, match="empty"):
            analyze_sentiment("")

    def test_none_text_raises_type_error(self):
        """None as input raises TypeError."""
        with pytest.raises((TypeError, ValueError)):
            analyze_sentiment(None)

    @patch("app.sentiment.client.chat.completions.create")
    def test_handles_malformed_json_from_llm(self, mock_create):
        """If the LLM returns malformed JSON, the function handles the error."""
        # ─── ARRANGE ───
        mock_create.return_value = create_sentiment_mock.__wrapped__ if hasattr(
            create_sentiment_mock, '__wrapped__') else MagicMock()
        bad_response = MagicMock()
        bad_response.choices[0].message.content = "This is not JSON at all"
        mock_create.return_value = bad_response

        # ─── ACT / ASSERT ───
        # The function must handle the error gracefully
        # (raise a controlled exception, not crash with AttributeError)
        with pytest.raises((ValueError, KeyError)):
            analyze_sentiment("Some text")

Verifying interactions with the LLM

Tests don't only verify the result — they can also verify how the LLM was called:

@patch("app.sentiment.client.chat.completions.create")
def test_uses_correct_model(mock_create):
    """The function uses the correct model."""
    mock_create.return_value = create_sentiment_mock("positive", 0.9)

    analyze_sentiment("text")

    # Verify the arguments the LLM was called with
    call_kwargs = mock_create.call_args.kwargs
    assert call_kwargs.get("model") == "gpt-4o-mini", (
        f"Expected gpt-4o-mini, got: {call_kwargs.get('model')}"
    )


@patch("app.sentiment.client.chat.completions.create")
def test_uses_low_temperature_for_consistency(mock_create):
    """Temperature must be low for greater consistency in analysis."""
    mock_create.return_value = create_sentiment_mock("positive", 0.9)

    analyze_sentiment("text")

    call_kwargs = mock_create.call_args.kwargs
    temperature = call_kwargs.get("temperature", 1.0)
    assert temperature <= 0.3, (
        f"Temperature should be ≤0.3 for consistency, got: {temperature}"
    )


@patch("app.sentiment.client.chat.completions.create")
def test_does_not_call_llm_for_empty_input(mock_create):
    """If the input is invalid, the LLM should NOT be called (cost saving)."""
    with pytest.raises(ValueError):
        analyze_sentiment("")

    mock_create.assert_not_called()

Comparison: well vs poorly structured test

Troubleshooting

Problem: The test passes but when I run the real code it fails. Cause: The mock doesn't reflect the real API structure. The code does response.choices[0].message.content but the mock returns a dict instead of an object with attributes. Solution: Use MagicMock() and configure the attributes: mock.choices[0].message.content = "...". Don't use plain dicts as API mocks.

Problem: AssertionError but I don't know what value result had. Solution: Add the value to the assertion message:

assert "sentiment" in result, f"Full result: {result}"

Problem: The test fails with AttributeError: 'MagicMock' object has no attribute 'content'. Solution: MagicMock() creates attributes automatically but chained accesses to lists require explicit configuration:

mock_response.choices = [MagicMock()]          # Real list
mock_response.choices[0].message.content = "..." # Configure the attribute

Problem: assert_called_once() fails even though the code visibly calls the function. Cause: You're verifying a different mock than the one used. The @patch path doesn't match where the function is used. Solution: The path must be "module_that_uses.attribute". If app.sentiment imports with from openai import OpenAI; client = OpenAI(), the patch is @patch("app.sentiment.client.chat.completions.create").

Problem: Slow tests due to mocks that take time to initialize. Solution: MagicMock() is instantaneous. If the tests are slow, there's probably a real un-mocked LLM call. Use pytest -s to see the output and detect unexpected calls.

Exercises

Exercise 1: Identify AAA

In this test, clearly identify where Arrange ends, where Act is, and where the Asserts are:

def test_parse_summary():
    import json
    raw = '```json\n{"summary": "Short text", "points": ["a", "b", "c"]}\n```'
    result = parse_summary_response(raw)
    assert isinstance(result, dict)
    assert result["summary"] == "Short text"
    assert len(result["points"]) == 3

def test_parse_summary():
    # ─── ARRANGE ───────────────────────────────────────────
    import json
    raw = '```json\n{"summary": "Short text", "points": ["a", "b", "c"]}\n```'
    # No mock because parse_summary_response is deterministic

    # ─── ACT ────────────────────────────────────────────────
    result = parse_summary_response(raw)

    # ─── ASSERT ─────────────────────────────────────────────
    assert isinstance(result, dict)
    assert result["summary"] == "Short text"
    assert len(result["points"]) == 3

Exercise 2: Rewrite with AAA and specific assertions

Refactor this test so it's more readable and the assertions are more informative:

def test_x():
    r = summarize("text")
    assert r
    assert r.get("s")

@patch("app.summarizer.client.chat.completions.create")
def test_summarize_returns_non_empty_summary(mock_create):
    # ─── ARRANGE ───
    mock_response = MagicMock()
    mock_response.choices[0].message.content = (
        '{"summary": "Summary of the text", "points": ["Point 1", "Point 2", "Point 3"]}'
    )
    mock_create.return_value = mock_response
    input_text = "test text to summarize"

    # ─── ACT ───
    result = summarize(input_text)

    # ─── ASSERT ───
    assert result is not None, "summarize must not return None"
    assert "summary" in result, f"Missing 'summary' key. Result: {result}"
    assert len(result["summary"]) > 0, "The summary must not be empty"

Exercise 3: Realistic OpenAI mock

Create a response mock that exactly replicates the real structure of OpenAI's ChatCompletion object (including usage, model, id, and choices[0].finish_reason).

from unittest.mock import MagicMock

def create_realistic_openai_mock(content: str, tokens: int = 150) -> MagicMock:
    """
    Mock that replicates the exact structure of OpenAI's ChatCompletion.
    Based on the real API structure (verify with the OpenAI docs).
    """
    response = MagicMock()

    # Response metadata
    response.id = "chatcmpl-test-abc123"
    response.model = "gpt-4o-mini"
    response.object = "chat.completion"

    # The main Choice
    choice = MagicMock()
    choice.index = 0
    choice.finish_reason = "stop"  # Or "length" if it was truncated
    choice.message.role = "assistant"
    choice.message.content = content
    response.choices = [choice]

    # Token usage (important for cost tracking)
    response.usage.prompt_tokens = tokens // 3
    response.usage.completion_tokens = tokens * 2 // 3
    response.usage.total_tokens = tokens

    return response

# Verify that it works the same as the real API:
# result = response.choices[0].message.content  ← identical to the real API

Exercise 4: Assertions for invariant properties

For a function extract_key_entities(text) -> list[dict] that extracts entities from text with the LLM, write 5 assertions that verify invariant properties of the output (without comparing the exact content of the entities).

@patch("app.entities.client.chat.completions.create")
def test_extract_key_entities_structure(mock_create):
    # ─── ARRANGE ───
    mock_create.return_value = create_realistic_openai_mock(
        '[{"name": "Apple", "type": "company"}, {"name": "Tim Cook", "type": "person"}]'
    )

    # ─── ACT ───
    entities = extract_key_entities("Apple CEO Tim Cook announced new products.")

    # ─── ASSERT — 5 invariant properties ───
    # 1. The result is a list
    assert isinstance(entities, list), f"Expected list, got {type(entities)}"

    # 2. The list isn't empty for text with obvious entities
    assert len(entities) > 0, "The entities list must not be empty"

    # 3. Each entity is a dict
    for entity in entities:
        assert isinstance(entity, dict), f"Each entity must be a dict: {entity}"

    # 4. Each entity has the required keys
    required_keys = {"name", "type"}
    for entity in entities:
        assert required_keys.issubset(entity.keys()), (
            f"Entity missing keys {required_keys - entity.keys()}: {entity}"
        )

    # 5. The 'name' values aren't empty
    for entity in entities:
        assert entity["name"].strip(), f"Empty entity name: {entity}"

Exercise 5: Verify that the LLM isn't called in error cases

Write a test that verifies analyze_sentiment("") doesn't call the LLM (to verify that input validation happens BEFORE the expensive API call).

@patch("app.sentiment.client.chat.completions.create")
def test_empty_text_does_not_call_llm(mock_create):
    """
    If the input is invalid, the LLM must NOT be called.
    This is important: calling the LLM with invalid input
    wastes money and doesn't produce useful results.
    """
    # ─── ARRANGE ───
    # We don't configure return_value because we do NOT expect it to be called

    # ─── ACT / ASSERT ───
    with pytest.raises(ValueError):
        analyze_sentiment("")  # Must fail on validation, not on the LLM

    # Verify that the LLM was never called
    mock_create.assert_not_called(), (
        "The LLM should not be called if the input is invalid"
    )

Summary

• Arrange for LLM: configure the mock BEFORE the Act, replicate the real API structure with MagicMock()
• Act is a single thing: the function under test. Multiple Acts → multiple tests
• Assert specifically: informative messages + verify types + verify ranges
• For non-deterministic outputs: assertions about invariant properties, not exact values
• Descriptive names: test_[function]_[condition]_[expected_result] — when it fails, the name diagnoses
• Verify not only the result, but also the interactions: assert_called_once(), assert_called_with(), assert_not_called()

Additional resources

1. Arrange-Act-Assert (Python Testing with pytest) — Reference book on testing in Python
2. pytest assert introspection — How pytest improves asserts with context information
3. unittest.mock — Mock objects — Official documentation for MagicMock and its verification methods
4. Where to patch (Python docs) — Critical guide on the correct path for @patch
5. Test Naming Best Practices — Roy Osherove on naming conventions
6. pytest parametrize with ids — How to make descriptive names in parametrized tests