8. Module 2 Summary and Troubleshooting

Description

Wrap-up for Module 2: common errors with their solutions, a closing checklist, a summary table of key concepts, and preparation for Module 3 (Integration Testing). If something went wrong during the project, this capsule has the answer. If everything went well, use it to consolidate what you learned before moving on.

The 8 most common Module 2 errors

Error 1: A trivial mock that doesn't reflect the real API

Symptom:

# Your mock:
client.chat.completions.create.return_value = "Hello"

# Error in production:
AttributeError: 'str' object has no attribute 'choices'
# Or worse: the test passes but production fails with an unexpected structure

Cause: The mock returns a simple string instead of an object that replicates the OpenAI API structure.

Solution:

# Always use create_openai_chat_response:
from tests.helpers import create_openai_chat_response

client.chat.completions.create.return_value = create_openai_chat_response(
    '{"sentiment": "positive", "score": 0.9}'
)

# Verify that your mock has the same structure as the real object:
# response.choices[0].message.content → string
# response.choices[0].finish_reason   → "stop" / "length"
# response.usage.total_tokens         → int

Rule: If your code accesses response.X.Y.Z, your mock must have response.X.Y.Z configured. MagicMock() creates attributes on access, but if the code performs operations on them (like len(), in, indexing), you need real values.

Error 2: Patch in the wrong place

Symptom:

@patch("openai.OpenAI")
def test_something(mock_openai):
    result = analyze_sentiment("text")
    # The mock is NOT applied — the function uses the real client
    # The tests pass but it makes calls to the real API

Cause: You patch where the object is DEFINED, not where it's USED.

Solution:

# Identify where it's imported in your module:

# If app/sentiment.py has: import openai; client = openai.OpenAI()
@patch("app.sentiment.openai.OpenAI")   # ← Patch where it's used

# If app/sentiment.py has: from openai import OpenAI; client = OpenAI()
@patch("app.sentiment.OpenAI")          # ← Patch where it's imported

# If app/sentiment.py has: client = openai.OpenAI() at module level
@patch("app.sentiment.client")          # ← Patch the object directly

# The cleanest way: dependency injection
def analyze_sentiment(text: str, client=None):
    if client is None:
        client = openai.OpenAI()
    # Now you don't need patch — you pass the mock directly in tests

Golden rule: Read the first line of app/sentiment.py. How does it import the client? That determines the patch path.

Error 3: Vague contracts that protect nothing

Symptom:

def test_contract():
    result = analyze_sentiment("text", client=mock_client)
    assert result is not None  # ← This "contract" protects nothing
    assert "sentiment" in result  # ← Minimally useful, but insufficient

Cause: The contract verifies that the result exists but not that it meets the prompt's specifications.

Solution:

def test_contract_complete():
    result = analyze_sentiment("text", client=mock_client)
    
    # Structure
    assert "sentiment" in result
    assert "score" in result
    assert "keywords" in result
    assert "explanation" in result
    
    # Types
    assert isinstance(result["sentiment"], str)
    assert isinstance(result["score"], (int, float))
    assert isinstance(result["keywords"], list)
    
    # Constraints (this is where the contract's real value is)
    assert result["sentiment"] in ["positive", "negative", "neutral"]
    assert 0.0 <= result["score"] <= 1.0
    assert len(result["explanation"]) <= 500
    assert all(isinstance(k, str) for k in result["keywords"])

Rule: A contract without constraints is just a "structure smoke test". The real value is in the constraints: ranges, allowed values, lengths.

Error 4: Snapshot updated without reviewing the diff

Symptom:

# The test failed, so you update:
pytest --snapshot-update  # For all snapshots

# A week later, the bug is in production
# The snapshot hid the regression

Cause: Updating snapshots in batch without reviewing each diff.

Solution:

# Step 1: See what failed
pytest tests/test_snapshots.py -v

# Step 2: See the exact diff of the snapshot that failed
# (the test output shows the diff)

# Step 3: Analyze the diff
# Is the change intentional? → Update that specific snapshot
# Is it a bug? → Fix the code

# Step 4: If it's intentional, update only that snapshot
pytest tests/test_snapshots.py::test_specific -v --snapshot-update
git diff tests/__snapshots__/  # Review exactly what changed

Rule: Never pytest --snapshot-update in batch. Always one by one, always with a diff review.

Error 5: A parser test that depends on the LLM

Symptom:

def test_parser():
    # Calls the real LLM to test the parser
    client = openai.OpenAI()
    response = client.chat.completions.create(...)
    raw = response.choices[0].message.content
    
    result = parse_json_response(raw)
    assert "sentiment" in result
    # This test: is slow, costly, and can fail because of the LLM

Cause: Confusion between testing the parser and testing the LLM. They're different responsibilities.

Solution:

# Test the parser directly with controlled inputs
@pytest.mark.parametrize("raw,expected", [
    ('{"sentiment": "positive"}', {"sentiment": "positive"}),
    ('```json\n{"sentiment": "negative"}\n```', {"sentiment": "negative"}),
    ('The analysis: {"sentiment": "neutral"}', {"sentiment": "neutral"}),
])
def test_parser_isolated(raw, expected):
    # No LLM, no mocks: the parser receives strings directly
    result = parse_json_response(raw)
    assert result == expected

Rule: Parsers are 100% deterministic. Test them with direct strings, not with LLM outputs.

Error 6: A factory fixture that mixes responsibilities

Symptom:

@pytest.fixture
def super_factory():
    def _create(
        sentiment=None,
        summary=None,
        classification=None,
        error_type=None,
        async_mode=False,
        stream=False,
        # ... 15 more parameters
    ):
        # 100 lines of code...
    return _create

Cause: A factory that tries to handle every possible case becomes impossible to understand and maintain.

Solution:

# Small, specific factories:
@pytest.fixture
def make_sentiment_client():
    """Only for sentiment responses."""
    def _create(sentiment="neutral", score=0.5, **kwargs):
        ...
    return _create

@pytest.fixture
def make_error_client():
    """Only for simulating LLM errors."""
    def _create(error_type="rate_limit"):
        ...
    return _create

@pytest.fixture
def make_summary_client():
    """Only for summary responses."""
    def _create(summary="Test summary", confidence=0.8):
        ...
    return _create

Rule: One factory, one responsibility. If you need to combine, use multiple factories in the test.

Error 7: Not verifying that the LLM was called correctly

Symptom:

def test_sentiment():
    result = analyze_sentiment("text", client=mock_client)
    assert result["sentiment"] == "positive"
    # Passed — but was the LLM called with the correct prompt?
    # Was the text passed in the message? Was temperature=0 used?

Cause: The tests verify the output but not the interaction with the LLM.

Solution:

def test_sentiment_verifies_call(mock_openai_client):
    result = analyze_sentiment("Important text", client=mock_openai_client)
    
    # Verify it was called
    mock_openai_client.chat.completions.create.assert_called_once()
    
    # Verify the arguments
    call_kwargs = mock_openai_client.chat.completions.create.call_args.kwargs
    
    assert call_kwargs["model"] == "gpt-4o-mini"
    assert call_kwargs["temperature"] == 0.0
    
    messages = call_kwargs["messages"]
    user_message = next(m for m in messages if m["role"] == "user")
    assert "Important text" in user_message["content"]

Rule: For critical logic, verify both the output and the interactions with the mock (assert_called_once, call_args).

Error 8: Forgotten AsyncMock for async functions

Symptom:

async def analyze_async(text, client):
    response = await client.chat.completions.create(...)  # ← It's await

# In the test:
def test_analyze_async():
    client = MagicMock()
    client.chat.completions.create.return_value = ...  # ← Normal MagicMock

# Error: TypeError: object MagicMock can't be used in 'await' expression

Cause: MagicMock is not awaitable. For async functions, you need AsyncMock.

Solution:

from unittest.mock import AsyncMock, MagicMock

@pytest.mark.asyncio
async def test_analyze_async_correct():
    client = MagicMock()
    # AsyncMock for the function that will be awaited
    client.chat.completions.create = AsyncMock(
        return_value=create_openai_chat_response('{"sentiment": "positive", "score": 0.9}')
    )
    
    result = await analyze_async("text", client=client)
    
    assert result["sentiment"] == "positive"
    client.chat.completions.create.assert_called_once()

Rule: If your function does await something(), the mock for something must be AsyncMock, not MagicMock.

Quick diagnosis: decision tree

Test fails with AttributeError on the mock
  → Verify that create_openai_chat_response is configured correctly
  → Make sure choices[0].message.content is defined

Test doesn't use the mock (calls the real LLM)
  → Patch in the wrong place
  → Verify the path: patch("module.where.it.is.used.name")

Test passes but coverage is low
  → Missing parametrize for multiple formats
  → Missing edge case tests (empty, malformed, error)

Contract test passes with a mock but fails with the real LLM
  → The mock doesn't reflect the LLM's real variability
  → Capture a real output and use it as a test case
  → Adjust the contract or the parser to be more flexible

Snapshot updates on its own in CI
  → NEVER update snapshots automatically in CI
  → The snapshot failed for a reason — investigate before updating

Factory too complex and impossible to understand
  → Split it into smaller, more specific factories
  → One factory, one responsibility

AsyncMock error
  → Use AsyncMock for async functions, MagicMock for sync

Module 2 closing checklist

Before moving on to Module 3, verify that you have everything:

Tests written

• All the app's prompts have at least one contract test
• The contract includes: structure + types + constraints
• The parsers are tested with 5+ formats (including markdown, preceding text)
• The output processors have tests for: normal values, out of range, missing fields
• Error handling is covered: rate limit, timeout, empty response, malformed JSON
• Regression tests for the 3 main cases (positive, negative, neutral)

Test quality

• The mocks use create_openai_chat_response (realistic structure)
• The patch is in the right place (where it's used, not where it's defined)
• The contracts have specific constraints (not just "key exists")
• The snapshots have documentation on when to update

Performance

• pytest -m unit finishes in under 10 seconds
• 0 calls to the real OpenAI API in unit tests
• pytest --cov=app.parsers shows >90% coverage

Organization

• Factories in conftest.py (not duplicated in each test)
• Contract tests in tests/unit/contracts/
• Parser tests in tests/unit/parsers/
• Regression tests in tests/unit/regression/

Module concepts summary

Module success metrics

When you complete Module 2 correctly, you should see:

# Expected result at the end of Module 2:
$ pytest tests/unit/ -v --tb=short

=== 50+ passed in 3.2s ===

$ pytest --cov=app --cov-report=term-missing tests/unit/
Name                    Stmts   Miss  Cover
-------------------------------------------
app/parsers.py             45      2    96%
app/processors.py          38      1    97%
app/sentiment.py           52     12    77%
-------------------------------------------
TOTAL                     135     15    89%

$ pytest -m unit --tb=short  # Only unit tests
=== 50+ passed in 3.1s ===

If your numbers are similar, you completed the module correctly.

Next module: Integration Testing

In Module 3 you'll face the reality that mocks avoid:

The Module 3 challenge

Unit tests (M2):     Mock LLM → deterministic output → easy to test
Integration tests (M3):  Real LLM → variable output → requires new strategies

What you'll learn in Module 3

1. Semantic similarity assertions: Instead of assert result == "exact", use semantic similarity: assert similarity(result, expected) > 0.8. When meaning matters more than the exact words.

2. Property-based testing with Hypothesis: Define properties that must always hold (invariants) and let Hypothesis generate the inputs. E.g.: "For any input text, the score is always 0-1".

3. Flaky test management: Strategies for real-LLM tests that sometimes fail: retry logic, tolerance in assertions, categorizing as "flaky" vs "real failure".

4. Budget controls: How to test with the real LLM without spending a fortune. Token limits, caching responses in tests, when the real LLM is necessary vs when the mock is enough.

5. Decision framework: When to use a mock? When to use a local model (Ollama)? When to use the real LLM? The decision framework based on the test type and the development stage.

The key transition

Module 2: "My app is tested — all tests pass with mocks"
          ↓
Module 3: "But does it work with the real LLM? What happens when the model
           produces variations? How do I test semantic quality?"

Final module exercises

Exercise 1: Error diagnosis

Analyze this test that always passes but has a bug in the mock. Identify the problem:

def test_sentiment_result(mocker):
    mock_create = mocker.patch("app.sentiment.client.chat.completions.create")
    
    mock_response = MagicMock()
    mock_response.choices[0].message.content = {"sentiment": "positive", "score": 0.9}
    mock_create.return_value = mock_response
    
    result = analyze_sentiment("Positive text")
    
    assert result["sentiment"] == "positive"  # ← Passes

mock_response.choices[0].message.content = '{"sentiment": "positive", "score": 0.9}'
# ← JSON string, not a dict

Exercise 2: A contract that got broken

A developer changed the sentiment prompt to add an "urgency" field. The prompt contract now needs to be updated. Describe the steps:

Exercise 3: Module 2 ROI

Calculate the approximate savings if you have 50 unit tests that used to call the real LLM and now use mocks. Assume:

• Each test made 1 LLM call
• Each call costs $0.001 (gpt-4o-mini)
• The tests run 10 times a day (CI + local)
• You work 20 days a month

Without mocks: 50 tests × 1 call × $0.001 × 10 runs × 20 days = $10/month

With mocks: $0.00/month

Savings: $10/month × 12 months = $120/year on this project alone

But the real savings include:
- Waiting time: 50 tests × 2s × 10 runs × 20 days = 20,000s ≈ 5.6 hours/month
- If you run the tests 100 times a day: 200,000s ≈ 56 hours/month saved
- Not to mention tests you didn't run because of the cost → undetected bugs → more expensive

Conclusion: The cost of mocking is 0 — the investment is writing the tests well.

Exercise 4: Getting ready for Module 3

Before starting Module 3, identify in your app:

1. Which parts really need the real LLM to be tested?
2. Which of the tests you wrote in M2 gave you the most confidence?
3. What's a case where a mock would NOT be enough?

Additional resources

1. pytest-mock — Documentation — pytest plugin for cleaner mocking
2. Pydantic v2 — Validators — For executable contracts
3. unittest.mock — Where to patch — The critical guide for understanding patch scope
4. syrupy — Snapshot testing — The best snapshot library for pytest
5. pytest-cov — To measure and enforce minimum coverage
6. Module 3: Integration Testing — The next step: real LLM + semantic assertions
7. OpenAI API Reference — To create mocks with the exact structure
8. Testing Anti-patterns — The most common testing errors (many apply to AI apps)