8. Troubleshooting and Module 1 Summary

Description

This is the closing session of Module 1. It consolidates the most common errors teams run into when setting up testing for LLM apps, with diagnosis and concrete solutions. It also includes a summary of the module's key concepts and a clear transition to Module 2.

This session is meant as a quick reference: when something fails in your suite, come here before searching on Google. The problems are ordered by frequency, with root-cause diagnosis and a specific solution.

Most common errors — Diagnosis and solution

Error 1: ModuleNotFoundError: No module named 'app'

Frequency: Very common on the first setup.

Symptom:

ERRORS
tests/unit/test_parsers.py - ModuleNotFoundError: No module named 'app'

Cause: Python can't find the app package because src/ isn't on the PYTHONPATH.

Solutions (choose one):

# SOLUTION A: sys.path in conftest.py (the simplest)
# tests/conftest.py
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))
# This sys.path.insert makes 'import app' work in all tests

# SOLUTION B: pyproject.toml with an editable package setup
# pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.backends.legacy:build"

[tool.setuptools.packages.find]
where = ["src"]

# Install in editable mode:
# pip install -e .

# SOLUTION C: pytest.ini with pythonpath (pytest >= 7.0)
# pytest.ini
[pytest]
pythonpath = src

Which to use: Solution C with pythonpath = src in pytest.ini is the cleanest for new projects. Solution A is the fastest if you already have the project configured.

Error 2: AttributeError: Mock object has no attribute 'choices'

Frequency: Very common when creating mocks for the first time.

Symptom:

AttributeError: Mock object has no attribute 'choices'
# Or:
AttributeError: 'dict' object has no attribute 'choices'

Cause 1: The fixture returns a Python dictionary, but the app code accesses the response with dot notation (response.choices).

# ❌ Incorrect fixture (returns a dict)
@pytest.fixture
def bad_response():
    return {"choices": [{"message": {"content": "..."}}]}

# The code does:
content = response.choices[0].message.content  # AttributeError!
# Dicts use bracket notation, not dot notation

# ✅ Correct fixture (returns a MagicMock)
@pytest.fixture
def good_response():
    mock = MagicMock()
    mock.choices[0].message.content = '{"sentiment": "positive"}'
    return mock

Cause 2: MagicMock() creates attributes automatically but empty lists aren't automatic.

# ❌ Problem: choices[0] accesses an empty list
response = MagicMock()
response.choices[0].message.content = "..."  # Doesn't work directly

# ✅ Solution: assign a real list with a MagicMock
response = MagicMock()
choice = MagicMock()
choice.message.content = '{"sentiment": "positive"}'
response.choices = [choice]  # Real list with a MagicMock

Quick diagnosis:

# Run in interactive Python to verify your mock:
from unittest.mock import MagicMock

mock = MagicMock()
choice = MagicMock()
choice.message.content = "test content"
mock.choices = [choice]

print(mock.choices[0].message.content)  # Should print "test content"

Error 3: The mock isn't applied — the code calls the real API

Frequency: Very common when @patch has the wrong path.

Symptom:

# The test passes but you see charges on the API key
# Or the test is slow (3-5 seconds instead of <100ms)

Cause: The @patch path doesn't match where the object is imported.

# ❌ Wrong path: patches where it's DEFINED
@patch("openai.Client.chat.completions.create")
def test_something(mock_create):
    ...  # Doesn't work: the code imports from its module, not from openai

# ✅ Correct path: patches where it's USED
@patch("app.sentiment.client.chat.completions.create")
def test_something(mock_create):
    ...  # Works: intercepts the call in the module that makes it

Golden rule: The @patch path must be "module_where_it_is_used.object.method".

How to find the correct path:

# 1. Open the module that calls the LLM (e.g., app/sentiment.py)
# 2. Find the line where the client is imported:
from openai import OpenAI
client = OpenAI()  # The 'client' object is in app.sentiment

# 3. The correct path is:
# @patch("app.sentiment.client.chat.completions.create")

Verification:

@patch("app.sentiment.client.chat.completions.create")
def test_no_real_api_calls(mock_create, sentiment_positive_response):
    mock_create.return_value = sentiment_positive_response
    analyze_sentiment("test")
    # If this passes in <100ms, the mock is working
    # If it takes >2 seconds, the mock wasn't applied
    mock_create.assert_called_once()

Error 4: Flaky tests — they pass sometimes, fail others

Frequency: Common when using real LLMs in unit tests.

Symptom:

FAILED tests/unit/test_sentiment.py::test_positive_sentiment
# Passes 7 out of 10 times

Cause: The test calls the real LLM (not mocked) and the output varies.

Diagnosis:

# Run the same test multiple times
pytest tests/unit/test_sentiment.py::test_positive_sentiment -v --count=5
# If some pass and others fail → flaky test due to a real LLM

Solution:

# ❌ Flaky test: calls the real LLM
def test_positive_sentiment_flaky():
    result = analyze_sentiment("I love this!")
    assert result["sentiment"] == "positive"  # Fails when the LLM says "neutral"

# ✅ Robust test: mocks the LLM
@patch("app.sentiment.client.chat.completions.create")
def test_positive_sentiment_robust(mock_create, make_llm_response):
    mock_create.return_value = make_llm_response(
        '{"sentiment": "positive", "confidence": 0.9}'
    )
    result = analyze_sentiment("I love this!")
    assert result["sentiment"] == "positive"  # Always passes

Rule: In unit, contract, behavioral, and regression tests: ALWAYS mock the LLM. Only tests marked with @pytest.mark.integration should call the real API.

Error 5: fixture 'fixture_name' not found

Frequency: Moderate.

Symptom:

ERRORS
tests/unit/test_something.py::test_something - fixture 'make_llm_response' not found

Possible cause 1: The conftest.py has a syntax error and doesn't load.

# Verify that conftest.py has no errors:
python -c "import tests.conftest"
# If it prints nothing and doesn't raise → OK

Possible cause 2: The conftest.py is in the wrong directory.

tests/
├── conftest.py      ← Available to all tests in tests/
└── unit/
    ├── conftest.py  ← Available only to tests in tests/unit/
    └── test_something.py ← Can use fixtures from BOTH conftest.py files

Possible cause 3: The fixture has a typo in its name.

# conftest.py
@pytest.fixture
def make_llm_resonse():  # Typo: "resonse" instead of "response"
    ...

# test_something.py
def test_x(make_llm_response):  # Looks for "response" — doesn't find the fixture
    ...

Error 6: Slow tests — the suite takes >30 seconds

Frequency: Moderate.

Symptom: pytest -m "not integration" takes >30 seconds.

Main cause: Some test is calling the real API (without a mock).

Diagnosis:

# See the slowest tests:
pytest -m "not integration" --durations=10
# The output shows the 10 slowest tests with their time
# If any takes >1 second → it probably calls the API

Solution:

# Identify which test is slow
pytest -m "not integration" -v --durations=10

# Run that specific test with -s to see the output
pytest tests/unit/test_something.py::test_slow -s
# If it makes calls to external URLs → it isn't mocked correctly

Prevention: Configure a timeout for unit tests:

pip install pytest-timeout

# In pytest.ini:
# timeout = 5  # Fails any test that takes >5 seconds

Error 7: PytestUnknownMarkWarning: Unknown pytest.mark.contract

Frequency: Low but confusing.

Symptom:

PytestUnknownMarkWarning: Unknown pytest.mark.contract - is this a typo?

Cause: The marker is used in the code but isn't registered in pytest.ini.

Solution:

# pytest.ini — add all the markers used
[pytest]
markers =
    smoke: Smoke tests
    contract: Contract tests
    behavioral: Behavior tests
    regression: Regression tests
    unit: Unit tests
    integration: Integration tests

Error 8: Coverage shows 0% for modules that do have tests

Symptom:

pytest --cov=app --cov-report=term-missing
# Output: "No data to report"
# Or coverage shows 0% for tested modules

Cause: The module specified in --cov= doesn't match the package name.

Diagnosis:

# Verify that app is the correct package name
ls src/
# app/   ← The directory must have __init__.py

python -c "import app; print(app.__file__)"
# Must print the path to src/app/__init__.py

Solution:

# Specify the correct path
pytest --cov=src/app --cov-report=term-missing

# Or configure in pyproject.toml:
# [tool.coverage.run]
# source = ["app"]

Module closing checklist

Before moving to Module 2, verify that you have:

Base configuration:
├── [ ] pytest.ini with registered markers and testpaths configured
├── [ ] conftest.py with the make_llm_response fixture (factory)
├── [ ] conftest.py with predefined response fixtures
├── [ ] conftest.py with mock_openai_client
├── [ ] helpers.py with create_openai_chat_response()
└── [ ] sys.path configured to import 'app'

Tests written:
├── [ ] ≥3 smoke tests (modules import, API responds)
├── [ ] ≥5 contract tests (parser output structure)
├── [ ] ≥3 contract tests (main function output structure)
├── [ ] ≥3 behavioral tests (invariant properties)
└── [ ] ≥2 regression tests

Verification:
├── [ ] pytest -m smoke                          → ✅ Everything passes
├── [ ] pytest -m contract                       → ✅ Everything passes
├── [ ] pytest -m "not integration"              → ✅ Everything passes
├── [ ] pytest -m "not integration" --durations=5 → All <1 second
└── [ ] pytest --cov=app -m "not integration"   → Coverage >70%

Module summary: key concepts

What you learned and what you can do now

By completing this module you can:

Explain:

• Why AI apps need different testing from traditional software
• The difference between testing and evaluation of LLMs
• When to use each test type (smoke, contract, behavioral, regression)

Configure:

• pytest with markers, conftest.py, parametrize, and coverage
• Fixtures for LLM mocking with a realistic structure
• A 3-level test strategy (unit/integration/regression)

Write:

• Smoke tests to verify that the system starts up
• Contract tests for prompts and parsers (output structure)
• Behavioral tests for invariant properties
• Regression tests for known bugs

Run:

• The complete cost-free suite in <30 seconds
• Subsets depending on the context (development, PR, weekly)

Transition to Module 2: Unit Testing LLM Applications

Module 1 set up the infrastructure. Module 2 deepens it with three advanced techniques:

1. Advanced mocking of LLM responses

Module 1: basic create_openai_chat_response()
Module 2: Mocking of streams, of multiple calls,
          of specific OpenAI API errors

2. Prompt Contract Tests

Module 1: Basic contract tests (does it have the keys?)
Module 2: Complete contract tests:
          - Validate that the prompt produces the promised structure
          - Testing multiple variations of the same prompt
          - Detect regressions when the prompt changes

3. Snapshot Testing

Module 1: assert result == expected (fixed value)
Module 2: Snapshot testing — save the "golden" output of a function
          and verify that it doesn't change over time

4. Testing parsers and output processors

Module 1: Some parser tests
Module 2: Exhaustive testing of parsers:
          - With every possible LLM output format
          - API edge cases (JSON in markdown, with extra text)
          - Specialized fixture factories for each parser

The transition in one sentence: "You have pytest configured and you understand what to test → now learn to mock with surgical precision and to treat prompts as formal contracts."

Closing exercises

Exercise 1: Diagnose an existing suite

Your suite has 20 tests. pytest -m "not integration" takes 45 seconds and 3 tests fail intermittently. What diagnosis would you do, and in what order?

pytest -m "not integration" --durations=10

pytest tests/unit/test_something.py::test_flaky -v --count=5

pytest tests/ -v --tb=long

Exercise 2: Improve a generic fixture

This fixture only covers one case. How would you improve it to cover more without repeating code?

@pytest.fixture
def llm_response():
    mock = MagicMock()
    mock.choices[0].message.content = '{"sentiment": "positive", "confidence": 0.9}'
    return mock

# BEFORE: generic fixture with a fixed value
@pytest.fixture
def llm_response():
    mock = MagicMock()
    mock.choices[0].message.content = '{"sentiment": "positive", "confidence": 0.9}'
    return mock

# AFTER: factory fixture + predefined fixtures for common cases
from tests.helpers import create_openai_chat_response

@pytest.fixture
def make_llm_response():
    """Factory: returns the function to create custom responses."""
    return create_openai_chat_response

@pytest.fixture
def positive_response(make_llm_response):
    """Shortcut for the most common case."""
    return make_llm_response('{"sentiment": "positive", "confidence": 0.9}')

@pytest.fixture
def negative_response(make_llm_response):
    return make_llm_response('{"sentiment": "negative", "confidence": 0.85}')

# Tests use the correct fixture for their case:
# def test_positive_sentiment(positive_response):  ← Concise
# def test_custom_sentiment(make_llm_response):     ← Flexible
#     response = make_llm_response('{"sentiment": "neutral", "confidence": 0.1}')

Exercise 3: Write a preventive regression test

Identify a possible failure in the reference app and write a regression test to prevent it (even if the bug hasn't occurred yet).

@pytest.mark.regression
def test_parser_preventive_handles_json_with_boolean_values():
    """
    Preventive: the parser must correctly handle boolean values in JSON.
    The LLM could return 'true'/'false' (JSON), which are valid but
    must be preserved as bool in Python, not as a string.
    """
    raw = '{"is_spam": true, "confidence": 0.9}'
    result = parse_json_from_llm_output(raw)
    assert result["is_spam"] is True  # bool, not the string "true"
    assert isinstance(result["is_spam"], bool)


@pytest.mark.regression
@patch("app.sentiment.client.chat.completions.create")
def test_analyze_sentiment_preventive_handles_null_confidence(mock_create, make_llm_response):
    """
    Preventive: if the LLM returns 'null' for confidence, the function
    must handle it gracefully (not raise TypeError in the range validation).
    """
    mock_create.return_value = make_llm_response(
        '{"sentiment": "positive", "confidence": null}'
    )
    # Must raise ValueError with a clear message, not TypeError
    with pytest.raises((ValueError, TypeError)):
        analyze_sentiment("text")

Exercise 4: A testing plan for Module 2

Based on what you learned in Module 1, what would you add to your test suite in Module 2 that you can't do yet?

What you CANNOT do with the Module 1 tools:

1. Snapshot testing
   - You need to save the "golden" output of a function
   - Verify that it doesn't change between runs
   - Tool: pytest-snapshot or similar

2. Testing multiple variations of the same prompt
   - Test matrix: prompt v1 vs v2 vs v3
   - Detect when a prompt change introduces regressions
   - Requires a more sophisticated fixture factory strategy

3. Exhaustive testing of API errors
   - RateLimitError, APITimeoutError, AuthenticationError
   - Each one requires a different mock with side_effect
   - You need to understand the openai exception hierarchy

4. Streaming testing
   - If your app uses stream=True, the mock is very different
   - Requires AsyncMock with generators

5. Advanced chain testing
   - LangChain chains with multiple steps
   - Each chain step may need a different mock

Exercise 5: Final self-assessment

Answer honestly: how many of these statements can you confirm?

[ ] I can explain why non-deterministic LLM outputs are not
    an excuse for not testing.

[ ] I can configure pytest.ini with markers, testpaths, and addopts correctly.

[ ] I can create a conftest.py with a factory fixture for LLM mocks.

[ ] I understand the difference between smoke, contract, behavioral, and regression tests.

[ ] I can write a contract test for a prompt that produces JSON.

[ ] I know how to use @patch with the correct path to intercept API calls.

[ ] I can run pytest -m "not integration" and have it take <30 seconds.

[ ] I know how to diagnose why a test is flaky and how to make it robust.

If you checked 7-8: you're ready for Module 2. If you checked 5-6: review sessions 03-05 before continuing. If you checked <5: complete Project 07 (Test Suite Setup) before moving on.

Additional resources

1. pytest FAQ — Answers to frequent configuration questions
2. Where to patch — The most important guide for understanding @patch
3. pytest-timeout — Detect slow tests automatically
4. pytest --durations — Suite profiling
5. Module 2: Unit Testing LLM Applications — Next module
6. Guide #12 Evaluation Frameworks — Complement to testing for measuring output quality