1. Module Introduction: Testing Fundamentals for AI

Description

This is the first module of the Production Best Practices guide. Before implementing guardrails, logging, or reliability, you need to internalize an uncomfortable truth: your AI application probably has no tests, and that's a problem. Most AI Engineers come from a world of rapid prototyping where "it works" = "it's ready." This module breaks that mindset with concrete facts and runnable code from the very first session.

The goal isn't to give you a pytest tutorial. You already know that. The goal is to teach you to adapt testing to the particularities of LLM systems: non-determinism in outputs, cost per API call, prompt fragility, and model drift. These characteristics make testing AI different — not impossible, just different.

By the end of this module you'll have a working testing infrastructure for your own LLM app, with pytest configured correctly, fixtures for LLM mocking, markers to separate fast tests from slow ones, and your first smoke tests and contract tests written. Not 100% coverage — a solid foundation to build on.

Context: Where are we in the guide?

Production Best Practices Guide
│
├── Phase 1: Testing AI Systems  ← YOU ARE HERE
│   ├── Module 1: Testing Fundamentals  ← THIS MODULE
│   ├── Module 2: Unit Testing LLM Applications
│   └── Module 3: Integration Testing & Non-Deterministic Strategies
│
├── Phase 2: Safety & Quality
│   ├── Module 4: Guardrails
│   ├── Module 5: Structured Logging
│   └── Module 6: Code Quality Patterns
│
└── Phase 3: Production Readiness
    ├── Module 7: Reliability Patterns
    └── Module 8: Production-Ready Capstone Project

This module is the foundation of everything that follows. The testing infrastructure you configure here is reused in every later module. When you implement guardrails (module 4), you'll test them with pytest. When you build the logging system (module 5), you'll validate that it works with tests. When you implement the reliability layer (module 7), you'll verify its behavior with automated tests.

You can't skip this module.

Prerequisites for this guide

Before you start, verify that you meet these requirements:

Technical:

• ✅ Python 3.10+ installed
• ✅ Basic familiarity with pytest (you can run pytest, you know what a fixture is)
• ✅ Experience with the OpenAI API or similar (you've used client.chat.completions.create)
• ✅ You know Pydantic (you'll use it in modules 2 and 4)
• ✅ You completed guide #12 (Evaluation Frameworks) or you understand the difference between LLM evaluation and code testing

Experience:

• ✅ You have at least one app that uses an LLM (chatbot, RAG, agent — anything)
• ✅ You understand that LLM outputs are non-deterministic
• ✅ You've seen a system fail in production (or you can sense it)

If you don't have your own LLM app, don't worry. In the Module 1 Project I provide a complete reference app for you to work on.

Comparison: Traditional testing vs AI testing

This is the most important difference you need to understand before writing your first test:

The most critical point: AI failures are silent. If a function adds incorrectly, the program crashes. If a prompt produces output with an incorrect structure, the system can "work" but return incorrect data to real users for days.

Why testing FIRST

The natural temptation is "I'll finish the feature first, then add tests." For AI apps, this is especially dangerous for three reasons:

Reason 1: Prompt fragility A minimal change in a prompt can completely change the output. Without tests, you never know whether your change broke something until a user complains.

# Original prompt → Output: {"sentiment": "positive", "confidence": 0.92}
prompt_v1 = "Analyze the sentiment of this text and return JSON."

# Modified prompt (small change) → Output: "The sentiment is positive." (string, not JSON)
prompt_v2 = "Analyze the sentiment of this text."

Without a contract test that verifies the output is JSON with the correct keys, this change goes unnoticed.

Reason 2: Model drift LLM providers update their models periodically. gpt-3.5-turbo from January 2025 is not identical to gpt-3.5-turbo from November 2025. Without regression tests, you won't know whether a model update changed your app's behavior.

Reason 3: Chain of effects In RAG systems and agents, a change in one component can affect several downstream. A test that fails in the LLM output parser can indicate that the problem is three steps earlier in the pipeline.

The non-determinism excuse

The most common objection: "I can't test LLM outputs because they're non-deterministic."

It's an excuse, not a technical limitation.

The reality:

Your typical LLM app
│
├── Deterministic (70-80%):
│   ├── LLM output parsers
│   ├── Validators (Pydantic, custom)
│   ├── Business logic that processes the output
│   ├── Config loading
│   ├── Chain/pipeline logic
│   └── Error handling
│
└── Non-deterministic (20-30%):
    └── The LLM output itself
        (but its STRUCTURE can be deterministic)

70-80% of your code is completely deterministic and testable with normal tests. For the remaining 20-30%, there are three strategies you'll see in modules 2 and 3:

1. Mock the LLM → Fully deterministic test (module 2)
2. Semantic similarity assertions → Verify output properties without an exact match (module 3)
3. Property-based testing → Verify output invariants (module 3)

You don't have to give up testing because of non-determinism. You need more sophisticated strategies.

Difference from Evaluation Frameworks (Guide #12)

If you completed guide #12, this distinction is critical:

They're complementary. Evaluation tells you whether the LLM responds well. Testing tells you whether your code handles that response correctly.

Initial technical setup

Before starting the technical sessions, set up your environment:

Base installation

# Create a virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install testing dependencies
pip install pytest pytest-asyncio pytest-mock pytest-cov

# Project dependencies (if you don't have them)
pip install openai pydantic python-dotenv

# Verify installation
pytest --version
# pytest 8.x.x

Environment variables

# .env (create in the project root)
OPENAI_API_KEY=sk-proj-...

# Verify it loads
python -c "from dotenv import load_dotenv; import os; load_dotenv(); print(os.getenv('OPENAI_API_KEY', 'NOT SET')[:10])"

Target directory structure

By the end of Module 1 you'll have this structure:

my-ai-app/
├── src/
│   └── app/
│       ├── __init__.py
│       ├── llm.py          # LLM client
│       └── parsers.py      # Output parsers
├── tests/
│   ├── conftest.py         # Shared fixtures
│   ├── unit/
│   │   ├── test_parsers.py
│   │   └── test_llm_chain.py
│   └── integration/
│       └── test_e2e.py
├── pytest.ini              # pytest configuration
├── .env                    # API keys
└── requirements.txt

Module objectives

By the end of this module you'll be able to:

1. Articulate why AI apps need different testing (prompt fragility, non-determinism, cost)
2. Configure pytest with conftest.py, markers, and parametrize adapted for AI projects
3. Write tests following the Arrange-Act-Assert pattern adapted for LLM apps
4. Categorize tests by type: smoke, contract, behavioral, regression
5. Mock LLM responses to make deterministic tests
6. Structure reusable fixtures for AI projects
7. Have a working Test Suite Setup running for your LLM app

Roadmap: the module's 8 sessions

Total estimated duration: 7 hours (reading + practice)

What you'll build in this module

The Module 1 mini-project is the Test Suite Setup: take an LLM app (yours or the provided reference one) and configure:

• pytest.ini with markers and appropriate configuration
• conftest.py with fixtures for LLM mocking and shared utilities
• tests/unit/ with basic smoke tests and contract tests
• tests/integration/ with a ready-made structure (the integration tests come in module 3)
• A first successful run of pytest -m "not integration" with all tests passing

It's not the most glamorous project in the guide, but it's the infrastructure that makes everything else possible.

What this module does NOT cover

To manage expectations:

• ❌ Semantic similarity assertions → Module 3 (requires advanced strategies for non-determinism)
• ❌ Property-based testing with Hypothesis → Module 3
• ❌ End-to-end tests with a real LLM → Module 3 (they have cost and strategy implications)
• ❌ Testing FastAPI endpoints → Mentioned but not the focus
• ❌ CI/CD pipeline → The CI command is mentioned but the GitHub Actions configuration is in module 7
• ❌ Advanced snapshot testing → Module 2

This module is "foundations": the basic tools so everything else works.

Evidence of success at the end of this module

You'll know you completed the module successfully if:

• ✅ You can run pytest -m unit and all tests pass (no import errors)
• ✅ You can run pytest -m smoke and the project's smoke tests pass
• ✅ Your conftest.py has at least one reusable LLM mock
• ✅ You can explain the difference between a smoke test, a contract test, and a behavioral test
• ✅ You can distinguish which parts of your app are deterministic vs non-deterministic

Exercises

Exercise 1: Inventory of your app — what's deterministic?

Take your current LLM app (or the project's reference one). Draw a simple diagram of the data flow and mark each component as D (deterministic) or ND (non-deterministic).

Input: "Analyze this contract"
│
├── [D] validate_input()           → checks that it's not empty
├── [D] build_prompt()             → builds the prompt with a template
├── [ND] call_llm()                → calls the OpenAI API
├── [D] parse_llm_response()       → extracts JSON from the output
├── [D] validate_response_schema() → checks required keys
└── [D] format_for_user()          → formats for the user

Exercise 2: What would break your app?

Imagine you change one word in your main prompt. List 3 things that could break and how you'd find out without tests vs with tests.

Exercise 3: Testing ROI — the concrete math

Estimate: if a prompt change breaks something in production, how much time do you lose debugging? Compare it to the time to write a contract test.

Debugging a silent production bug:
- Detect the problem: 30 min - 2 hours (if you have logging)
- Reproduce locally: 1-3 hours
- Identify root cause: 1-4 hours
- Fix + verify: 30 min
- Total: 3-10 hours (more if it was on a weekend)

Contract test that would have caught it:
- Write the test: 10-15 minutes
- Future maintenance: ~2 min per change
- Amortized total: 15-30 minutes

Exercise 4: Difference between Testing and Evaluation

Describe in your own words the difference between:

1. A test that verifies the LLM output is JSON with the keys sentiment and confidence
2. An evaluation metric that measures whether the detected sentiment is correct

def test_sentiment_output_structure(mock_llm):
    result = analyze_sentiment("I love this product")
    # Verify STRUCTURE, not quality
    assert isinstance(result, dict)
    assert "sentiment" in result
    assert "confidence" in result
    assert isinstance(result["confidence"], float)
    assert 0.0 <= result["confidence"] <= 1.0

# Evaluation batch (offline)
correct = 0
for text, expected_sentiment in golden_dataset:
    result = analyze_sentiment(text)
    if result["sentiment"] == expected_sentiment:
        correct += 1
accuracy = correct / len(golden_dataset)
# "Accuracy on golden dataset: 87%"

Exercise 5: First mental test

Without writing code, describe in words what the first test you'd write for your LLM app would look like. Consider: what do you mock? what do you assert? is it smoke, contract, behavioral, or regression?

Test: "The sentiment analyzer returns the correct structure"
Type: Contract test

Arrange:
- Mock of the LLM that returns '{"sentiment": "positive", "confidence": 0.9}'

Act:
- Call analyze_sentiment("I love this!")

Assert:
- The result is a dict
- It has keys "sentiment" and "confidence"
- "sentiment" is a string (doesn't verify the semantic value)
- "confidence" is a float between 0 and 1

Why it's a contract test: It verifies that the prompt "fulfills its contract"
of returning a specific structure, without evaluating whether the detected
sentiment is correct.

Test: "The app imports correctly"
Type: Smoke test

Arrange: nothing

Act: import app.main

Assert: doesn't raise ImportError, AttributeError, or NameError

Summary

• Testing AI isn't the same as testing traditional software: non-determinism, cost per test, and silent failures change the strategy
• 70-80% of your LLM app is completely deterministic and testable normally
• The LLM's non-determinism is handled with mocking (module 2) or property assertions (module 3)
• Testing and Evaluation are complementary: testing verifies code behavior, evaluation measures LLM output quality
• This module configures the testing infrastructure you'll use in all following modules
• Start simple: smoke tests first, contract tests second, behavioral tests later

Additional resources

1. pytest Documentation — Official pytest documentation (the foundation of everything)
2. unittest.mock — Mocking in standard Python
3. pytest-asyncio — Async testing (critical for apps using async LLM calls)
4. Testing Best Practices (Google) — Google Engineering's testing blog
5. Guide #12: Evaluation Frameworks — Prerequisite of this guide (evaluation vs testing difference)
6. The Practical Test Pyramid (Martin Fowler) — Layered testing strategy (applies to AI with adaptations)
7. pytest-mock — Cleaner mocking integration with pytest