1. Introduction: Integration Testing for AI

Description

Modules 1-2 solved testing deterministically with mocks. But there are scenarios where you need the real LLM: validating that the model produces useful outputs, detecting regressions when you change models, verifying the end-to-end flow. The problem: with a real LLM, the same input can produce different outputs. This module teaches the proven strategies for handling that — without losing confidence in your test suite.

The limit of mocks

In Module 2 you learned that mocks are powerful: fast, free, deterministic. But they have a fundamental limit:

# With a mock, you know that:
# ✅ Your parser handles the JSON well
# ✅ Your processor normalizes the score correctly
# ✅ The output structure meets the contract
# ✅ API errors are handled gracefully

# With a mock, you do NOT know:
# ❌ Does the LLM actually understand the prompt?
# ❌ Is the summary coherent with the original text?
# ❌ Does the model keep producing the same quality after an update?
# ❌ Does the prompt work well for all languages and edge inputs?

Integration tests with the real LLM fill this gap. They don't replace unit tests — they complement them.

The core challenge: non-determinism

The fundamental problem of testing with a real LLM:

# Same input, two different runs:
text = "Python is an interpreted, high-level programming language."

result_1 = summarize(text)
# → {"summary": "Python is an interpreted high-level language.", "score": 0.9}

result_2 = summarize(text)
# → {"summary": "Python: an interpreted high-level language.", "score": 0.92}

# Both are CORRECT. But they're DIFFERENT.
# assert result_1 == result_2  → FAILS
# assert result_1["summary"] == "Python is an interpreted language."  → FAILS too

The strategies in this module solve exactly this problem: how to make assertions that are meaningful when the output varies.

The assertion spectrum

There's a continuum of strategies, from the strictest to the most flexible:

Stricter                                                  More flexible
─────────────────────────────────────────────────────────────────►

[Exact equality]  [Contains]  [Regex]  [Properties]  [Semantic]
     result ==        kw in      re.     len in range   similarity
    "exact text"     result    match()      0.85-0.95

The rule: Use the strictest assertion that's stable. If the exact output can vary, move up a level in the spectrum until the assertion is robust without losing detection value.

When to use integration tests

Proportion rule: 80% unit tests (mock), 20% integration tests (real LLM). Integration tests are the strategic complement, not the base.

Cost as a real constraint

Integration tests cost money. This is a design constraint, not a detail:

Real example:
- 100 integration tests
- Each test: 500 tokens input + 200 tokens output
- Model: gpt-4o-mini ($0.15/1M input, $0.60/1M output)

Cost per run:
  Input:  100 × 500 × $0.15/1M = $0.0075
  Output: 100 × 200 × $0.60/1M = $0.0120
  Total:  ~$0.02 per run

With 50 commits/day: $1/day = $30/month IN TESTS ALONE

With gpt-4o ($2.50/1M input, $10/1M output):
  Same setup: ~$0.25 per run × 50 = $12.50/day = $375/month

This is why budget controls are not optional — they're part of the test suite's design.

The module's five strategies

This module teaches five concrete strategies:

Strategy 1: End-to-End Tests (Capsule 2)

Run the complete flow with the real LLM. Flexible assertions on the output. Budget controls mandatory.

Strategy 2: Mock vs Real Decision Framework (Capsule 3)

Clear criteria for deciding when each test type adds the most value. Per-environment configuration.

Strategy 3: Semantic Similarity Assertions (Capsule 4)

Compare the meaning of the output with embeddings. Eliminates fragile string equality while keeping meaningful assertions.

Strategy 4: Property-Based Testing (Capsule 5)

Define invariants that must always hold. Hypothesis generates inputs automatically. More effective than manual test cases.

Strategy 5: Flaky Test Management (Capsule 6)

Retry logic, tolerance thresholds, quarantine. For when the LLM's variance makes tests fail occasionally.

The module's mindset shift

Before this module:
  → "I can't test with the real LLM — the output is always different"
  → "Integration tests are too expensive"
  → "My AI tests are flaky, it's normal"

After this module:
  → "Non-determinism is manageable with the right strategies"
  → "Integration tests are strategic — few, well chosen"
  → "Flakiness is a signal that requires attention, not something to ignore"

Module prerequisites

Before starting, make sure you have from Module 2:

# 1. Working unit test suite
pytest -m unit -v  # Must pass 100%

# 2. Working reference app
python -c "from app.sentiment import analyze_sentiment; print('OK')"

# 3. API key configured (for integration tests)
echo $OPENAI_API_KEY  # Must show the key

# 4. Additional dependencies for this module
pip install pytest-rerunfailures hypothesis sentence-transformers numpy

Module roadmap

Connection with the previous modules

Module 1: Configured pytest, defined markers, smoke tests
          ↓
Module 2: Unit tests with mocks, contract tests, parsers
          ↓
Module 3: Integration tests, real LLM, non-determinism strategies
          ↓
Module 4: Guardrails implemented and tested with M3 strategies

Module 3 doesn't just add integration tests — it also serves as the foundation for the following modules, where you'll test guardrails, input/output validations, and security behaviors that require the real LLM to be validated correctly.

Exercises

Exercise 1: Identify the mocks' gap

For your current app, identify 3 things the mock CANNOT validate but that are important for production:

Exercise 2: Calculate your test suite's cost

Estimate the monthly cost of running integration tests for your app:

• 20 integration tests
• Each test: 600 tokens input + 150 tokens output
• Model: gpt-4o-mini
• Frequency: 2 times a day (CI on main)

Per run:
  Input:  20 × 600 × $0.15/1M  = $0.0018
  Output: 20 × 150 × $0.60/1M  = $0.0018
  Total:  ~$0.0036 per run

Per month (30 days × 2 runs/day):
  $0.0036 × 60 = ~$0.22/month

With gpt-4o (17x more expensive):
  $0.22 × 17 ≈ $3.74/month

Conclusion: With gpt-4o-mini it's very affordable.
With gpt-4o, consider running less frequently.

Exercise 3: Design the assertion spectrum

For an E2E text-summarization test, design an assertion at each level of the spectrum:

# Level 1: Exact equality (fragile with the real LLM)
assert result["summary"] == "Python is an interpreted, high-level language."

# Level 2: Contains a keyword (more stable)
assert "python" in result["summary"].lower()

# Level 3: Regex (specific format)
assert re.search(r'\w+ is \w+', result["summary"])  # Has the form "X is Y"

# Level 4: Properties (invariants)
assert 10 <= len(result["summary"].split()) <= 100  # Between 10 and 100 words
assert 0 <= result["confidence"] <= 1

# Level 5: Semantic similarity
assert_semantically_similar(
    result["summary"],
    "Python is a high-level programming language",
    threshold=0.8
)

Exercise 4: Non-determinism in your project

What's the most "fragile" test you currently have? Which strategy from the module would make it more robust?

Exercise 5: Plan your integration test suite

Before writing the code, plan:

1. How many integration tests will your suite have?
2. When will they run (every commit / on main / nightly)?
3. What's the maximum budget per run?
4. Which strategy will you use for each test?

Integration Test Suite Plan:

Tests: 10-15 (rule: few and well chosen)
Execution: On merge to main (not on every commit)
Budget: $0.10 maximum per run

Test 1: E2E main flow (summary)
  → Strategy: properties (length, structure)
  → Model: gpt-4o-mini
  → Estimated tokens: 800

Test 2: Semantic quality of the summary
  → Strategy: semantic similarity threshold=0.8
  → Model: gpt-4o-mini
  → Estimated tokens: 1200 (includes embeddings)

Test 3: Flow with edge input (very short text)
  → Strategy: properties + error handling
  → Model: gpt-4o-mini
  → Estimated tokens: 300

...

Summary

• Integration tests complement (don't replace) unit tests with mocks
• Non-determinism is manageable with the right strategies: semantic similarity, property-based, flaky management
• The assertion spectrum: from exact equality to semantic similarity — use the strictest one that's stable
• Cost is a real constraint — budget controls from the design, not as an afterthought
• Proportion: 80% unit tests, 20% strategic integration tests

Additional resources

1. Testing ML Systems — Google — Practices from large teams
2. OpenAI Pricing — Current prices to calculate your budget
3. The Test Pyramid — Martin Fowler — Proportions of test types
4. Non-determinism in Tests — Martin Fowler — General strategies
5. Module 2: Unit Testing LLM Applications — Prerequisite
6. Hypothesis Documentation — For property-based testing (Capsule 5)
7. sentence-transformers — Local embeddings for semantic assertions