Module 3: Integration Testing & Non-Deterministic Strategies

3. Real LLM vs Mocks: Decision Framework

Description

When to use mocks and when to use the real LLM? This capsule presents a practical decision framework based on four dimensions: cost, speed, test value, and execution context (development, CI, pre-release). Without clear criteria, teams end up with two equally problematic extremes: costly tests that fail due to LLM variance, or mock suites that give false confidence without detecting real problems.

The real dilemma

Two teams with the same sentiment analysis app:

Team A (mocks only):

# Everything with mocks — 200 tests, all pass
# Speed: 5 seconds total
# Cost: $0
# Problem: In production, the real LLM produces outputs
# with a slightly different format that the parser doesn't handle.
# The bug reached production. The mocks didn't detect it.

Team B (all real):

# Everything with the real LLM — 200 tests
# Speed: 10 minutes
# Cost: $2 per run × 50 runs/day = $100/day
# Tests fail 20% of the time due to variance → the team ignores the failures
# "AI tests are always like this" → confidence destroyed

The solution: A framework that combines both depending on the context.

The decision framework

Question 1: Do I need to validate the output's STRUCTURE?
  └── Yes → MOCK (contract test, deterministic, free)
  └── No → next question

Question 2: Do I need to validate the output's SEMANTIC QUALITY?
  └── Yes → REAL LLM (integration test, with a budget)
  └── No → next question

Question 3: What context am I in?
  ├── Daily development → MOCK (speed, free)
  ├── CI on every commit → MOCK (don't block, don't spend)
  ├── CI on main/PR → MOCK + integration subset (if there's budget)
  └── Pre-release → MOCK + full integration (critical validation)

The diagnostic question

The fastest way to decide:

"If the LLM returns structured garbage, would the test fail?"

Yes → Use a mock (the test validates structure, not quality)

No → You need the real LLM (the test validates semantic quality)

Complete decision table

Test to write	Mock	Real LLM	Reason
The output has a "sentiment" key	✅	❌	Structure — contract test
score is between 0 and 1	✅	❌	Numeric constraint
The parser handles JSON in markdown	✅	❌	Deterministic logic
The error handler works	✅	❌	Error handling, no LLM
The summary is coherent with the original	❌	✅	Semantic quality
The prompt works in English and Spanish	❌	✅	Real LLM behavior
Model drift post-update	❌	✅	Requires the real LLM
The complete flow produces usable output	✅ + ❌	✅	E2E combines both
Tests on every CI commit	✅	❌	Speed and cost
Pre-release validation	✅	✅	Both, with a budget

The three test environments

Environment 1: Mock (Development/Fast CI)

# Characteristics:
# - No API calls
# - Milliseconds
# - Deterministic
# - Cost: $0
# - When: daily development, every commit

# Configuration:
@pytest.mark.unit
def test_contract_with_mock(make_sentiment_client):
    client = make_sentiment_client(sentiment="positive", score=0.9)
    result = analyze_sentiment("text", client=client)
    assert result["sentiment"] == "positive"

# pytest.ini:
# [pytest]
# addopts = -m "not integration"  # By default, only unit

Environment 2: Sandbox (Budget-Capped Integration)

# Characteristics:
# - Real LLM with a maximum budget
# - Economical model (gpt-4o-mini)
# - Subset of critical tests
# - When: CI on main, pre-release

# Configuration:
E2E_CONFIG = {
    "model": "gpt-4o-mini",
    "max_budget_usd": 0.25,
    "max_tests": 15,  # Only the most critical
    "timeout_sec": 30
}

@pytest.mark.integration
@pytest.mark.sandbox
def test_e2e_sandbox(sandbox_client):
    # sandbox_client checks the budget and skips if it was exceeded
    result = analyze_sentiment("real test text", client=sandbox_client)
    assert_quality_properties(result)

Environment 3: Real (Full Validation)

# Characteristics:
# - Real LLM with no model restrictions
# - Can use gpt-4o if there's a reason
# - Complete suite of integration tests
# - When: release, manual pre-deploy, nightly

# Configuration:
@pytest.mark.integration
@pytest.mark.real
@pytest.mark.skipif(not os.getenv("FULL_VALIDATION"), reason="Only in full validation")
def test_full_quality_validation(real_client):
    result = analyze_sentiment("critical text", client=real_client)
    assert_full_quality(result)

Per-environment configuration: complete code

# tests/config.py
import os
from enum import Enum

class TestEnvironment(Enum):
    MOCK_ONLY = "mock_only"
    SANDBOX = "sandbox"
    FULL = "full"

def get_test_env() -> TestEnvironment:
    """
    Determines the test environment based on environment variables.
    
    Variables:
        OPENAI_API_KEY: Required for sandbox and full
        RUN_INTEGRATION: "true" to enable integration tests
        FULL_VALIDATION: "true" to run the complete suite
    
    Logic:
        - No API key: mock_only
        - API key + RUN_INTEGRATION=true: sandbox
        - API key + FULL_VALIDATION=true: full
    """
    has_api_key = bool(os.getenv("OPENAI_API_KEY"))
    run_integration = os.getenv("RUN_INTEGRATION", "false").lower() == "true"
    full_validation = os.getenv("FULL_VALIDATION", "false").lower() == "true"
    
    if not has_api_key:
        return TestEnvironment.MOCK_ONLY
    if full_validation:
        return TestEnvironment.FULL
    if run_integration:
        return TestEnvironment.SANDBOX
    return TestEnvironment.MOCK_ONLY

def should_run_integration() -> bool:
    return get_test_env() in (TestEnvironment.SANDBOX, TestEnvironment.FULL)

def should_run_full() -> bool:
    return get_test_env() == TestEnvironment.FULL

# In conftest.py:
@pytest.fixture(scope="session")
def test_env():
    return get_test_env()

# Convenience decorators:
skip_unless_integration = pytest.mark.skipif(
    not should_run_integration(),
    reason="Requires RUN_INTEGRATION=true and OPENAI_API_KEY"
)

skip_unless_full = pytest.mark.skipif(
    not should_run_full(),
    reason="Requires FULL_VALIDATION=true and OPENAI_API_KEY"
)

Cost vs value: being strategic

Not all integration tests have the same value. Prioritize:

# HIGH priority: tests that detect frequent real bugs
# - Main flow (80% of traffic)
# - Prompts that have had bugs before
# - Validate a model change

@pytest.mark.integration
@pytest.mark.priority("high")
def test_main_flow_e2e():
    """The main flow works with the real LLM."""
    result = analyze_sentiment("Text representative of real traffic")
    assert_quality_properties(result)

# MEDIUM priority: robustness tests
# - Edge inputs (very short, very long, another language)
# - LLM error handling

@pytest.mark.integration
@pytest.mark.priority("medium")
def test_edge_case_short_input():
    result = analyze_sentiment("ok")
    assert result["sentiment"] in ["positive", "negative", "neutral"]

# LOW priority: fine-grained quality tests
# - Semantic similarity threshold
# - Explanation quality
# Only in full validation

@pytest.mark.integration
@pytest.mark.priority("low")
@skip_unless_full
def test_explanation_quality():
    result = analyze_sentiment("I love this product")
    assert len(result["explanation"]) > 20  # Informative explanation

The trap: tests that "validate structure" with the real LLM

A frequent mistake is running structure tests with the real LLM when the mock is enough:

# ❌ Unnecessary cost: uses the real LLM to validate structure
@pytest.mark.integration
def test_output_has_sentiment_key():
    client = openai.OpenAI()  # real LLM
    result = analyze_sentiment("text", client=client)
    assert "sentiment" in result  # ← This doesn't require the real LLM

# ✅ Correct: structure with a mock
@pytest.mark.unit
def test_output_has_sentiment_key(make_sentiment_client):
    client = make_sentiment_client()
    result = analyze_sentiment("text", client=client)
    assert "sentiment" in result  # ← Same assertion, no cost

The rule: If the test would fail even if the LLM returned a "valid random response," use a mock. Only use the real LLM when you need the content to make sense.

Per-environment CI/CD configuration

# .github/workflows/tests.yml

name: Tests

on: [push, pull_request]

jobs:
  # ─── Always: Unit Tests ───
  unit-tests:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: pytest -m "not integration" -v --tb=short
        # No API key, no cost, deterministic, fast

  # ─── Only on main: Integration (Sandbox) ───
  integration-sandbox:
    name: Integration Tests (Sandbox)
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      RUN_INTEGRATION: "true"
      E2E_BUDGET_USD: "0.25"
    steps:
      - uses: actions/checkout@v3
      - run: pytest -m integration --timeout=60 -v
        # With API key, limited budget, only on main

  # ─── Manual: Full Validation ───
  full-validation:
    name: Full Validation
    runs-on: ubuntu-latest
    if: github.event_name == 'workflow_dispatch'  # Manual only
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      FULL_VALIDATION: "true"
    steps:
      - uses: actions/checkout@v3
      - run: pytest -m "integration or e2e" -v
        # Manual only, no budget limit, for pre-release

Exercises

Exercise 1: Classify your tests

For the following tests, decide: mock, sandbox, or full validation:

Verify that the output has a "summary" key
Verify that the summary doesn't talk about a different topic than the input
Verify that the parser handles JSON with markdown code blocks
Verify that the English prompt produces outputs of equivalent quality to Spanish
Verify that the score is a float between 0 and 1

See solution

Mock — structure, pure contract test
Sandbox/Full — semantic relevance, requires the real LLM
Mock — deterministic parser logic
Full — comparative quality, requires the real LLM in two languages
Mock — numeric constraint, contract test

Exercise 2: Design the CI pipeline

Design the CI pipeline for a team with:

100 unit tests
15 critical integration tests
5 advanced quality validation tests

When does each group run? What environment variables do you need?

See solution

Push to any branch:
  → 100 unit tests (no API key, always, <30s)
  
Merge to main:
  → 100 unit tests
  → 15 integration tests (RUN_INTEGRATION=true, budget $0.15)
  
Manual pre-release:
  → 100 unit tests
  → 15 integration tests
  → 5 quality tests (FULL_VALIDATION=true)

Environment variables:
  - OPENAI_API_KEY (secret)
  - RUN_INTEGRATION (default: false)
  - FULL_VALIDATION (default: false)
  - E2E_BUDGET_USD (default: 0.25)

Exercise 3: The model drift case

You're switching from gpt-4o-mini-2024-07-18 to gpt-4o-mini-2025-01-15. Which tests would you run to validate there's no regression?

See guide

All unit tests (they should already pass — they're mocks, model-independent)
Structure integration tests (the output still has the expected keys)
Quality integration tests (semantic similarity >= previous threshold)
Compare directly: run the same inputs with both models and compare scores
Specific prompt tests that are known to sometimes change behavior between versions

Specific drift test:

@pytest.mark.integration
def test_model_drift_detection(integration_client):
    """Detects whether the new model changes the behavior."""
    test_cases = [
        ("Very positive text", "positive"),
        ("Very negative text", "negative"),
        ("Neutral factual text", "neutral")
    ]
    
    for text, expected_sentiment in test_cases:
        result = analyze_sentiment(text, client=integration_client)
        assert result["sentiment"] == expected_sentiment, \
            f"Model drift detected for: '{text[:50]}'"

Exercise 4: Framework ROI

Calculate the monthly savings of switching from "all real LLM" to "80% mock + 20% integration":

Data:

200 tests total
Each test: 600 tokens on average
gpt-4o-mini
CI: 30 runs/day (team commits)

See calculation

Without a framework (all real):
  200 tests × 600 tokens × $0.00000015/token = $0.018/run
  $0.018 × 30 runs/day = $0.54/day
  $0.54 × 30 days = $16.20/month

With a framework (80% mock, 20% integration, integration only on main):
  Integration: 40 tests × 600 tokens × $0.00000015 = $0.0036/run
  Integration on main: ~5 runs/day (merges to main)
  $0.0036 × 5 = $0.018/day
  $0.018 × 30 days = $0.54/month

Savings: $16.20 - $0.54 = $15.66/month
Percentage: 97% cost reduction

Bonus: the unit tests run in <5s instead of ~20 minutes → 
  30 devs × 30 min saved/day × $80/hour = $1,200/day in time

Exercise 5: Document your decision

Write the "Testing Strategy" section for your project's README. It must explain:

Which tests use a mock vs the real LLM
How to run each type
Which environment variables are needed

See template

## Testing Strategy

### Test types

| Type | Mock/Real | When it runs | Command |
|------|-----------|--------------|---------|
| Unit tests | Mock | Always | `pytest -m unit` |
| Contract tests | Mock | Always | `pytest -m contract` |
| Integration (sandbox) | Real LLM | On main | `pytest -m integration` |
| Full validation | Real LLM | Manual | `FULL_VALIDATION=true pytest -m integration` |

### Environment variables

- `OPENAI_API_KEY`: Required for integration tests
- `RUN_INTEGRATION=true`: Enables integration tests in CI
- `FULL_VALIDATION=true`: Enables full validation
- `E2E_BUDGET_USD=0.25`: Maximum budget per run (default: $0.25)

### Running tests locally

\`\`\`bash
# Only unit tests (always available, no API key)
pytest -m "not integration" -v

# With integration tests (requires an API key)
export OPENAI_API_KEY=sk-...
export RUN_INTEGRATION=true
pytest -m integration -v --timeout=60
\`\`\`

Summary

Mock for structure/logic — always, fast, free, deterministic
Real LLM for semantic quality — strategic, with a budget, on main or manual
Three environments: Mock-only (development), Sandbox (CI on main), Full (pre-release)
The trap: don't use the real LLM where a mock is enough — it's cost with no added value
CI configured per environment: only unit on every push; integration only on main

Additional resources

Test Pyramid — Martin Fowler — Correct proportions of test types
GitHub Actions — Conditional execution — Run jobs conditionally
pytest markers — Test organization
OpenAI Pricing — Calculate costs
The Testing Trophy — Another perspective on proportions
Effective Software Testing — Reference book on strategic testing

← Previous2. End-to-End Tests Next →4. Semantic Similarity Assertions