2. Why AI Needs Different Testing

Description

LLM applications have four particularities that make traditional testing insufficient: non-determinism in outputs, prompt fragility, model drift, and cost per test. This session explores each one with concrete examples of how it manifests in production and which testing strategy corresponds to it.

The conclusion isn't "you can't test AI." The conclusion is: 70-80% of your AI app is completely deterministic and testable with normal asserts. For the remaining 20-30% there are specific strategies you'll learn in modules 2 and 3. This session gives you the complete map.

By the end you'll understand exactly which parts of your app you can test today (with standard tools), which parts require mocking, and which parts require more advanced strategies.

Problem 1: Non-determinism

What it is

The same prompt with the same input can produce different outputs on each call:

# tests/demos/non_determinism_demo.py
import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
client = OpenAI()

def call_llm(prompt: str, temperature: float = 0.7) -> str:
    """Direct LLM call for demonstration."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
    )
    return response.choices[0].message.content

# Same prompt, multiple responses
prompt = "Describe Python in exactly 2 words."

for i in range(3):
    result = call_llm(prompt)
    print(f"Run {i+1}: {result!r}")

# Possible output:
# Run 1: 'Versatile language.'
# Run 2: 'Simple powerful.'
# Run 3: 'Multipurpose flexible.'

Run this demo: python tests/demos/non_determinism_demo.py

Why this breaks traditional tests

# THIS FAILS INTERMITTENTLY — DON'T DO THIS
def test_llm_describes_python():
    result = call_llm("Describe Python in exactly 2 words.")
    assert result == "Versatile language."  # ❌ Fails 2 out of 3 times

The test passes sometimes and fails other times. This is called a flaky test and it's the worst kind of test: it makes you lose confidence in your entire suite because you never know whether a failure is real or random.

The correct solution

# THIS IS ROBUST — VERIFY PROPERTIES, NOT AN EXACT VALUE
def test_llm_describes_python_structure():
    result = call_llm("Describe Python in exactly 2 words.")
    # Verify invariant properties, not the exact value
    words = result.strip().rstrip(".").split()
    assert len(words) == 2, f"Expected 2 words, got: {result!r}"
    assert all(word.isalpha() for word in words), f"Expected words only: {result!r}"

But even this can be fragile. The definitive solution for unit tests is to mock the LLM (module 2), so the output is 100% predictable.

The 70/30 rule

Your typical LLM app:

┌─────────────────────────────────────────┐
│ 70-80% DETERMINISTIC                    │
│                                         │
│  - validate_input()     → exact assert  │
│  - build_prompt()       → exact assert  │
│  - parse_llm_output()   → exact assert  │
│  - validate_schema()    → exact assert  │
│  - format_response()    → exact assert  │
│  - config loading       → exact assert  │
│  - chain routing logic  → mock + assert │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│ 20-30% NON-DETERMINISTIC                │
│                                         │
│  - call_llm()           → mock          │
│  - streaming output     → mock          │
│                                         │
│ For tests that DO need a real LLM:      │
│  - semantic assertions (M3)             │
│  - property-based testing (M3)          │
└─────────────────────────────────────────┘

Problem 2: Prompt Fragility

What it is

Prompts are fragile code. A minimal change — adding a word, changing punctuation, rephrasing a sentence — can completely change the LLM's behavior.

Concrete example: changes that break the system

# src/app/prompts.py

# VERSION 1: Works perfectly
PROMPT_V1 = """Analyze the sentiment of the following text.
Return ONLY a JSON object with this exact structure:
{{"sentiment": "positive|negative|neutral", "confidence": 0.0-1.0}}

Text: {text}"""

# VERSION 2: We add a useful instruction... that breaks everything
PROMPT_V2 = """Analyze the sentiment of the following text.
Return ONLY a JSON object with this exact structure:
{{"sentiment": "positive|negative|neutral", "confidence": 0.0-1.0}}
If the text is in Spanish, analyze accordingly.

Text: {text}"""

# VERSION 3: Small format change
PROMPT_V3 = """Analyze the sentiment. Return JSON:
{{"sentiment": "positive|negative|neutral", "confidence": 0.0-1.0}}

Text: {text}"""

How these changes manifest in production:

# tests/unit/test_prompt_fragility_demo.py
import pytest
from unittest.mock import MagicMock, patch

def parse_sentiment_response(raw: str) -> dict:
    """Parser that assumes clean JSON."""
    import json
    return json.loads(raw.strip())

# ❌ V2 could produce this → breaks the parser:
problematic_outputs = [
    # The LLM adds a note at the end
    '{"sentiment": "positive", "confidence": 0.9}\nNote: Text appears to be in English.',
    # The LLM adds markdown
    '```json\n{"sentiment": "positive", "confidence": 0.9}\n```',
    # The LLM adds introductory text
    'Here is the analysis: {"sentiment": "positive", "confidence": 0.9}',
]

@pytest.mark.parametrize("output", problematic_outputs)
def test_parser_handles_problematic_outputs(output):
    """Verifies that the parser handles problematic outputs."""
    try:
        result = parse_sentiment_response(output)
        # If it gets here without an exception, we verify the structure
        assert "sentiment" in result
        assert "confidence" in result
    except Exception as e:
        # If it fails, the test documents the type of output that isn't handled
        pytest.fail(f"Parser failed on output: {output!r}\nError: {e}")

Prompt Contract Tests

The solution is to treat prompts as behavioral contracts and test them:

# tests/unit/test_prompt_contracts.py
import pytest
from unittest.mock import patch, MagicMock
from app.sentiment import analyze_sentiment  # Function that builds the prompt and calls the LLM

def create_mock_response(content: str):
    """Factory for realistic mock responses."""
    mock = MagicMock()
    mock.choices = [MagicMock()]
    mock.choices[0].message.content = content
    return mock

class TestSentimentPromptContract:
    """The sentiment prompt must ALWAYS produce this structure."""

    @patch("app.sentiment.client.chat.completions.create")
    def test_returns_dict(self, mock_create):
        """Contract: the result is a dict."""
        mock_create.return_value = create_mock_response(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("I love this!")
        assert isinstance(result, dict)

    @patch("app.sentiment.client.chat.completions.create")
    def test_has_required_keys(self, mock_create):
        """Contract: the result has the required keys."""
        mock_create.return_value = create_mock_response(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("I love this!")
        assert "sentiment" in result
        assert "confidence" in result

    @patch("app.sentiment.client.chat.completions.create")
    def test_sentiment_is_valid_value(self, mock_create):
        """Contract: sentiment is one of the valid values."""
        mock_create.return_value = create_mock_response(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("I love this!")
        assert result["sentiment"] in ["positive", "negative", "neutral"]

    @patch("app.sentiment.client.chat.completions.create")
    def test_confidence_is_valid_range(self, mock_create):
        """Contract: confidence is in the range [0.0, 1.0]."""
        mock_create.return_value = create_mock_response(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("I love this!")
        assert 0.0 <= result["confidence"] <= 1.0

These tests don't verify whether the detected sentiment is correct (that's evaluation). They verify that the prompt's structural contract is met.

Problem 3: Model Drift

What it is

LLM providers update their models constantly. gpt-4o-mini from January 2025 and gpt-4o-mini from July 2025 can have slightly different behaviors for the same prompt.

Real cases of model drift

Scenario 1: Silent model update
- Your app uses "gpt-3.5-turbo" (an alias that always points to the latest)
- OpenAI updates which model is "latest"
- Your prompt worked with the previous model
- The new model produces slightly different JSON formatting
- Your parser fails silently
- Users receive errors for 6 hours before anyone notices

Scenario 2: Model migration
- You decide to migrate from gpt-3.5-turbo to gpt-4o-mini for cost
- Apparently it's a drop-in replacement
- 3 prompts out of 12 produce output with a different format
- Without regression tests you don't know which 3 until production

Regression tests for model drift

# tests/regression/test_model_regression.py
"""
Regression tests to detect model drift.
They run weekly or before migrating a model.
They're slower (they call the real API) and cost money.
"""
import pytest
import json

# Only run in CI weekly or with a special flag
pytestmark = pytest.mark.regression

@pytest.fixture
def real_sentiment_app():
    """Real app with an LLM, for regression tests."""
    from app.sentiment import SentimentAnalyzer
    return SentimentAnalyzer()  # Uses a real API key

class TestSentimentRegression:
    """
    These tests verify that the current model still behaves
    the same as when we configured the prompts.
    """

    @pytest.mark.parametrize("text,expected_sentiment", [
        ("I absolutely love this product!", "positive"),
        ("This is terrible, worst purchase ever.", "negative"),
        ("The package arrived on Tuesday.", "neutral"),
    ])
    def test_sentiment_matches_golden_set(self, real_sentiment_app, text, expected_sentiment):
        """
        Golden set: cases where the sentiment is unambiguous.
        If the LLM fails on these, there's significant model drift.
        """
        result = real_sentiment_app.analyze(text)
        assert result["sentiment"] == expected_sentiment, (
            f"Model drift detected for text: {text!r}\n"
            f"Expected: {expected_sentiment}, Got: {result['sentiment']}"
        )

Run regression tests:

# Weekly (GitHub Actions cron)
pytest -m regression -v --tb=short

# Before migrating a model
pytest -m regression -v

Problem 4: Cost Per Test

The real math

# Example: document analysis app
# Model: gpt-4o-mini
# Tokens per test: ~500 input + ~200 output

# Current prices (check at platform.openai.com/pricing):
INPUT_PRICE_PER_1M = 0.15   # $0.15 per 1M input tokens
OUTPUT_PRICE_PER_1M = 0.60  # $0.60 per 1M output tokens

TOKENS_INPUT = 500
TOKENS_OUTPUT = 200

cost_per_test = (
    (TOKENS_INPUT / 1_000_000) * INPUT_PRICE_PER_1M +
    (TOKENS_OUTPUT / 1_000_000) * OUTPUT_PRICE_PER_1M
)
print(f"Cost per test: ${cost_per_test:.6f}")  # $0.000195

# Suite of 100 tests
cost_100_tests = cost_per_test * 100
print(f"100 tests: ${cost_100_tests:.4f}")  # $0.0195

# If you do 20 PRs per day, each one runs the suite:
daily_cost = cost_100_tests * 20
print(f"Daily cost: ${daily_cost:.2f}")  # $0.39/day

# Monthly:
monthly_cost = daily_cost * 22  # business days
print(f"Monthly cost: ${monthly_cost:.2f}")  # $8.58/month

It seems like little. But consider:

• Suite grows to 500 tests = $43/month
• More expensive model (gpt-4o): ~30x more expensive = ~$1,290/month

3-level strategy

Level 1: Unit tests with mocks (fast, cheap)
├── Cost: $0 (they don't call the API)
├── Speed: <1 second per test
├── When to run: on every save (watch mode) and every PR
└── Cover: 80% of the functionality

Level 2: Integration tests with a real LLM (slow, have a cost)
├── Cost: $X per run (depends on the suite)
├── Speed: 2-10 seconds per test
├── When to run: on every PR (with a budget limit) or nightly
└── Cover: critical end-to-end flows

Level 3: Regression tests (slow, have a cost)
├── Cost: $XX per run
├── Speed: variable
├── When to run: weekly or pre-deploy
└── Cover: golden set to detect model drift

# pytest.ini — configuration of the 3-level strategy
"""
[pytest]
testpaths = tests
markers =
    unit: Tests with mocks. Fast, no cost. (default in CI)
    integration: Tests with a real LLM. They have a cost. Run on PR.
    regression: Regression tests. Expensive. Run weekly.
    smoke: Smoke tests. Verify that the system boots.
addopts = -v --tb=short
"""

# CI on every commit: only unit + smoke (no cost)
pytest -m "unit or smoke"

# CI on every PR: unit + smoke + integration (with a budget limit)
pytest -m "not regression" --max-time=120

# Weekly cron: everything including regression
pytest --all

Why not testing costs more than testing

The following table summarizes the real cost of not having tests:

The monthly cost of a well-designed test suite (unit with free mocks + controlled integration) is less than one hour of debugging by a senior engineer.

Complete map: what to test and how

# Classification of testing strategy by component

TESTING_MAP = {
    # DETERMINISTIC → exact assert, no cost, fast
    "input_validator": {
        "strategy": "exact assert",
        "mock_llm": False,
        "example": "assert validate_input('') raises ValueError"
    },
    "prompt_builder": {
        "strategy": "exact assert on string",
        "mock_llm": False,
        "example": "assert '{text}' in build_prompt(text='hello')"
    },
    "output_parser": {
        "strategy": "exact assert with fixed inputs",
        "mock_llm": False,
        "example": "assert parse_json_output('{\"x\": 1}') == {'x': 1}"
    },
    "output_validator": {
        "strategy": "exact assert with schema",
        "mock_llm": False,
        "example": "assert validate_schema({'sentiment': 'positive'}) is True"
    },

    # CHAIN/PIPELINE → mock LLM, assert flow
    "llm_chain": {
        "strategy": "mock LLM + assert result",
        "mock_llm": True,
        "example": "mock llm returns JSON, assert chain returns parsed dict"
    },
    "orchestrator": {
        "strategy": "mock external services + assert flow",
        "mock_llm": True,
        "example": "mock llm + db, assert orchestrator calls in right order"
    },

    # NON-DETERMINISTIC → semantic assertions (module 3)
    "llm_output_quality": {
        "strategy": "semantic similarity / property-based (see M3)",
        "mock_llm": False,
        "example": "assert len(response) > 50 and 'Python' in response"
    },
}

Integrated example: summarization app

To anchor everything above, here's the complete map of a real app:

# src/app/summarizer.py
import json
from openai import OpenAI

client = OpenAI()

# COMPONENT 1: Deterministic (testable with an exact assert)
def validate_input(text: str, max_chars: int = 10000) -> str:
    """Validates and cleans the input before sending it to the LLM."""
    if not text or not text.strip():
        raise ValueError("Input cannot be empty")
    if len(text) > max_chars:
        raise ValueError(f"Input exceeds {max_chars} characters")
    return text.strip()

# COMPONENT 2: Deterministic (testable with an exact assert)
def build_summary_prompt(text: str, language: str = "en") -> str:
    """Builds the prompt to summarize."""
    return f"""Summarize the following text into 3 key points.
Respond ONLY with JSON: {{"points": ["point1", "point2", "point3"]}}
Response language: {language}

Text: {text}"""

# COMPONENT 3: Non-deterministic (mock in unit tests)
def call_llm(prompt: str) -> str:
    """Calls the LLM. Non-deterministic."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.3,  # A low temperature reduces variability
    )
    return response.choices[0].message.content

# COMPONENT 4: Deterministic (testable with an exact assert)
def parse_summary_response(raw: str) -> dict:
    """Parses the LLM's JSON response."""
    # Extract JSON even if there's surrounding text
    start = raw.find("{")
    end = raw.rfind("}") + 1
    if start == -1 or end == 0:
        raise ValueError(f"No JSON found in: {raw!r}")
    return json.loads(raw[start:end])

# COMPONENT 5: Deterministic (testable with an exact assert)
def validate_summary(parsed: dict) -> dict:
    """Validates that the summary has the correct structure."""
    if "points" not in parsed:
        raise ValueError("Missing key 'points' in response")
    if not isinstance(parsed["points"], list):
        raise ValueError("'points' must be a list")
    if len(parsed["points"]) != 3:
        raise ValueError(f"Expected 3 points, got {len(parsed['points'])}")
    return parsed

# MAIN FUNCTION: Orchestrates everything (testable by mocking the LLM)
def summarize(text: str, language: str = "en") -> dict:
    """Main function: validate → build prompt → call LLM → parse → validate."""
    validated_text = validate_input(text)
    prompt = build_summary_prompt(validated_text, language)
    raw_response = call_llm(prompt)  # ← the only non-deterministic point
    parsed = parse_summary_response(raw_response)
    return validate_summary(parsed)

Tests for each component:

# tests/unit/test_summarizer.py
import pytest
from unittest.mock import patch, MagicMock
from app.summarizer import (
    validate_input, build_summary_prompt,
    parse_summary_response, validate_summary, summarize
)

# Tests for validate_input — completely deterministic
class TestValidateInput:
    def test_valid_input(self):
        assert validate_input("Hello world") == "Hello world"

    def test_strips_whitespace(self):
        assert validate_input("  Hello  ") == "Hello"

    def test_empty_raises(self):
        with pytest.raises(ValueError, match="empty"):
            validate_input("")

    def test_too_long_raises(self):
        with pytest.raises(ValueError, match="exceeds"):
            validate_input("x" * 10001)

# Tests for build_summary_prompt — completely deterministic
class TestBuildSummaryPrompt:
    def test_includes_text(self):
        prompt = build_summary_prompt("my text")
        assert "my text" in prompt

    def test_includes_language(self):
        prompt = build_summary_prompt("text", language="en")
        assert "en" in prompt

    def test_includes_json_structure(self):
        prompt = build_summary_prompt("text")
        assert '"points"' in prompt

# Tests for parse_summary_response — completely deterministic
class TestParseSummaryResponse:
    def test_valid_json(self):
        result = parse_summary_response('{"points": ["a", "b", "c"]}')
        assert result == {"points": ["a", "b", "c"]}

    def test_json_with_surrounding_text(self):
        raw = 'Here it is: {"points": ["a", "b", "c"]} end.'
        result = parse_summary_response(raw)
        assert result["points"] == ["a", "b", "c"]

    def test_no_json_raises(self):
        with pytest.raises(ValueError, match="No JSON found"):
            parse_summary_response("No JSON here")

# Test for summarize — mocks the LLM
class TestSummarize:
    @patch("app.summarizer.client.chat.completions.create")
    def test_summarize_returns_valid_structure(self, mock_create):
        """With a mocked LLM, verify that the function orchestrates correctly."""
        mock_response = MagicMock()
        mock_response.choices[0].message.content = (
            '{"points": ["Point 1", "Point 2", "Point 3"]}'
        )
        mock_create.return_value = mock_response

        result = summarize("A long text about Python and AI.")

        assert isinstance(result, dict)
        assert "points" in result
        assert len(result["points"]) == 3
        mock_create.assert_called_once()  # Verify it called the LLM exactly once

Comparison of testing strategies

Troubleshooting

Problem: My test fails intermittently with the same code. Solution: It's a flaky test — you're calling the real LLM in a test that should mock it. Identify the un-mocked call with --tb=long and add @patch.

Problem: I mock the LLM but the test still calls the API (I see charges). Solution: The patch path is wrong. The patch must point to where the object is used, not where it's defined. If app.summarizer does from openai import OpenAI; client = OpenAI(), the correct patch is @patch("app.summarizer.client.chat.completions.create").

Problem: The regression tests are very slow and expensive. Solution: Use a small golden set (10-20 unambiguous cases) instead of testing everything. The goal is to detect significant drift, not perfection.

Problem: I don't know how to tell a non-determinism bug from a real bug. Solution: Run the test 3 times. If it fails consistently → real bug. If it fails 1 out of 3 → flaky test or non-determinism. The solution is always to mock the LLM in unit tests.

Problem: The LLM produces valid JSON but with different keys depending on the day. Solution: Your prompt doesn't specify the schema enough. Add an explicit JSON example in the prompt and/or use OpenAI structured outputs (function calling or response_format={"type": "json_object"}).

Exercises

Exercise 1: Classify your app's components

Take your LLM app and classify each function into: (a) purely deterministic, (b) requires an LLM mock, (c) requires semantic assertions.

# (a) Purely deterministic
def build_prompt(context: str) -> str: ...
def parse_json_response(raw: str) -> dict: ...
def validate_schema(data: dict) -> bool: ...

# (b) Requires an LLM mock
def generate_summary(text: str) -> dict:
    prompt = build_prompt(text)
    raw = call_llm(prompt)  # ← mock point
    return parse_json_response(raw)

# (c) Requires semantic assertions
# Tests that verify the summary is "correct"
# → Module 3

Exercise 2: Calculate the cost of your current (or proposed) suite

Estimate the monthly cost if your suite of 50 tests called the real API on each PR (20 PRs/business day).

# Calculation
tests = 50
tokens_input = 500
tokens_output = 200
prs_per_day = 20
days_per_month = 22

# gpt-4o-mini (check current prices)
cost_per_input_token = 0.15 / 1_000_000
cost_per_output_token = 0.60 / 1_000_000

cost_per_test = (
    tokens_input * cost_per_input_token +
    tokens_output * cost_per_output_token
)
cost_per_run = cost_per_test * tests
cost_monthly = cost_per_run * prs_per_day * days_per_month

print(f"Cost per test: ${cost_per_test:.6f}")
print(f"Cost per run: ${cost_per_run:.4f}")
print(f"Monthly cost: ${cost_monthly:.2f}")

# With mocks: $0 for unit tests
# Only integration tests (5-10% of the total) use the real API

Exercise 3: Write a contract test

For the summarize() function from this session's example, write a contract test that verifies the output has exactly 3 points and each one is a non-empty string.

# tests/unit/test_summarize_contract.py
import pytest
from unittest.mock import patch, MagicMock
from app.summarizer import summarize

@patch("app.summarizer.client.chat.completions.create")
def test_summarize_three_non_empty_points(mock_create):
    """
    Contract: summarize() always returns exactly 3 non-empty points.
    This test verifies the structure, not the quality of the content.
    """
    # Arrange
    mock_response = MagicMock()
    mock_response.choices[0].message.content = (
        '{"points": ["Python is versatile", "Python is popular in AI", "Python has a large ecosystem"]}'
    )
    mock_create.return_value = mock_response

    # Act
    result = summarize("Text about Python for demonstration.")

    # Assert — structural contract
    assert "points" in result
    assert len(result["points"]) == 3
    for point in result["points"]:
        assert isinstance(point, str), f"Point must be a string: {point!r}"
        assert len(point.strip()) > 0, f"Point cannot be empty: {point!r}"

Exercise 4: Identify prompt fragility

The following prompt has 2 ways to fail silently. Identify them and propose solutions.

prompt = """Analyze this customer feedback and provide insights.
Return your analysis as JSON.
Feedback: {text}"""

prompt = """Analyze this customer feedback.
Return ONLY this JSON (no other text):
{{"sentiment": "positive|negative|neutral", "key_issue": "string", "recommendation": "string"}}

Feedback: {text}"""

@pytest.mark.parametrize("output", [
    '{"sentiment": "positive", "key_issue": "pricing", "recommendation": "add discount"}',
])
def test_feedback_prompt_contract(output, mocker):
    mocker.patch("app.feedback.call_llm", return_value=output)
    result = analyze_feedback("Great product but too expensive!")
    assert all(key in result for key in ["sentiment", "key_issue", "recommendation"])

Exercise 5: Design the testing strategy for your app

For your LLM app (or the project's reference one), define:

1. How many unit tests (with mocks) do you need? What do they cover?
2. How many integration tests (real LLM) do you need? When do they run?
3. Do you need regression tests? With what golden set?

Unit tests (with mocks):
- One test per deterministic function (validators, parsers, builders)
- One test per execution path in the main function (happy path, error paths)
- Goal: cover 80% of the code without API calls
- When: on every save, on every PR

Integration tests (real LLM):
- 1-3 tests per critical user flow
- Only the most-used flows in production
- Goal: verify that the end-to-end system works with a real LLM
- When: on every PR with a budget limit ($0.50-1 per PR)

Regression tests:
- Only if the model changes frequently or if you have an SLA
- Small golden set (10-20 unambiguous cases)
- When: weekly or before an important deploy

Summary

• Non-determinism only affects the LLM output (~20-30% of the code); the rest is deterministic and testable with normal asserts
• Prompt fragility: a minimal change can break parsing — contract tests catch this before deploy
• Model drift: models change; regression tests with golden sets detect behavioral changes
• Cost per test: unit tests should use mocks ($0); only integration tests use a real LLM (with budget controls)
• The optimal strategy has 3 levels: unit (mocks, always), integration (real LLM, on PR), regression (real LLM, weekly)
• The ROI of testing in AI is even higher than in traditional software because of the silent nature of the failures

Additional resources

1. Testing ML Systems (Google) — Testing for ML/AI systems in production
2. Non-Determinism in Testing (Martin Fowler) — Why flaky tests are so harmful and how to eliminate them
3. OpenAI Structured Outputs — response_format to eliminate the JSON parsing problem
4. pytest-mock — Cleaner mocking integration with pytest
5. Property-Based Testing with Hypothesis — Strategy to handle non-determinism (covered in depth in module 3)
6. OpenAI API Pricing — To calculate the real cost of your test suite
7. Evals for LLM Applications (Hamel Husain) — Mental framework to decide what to evaluate vs what to test