Module 1: Testing Fundamentals for AI

6. Taxonomy of Tests: Smoke, Contract, Behavioral, Regression

Description

Not all tests are equal, and not all are worth the same amount of writing time. Without a clear taxonomy, teams tend to write tests at random, duplicate effort, and never have coverage where it matters most. This session defines four types of tests specific to AI apps — smoke, contract, behavioral, and regression — with clear criteria for what each one writes and when to run it.

The taxonomy isn't academic theory. It's a decision framework: when you have 2 hours to write tests before a deploy, what do you write first? When a bug reaches production, what test would have caught it? When the LLM changes version, what do you run to validate that nothing broke?

By the end you'll have clarity on which type of test corresponds to each situation, how to organize them with pytest markers, and what the writing priority is to maximize value with the least time invested.

The four types of tests for AI

┌─────────────────────────────────────────────────────────────────┐
│                    TEST PYRAMID FOR AI                          │
│                                                                  │
│          ┌───────────────────────────────┐                      │
│          │      REGRESSION TESTS         │  → Add when          │
│          │  "Did any bug come back?"     │    you find bugs     │
│          └───────────────────────────────┘                      │
│        ┌─────────────────────────────────────┐                  │
│        │       BEHAVIORAL TESTS              │  → After         │
│        │  "Does the output have the          │    contracts     │
│        │   expected properties?"             │                  │
│        └─────────────────────────────────────┘                  │
│      ┌─────────────────────────────────────────────┐            │
│      │           CONTRACT TESTS                    │  → Second  │
│      │  "Does the output meet the expected structure?"│  step   │
│      └─────────────────────────────────────────────┘            │
│    ┌─────────────────────────────────────────────────────┐      │
│    │                  SMOKE TESTS                        │      │
│    │  "Does the system start up and respond?"            │  → First │
│    └─────────────────────────────────────────────────────┘      │
└─────────────────────────────────────────────────────────────────┘

Type 1: Smoke Tests

Definition

Smoke tests verify that the system is alive and responds. They don't prove that the logic is correct — only that the code starts up, the modules import without errors, and the main endpoints respond.

The name comes from hardware: "Does it smoke when you turn it on?" If the system smokes (crashes on startup), nothing else matters.

Characteristics

Speed: Very fast (<1 second per test)
Cost: $0 (they don't call the real LLM)
Deterministic: 100%
When to run: Always — on every commit, on every PR, locally before starting work

Examples

# tests/unit/test_smoke.py
import pytest
from app import llm, parsers, sentiment


@pytest.mark.smoke
def test_app_modules_import_without_errors():
    """The app package and its modules import without errors."""
    # If any import fails (NameError, ImportError, SyntaxError),
    # this test fails and indicates a critical problem in the code
    assert llm is not None
    assert parsers is not None
    assert sentiment is not None


@pytest.mark.smoke
def test_main_functions_are_callable():
    """The system's main functions are callable."""
    from app.sentiment import analyze_sentiment
    from app.summarizer import summarize
    from app.parsers import parse_json_from_llm_output

    assert callable(analyze_sentiment)
    assert callable(summarize)
    assert callable(parse_json_from_llm_output)


@pytest.mark.smoke
def test_config_loads_without_errors():
    """The system configuration loads correctly."""
    from app.config import settings
    # Only verifies that it exists, not that the values are correct
    assert settings is not None
    assert hasattr(settings, "model_name")
    assert hasattr(settings, "max_tokens")


# For apps with FastAPI:
@pytest.mark.smoke
def test_api_health_check_returns_200(test_client):
    """The /health endpoint responds with 200."""
    response = test_client.get("/health")
    assert response.status_code == 200


@pytest.mark.smoke
def test_api_docs_accessible(test_client):
    """The Swagger documentation is accessible."""
    response = test_client.get("/docs")
    assert response.status_code == 200


@pytest.mark.smoke
def test_main_endpoint_accepts_valid_request(test_client):
    """The main endpoint accepts a valid request (no 404, no 422)."""
    response = test_client.post("/analyze", json={"text": "Test input"})
    # We don't verify the result — only that the endpoint exists and accepts the format
    assert response.status_code != 404, "Endpoint /analyze doesn't exist"
    assert response.status_code != 422, "Endpoint /analyze rejects the request format"

How many smoke tests you need

Minimal LLM app: 3-5 smoke tests
├── test_modules_import           ← ALWAYS
├── test_main_function_callable   ← ALWAYS
└── test_config_loads             ← If you have config

App with FastAPI: 5-8 smoke tests
├── test_modules_import
├── test_health_check_200
├── test_main_endpoint_accepts_request
├── test_docs_accessible          ← Optional
└── test_db_connection            ← If you have a DB

Type 2: Contract Tests

Definition

Contract tests verify that a component fulfills its "contract": the structure and types of the output it promised to produce. For LLM apps, the most important contract is the prompt's: "this prompt always produces JSON with these keys and these types."

The name comes from Design by Contract: if the function promises to return {"sentiment": str, "confidence": float}, the contract test verifies exactly that.

Characteristics

Speed: Fast with mocks (<100ms per test)
Cost: $0 with mocks
Deterministic: 100% with mocks
When to run: Always — on every commit and PR

Examples

# tests/unit/test_contracts.py
import pytest
from unittest.mock import patch, MagicMock
from app.sentiment import analyze_sentiment
from app.summarizer import summarize


def create_mock(content: str) -> MagicMock:
    mock = MagicMock()
    mock.choices[0].message.content = content
    return mock


@pytest.mark.contract
class TestSentimentPromptContract:
    """
    The sentiment analysis prompt must ALWAYS produce:
    - A dict
    - With keys "sentiment" and "confidence"
    - "sentiment" is one of ["positive", "negative", "neutral"]
    - "confidence" is a float in [0.0, 1.0]
    """

    @patch("app.sentiment.client.chat.completions.create")
    def test_returns_dict(self, mock_create):
        mock_create.return_value = create_mock(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("text")
        assert isinstance(result, dict), f"Expected dict, got {type(result)}"

    @patch("app.sentiment.client.chat.completions.create")
    def test_has_sentiment_key(self, mock_create):
        mock_create.return_value = create_mock(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("text")
        assert "sentiment" in result, f"Missing 'sentiment' key. Got: {result}"

    @patch("app.sentiment.client.chat.completions.create")
    def test_has_confidence_key(self, mock_create):
        mock_create.return_value = create_mock(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("text")
        assert "confidence" in result, f"Missing 'confidence' key. Got: {result}"

    @patch("app.sentiment.client.chat.completions.create")
    def test_sentiment_is_valid_value(self, mock_create):
        mock_create.return_value = create_mock(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("text")
        valid_sentiments = {"positive", "negative", "neutral"}
        assert result["sentiment"] in valid_sentiments, (
            f"sentiment must be one of {valid_sentiments}, got: {result['sentiment']!r}"
        )

    @patch("app.sentiment.client.chat.completions.create")
    def test_confidence_is_numeric(self, mock_create):
        mock_create.return_value = create_mock(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("text")
        assert isinstance(result["confidence"], (int, float)), (
            f"confidence must be numeric, got: {type(result['confidence'])}"
        )

    @patch("app.sentiment.client.chat.completions.create")
    def test_confidence_in_valid_range(self, mock_create):
        mock_create.return_value = create_mock(
            '{"sentiment": "positive", "confidence": 0.9}'
        )
        result = analyze_sentiment("text")
        assert 0.0 <= result["confidence"] <= 1.0, (
            f"confidence out of range [0,1]: {result['confidence']}"
        )


@pytest.mark.contract
class TestSummarizerPromptContract:
    """
    The summarization prompt must ALWAYS produce:
    - A dict with key "points"
    - "points" is a list of exactly 3 non-empty strings
    """

    @patch("app.summarizer.client.chat.completions.create")
    def test_returns_dict_with_points(self, mock_create):
        mock_create.return_value = create_mock(
            '{"points": ["Point 1", "Point 2", "Point 3"]}'
        )
        result = summarize("long text")
        assert isinstance(result, dict)
        assert "points" in result
        assert isinstance(result["points"], list)

    @patch("app.summarizer.client.chat.completions.create")
    def test_returns_exactly_three_points(self, mock_create):
        mock_create.return_value = create_mock(
            '{"points": ["Point 1", "Point 2", "Point 3"]}'
        )
        result = summarize("long text")
        assert len(result["points"]) == 3, (
            f"Expected 3 points, got {len(result['points'])}: {result['points']}"
        )

    @patch("app.summarizer.client.chat.completions.create")
    def test_each_point_is_non_empty_string(self, mock_create):
        mock_create.return_value = create_mock(
            '{"points": ["Point 1", "Point 2", "Point 3"]}'
        )
        result = summarize("long text")
        for i, point in enumerate(result["points"]):
            assert isinstance(point, str), f"Point {i} must be str: {point!r}"
            assert len(point.strip()) > 0, f"Point {i} cannot be empty"

Contract tests for parsers

@pytest.mark.contract
class TestJsonParserContract:
    """
    parse_json_from_llm_output must ALWAYS:
    - Return a dict for valid JSON (in any format)
    - Raise ValueError for input without JSON
    - Raise JSONDecodeError for malformed JSON
    """

    @pytest.mark.parametrize("raw_json,expected", [
        ('{"x": 1}', {"x": 1}),
        ('```json\n{"x": 1}\n```', {"x": 1}),
        ('Result: {"x": 1}', {"x": 1}),
    ])
    def test_returns_dict_for_valid_json(self, raw_json, expected):
        result = parse_json_from_llm_output(raw_json)
        assert result == expected

    def test_raises_for_no_json(self):
        with pytest.raises(ValueError):
            parse_json_from_llm_output("No JSON here")

    def test_raises_for_malformed_json(self):
        import json
        with pytest.raises(json.JSONDecodeError):
            parse_json_from_llm_output('{"incomplete":')

Type 3: Behavioral Tests

Definition

Behavioral tests verify that the output has the expected properties without comparing the exact value. They're more flexible than contract tests (which verify structure) and are used when there's something that can't be completely deterministic or when the specification is "within a range" instead of "exactly X".

Characteristics

Speed: Fast with mocks; slow with a real LLM
Cost: $0 with mocks; $X with a real LLM
Deterministic: With mocks yes; with a real LLM, partially
When to run: Always with mocks; only on PR/nightly with a real LLM

Examples

# tests/unit/test_behavioral.py
import pytest
from unittest.mock import patch, MagicMock


@pytest.mark.behavioral
class TestSentimentBehavior:
    """Behavior tests — output properties, not exact values."""

    @patch("app.sentiment.client.chat.completions.create")
    def test_confidence_reflects_certainty(self, mock_create):
        """
        For unambiguous input (clearly positive or negative),
        confidence must be high (>0.7).
        This test uses an LLM mock with an appropriate response for strong text.
        """
        mock_create.return_value = create_mock(
            '{"sentiment": "positive", "confidence": 0.95}'
        )
        result = analyze_sentiment("I absolutely LOVE this! Best product EVER!")
        # Behavioral: confidence must be high for strong text
        assert result["confidence"] > 0.7, (
            f"For very positive text we expect high confidence. Got: {result['confidence']}"
        )

    @patch("app.sentiment.client.chat.completions.create")
    def test_response_is_deterministic_for_same_input(self, mock_create):
        """With mocks, the same input always produces the same output."""
        mock_response = create_mock('{"sentiment": "positive", "confidence": 0.9}')
        mock_create.return_value = mock_response

        result1 = analyze_sentiment("Same text")
        result2 = analyze_sentiment("Same text")

        assert result1 == result2


@pytest.mark.behavioral
class TestSummarizerBehavior:
    """Behavior tests for the summarizer."""

    @patch("app.summarizer.client.chat.completions.create")
    def test_summary_points_are_different_from_each_other(self, mock_create):
        """The summary points must not be duplicated."""
        mock_create.return_value = create_mock(
            '{"points": ["Python is versatile", "Python dominates AI", "Python has a large ecosystem"]}'
        )
        result = summarize("Long text about Python and AI in the industry")

        # Behavioral: the points must not be duplicated
        points = result["points"]
        unique_points = set(points)
        assert len(unique_points) == len(points), (
            f"The summary points contain duplicates: {points}"
        )

    @patch("app.summarizer.client.chat.completions.create")
    def test_summary_points_have_minimum_length(self, mock_create):
        """Each summary point must have a minimum length."""
        mock_create.return_value = create_mock(
            '{"points": ["Point with enough information to be useful", '
            '"Another point with significant content", '
            '"The third point also has relevant content"]}'
        )
        result = summarize("Long text")

        for i, point in enumerate(result["points"]):
            assert len(point) >= 10, (
                f"Point {i} too short ({len(point)} chars): {point!r}"
            )


# Integration behavioral tests (with a real LLM)
@pytest.mark.integration
@pytest.mark.behavioral
class TestSentimentBehaviorIntegration:
    """Behavior tests with a real LLM — they have a cost."""

    def test_positive_text_detected_correctly(self):
        """For clearly positive text, the sentiment must be positive."""
        result = analyze_sentiment("I absolutely love this product! Amazing quality!")
        # Behavioral: for unambiguously positive text, the LLM must detect positive
        assert result["sentiment"] == "positive", (
            f"Expected positive sentiment for clearly positive text. Got: {result['sentiment']}"
        )

    def test_negative_text_detected_correctly(self):
        """For clearly negative text, the sentiment must be negative."""
        result = analyze_sentiment("Terrible experience. Worst product ever. Never again.")
        assert result["sentiment"] == "negative"

Type 4: Regression Tests

Definition

Regression tests verify that known bugs don't reappear. They're created AFTER finding and fixing a bug: when you fix the bug, you add a test with the exact input that caused it. If someone inadvertently reintroduces the same bug, the test fails.

Characteristics

Speed: Variable (depends on the bug)
Cost: $0 if the bug was in parsers; $X if it was in the LLM response
Deterministic: Yes (with mocks for LLM bugs)
When to run: Always — on every commit and PR

Process for creating a regression test

Bug found in production:
"Input with emojis causes a UnicodeDecodeError in the parser"

1. Reproduce the bug:
   >>> parse_json_from_llm_output('{"text": "I love 🎉 this!"}')
   UnicodeDecodeError: ...  ← Confirmed

2. Create the regression test BEFORE the fix:
   def test_parser_regression_handles_emoji_in_json():
       raw = '{"text": "I love 🎉 this!"}'
       result = parse_json_from_llm_output(raw)  ← Fails (expected)
       assert result["text"] == "I love 🎉 this!"

3. Implement the fix

4. Verify that the regression test passes:
   pytest tests/regression/test_parser_regression.py -v  ← Now passes ✅

5. The test stays in the suite permanently

Examples

# tests/regression/test_parser_regression.py
"""
Regression tests: bugs that have been found and fixed.
These tests must NOT be deleted — they prevent the bugs from returning.
Each test must have a comment with the date and description of the bug.
"""
import pytest
from app.parsers import parse_json_from_llm_output


@pytest.mark.regression
class TestParserRegressions:

    def test_handles_emojis_in_json_content(self):
        """
        Regression [2026-01-15]: The parser raised UnicodeDecodeError
        when the JSON contained emojis. Fix: use encoding='utf-8' in json.loads.
        """
        raw = '{"text": "I love 🎉 this product! Amazing! 🚀"}'
        result = parse_json_from_llm_output(raw)
        assert "🎉" in result["text"]
        assert "🚀" in result["text"]

    def test_handles_unicode_characters(self):
        """
        Regression [2026-01-20]: Parser failed with Spanish characters ñ, ü, é.
        Bug related to the emoji one.
        """
        raw = '{"resumen": "El niño aprendió inglés y matemáticas"}'
        result = parse_json_from_llm_output(raw)
        assert "ñ" in result["resumen"]
        assert "é" in result["resumen"]

    def test_handles_nested_quotes_in_json(self):
        """
        Regression [2026-02-03]: The parser failed when the JSON content
        had escaped double quotes (\\"). The LLM sometimes produces this.
        """
        raw = '{"quote": "She said \\"hello\\" to me"}'
        result = parse_json_from_llm_output(raw)
        assert 'hello' in result["quote"]

    def test_handles_newlines_in_json_values(self):
        """
        Regression [2026-02-10]: The parser failed when the JSON values
        contained literal newlines (not escaped \\n).
        """
        # The LLM sometimes produces JSON with literal newlines in the values
        raw = '{"text": "First line\\nSecond line\\nThird line"}'
        result = parse_json_from_llm_output(raw)
        assert "\n" in result["text"]


@pytest.mark.regression
class TestSentimentRegressions:

    @patch("app.sentiment.client.chat.completions.create")
    def test_handles_very_long_input_without_timeout(self, mock_create):
        """
        Regression [2026-01-25]: The function "hung" with very long inputs
        because there was no token limit. Fix: truncate the input to 5000 chars.
        """
        mock_create.return_value = create_mock(
            '{"sentiment": "neutral", "confidence": 0.5}'
        )
        very_long_text = "word " * 10000  # 50,000 characters

        # Must complete in a reasonable time (no timeout)
        result = analyze_sentiment(very_long_text)
        assert result["sentiment"] in ["positive", "negative", "neutral"]

    @patch("app.sentiment.client.chat.completions.create")
    def test_does_not_leak_api_key_in_error_message(self, mock_create):
        """
        Regression [2026-02-01]: An error in exception handling
        included the API key in the error message. Fix: sanitize error messages.
        """
        mock_create.side_effect = Exception("Error with key sk-proj-abc123xyz")

        with pytest.raises(Exception) as exc_info:
            analyze_sentiment("text")

        # The error message must NOT contain credentials
        error_message = str(exc_info.value)
        assert "sk-proj" not in error_message, (
            "The error message must not contain the API key"
        )

When to write each type

Decision framework

Situation → Type of test to write

"I'm going to deploy in 1 hour and there are no tests"
→ Smoke tests first (5 min), then contract tests for the critical flow

"I just changed the main prompt"
→ Contract tests to verify that the output structure didn't change

"The LLM produces outputs with different formats depending on the day"
→ Behavioral tests on invariant properties (length, range, type)
→ Contract tests with mocks that cover the possible formats

"I found a bug in production"
→ Regression test with the exact input that caused the bug
→ Fix → regression test passes → integrate into the suite

"I'm going to migrate from gpt-3.5-turbo to gpt-4o-mini"
→ Regression tests with a golden set (the most important behaviors)
→ Run before and after the migration

"I want to know if my refactoring didn't break anything"
→ All existing tests (smoke + contract + behavioral + regression)

Priority with limited time

Time available	What to write
30 minutes	2 smoke tests + 1 contract test for the most critical flow
2 hours	Complete smoke + contract tests for all prompts
1 day	All of the above + behavioral tests + regression for known bugs
1 week	Complete suite with all types for all components

pytest markers for each type

Configuration in pytest.ini

[pytest]
testpaths = tests
markers =
    smoke: Smoke tests. No cost. Verify that the system starts up and responds.
    contract: Contract tests. No cost. Verify the structure and types of the output.
    behavioral: Behavior tests. Variable cost. Verify output properties.
    regression: Regression tests. No cost (mocks). Prevent known bugs.
    unit: Tests with mocks. No cost. Fast.
    integration: Tests with a real LLM. They have a cost. Slow.
addopts = -v --tb=short

Execution strategy by context

# Local development — always run:
pytest -m "smoke or contract or regression"

# Pre-commit — fast tests:
pytest -m "smoke or unit" --no-header -q

# CI on every PR — no cost:
pytest -m "not integration" -v

# CI on PR for the main branch — include integration:
pytest -m "not regression" -v  # Integration yes, regression no (expensive)

# Weekly CI / pre-release — everything:
pytest --all -v

# Verify after a prompt change:
pytest -m "contract" -v

# Verify after a model migration:
pytest -m "regression or behavioral" -v

File organization

tests/
├── conftest.py                     # Global fixtures
├── smoke/
│   └── test_smoke.py               # All the smoke tests
├── unit/
│   ├── contracts/
│   │   ├── test_sentiment_contract.py
│   │   ├── test_summarizer_contract.py
│   │   └── test_parser_contract.py
│   ├── behavioral/
│   │   └── test_behavioral.py
│   └── regression/
│       ├── test_parser_regression.py
│       └── test_sentiment_regression.py
└── integration/
    └── test_e2e.py

Complete comparison of the four types

Aspect	Smoke	Contract	Behavioral	Regression
Question	Does it start?	Correct structure?	Properties OK?	Did bug X come back?
When to write	First	Second	Third	On finding a bug
Uses LLM mock	No (no LLM)	Yes	Yes/no	Yes
Speed	Very fast	Fast	Variable	Variable
Cost	$0	$0	$0 (mock)	$0 (mock)
Determ.	100%	100%	100% (mock)	100%
When to run	Always	Always	Always	Always
Typical number	3-8	5-20	5-15	Grows over time

Troubleshooting

Problem: I don't know whether a test is "contract" or "behavioral". Solution: Contract = verifies exact structure (keys, types, valid values). Behavioral = verifies a property (range, length, relationship between values). If you compare with == → contract. If you compare with >, <, in, isinstance → behavioral.

Problem: I have many regression tests and some are slow. Solution: Regression tests for parsers/validators are super fast (no LLM). Those that require a real LLM must be marked with @pytest.mark.integration in addition to @pytest.mark.regression so you can exclude them from the fast CI.

Problem: A behavioral test with a real LLM fails intermittently. Solution: If the behavioral test uses a real LLM and verifies something like "the sentiment is positive for positive text", it can fail if the LLM changes behavior. Two options: (1) convert it to a unit test with a mock (more robust), or (2) accept that it can be flaky and run it only in the weekly regression.

Problem: I don't know how to handle a smoke test that requires a real connection. Solution: Smoke tests must be as fast and cheap as possible. If the smoke test requires a real LLM, create two versions: smoke with a mock (always runs) and an integration smoke (only on PR).

Problem: How do I organize when a test seems to be of multiple types? Solution: Apply multiple markers. @pytest.mark.regression @pytest.mark.behavioral is perfectly valid. The test is a regression because it arose from a bug, and behavioral because it verifies output properties.

Exercises

Exercise 1: Classify tests

Classify these tests as smoke, contract, behavioral, or regression:

a) test_app_imports_without_errors()
b) test_summarize_returns_dict_with_points_key()
c) test_summary_length_is_at_least_50_chars()
d) test_parser_handles_emoji_input()
e) test_health_endpoint_returns_200()
f) test_confidence_is_between_0_and_1()

See solution

a) test_app_imports_without_errors()          → SMOKE
   "The system starts up" — basic, before everything.

b) test_summarize_returns_dict_with_points_key()  → CONTRACT
   Verifies exact structure: does the "points" key exist?

c) test_summary_length_is_at_least_50_chars()  → BEHAVIORAL
   Verifies a property (minimum length), not an exact value.

d) test_parser_handles_emoji_input()           → REGRESSION
   "handles" + a specific case (emoji) → Arose from a specific bug.

e) test_health_endpoint_returns_200()          → SMOKE
   Verifies that the endpoint exists and responds — basic.

f) test_confidence_is_between_0_and_1()        → CONTRACT
   Verifies a valid range for a specific type — part of the output contract.
   (It could be behavioral if you verify a semantic behavior property)

Exercise 2: Create a contract test from a specification

The /classify endpoint must return:

{
  "category": "technology|sports|politics|entertainment",
  "subcategory": "string (optional)",
  "confidence": "float between 0 and 1"
}

Write a complete contract test.

See solution

@pytest.mark.contract
@patch("app.classifier.client.chat.completions.create")
def test_classify_endpoint_contract(mock_create):
    """
    Contract of the /classify endpoint:
    - Returns a dict with key 'category'
    - 'category' is one of the valid values
    - 'confidence' is in [0.0, 1.0]
    - 'subcategory' is optional but if it exists it's a string
    """
    mock_create.return_value = create_mock(
        '{"category": "technology", "subcategory": "AI", "confidence": 0.92}'
    )

    result = classify_text("OpenAI releases new model with enhanced reasoning capabilities.")

    # 1. It's a dict
    assert isinstance(result, dict)

    # 2. Has key 'category'
    assert "category" in result, f"Missing 'category'. Got: {result}"

    # 3. 'category' has a valid value
    valid_categories = {"technology", "sports", "politics", "entertainment"}
    assert result["category"] in valid_categories, (
        f"Invalid category: {result['category']!r}. Valid ones: {valid_categories}"
    )

    # 4. Has key 'confidence'
    assert "confidence" in result

    # 5. 'confidence' is a float in a valid range
    assert isinstance(result["confidence"], float)
    assert 0.0 <= result["confidence"] <= 1.0

    # 6. If it has 'subcategory', it must be a string
    if "subcategory" in result and result["subcategory"] is not None:
        assert isinstance(result["subcategory"], str)

Exercise 3: From bug to regression test

You describe this bug: "When the input has single quotes ('), the JSON parsing fails with JSONDecodeError because the LLM interprets them as string delimiters."

Write the regression test for this bug.

See solution

@pytest.mark.regression
def test_parser_regression_handles_single_quotes_in_input():
    """
    Regression [2026-02-15]: The function raised JSONDecodeError when the
    user input contained single quotes.

    Cause: The prompt built the JSON with an f-string and the single quotes
    in the input were escaped incorrectly.

    Fix: Use json.dumps() to serialize the user's text in the prompt.
    """
    # Input that caused the bug
    problematic_input = "I'm really happy with this product! It's amazing!"

    # Must process correctly, without an exception
    mock_response_content = '{"sentiment": "positive", "confidence": 0.95}'

    with patch("app.sentiment.client.chat.completions.create") as mock_create:
        mock_create.return_value = create_mock(mock_response_content)
        result = analyze_sentiment(problematic_input)

    # Must not have raised an exception
    assert result is not None
    assert "sentiment" in result
    assert result["sentiment"] == "positive"

Exercise 4: Behavioral test for a distribution property

Write a behavioral test that verifies that, given a long text (>500 words), the summary is significantly shorter (less than 50% of the original length).

See solution

@pytest.mark.behavioral
@patch("app.summarizer.client.chat.completions.create")
def test_summarizer_compresses_long_text(mock_create):
    """
    Behavior property: the summary must be shorter than the original.
    For text >500 words, the summary must be <50% of the length.
    """
    # ─── ARRANGE ───
    long_text = " ".join(["word"] * 600)  # 600 words, ~3000 chars
    mock_create.return_value = create_mock(
        '{"points": ["Summarized point 1", "Summarized point 2", "Summarized point 3"]}'
    )

    # ─── ACT ───
    result = summarize(long_text)

    # ─── ASSERT (behavioral) ───
    # The sum of the points must not exceed 50% of the original
    summary_length = sum(len(p) for p in result["points"])
    original_length = len(long_text)

    assert summary_length < original_length * 0.5, (
        f"The summary ({summary_length} chars) isn't short enough. "
        f"Original: {original_length} chars. "
        f"Ratio: {summary_length/original_length:.1%} (expected <50%)"
    )

Exercise 5: Prioritizing tests

You have 4 hours before an important deploy. Your app has:

3 different prompts
5 output parsers
1 main FastAPI endpoint
2 known bugs that have already been fixed

Which tests do you prioritize? Justify the order.

See guide

Hour 1: Smoke tests (5 min) + Contract tests for the 3 prompts (45 min)
─────────────────────────────────────────────────────────────────
- 1 smoke test: the main endpoint responds without errors
- 1 smoke test: modules import correctly
- 3 contract tests: one per prompt (output structure)

Hour 2: Contract tests for the 5 parsers (60 min)
─────────────────────────────────────────────────
- 5 contract tests: one per parser
- They cover the happy path + 1 edge case each
- Total: ~10 tests in 60 min

Hour 3: Regression tests for the 2 known bugs (30 min)
────────────────────────────────────────────────────────────
- 2 exact regression tests (the inputs that caused the bugs)
- They're the easiest to write: you already know the problematic input

Hour 4: Behavioral tests for the most important behaviors (60 min)
──────────────────────────────────────────────────────────────────────────
- 2-3 behavioral tests for critical properties (length, range, format)

Result: ~18-20 tests in 4 hours
- Coverage: all critical flows have tests
- Zero known risks: smoke + contracts
- Previous bugs documented: regression tests

Summary

Smoke tests: "Does it start?" → Always first, always run, 3-8 tests
Contract tests: "Correct structure?" → Second step, with mocks, one per prompt/parser
Behavioral tests: "Expected properties?" → Third, flexible assertions on invariant properties
Regression tests: "Did the bug come back?" → Created when you find bugs, never deleted
Writing order: smoke → contract → behavioral → regression (when there are bugs)
Each type has its marker: pytest -m contract to verify prompts, pytest -m smoke for a fast sanity check
Contract tests are the heart of LLM app testing: they protect against prompt fragility and model drift

Additional resources

Test Pyramid — Martin Fowler — The original test pyramid and its relevance for AI
Consumer-Driven Contract Testing — Pact — Contract testing in distributed systems (applicable concept)
pytest markers documentation — How to use markers to organize and run subsets
Regression Testing — Wikipedia — Fundamentals of regression testing
Testing ML Systems — Google — Testing strategies for ML systems in production
Property-Based Testing with Hypothesis — For advanced behavioral tests (covered in depth in module 3)

← Previous5. Fixtures for LLM Apps Next →7. Project: Test Suite Setup