7. Project: Prompt Contract Tests

Description

This is the hands-on project for Module 2. You'll build a complete suite of prompt contract tests for the Module 1 app. When you finish, you'll have tests that validate all of the app's prompts without making a single call to the real LLM: 0 API calls, execution in <10 seconds, significant coverage.

Starting point: the Module 1 app

You'll work on the sentiment analysis app from Module 1. If you don't have it, here's the reference version:

src/
├── app/
│   ├── __init__.py
│   ├── config.py          ← LLM configuration
│   ├── parsers.py         ← Parser for the LLM's JSON output
│   ├── processors.py      ← Output processor (normalization)
│   ├── sentiment.py       ← Sentiment analysis logic
│   └── main.py            ← FastAPI app
tests/
├── conftest.py            ← Shared fixtures (from Module 1)
├── helpers.py             ← create_openai_chat_response
├── smoke/
│   └── test_smoke.py      ← Smoke tests (from Module 1)
└── unit/
    ├── contracts/
    │   └── test_contracts.py
    ├── parsers/
    │   └── test_parsers.py
    └── regression/
        └── test_regression.py
pytest.ini
requirements.txt

The reference app: complete code

src/app/__init__.py

# Empty

src/app/config.py

import os
from dataclasses import dataclass

@dataclass
class Config:
    openai_api_key: str
    model: str = "gpt-4o-mini"
    temperature: float = 0.0
    max_tokens: int = 500
    
def get_config() -> Config:
    api_key = os.getenv("OPENAI_API_KEY", "test-key-placeholder")
    return Config(openai_api_key=api_key)

src/app/parsers.py

import json
import re
from typing import Any

def parse_json_response(raw: str) -> dict:
    """
    Extracts and parses JSON from the LLM output.
    
    Supports:
    - Direct JSON: '{"key": "value"}'
    - JSON in markdown: '```json\\n{...}\\n```'
    - JSON with preceding text: 'Result: {...}'
    
    Raises:
        ValueError: If no valid JSON is found in the string
    """
    if not raw or not raw.strip():
        raise ValueError("The LLM response is empty")
    
    # Attempt 1: parse directly
    try:
        return json.loads(raw.strip())
    except json.JSONDecodeError:
        pass
    
    # Attempt 2: extract from a markdown code block
    markdown_match = re.search(r'```(?:json)?\s*\n?(.*?)\n?```', raw, re.DOTALL)
    if markdown_match:
        try:
            return json.loads(markdown_match.group(1).strip())
        except json.JSONDecodeError:
            pass
    
    # Attempt 3: look for any JSON object in the text
    json_matches = re.findall(r'\{[^{}]*(?:\{[^{}]*\}[^{}]*)*\}', raw, re.DOTALL)
    for match in json_matches:
        try:
            return json.loads(match)
        except json.JSONDecodeError:
            continue
    
    raise ValueError(f"No valid JSON found in: {raw[:100]!r}")

src/app/processors.py

VALID_SENTIMENTS = {"positive", "negative", "neutral"}

def process_sentiment_output(raw_dict: dict) -> dict:
    """
    Normalizes and validates the sentiment parser's output.
    
    Guarantees:
    - sentiment always in VALID_SENTIMENTS
    - score always a float 0.0-1.0
    - keywords always a list (may be empty)
    - explanation always a string (may be empty)
    """
    # Sentiment with normalization and fallback
    raw_sentiment = str(raw_dict.get("sentiment", "")).strip().lower()
    sentiment = raw_sentiment if raw_sentiment in VALID_SENTIMENTS else "neutral"
    
    # Score with clamping
    try:
        score = float(raw_dict.get("score", 0.5))
        score = max(0.0, min(1.0, score))
    except (ValueError, TypeError):
        score = 0.5
    
    # Keywords: normalize to a clean list
    raw_keywords = raw_dict.get("keywords", [])
    if isinstance(raw_keywords, str):
        keywords = [k.strip() for k in raw_keywords.split(",") if k.strip()]
    elif isinstance(raw_keywords, list):
        keywords = [str(k).strip() for k in raw_keywords if k and str(k).strip()]
    else:
        keywords = []
    
    # Explanation: normalized string
    explanation = str(raw_dict.get("explanation", "")).strip()[:500]
    
    return {
        "sentiment": sentiment,
        "score": score,
        "keywords": keywords,
        "explanation": explanation
    }

src/app/sentiment.py

import openai
from app.config import get_config
from app.parsers import parse_json_response
from app.processors import process_sentiment_output

SENTIMENT_PROMPT = """Analyze the sentiment of the following text.

Respond ONLY with valid JSON matching this exact structure:
{{
    "sentiment": "<positive|negative|neutral>",
    "score": <number between 0.0 and 1.0>,
    "explanation": "<brief explanation in at most 200 characters>",
    "keywords": ["<word1>", "<word2>"]
}}

Text to analyze:
{text}"""

def analyze_sentiment(text: str, client=None) -> dict:
    """
    Analyzes the sentiment of a text using an LLM.
    
    Args:
        text: Text to analyze (non-empty)
        client: OpenAI client (if None, creates one from the configuration)
    
    Returns:
        dict with: sentiment, score, keywords, explanation
    
    Raises:
        ValueError: If text is empty
    """
    if not text or not text.strip():
        raise ValueError("The text cannot be empty")
    
    if client is None:
        config = get_config()
        client = openai.OpenAI(api_key=config.openai_api_key)
    
    try:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {
                    "role": "system",
                    "content": "You are a sentiment analyzer. Always respond with valid JSON."
                },
                {
                    "role": "user",
                    "content": SENTIMENT_PROMPT.format(text=text)
                }
            ],
            temperature=0.0,
            max_tokens=500,
            response_format={"type": "json_object"}
        )
        
        raw = response.choices[0].message.content
        parsed = parse_json_response(raw)
        return process_sentiment_output(parsed)
        
    except Exception as e:
        return {
            "sentiment": "unknown",
            "score": 0.0,
            "keywords": [],
            "explanation": "",
            "error": str(e)
        }

src/app/main.py

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field

from app.sentiment import analyze_sentiment

app = FastAPI(
    title="Sentiment Analysis API",
    description="Analyzes the sentiment of texts using an LLM",
    version="1.0.0"
)

class AnalyzeRequest(BaseModel):
    text: str = Field(min_length=1, max_length=5000)

class SentimentResponse(BaseModel):
    sentiment: str
    score: float
    keywords: list[str]
    explanation: str

@app.post("/analyze", response_model=SentimentResponse)
def analyze_endpoint(request: AnalyzeRequest):
    result = analyze_sentiment(request.text)
    if "error" in result:
        raise HTTPException(status_code=500, detail="Error analyzing the text")
    return result

@app.get("/health")
def health_check():
    return {"status": "ok", "service": "sentiment-analysis"}

Step 1: Verify the structure

Before writing tests, make sure the structure exists:

# Create the structure if it doesn't exist
mkdir -p tests/unit/contracts
mkdir -p tests/unit/parsers
mkdir -p tests/unit/regression
touch tests/unit/contracts/__init__.py
touch tests/unit/parsers/__init__.py
touch tests/unit/regression/__init__.py

# Verify that pytest can import the app
python -c "from app.sentiment import analyze_sentiment; print('Import OK')"

# Verify that pytest collects the tests
pytest --collect-only -q

Step 2: Update tests/helpers.py

# tests/helpers.py
from unittest.mock import MagicMock
import json

def create_openai_chat_response(
    content: str,
    model: str = "gpt-4o-mini",
    prompt_tokens: int = 45,
    completion_tokens: int = 25,
    finish_reason: str = "stop"
) -> MagicMock:
    """
    Creates a mock that exactly replicates the structure of openai.ChatCompletion.
    
    IMPORTANT: The content must be a JSON string if the app expects JSON.
    """
    mock_message = MagicMock()
    mock_message.role = "assistant"
    mock_message.content = content
    mock_message.tool_calls = None
    
    mock_choice = MagicMock()
    mock_choice.index = 0
    mock_choice.message = mock_message
    mock_choice.finish_reason = finish_reason
    
    mock_usage = MagicMock()
    mock_usage.prompt_tokens = prompt_tokens
    mock_usage.completion_tokens = completion_tokens
    mock_usage.total_tokens = prompt_tokens + completion_tokens
    
    mock_response = MagicMock()
    mock_response.id = "chatcmpl-mock-test"
    mock_response.model = model
    mock_response.choices = [mock_choice]
    mock_response.usage = mock_usage
    
    return mock_response

def create_sentiment_response(
    sentiment: str = "neutral",
    score: float = 0.5,
    explanation: str = "Test analysis",
    keywords: list = None
) -> MagicMock:
    """Specialized helper for sentiment responses."""
    if keywords is None:
        keywords = []
    
    content = json.dumps({
        "sentiment": sentiment,
        "score": score,
        "explanation": explanation,
        "keywords": keywords
    })
    
    return create_openai_chat_response(content)

Step 3: Update tests/conftest.py

# tests/conftest.py
import pytest
import json
from unittest.mock import MagicMock, AsyncMock
from tests.helpers import create_openai_chat_response, create_sentiment_response

# =============================================
# STATIC FIXTURES: common cases
# =============================================

@pytest.fixture
def mock_sentiment_client_positive():
    """Mock client that returns positive sentiment (standard case)."""
    client = MagicMock()
    client.chat.completions.create.return_value = create_sentiment_response(
        sentiment="positive",
        score=0.92,
        explanation="The text uses clearly positive language.",
        keywords=["excellent", "fantastic"]
    )
    return client

@pytest.fixture
def mock_sentiment_client_negative():
    """Mock client that returns negative sentiment."""
    client = MagicMock()
    client.chat.completions.create.return_value = create_sentiment_response(
        sentiment="negative",
        score=0.08,
        explanation="The text uses clearly negative language.",
        keywords=["terrible", "horrible"]
    )
    return client

@pytest.fixture
def mock_sentiment_client_neutral():
    """Mock client that returns neutral sentiment."""
    client = MagicMock()
    client.chat.completions.create.return_value = create_sentiment_response(
        sentiment="neutral",
        score=0.5,
        explanation="The text doesn't express a clear sentiment.",
        keywords=[]
    )
    return client

# =============================================
# FIXTURE FACTORIES: dynamic variations
# =============================================

@pytest.fixture
def make_sentiment_client():
    """
    Factory to create mock clients with configurable sentiment responses.
    
    Usage:
        def test_x(make_sentiment_client):
            client = make_sentiment_client(sentiment="positive", score=0.9)
    """
    def _create(
        sentiment: str = "neutral",
        score: float = 0.5,
        explanation: str = "Test analysis",
        keywords: list = None
    ) -> MagicMock:
        if keywords is None:
            keywords = []
        client = MagicMock()
        client.chat.completions.create.return_value = create_sentiment_response(
            sentiment=sentiment,
            score=score,
            explanation=explanation,
            keywords=keywords
        )
        return client
    return _create

@pytest.fixture
def make_error_client():
    """
    Factory to create mock clients that simulate errors.
    
    Error types:
        "rate_limit", "timeout", "connection", "empty_response"
    """
    import openai
    
    def _create(error_type: str = "generic") -> MagicMock:
        client = MagicMock()
        
        if error_type == "rate_limit":
            client.chat.completions.create.side_effect = openai.RateLimitError(
                message="Rate limit exceeded",
                response=MagicMock(status_code=429),
                body={}
            )
        elif error_type == "timeout":
            client.chat.completions.create.side_effect = openai.APITimeoutError(
                request=MagicMock()
            )
        elif error_type == "connection":
            client.chat.completions.create.side_effect = openai.APIConnectionError(
                request=MagicMock()
            )
        elif error_type == "empty_response":
            client.chat.completions.create.return_value = create_openai_chat_response("")
        elif error_type == "malformed_json":
            client.chat.completions.create.return_value = create_openai_chat_response(
                '{"sentiment": "positive", "score": 0.9'  # Incomplete JSON
            )
        else:
            client.chat.completions.create.side_effect = Exception(f"Generic error: {error_type}")
        
        return client
    return _create

Step 4: Prompt contract tests

# tests/unit/contracts/test_contracts.py
import pytest
from app.sentiment import analyze_sentiment

VALID_SENTIMENTS = ["positive", "negative", "neutral"]

class TestSentimentPromptContract:
    """Tests for the sentiment analysis prompt contract."""
    
    # DEFINED CONTRACT:
    # - Output is a dict with: sentiment, score, keywords, explanation
    # - sentiment: one of ["positive", "negative", "neutral"]
    # - score: float between 0.0 and 1.0
    # - keywords: list of strings (may be empty)
    # - explanation: string (may be empty)
    
    def test_contract_structure(self, make_sentiment_client):
        """The output has all the contract's required keys."""
        client = make_sentiment_client()
        result = analyze_sentiment("test text", client=client)
        
        assert isinstance(result, dict), f"Expected dict, got {type(result)}"
        assert "sentiment" in result, "Missing key 'sentiment'"
        assert "score" in result, "Missing key 'score'"
        assert "keywords" in result, "Missing key 'keywords'"
        assert "explanation" in result, "Missing key 'explanation'"
    
    def test_contract_types(self, make_sentiment_client):
        """Each contract field has the correct type."""
        client = make_sentiment_client(
            sentiment="positive", score=0.85, keywords=["good"], explanation="Positive"
        )
        result = analyze_sentiment("text", client=client)
        
        assert isinstance(result["sentiment"], str)
        assert isinstance(result["score"], (int, float))
        assert isinstance(result["keywords"], list)
        assert isinstance(result["explanation"], str)
    
    def test_contract_sentiment_values(self, make_sentiment_client):
        """The sentiment field can only be one of the three allowed values."""
        client = make_sentiment_client(sentiment="positive")
        result = analyze_sentiment("positive text", client=client)
        
        assert result["sentiment"] in VALID_SENTIMENTS, \
            f"sentiment '{result['sentiment']}' is not in {VALID_SENTIMENTS}"
    
    def test_contract_score_range(self, make_sentiment_client):
        """The score is always in the range [0, 1]."""
        client = make_sentiment_client(score=0.75)
        result = analyze_sentiment("text", client=client)
        
        assert 0.0 <= result["score"] <= 1.0, \
            f"score {result['score']} outside the range [0, 1]"
    
    def test_contract_keywords_is_list(self, make_sentiment_client):
        """keywords is always a list (may be empty)."""
        client = make_sentiment_client(keywords=["word1", "word2"])
        result = analyze_sentiment("text", client=client)
        
        assert isinstance(result["keywords"], list)
    
    def test_contract_all_keywords_are_strings(self, make_sentiment_client):
        """All keyword elements are strings."""
        client = make_sentiment_client(keywords=["good", "excellent", "fantastic"])
        result = analyze_sentiment("text", client=client)
        
        assert all(isinstance(k, str) for k in result["keywords"]), \
            f"Some keywords are not strings: {result['keywords']}"

class TestSentimentContractVariations:
    """Contract tests for different input types."""
    
    @pytest.mark.parametrize("sentiment,score", [
        ("positive", 0.95),
        ("negative", 0.05),
        ("neutral", 0.5),
        ("positive", 0.6),   # Positive with medium confidence
        ("negative", 0.4),   # Negative with medium confidence
    ])
    def test_contract_multiple_sentiments(self, make_sentiment_client, sentiment, score):
        """The contract holds for all sentiment types."""
        client = make_sentiment_client(sentiment=sentiment, score=score)
        result = analyze_sentiment("test text", client=client)
        
        assert result["sentiment"] in VALID_SENTIMENTS
        assert 0 <= result["score"] <= 1
        assert isinstance(result["keywords"], list)
    
    def test_contract_with_empty_keywords(self, make_sentiment_client):
        """The contract holds when the LLM returns no keywords."""
        client = make_sentiment_client(keywords=[])
        result = analyze_sentiment("short text", client=client)
        
        assert result["keywords"] == []
    
    def test_contract_with_many_keywords(self, make_sentiment_client):
        """The contract holds with many keywords."""
        many_keywords = ["word1", "word2", "word3", "word4", "word5"]
        client = make_sentiment_client(keywords=many_keywords)
        result = analyze_sentiment("long text with many words", client=client)
        
        assert isinstance(result["keywords"], list)
        assert all(isinstance(k, str) for k in result["keywords"])

Step 5: Parser tests

# tests/unit/parsers/test_parsers.py
import pytest
import json
from app.parsers import parse_json_response

class TestParseJsonResponse:
    """Tests for the LLM's JSON parser."""
    
    @pytest.mark.parametrize("raw_input,expected", [
        (
            '{"sentiment": "positive", "score": 0.9}',
            {"sentiment": "positive", "score": 0.9},
        ),
        (
            '```json\n{"sentiment": "negative", "score": 0.1}\n```',
            {"sentiment": "negative", "score": 0.1},
        ),
        (
            '```\n{"sentiment": "neutral"}\n```',
            {"sentiment": "neutral"},
        ),
        (
            'The analysis is: {"sentiment": "positive", "score": 0.85}',
            {"sentiment": "positive", "score": 0.85},
        ),
        (
            '\n\n{"sentiment": "neutral", "score": 0.5}\n\n',
            {"sentiment": "neutral", "score": 0.5},
        ),
    ])
    def test_valid_formats(self, raw_input, expected):
        """The parser handles all valid LLM formats."""
        assert parse_json_response(raw_input) == expected
    
    def test_empty_raises_value_error(self):
        with pytest.raises(ValueError, match="empty"):
            parse_json_response("")
    
    def test_whitespace_only_raises_value_error(self):
        with pytest.raises(ValueError):
            parse_json_response("   \n\t  ")
    
    def test_no_json_raises_value_error(self):
        with pytest.raises(ValueError):
            parse_json_response("This text contains no JSON")
    
    def test_malformed_json_raises(self):
        with pytest.raises((json.JSONDecodeError, ValueError)):
            parse_json_response('{"sentiment": "positive", "score": 0.9')
    
    def test_unicode_handled_correctly(self):
        raw = '{"sentiment": "positivo", "keywords": ["fantástico", "excelente"]}'
        result = parse_json_response(raw)
        assert "fantástico" in result["keywords"]
    
    def test_nested_json_parsed(self):
        raw = '{"result": {"sentiment": "positive"}, "score": 0.9}'
        result = parse_json_response(raw)
        assert result["result"]["sentiment"] == "positive"

class TestProcessSentimentOutput:
    """Tests for the sentiment output processor."""
    
    def test_normalizes_uppercase_sentiment(self):
        from app.processors import process_sentiment_output
        result = process_sentiment_output({"sentiment": "POSITIVE", "score": 0.9})
        assert result["sentiment"] == "positive"
    
    def test_invalid_sentiment_defaults_to_neutral(self):
        from app.processors import process_sentiment_output
        result = process_sentiment_output({"sentiment": "very_positive", "score": 0.9})
        assert result["sentiment"] == "neutral"
    
    def test_clamps_score_above_one(self):
        from app.processors import process_sentiment_output
        result = process_sentiment_output({"sentiment": "positive", "score": 1.5})
        assert result["score"] == 1.0
    
    def test_clamps_score_below_zero(self):
        from app.processors import process_sentiment_output
        result = process_sentiment_output({"sentiment": "negative", "score": -0.1})
        assert result["score"] == 0.0
    
    def test_missing_fields_use_defaults(self):
        from app.processors import process_sentiment_output
        result = process_sentiment_output({})
        assert result["sentiment"] == "neutral"
        assert result["score"] == 0.5
        assert result["keywords"] == []
        assert result["explanation"] == ""

Step 6: Error handling tests

# tests/unit/contracts/test_error_handling.py
import pytest
import openai
from app.sentiment import analyze_sentiment

class TestSentimentErrorHandling:
    """Tests that verify the behavior in error cases."""
    
    def test_rate_limit_returns_error_response(self, make_error_client):
        """The app handles rate limit gracefully."""
        client = make_error_client("rate_limit")
        result = analyze_sentiment("text", client=client)
        
        assert result is not None, "The app must not return None"
        assert "error" in result or result.get("sentiment") == "unknown"
    
    def test_timeout_returns_error_response(self, make_error_client):
        """The app handles timeout gracefully."""
        client = make_error_client("timeout")
        result = analyze_sentiment("text", client=client)
        
        assert result is not None
        assert "error" in result or result.get("sentiment") == "unknown"
    
    def test_empty_response_handled_gracefully(self, make_error_client):
        """The app handles an empty LLM response without crashing."""
        client = make_error_client("empty_response")
        result = analyze_sentiment("text", client=client)
        
        assert result is not None
        # Must not raise an exception — must return something
    
    def test_malformed_json_handled_gracefully(self, make_error_client):
        """The app handles malformed JSON from the LLM."""
        client = make_error_client("malformed_json")
        result = analyze_sentiment("text", client=client)
        
        assert result is not None
    
    def test_empty_text_raises_value_error(self, make_sentiment_client):
        """Empty input raises ValueError before calling the LLM."""
        client = make_sentiment_client()
        
        with pytest.raises(ValueError, match="empty"):
            analyze_sentiment("", client=client)
    
    def test_empty_text_does_not_call_llm(self, make_sentiment_client):
        """With empty input, the LLM must not be called."""
        client = make_sentiment_client()
        
        with pytest.raises(ValueError):
            analyze_sentiment("", client=client)
        
        # Verify the LLM was not called
        client.chat.completions.create.assert_not_called()

Step 7: Regression tests

# tests/unit/regression/test_regression.py
import pytest
import json

# Manual snapshot of known behaviors
KNOWN_GOOD_CASES = [
    {
        "id": "r001",
        "input": "This product is absolutely fantastic",
        "mock_response": {
            "sentiment": "positive",
            "score": 0.97,
            "explanation": "Use of a superlative with a very strong positive connotation.",
            "keywords": ["fantastic", "absolutely"]
        },
        "expected": {
            "sentiment": "positive",
            "score_min": 0.8,
            "score_max": 1.0,
            "has_keywords": True
        }
    },
    {
        "id": "r002",
        "input": "Horrible experience, I would never return",
        "mock_response": {
            "sentiment": "negative",
            "score": 0.03,
            "explanation": "Very negative vocabulary with intent not to repeat.",
            "keywords": ["horrible", "never"]
        },
        "expected": {
            "sentiment": "negative",
            "score_min": 0.0,
            "score_max": 0.2,
            "has_keywords": True
        }
    },
    {
        "id": "r003",
        "input": "The product arrived on Tuesday",
        "mock_response": {
            "sentiment": "neutral",
            "score": 0.5,
            "explanation": "Factual description with no emotional charge.",
            "keywords": []
        },
        "expected": {
            "sentiment": "neutral",
            "score_min": 0.3,
            "score_max": 0.7,
            "has_keywords": False
        }
    }
]

@pytest.mark.parametrize("case", KNOWN_GOOD_CASES, ids=lambda c: c["id"])
def test_regression_known_cases(make_sentiment_client, case):
    """
    Regression tests: known behaviors that must not change.
    
    If these tests fail, it means something changed in the pipeline.
    Review before updating the regression cases.
    """
    from app.sentiment import analyze_sentiment
    
    client = make_sentiment_client(**case["mock_response"])
    result = analyze_sentiment(case["input"], client=client)
    
    expected = case["expected"]
    
    assert result["sentiment"] == expected["sentiment"], \
        f"Regression [{case['id']}]: sentiment changed from '{expected['sentiment']}' to '{result['sentiment']}'"
    
    assert expected["score_min"] <= result["score"] <= expected["score_max"], \
        f"Regression [{case['id']}]: score {result['score']} outside [{expected['score_min']}, {expected['score_max']}]"
    
    if expected["has_keywords"]:
        assert len(result["keywords"]) > 0, \
            f"Regression [{case['id']}]: keywords were expected but the list is empty"

Step 8: Final verification

# Run all the module's tests
pytest tests/unit/ -v --tb=short

# Expected result:
# tests/unit/contracts/test_contracts.py::TestSentimentPromptContract::test_contract_structure PASSED
# tests/unit/contracts/test_contracts.py::TestSentimentPromptContract::test_contract_types PASSED
# ... (all contract tests)
# tests/unit/parsers/test_parsers.py::TestParseJsonResponse::test_valid_formats[json_direct] PASSED
# ... (all parser tests)
# tests/unit/regression/test_regression.py::test_regression_known_cases[r001] PASSED
# ... (all regression tests)

# Run only contract tests
pytest -m contract -v

# Measure the module's coverage
pytest tests/unit/ --cov=app --cov-report=term-missing

# Verify speed (must be <10s)
time pytest tests/unit/ -q

# Verify 0 real API calls (if you have the flag configured)
pytest tests/unit/ -v --no-header
# The output must NOT show any HTTP call to api.openai.com

Delivery checklist

Before considering the module finished, verify:

• pytest tests/unit/ passes 100%
• All the app's prompts have at least one contract test
• The parser is tested with 5+ distinct input formats
• The error edge cases are covered (rate limit, timeout, empty response)
• The regression tests cover the 3 main cases (positive, negative, neutral)
• pytest -m unit finishes in under 10 seconds
• pytest --cov=app shows >80% coverage in parsers.py and processors.py
• 0 real OpenAI API calls (verify with pytest -s — it must not print API calls)

Optional extensions

If you completed the basics and want to go further:

Extension 1: API tests with FastAPI TestClient

# tests/unit/api/test_api.py
from fastapi.testclient import TestClient
from app.main import app

def test_analyze_endpoint_contract(make_sentiment_client, mocker):
    """The /analyze endpoint fulfills the HTTP contract."""
    mocker.patch("app.sentiment.client", make_sentiment_client(sentiment="positive", score=0.9))
    
    client = TestClient(app)
    response = client.post("/analyze", json={"text": "Test text"})
    
    assert response.status_code == 200
    data = response.json()
    assert "sentiment" in data
    assert "score" in data
    assert data["sentiment"] in ["positive", "negative", "neutral"]

Extension 2: Pre-commit hooks

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: unit-tests
        name: Run unit tests
        entry: pytest tests/unit/ -q --tb=short
        language: system
        pass_filenames: false

Final project exercises

Exercise 1: Add a new prompt

Add a second prompt to the app: summarize_text(text, client) that returns {summary: str, confidence: float}. Write the contract and the corresponding tests.

Exercise 2: Increase parser coverage

Run pytest tests/unit/parsers/ --cov=app.parsers --cov-report=term-missing. Identify the uncovered branches and add tests to cover them.

Exercise 3: Measure the execution time

Add a test that verifies the app's unit tests run in <10 seconds total. Describe how you'd implement it.

# Simple approach: with time
import time

def test_unit_suite_speed():
    """Verify that the unit test suite is fast."""
    import subprocess
    
    start = time.time()
    result = subprocess.run(
        ["pytest", "tests/unit/", "-q", "--tb=no"],
        capture_output=True
    )
    elapsed = time.time() - start
    
    assert elapsed < 10, f"The unit tests took {elapsed:.1f}s (max 10s)"
    assert result.returncode == 0, "The unit tests failed"

[pytest]
timeout = 30  # Maximum 30s per individual test

Project summary

When you finish this project you have:

Execution time: <10 seconds Real API calls: 0 Cost: $0.00

This is the power of prompt contract tests: significant coverage without spending a single cent on API calls.

Additional resources

1. pytest — Getting Started — pytest basics
2. FastAPI TestClient — For API tests
3. pytest-cov — To measure coverage
4. Pydantic v2 BaseModel — To define contracts
5. Python unittest.mock — Complete mocking reference
6. pytest markers — To organize tests by category
7. Module 3: Integration Testing — The next step