3. Prompt Contract Tests

Description

A prompt contract is the explicit specification of what the output must meet: structure, types, constraints, and format. This session teaches how to go from implicit contracts (in your head) to explicit, testable contracts. A vague contract ("it must be good") isn't testable; a specific one ("JSON with keys X, Y, Z, with X between 10-500 chars and Y in the range 0-1") is.

The problem: implicit contracts

Every developer who uses LLMs has implicit contracts. The problem is that they live in your head:

"My summary prompt should return... something useful. A summary. With its important parts."

When something fails, you don't know exactly what: was a key missing? did the JSON malform? is the confidence out of range? is the summary too long?

The solution: Make the contract explicit and turn it into code.

Anatomy of a complete contract

A good prompt contract has four parts:

1. Structure (What form does the output have?)

The output is a JSON object (dict) with the following required keys:
- summary
- confidence
- sources

2. Types (What data type is each part?)

- summary: string
- confidence: float (or int as a special case)
- sources: list of strings

3. Constraints (What are the limits and restrictions?)

- summary: between 10 and 500 characters
- confidence: between 0.0 and 1.0 (inclusive)
- sources: can be empty; each element is a non-empty string

4. Error invariants (What happens in edge cases?)

- If the input is empty: summary = "", confidence = 0.0, sources = []
- If the text is very short: summary can equal the original input
- If the LLM fails: ??? (define the expected behavior)

From contract to code: step by step

Step 1: Write the contract in prose

Sentiment analysis prompt:
GIVEN: an input text (string, not empty)
RETURNS: a JSON with:
  - "sentiment": one of ["positive", "negative", "neutral"]
  - "score": float between 0.0 and 1.0
  - "explanation": non-empty string, maximum 200 characters
  - "keywords": list of strings, between 1 and 5 elements
EDGE CASES:
  - Very short input (1 word): must work the same
  - Input in another language: must return the same structure

Step 2: Translate to assertions

def test_sentiment_prompt_contract(mock_openai_client):
    """
    The sentiment analysis prompt meets the defined contract.

    CONTRACT:
    - Output is a dict with keys: sentiment, score, explanation, keywords
    - sentiment: one of ["positive", "negative", "neutral"]
    - score: float 0.0 - 1.0
    - explanation: non-empty string, maximum 200 chars
    - keywords: list of 1-5 strings
    """
    # Arrange
    text = "I love this product, it's exactly what I needed."

    # Act
    result = analyze_sentiment(text, client=mock_openai_client)

    # Assert: structure
    assert isinstance(result, dict), \
        f"The result must be a dict, it is {type(result)}"
    assert "sentiment" in result, "Missing the 'sentiment' key"
    assert "score" in result, "Missing the 'score' key"
    assert "explanation" in result, "Missing the 'explanation' key"
    assert "keywords" in result, "Missing the 'keywords' key"

    # Assert: types
    assert isinstance(result["sentiment"], str), \
        f"sentiment must be a string, it is {type(result['sentiment'])}"
    assert isinstance(result["score"], (int, float)), \
        f"score must be numeric, it is {type(result['score'])}"
    assert isinstance(result["explanation"], str), \
        f"explanation must be a string, it is {type(result['explanation'])}"
    assert isinstance(result["keywords"], list), \
        f"keywords must be a list, it is {type(result['keywords'])}"

    # Assert: constraints
    assert result["sentiment"] in ["positive", "negative", "neutral"], \
        f"sentiment must be one of [positive, negative, neutral], it is '{result['sentiment']}'"
    assert 0.0 <= result["score"] <= 1.0, \
        f"score must be 0-1, it is {result['score']}"
    assert len(result["explanation"]) > 0, \
        "explanation cannot be empty"
    assert len(result["explanation"]) <= 200, \
        f"explanation must be at most 200 chars, it has {len(result['explanation'])}"
    assert 1 <= len(result["keywords"]) <= 5, \
        f"keywords must have 1-5 elements, it has {len(result['keywords'])}"
    assert all(isinstance(k, str) for k in result["keywords"]), \
        "All keywords must be strings"

Contract with Pydantic: the executable version

Pydantic turns the contract into executable documentation. If the contract lives in the Pydantic model, it's impossible for the code to produce an output that violates it (if you use the model correctly).

# app/schemas.py
from pydantic import BaseModel, Field, field_validator
from typing import Literal
from enum import Enum

class SentimentEnum(str, Enum):
    POSITIVE = "positive"
    NEGATIVE = "negative"
    NEUTRAL = "neutral"

class SentimentOutput(BaseModel):
    """
    Contract of the sentiment analysis prompt.

    This model defines exactly what the LLM must return.
    If the LLM returns something different, Pydantic raises ValidationError.
    """
    sentiment: SentimentEnum = Field(
        description="Detected sentiment: positive, negative, or neutral"
    )
    score: float = Field(
        ge=0.0, le=1.0,
        description="Model confidence, from 0.0 (low) to 1.0 (high)"
    )
    explanation: str = Field(
        min_length=1, max_length=200,
        description="Brief explanation of the detected sentiment"
    )
    keywords: list[str] = Field(
        min_length=1, max_length=5,
        description="Keywords that define the sentiment"
    )

    @field_validator("keywords")
    @classmethod
    def keywords_not_empty(cls, v: list[str]) -> list[str]:
        if any(not k.strip() for k in v):
            raise ValueError("keywords cannot be empty strings")
        return v

    @field_validator("explanation")
    @classmethod
    def explanation_not_empty_string(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("explanation cannot be only spaces")
        return v.strip()

# How to use the model in the app:
from app.schemas import SentimentOutput

def analyze_sentiment(text: str, client) -> dict:
    response = client.chat.completions.create(...)
    raw = response.choices[0].message.content
    parsed = json.loads(raw)
    validated = SentimentOutput(**parsed)  # Raises ValidationError if it doesn't meet the contract
    return validated.model_dump()

# Tests using the Pydantic model:
from pydantic import ValidationError
from app.schemas import SentimentOutput

def test_contract_via_pydantic(mock_openai_client):
    """The LLM output (mocked) passes Pydantic validation."""
    result = analyze_sentiment("Test text", client=mock_openai_client)

    # If it gets here, it already passed Pydantic validation (done internally)
    validated = SentimentOutput(**result)
    assert validated.sentiment in ["positive", "negative", "neutral"]
    assert 0 <= validated.score <= 1

def test_invalid_output_raises_validation_error():
    """Verifies that an invalid output raises ValidationError."""
    invalid_output = {
        "sentiment": "very_positive",  # Value not allowed
        "score": 1.5,                 # Out of range
        "explanation": "",             # Empty
        "keywords": []                # Empty list
    }

    with pytest.raises(ValidationError) as exc_info:
        SentimentOutput(**invalid_output)

    errors = exc_info.value.errors()
    error_fields = [e["loc"][0] for e in errors]

    # Verify that the errors are from the expected fields
    assert "sentiment" in error_fields
    assert "score" in error_fields

Multiple contracts in the same app

A real app has multiple prompts, each with its contract. The strategy: parametrize.

# Define all the contracts in a dictionary
PROMPT_CONTRACTS = {
    "sentiment": {
        "required_keys": ["sentiment", "score", "explanation", "keywords"],
        "types": {
            "sentiment": str,
            "score": (int, float),
            "explanation": str,
            "keywords": list
        },
        "constraints": {
            "sentiment": lambda v: v in ["positive", "negative", "neutral"],
            "score": lambda v: 0 <= v <= 1,
            "explanation": lambda v: 0 < len(v) <= 200,
            "keywords": lambda v: 1 <= len(v) <= 5
        }
    },
    "summary": {
        "required_keys": ["summary", "confidence", "sources"],
        "types": {
            "summary": str,
            "confidence": (int, float),
            "sources": list
        },
        "constraints": {
            "summary": lambda v: 10 <= len(v) <= 500,
            "confidence": lambda v: 0 <= v <= 1,
            "sources": lambda v: isinstance(v, list)
        }
    },
    "classification": {
        "required_keys": ["category", "confidence", "tags"],
        "types": {
            "category": str,
            "confidence": (int, float),
            "tags": list
        },
        "constraints": {
            "category": lambda v: len(v) > 0,
            "confidence": lambda v: 0 <= v <= 1,
            "tags": lambda v: len(v) <= 10
        }
    }
}

def validate_contract(result: dict, contract: dict) -> None:
    """Validates that a result meets a contract."""
    # Structure
    for key in contract["required_keys"]:
        assert key in result, f"Missing required key: '{key}'"

    # Types
    for field, expected_type in contract["types"].items():
        assert isinstance(result[field], expected_type), \
            f"'{field}' must be {expected_type}, it is {type(result[field])}"

    # Constraints
    for field, constraint_fn in contract["constraints"].items():
        assert constraint_fn(result[field]), \
            f"'{field}' with value '{result[field]}' violates the constraint"

# Parametrized test for all the prompts:
@pytest.mark.parametrize("prompt_name,input_text,mock_response", [
    (
        "sentiment",
        "I love the product",
        '{"sentiment": "positive", "score": 0.95, "explanation": "Expresses satisfaction", "keywords": ["love", "product"]}'
    ),
    (
        "summary",
        "Long text to summarize...",
        '{"summary": "Summary of the text", "confidence": 0.85, "sources": ["paragraph 1"]}'
    ),
    (
        "classification",
        "This is a technical article",
        '{"category": "technology", "confidence": 0.9, "tags": ["technical", "article"]}'
    ),
])
def test_all_prompts_fulfill_contract(
    mocker, prompt_name, input_text, mock_response
):
    """All the prompts meet their respective contracts."""
    mock_create = mocker.patch(f"app.prompts.{prompt_name}.client.chat.completions.create")
    mock_create.return_value = create_openai_chat_response(mock_response)

    result = run_prompt(prompt_name, input_text)

    contract = PROMPT_CONTRACTS[prompt_name]
    validate_contract(result, contract)

Contracts for edge cases

A good contract also defines the behavior in extreme cases:

# Contract for empty input
def test_contract_empty_input(mock_openai_client):
    """The contract is met even with empty input."""
    # Mock for an empty-input response
    mock_openai_client.chat.completions.create.return_value = create_openai_chat_response(
        '{"sentiment": "neutral", "score": 0.0, "explanation": "Empty input", "keywords": ["empty"]}'
    )

    result = analyze_sentiment("", client=mock_openai_client)

    # The contract must be met all the same
    assert result["sentiment"] in ["positive", "negative", "neutral"]
    assert 0 <= result["score"] <= 1
    assert len(result["explanation"]) > 0

# Contract for very long input
def test_contract_very_long_input(mock_openai_client):
    """The contract is met with very long inputs."""
    long_text = "text " * 5000  # 25,000 characters

    result = analyze_sentiment(long_text, client=mock_openai_client)

    # Same contract guarantees
    assert isinstance(result["sentiment"], str)
    assert 0 <= result["score"] <= 1

# Contract for input in another language
@pytest.mark.parametrize("language,input_text", [
    ("Spanish", "Este producto es fantástico"),
    ("English", "This product is fantastic"),
    ("French", "Ce produit est fantastique"),
])
def test_contract_multiple_languages(mocker, language, input_text):
    """The contract holds regardless of the input's language."""
    mock_create = mocker.patch("app.sentiment.client.chat.completions.create")
    mock_create.return_value = create_openai_chat_response(
        '{"sentiment": "positive", "score": 0.9, "explanation": "Positive text", "keywords": ["fantastic"]}'
    )

    result = analyze_sentiment(input_text)

    # The contract structure must be met for any language
    assert result["sentiment"] in ["positive", "negative", "neutral"]
    assert 0 <= result["score"] <= 1

Contract with optional fields

Some prompts have optional fields. The contract must specify it:

class AnalysisOutput(BaseModel):
    """
    Contract with optional fields.

    required: sentiment, score
    optional: explanation (None if the LLM doesn't provide it), metadata (additional dict)
    """
    sentiment: SentimentEnum
    score: float = Field(ge=0.0, le=1.0)

    # Optional fields
    explanation: str | None = Field(
        default=None,
        max_length=200,
        description="Optional explanation"
    )
    metadata: dict | None = Field(
        default=None,
        description="Additional metadata"
    )

def test_contract_with_optional_fields(mock_openai_client):
    """The optional fields must not fail if they're missing."""
    # Mock that omits the optional fields
    mock_openai_client.chat.completions.create.return_value = create_openai_chat_response(
        '{"sentiment": "positive", "score": 0.9}'  # Without explanation or metadata
    )

    result = analyze_sentiment_flexible("text", client=mock_openai_client)

    # The required fields must be present
    assert result["sentiment"] in ["positive", "negative", "neutral"]
    assert 0 <= result["score"] <= 1

    # The optional ones can be None
    assert result.get("explanation") is None or isinstance(result["explanation"], str)
    assert result.get("metadata") is None or isinstance(result["metadata"], dict)

Comparison: Contract Test vs Assertion about quality

This is the most common confusion. Concrete examples:

# ✅ CONTRACT TEST — verifies structure and constraints
def test_contract_correct():
    result = analyze_sentiment("text", client=mock_client)

    assert isinstance(result["sentiment"], str)           # type
    assert result["sentiment"] in VALID_SENTIMENTS        # allowed values
    assert 0 <= result["score"] <= 1                      # numeric range
    assert len(result["explanation"]) <= 200              # maximum length
    assert isinstance(result["keywords"], list)           # type

# ❌ EVALUATION (not a contract test)
def test_quality_incorrect_as_contract():
    result = analyze_sentiment("I hate this product", client=REAL_CLIENT)

    assert result["sentiment"] == "negative"  # ← This requires the real LLM + judgment
    assert result["score"] > 0.8              # ← Requires the LLM to be accurate
    assert "hate" in result["keywords"]       # ← Requires the LLM to extract well

# ✅ THE RIGHT WAY: separate contract from evaluation
def test_contract_structure_only(mock_client):
    result = analyze_sentiment("I hate this product", client=mock_client)
    # Only verifies structure with a mock
    assert result["sentiment"] in VALID_SENTIMENTS
    assert 0 <= result["score"] <= 1

# Separate evaluation test (in module-03 or in the evaluation guide)
@pytest.mark.integration
@pytest.mark.evaluation
def test_negative_sentiment_quality():
    result = analyze_sentiment("I hate this product", client=REAL_CLIENT)
    # Here we do verify quality with the real LLM
    assert result["sentiment"] == "negative"

Workflow: from prompt to contract to test

Step 1: Write the prompt
   "Analyze the sentiment of the text. Respond with JSON: {...}"
         ↓
Step 2: Define the contract (in prose first)
   "Output: JSON with sentiment (positive/negative/neutral),
    score (0-1), explanation (string, max 200 chars)"
         ↓
Step 3: Create the Pydantic model (executable contract)
   class SentimentOutput(BaseModel): ...
         ↓
Step 4: Write the contract test (with a mock)
   def test_sentiment_contract(mock_client): ...
         ↓
Step 5: Run the test
   pytest -m contract -v
         ↓
Step 6: If it fails → update the prompt or parser
   If the real LLM produces something different → adjust the contract or the prompt

Exercises

Exercise 1: Write a complete contract

For a prompt that extracts information from resumes (CVs), write the contract in prose and then as a Pydantic model:

The prompt: "Extract the main information from the resume: name, technical skills, years of experience, last position."

Output is JSON with:
- name: non-empty string
- skills: list of strings (can be empty if none are detected)
- years_experience: int >= 0 (can be 0 if junior or no info)
- last_position: string (can be None if there's no history)

from pydantic import BaseModel, Field

class CVExtraction(BaseModel):
    name: str = Field(min_length=1, description="Full name of the candidate")
    skills: list[str] = Field(
        default_factory=list,
        description="Extracted technical skills"
    )
    years_experience: int = Field(
        ge=0,
        description="Total years of experience (0 if there's no information)"
    )
    last_position: str | None = Field(
        default=None,
        description="Last job or position, None if there's no history"
    )

def test_cv_extraction_contract(mock_client):
    cv_text = "Juan García, 5 years of experience in Python and FastAPI. Last position: Senior Developer."

    mock_client.chat.completions.create.return_value = create_openai_chat_response(
        '{"name": "Juan García", "skills": ["Python", "FastAPI"], "years_experience": 5, "last_position": "Senior Developer"}'
    )

    result = extract_cv(cv_text, client=mock_client)
    validated = CVExtraction(**result)

    assert validated.name == "Juan García"
    assert len(validated.skills) == 2
    assert validated.years_experience == 5
    assert validated.last_position == "Senior Developer"

Exercise 2: Identify vague contracts

Identify the problems with these "contracts" and rewrite them correctly:

1. "The output must be useful and clear"
2. "The JSON must have the relevant information"
3. "The score must be high for positive texts"

Exercise 3: Contract for a list of items

Your prompt returns a list of product recommendations. Write the Pydantic model and the contract test.

The prompt returns something like this:

[
    {"product_id": "P001", "name": "Laptop", "score": 0.95, "reason": "Perfect for remote work"},
    {"product_id": "P002", "name": "Mouse", "score": 0.87, "reason": "Ideal complement"}
]

from pydantic import BaseModel, Field

class ProductRecommendation(BaseModel):
    product_id: str = Field(min_length=1)
    name: str = Field(min_length=1)
    score: float = Field(ge=0.0, le=1.0)
    reason: str = Field(min_length=1, max_length=300)

class RecommendationsOutput(BaseModel):
    recommendations: list[ProductRecommendation] = Field(
        min_length=1,
        max_length=10,
        description="List of 1 to 10 recommendations"
    )

def test_recommendations_contract(mock_client):
    mock_response = json.dumps([
        {"product_id": "P001", "name": "Laptop", "score": 0.95, "reason": "Perfect for work"},
        {"product_id": "P002", "name": "Mouse", "score": 0.87, "reason": "Ideal complement"}
    ])
    mock_client.chat.completions.create.return_value = create_openai_chat_response(mock_response)

    raw_result = get_recommendations("laptop for work", client=mock_client)
    output = RecommendationsOutput(recommendations=raw_result)

    assert len(output.recommendations) >= 1
    for rec in output.recommendations:
        assert 0 <= rec.score <= 1
        assert len(rec.reason) > 0

Exercise 4: Handling ValidationError

Write a test that verifies that when the LLM returns an output that violates the contract, your app handles it gracefully (doesn't expose the exception to the user):

def test_handles_contract_violation_gracefully(mock_openai_client):
    """When the LLM violates the contract, the app returns a controlled error."""
    # Mock that returns an invalid output (score > 1)
    mock_openai_client.chat.completions.create.return_value = create_openai_chat_response(
        '{"sentiment": "very_positive", "score": 1.5, "explanation": "", "keywords": []}'
    )

    result = analyze_sentiment("text", client=mock_openai_client)

    # The app must not crash — it must return a fallback result
    # Option A: returns None and logs the error
    assert result is None or "error" in result
    # Option B: returns default values
    # assert result["sentiment"] == "unknown"
    # assert result["score"] == 0.0

Exercise 5: Contract vs Evaluation in your project

For your own project (or a hypothetical one), write:

1. Three contract tests (structure and constraints)
2. An example of an evaluation test that should NOT go in the contract tests

# For a chatbot that answers questions:

def test_chatbot_contract_structure(mock_client):
    """The response has the expected structure."""
    result = chatbot_respond("How does Python work?", client=mock_client)
    assert "answer" in result
    assert "confidence" in result
    assert "sources" in result

def test_chatbot_contract_types(mock_client):
    """The types are correct."""
    result = chatbot_respond("What is FastAPI?", client=mock_client)
    assert isinstance(result["answer"], str)
    assert isinstance(result["confidence"], float)
    assert isinstance(result["sources"], list)

def test_chatbot_contract_constraints(mock_client):
    """The values meet the constraints."""
    result = chatbot_respond("What is an API?", client=mock_client)
    assert len(result["answer"]) > 0      # Not empty
    assert len(result["answer"]) <= 2000  # Maximum 2000 chars
    assert 0 <= result["confidence"] <= 1 # Valid range

# This requires the real LLM + semantic judgment:
@pytest.mark.evaluation
def test_chatbot_answer_quality():
    """The response correctly explains the concept."""
    result = chatbot_respond("What is a REST API?", client=REAL_CLIENT)
    # This is NOT a contract test — it requires the LLM to understand the concept
    assert "representational state transfer" in result["answer"].lower()
    assert result["confidence"] > 0.8  # Requires the LLM to be accurate

Summary

• Explicit contract = testable: from "it must be good" to "JSON with keys X, Y, Z of types A, B, C"
• Four parts of the contract: structure, types, constraints, error invariants
• Pydantic turns the contract into executable documentation and automatic validation
• Parametrize lets you validate all the app's prompts with a single test
• Contract test ≠ Evaluation: contract verifies form, evaluation verifies semantic quality
• Workflow: prompt → contract in prose → Pydantic model → contract test → run with a mock

Additional resources

1. Pydantic Documentation — Validators — Field validators with @field_validator
2. Pydantic v2 Migration Guide — If you're coming from Pydantic v1
3. OpenAI Structured Outputs — Force structured JSON from the LLM
4. Contract Testing (Pact) — Consumer-driven contract testing concepts (for microservices)
5. pytest parametrize — For testing multiple contracts
6. Python Enum — For defining allowed values in the contract
7. Testing Best Practices — Test organization in pytest