5. Property-Based Testing

Description

Instead of writing concrete test cases ("given input X, I expect output Y"), you define invariant properties that must hold for ANY valid input. Hypothesis automatically generates hundreds of test cases and looks for the ones that violate your property — often finding edge cases you would never have thought of. For LLM apps with deterministic mocks, it's the most effective strategy for finding bugs in parsers, processors, and business logic.

The problem it solves

With example-based testing, coverage is limited by your imagination:

# Example-based testing: you define each case
def test_parse_json_standard():
    assert parse_json_response('{"x": 1}') == {"x": 1}

def test_parse_json_markdown():
    assert parse_json_response('```json\n{"x": 1}\n```') == {"x": 1}

def test_parse_empty():
    with pytest.raises(ValueError):
        parse_json_response("")

# What about these cases you didn't write?
# '{"x": 1}  '  (trailing spaces)
# '{"x": 1}\n\n{"y": 2}'  (two JSONs)
# '{"x": "line1\nline2"}'  (newline in value)
# '{"x": 1.23456789012345}'  (high-precision float)
# — Hypothesis will find them for you

# Property-based testing: you define the property, Hypothesis does the rest
from hypothesis import given, settings
import hypothesis.strategies as st

@given(
    content=st.fixed_dictionaries({
        "sentiment": st.sampled_from(["positive", "negative", "neutral"]),
        "score": st.floats(min_value=0.0, max_value=1.0)
    })
)
def test_parse_json_roundtrip(content):
    """Property: any valid dict, when converted to JSON and parsed, reproduces the original."""
    raw = json.dumps(content)
    result = parse_json_response(raw)
    assert result == content
    # Hypothesis will generate hundreds of variations of content to verify this

Installation

pip install hypothesis

Fundamental Hypothesis concepts

Strategies: how to generate data

import hypothesis.strategies as st

# Texts
st.text()                          # Any Unicode text
st.text(min_size=10, max_size=500) # With length limits
st.from_regex(r'[a-z\s]+')        # Text that matches a regex

# Numbers
st.integers(min_value=0, max_value=100)
st.floats(min_value=0.0, max_value=1.0, allow_nan=False)

# Collections
st.lists(st.text(), min_size=1, max_size=10)
st.dictionaries(st.text(), st.integers())
st.sampled_from(["positive", "negative", "neutral"])  # From a list

# Composites
st.fixed_dictionaries({
    "sentiment": st.sampled_from(["positive", "negative", "neutral"]),
    "score": st.floats(0, 1),
    "keywords": st.lists(st.text(min_size=1), max_size=5)
})

# Optionals (None or the type)
st.one_of(st.none(), st.text())
st.text() | st.none()  # Equivalent

The Hypothesis workflow

1. You define the property with @given
2. Hypothesis generates examples automatically
3. If it finds an input that violates the property: SHRINKING
   → Hypothesis reduces the input to the minimum that reproduces the failure
   → Reports the minimal reproducible case
4. You fix the bug
5. Hypothesis saves the failed case in a database for future regression

Properties for deterministic logic (the most valuable application)

Parsers and processors are perfect candidates for property-based testing — they are 100% deterministic and have clear properties:

Parser properties

from hypothesis import given, settings, assume
import hypothesis.strategies as st
import json
import pytest

# Property 1: Roundtrip — json.dumps → parse_json_response
@given(
    content=st.fixed_dictionaries({
        "sentiment": st.sampled_from(["positive", "negative", "neutral"]),
        "score": st.floats(
            min_value=0.0,
            max_value=1.0,
            allow_nan=False,
            allow_infinity=False
        ),
        "keywords": st.lists(
            st.text(min_size=1, max_size=50, alphabet=st.characters(
                whitelist_categories=("L", "N", "Zs")  # Letters, numbers, spaces
            )),
            max_size=5
        )
    })
)
def test_parser_roundtrip_property(content):
    """
    PROPERTY: For any valid sentiment dict:
    json.dumps(d) → parse_json_response → result == original d
    """
    raw_json = json.dumps(content, ensure_ascii=False)
    result = parse_json_response(raw_json)
    
    assert result["sentiment"] == content["sentiment"]
    assert abs(result["score"] - content["score"]) < 1e-10  # Floats with tolerance
    assert result["keywords"] == content["keywords"]

# Property 2: Markdown wrapping — JSON in a code block is parsed the same
@given(
    content=st.fixed_dictionaries({
        "x": st.integers(),
        "y": st.text(min_size=1, max_size=50)
    })
)
def test_parser_markdown_unwrapping(content):
    """
    PROPERTY: JSON inside ```json ... ``` is parsed the same as direct JSON.
    """
    raw_direct = json.dumps(content)
    raw_markdown = f"```json\n{json.dumps(content)}\n```"
    
    result_direct = parse_json_response(raw_direct)
    result_markdown = parse_json_response(raw_markdown)
    
    assert result_direct == result_markdown

# Property 3: Error invariant — empty input always raises ValueError
@given(text=st.one_of(
    st.just(""),
    st.just("   "),
    st.just("\n\n\t"),
    st.text(max_size=5, alphabet=" \t\n")  # Whitespace only
))
def test_parser_empty_raises_value_error(text):
    """
    PROPERTY: Any empty or whitespace-only string raises ValueError.
    """
    with pytest.raises(ValueError):
        parse_json_response(text)

Processor properties

# Property 4: score clamping
@given(raw_score=st.floats(allow_nan=False, allow_infinity=False))
def test_processor_score_always_in_range(raw_score):
    """
    PROPERTY: Regardless of the input score, the output is always in [0, 1].
    """
    result = process_sentiment_output({"sentiment": "neutral", "score": raw_score})
    
    assert 0.0 <= result["score"] <= 1.0, \
        f"Score {raw_score} → {result['score']} — out of range [0, 1]"

# Property 5: sentiment normalization
@given(sentiment=st.text(max_size=100))
def test_processor_sentiment_always_valid(sentiment):
    """
    PROPERTY: Regardless of the input sentiment, the output is always one of the three valid ones.
    """
    result = process_sentiment_output({"sentiment": sentiment, "score": 0.5})
    
    assert result["sentiment"] in ["positive", "negative", "neutral"], \
        f"'{sentiment}' → '{result['sentiment']}' — not one of the valid values"

# Property 6: keywords always a list
@given(keywords=st.one_of(
    st.just(None),
    st.just(""),
    st.just([]),
    st.lists(st.text()),
    st.text(),
    st.integers()
))
def test_processor_keywords_always_list(keywords):
    """
    PROPERTY: Regardless of the type of the keywords input, the output is always a list.
    """
    result = process_sentiment_output({"sentiment": "neutral", "score": 0.5, "keywords": keywords})
    
    assert isinstance(result["keywords"], list), \
        f"keywords input {keywords!r} → type {type(result['keywords'])} — must be a list"

Properties for LLMs with mocks: the powerful combination

With deterministic mocks, property-based testing is especially potent:

@given(
    content=st.fixed_dictionaries({
        "sentiment": st.sampled_from(["positive", "negative", "neutral"]),
        "score": st.floats(min_value=0.0, max_value=1.0, allow_nan=False),
        "explanation": st.text(max_size=200),
        "keywords": st.lists(st.text(min_size=1, max_size=30), max_size=5)
    })
)
@settings(max_examples=50)  # Limit because each example creates a mock
def test_pipeline_property_with_mock(content):
    """
    PROPERTY: For any valid JSON from the LLM (mocked),
    the complete pipeline produces a result that satisfies the contract.
    """
    import json
    from unittest.mock import MagicMock
    from tests.helpers import create_openai_chat_response
    
    # Create the mock with the content generated by Hypothesis
    mock_client = MagicMock()
    mock_client.chat.completions.create.return_value = create_openai_chat_response(
        json.dumps(content)
    )
    
    # Run the pipeline
    result = analyze_sentiment("test text", client=mock_client)
    
    # Verify the contract on the result
    assert result["sentiment"] in ["positive", "negative", "neutral"]
    assert 0.0 <= result["score"] <= 1.0
    assert isinstance(result["keywords"], list)
    assert isinstance(result["explanation"], str)
    # The pipeline must not crash for any valid input from the LLM

Error handling properties

@given(error_content=st.one_of(
    st.just(""),
    st.just("{}"),
    st.just("{invalid json}"),
    st.text(max_size=200)  # Random text
))
@settings(max_examples=30)
def test_pipeline_handles_invalid_llm_output(error_content):
    """
    PROPERTY: The pipeline NEVER crashes — it always returns something or raises a known exception.
    It must not raise unexpected exceptions like KeyError, AttributeError, etc.
    """
    from unittest.mock import MagicMock
    from tests.helpers import create_openai_chat_response
    
    mock_client = MagicMock()
    mock_client.chat.completions.create.return_value = create_openai_chat_response(error_content)
    
    try:
        result = analyze_sentiment("text", client=mock_client)
        # If it doesn't raise, the result must be a dict with minimal keys
        assert isinstance(result, dict)
    except ValueError:
        pass  # ValueError is acceptable (invalid input)
    except Exception as e:
        # No other exception is acceptable
        pytest.fail(
            f"analyze_sentiment raised an unexpected exception for content={error_content!r}: "
            f"{type(e).__name__}: {e}"
        )

Advanced strategies

Strategy for Spanish text

# Spanish letters (including accents and ñ)
spanish_alphabet = st.characters(
    whitelist_categories=("L",),  # Letters only
    whitelist_characters=" .,;:!?¿¡"  # Spanish punctuation
)

spanish_text = st.text(
    alphabet=spanish_alphabet,
    min_size=5,
    max_size=500
)

@given(text=spanish_text)
def test_pipeline_handles_spanish_text(text):
    """The pipeline handles Spanish text without errors."""
    mock_client = make_sentiment_mock_for(text)
    result = analyze_sentiment(text, client=mock_client)
    assert isinstance(result, dict)

Strategy for simulating LLM responses

# Strategy that generates realistic LLM responses (well-formed JSON)
valid_llm_response = st.fixed_dictionaries({
    "sentiment": st.sampled_from(["positive", "negative", "neutral"]),
    "score": st.floats(0, 1, allow_nan=False),
    "keywords": st.lists(st.text(min_size=1, max_size=20), min_size=0, max_size=5),
    "explanation": st.text(min_size=0, max_size=200)
})

# Strategy that generates "dirty" LLM responses (with variable formatting)
dirty_llm_response = st.one_of(
    valid_llm_response.map(json.dumps),                            # Direct JSON
    valid_llm_response.map(lambda d: f"```json\n{json.dumps(d)}\n```"),  # Markdown
    valid_llm_response.map(lambda d: f"Response: {json.dumps(d)}")  # With a prefix
)

Configuring max_examples

from hypothesis import given, settings, HealthCheck

# Default: 100 examples
@given(content=st.text())
def test_default():
    ...

# For fast tests (simple logic):
@given(content=st.text())
@settings(max_examples=200)
def test_thorough():
    ...

# For tests with mocks (a bit slower):
@given(content=st.fixed_dictionaries({"x": st.integers()}))
@settings(max_examples=50)
def test_with_mock():
    ...

# For tests with the real LLM (very costly):
@given(content=st.sampled_from(PREDEFINED_CASES))
@settings(max_examples=5)
def test_with_real_llm():
    ...
    
# Suppress health checks if needed:
@settings(suppress_health_check=[HealthCheck.too_slow])

Comparison: example-based vs property-based

Exercises

Exercise 1: Your first property

Identify an invariant property of your format_summary_result function:

def format_summary_result(result: dict) -> str:
    """Formats the summary result as a string for the user."""
    return f"Summary: {result['summary']} (confidence: {result['confidence']:.0%})"

Write the test with @given:

@given(
    summary=st.text(min_size=1, max_size=200),
    confidence=st.floats(min_value=0.0, max_value=1.0, allow_nan=False)
)
def test_format_summary_result_properties(summary, confidence):
    """
    PROPERTIES:
    1. The result always contains the summary
    2. The result always starts with "Summary:"
    3. The result is always a non-empty string
    """
    result_dict = {"summary": summary, "confidence": confidence}
    formatted = format_summary_result(result_dict)
    
    assert isinstance(formatted, str)
    assert len(formatted) > 0
    assert formatted.startswith("Summary:")
    assert summary in formatted  # The original summary is in the result

Exercise 2: Clamping property

Write a property for the processor's score clamping. The score must always be in [0, 1] regardless of the input:

@given(raw_score=st.one_of(
    st.floats(allow_nan=False, allow_infinity=False),
    st.integers(),
    st.text(),  # Invalid type
    st.none()
))
def test_score_clamping_property(raw_score):
    """
    PROPERTY: For any score input (valid or invalid),
    the score in the output is always in [0.0, 1.0].
    """
    result = process_sentiment_output({"sentiment": "neutral", "score": raw_score})
    
    score = result["score"]
    assert isinstance(score, float), f"Score must be a float, it is {type(score)}"
    assert 0.0 <= score <= 1.0, f"Score {score} out of [0, 1]"
    assert not (score != score)  # Not NaN

Exercise 3: Shrinking in action

The following test has a bug. Identify which property it violates and explain how Hypothesis would find the minimal case:

def truncate_text(text: str, max_chars: int = 100) -> str:
    """Truncates the text to max_chars characters."""
    if len(text) > max_chars:
        return text[:max_chars]
    return text

What property could you write that would find an edge case?

@given(
    text=st.text(max_size=500),
    max_chars=st.integers(min_value=0, max_value=1000)
)
def test_truncate_text_properties(text, max_chars):
    """
    PROPERTIES:
    1. The result is never longer than max_chars
    2. The result is never longer than the input
    3. If the input <= max_chars, the result is identical to the input
    """
    result = truncate_text(text, max_chars)
    
    # Property 1: maximum length respected
    assert len(result) <= max_chars, f"len(result)={len(result)} > max_chars={max_chars}"
    
    # Property 2: doesn't add characters
    assert len(result) <= len(text)
    
    # Property 3: short input is not modified
    if len(text) <= max_chars:
        assert result == text, f"Short input was modified: '{text}' → '{result}'"

# Bug that Hypothesis would find:
# max_chars = 0 → text[:0] = "" ← Is this the expected behavior for max_chars=0?
# Hypothesis will find the minimal case: text="a", max_chars=0
# And will report that text[:0]="" is the result — you can decide whether it's correct

Exercise 4: Property with a mock

Write a property that verifies that for any valid LLM response (mocked), the pipeline returns a sentiment in the allowed list:

from hypothesis import given, settings
import hypothesis.strategies as st
from unittest.mock import MagicMock
from tests.helpers import create_openai_chat_response
import json

valid_sentiment_response = st.fixed_dictionaries({
    "sentiment": st.sampled_from(["positive", "negative", "neutral"]),
    "score": st.floats(min_value=0.0, max_value=1.0, allow_nan=False),
    "explanation": st.text(max_size=100),
    "keywords": st.lists(st.text(min_size=1, max_size=20), max_size=3)
})

@given(llm_output=valid_sentiment_response)
@settings(max_examples=50)
def test_pipeline_sentiment_always_valid(llm_output):
    """
    PROPERTY: For any valid LLM response,
    the pipeline produces a sentiment in the allowed list.
    """
    mock_client = MagicMock()
    mock_client.chat.completions.create.return_value = create_openai_chat_response(
        json.dumps(llm_output)
    )
    
    result = analyze_sentiment("any text", client=mock_client)
    
    assert result["sentiment"] in ["positive", "negative", "neutral"], \
        f"Sentiment '{result['sentiment']}' is not in the allowed list. LLM output: {llm_output}"

Exercise 5: When NOT to use Hypothesis

Describe 3 situations where property-based testing is not the right tool:

Summary

• Property-based testing = invariants that hold for any input — more robust than manual examples
• Hypothesis generates cases automatically and does shrinking to find the minimal reproducer
• Ideal for parsers and processors: 100% deterministic, clear properties, no API cost
• With mocks: combine with @given for massive coverage without calls to the real LLM
• With the real LLM: use a small max_examples to control cost
• Useful properties: range clamping, type invariants, parsing roundtrip, error handling

Additional resources

1. Hypothesis Documentation — Complete documentation
2. Hypothesis Strategies — All available strategies
3. In Praise of Property-Based Testing — Why it's powerful
4. Hypothesis Settings — Configuring max_examples and others
5. QuickCheck — Haskell — The origin of the concept
6. Property-Based Testing in Python — Real Python tutorial
7. Shrinking in Hypothesis — How Hypothesis minimizes failed cases