4. Snapshot Testing

Description

Snapshot testing captures the output of a test the first time and, in future runs, compares it with the saved snapshot. If the output changes, the test fails. In the context of AI apps with deterministic mocks, snapshots are an effective way to detect regressions: a change in the prompt, the parser, or the output processor that alters the output format is caught immediately. The key is not to update the snapshot without reviewing the diff.

The concept in 3 steps

Step 1: First run
  → Test runs, output generated
  → Snapshot saved to a file (tests/__snapshots__/...)
  → Test PASSES (creates the snapshot)

Step 2: Future runs
  → Test runs, output generated
  → Output compared with the saved snapshot
  → If equal: PASSES
  → If different: FAILS (shows the diff)

Step 3: When the test fails
  → Review the diff: is the change intentional?
  → Yes? → pytest --snapshot-update → updates the snapshot
  → No? → Bug → fix the code, not the snapshot

Why snapshot testing in AI apps?

In LLM apps with deterministic mocks, the snapshot seems redundant: "if the output is always the same, why save the snapshot?". The answer: protection against accidental changes.

Situation 1: You changed the prompt
  → The mock always returns the same string
  → But you changed how the prompt is built
  → Did you change the parser too? Did you forget something?
  → If the final output changed, the snapshot detects it

Situation 2: You refactored the parser
  → "I only simplified the code"
  → But the output has an extra space or a differently-named key
  → The snapshot captures the exact change

Situation 3: You updated a dependency
  → pydantic v1 → v2: dict() is now called model_dump()
  → The output format changed subtly
  → The snapshot detects the difference

Snapshot without a library: with a reference file

The simplest way: save the expected output as a JSON file and compare.

# tests/snapshots/sentiment_output.json (you create it the first time)
{
    "sentiment": "positive",
    "score": 0.95,
    "explanation": "The text clearly expresses satisfaction with the product.",
    "keywords": ["love", "product", "excellent"]
}

# test_sentiment_snapshot.py
import json
import pytest
from pathlib import Path
from tests.helpers import create_openai_chat_response

SNAPSHOTS_DIR = Path(__file__).parent / "snapshots"

def load_snapshot(name: str) -> dict:
    path = SNAPSHOTS_DIR / f"{name}.json"
    with open(path) as f:
        return json.load(f)

def save_snapshot(name: str, data: dict) -> None:
    SNAPSHOTS_DIR.mkdir(exist_ok=True)
    path = SNAPSHOTS_DIR / f"{name}.json"
    with open(path, "w") as f:
        json.dump(data, f, indent=2, ensure_ascii=False)

def test_sentiment_matches_snapshot(mocker):
    """The sentiment analyzer output didn't change relative to the snapshot."""
    mock_create = mocker.patch("app.sentiment.client.chat.completions.create")
    mock_create.return_value = create_openai_chat_response(
        '{"sentiment": "positive", "score": 0.95, "explanation": "The text clearly expresses satisfaction with the product.", "keywords": ["love", "product", "excellent"]}'
    )

    result = analyze_sentiment("I love this product, it's excellent", client=None)

    snapshot = load_snapshot("sentiment_output")
    assert result == snapshot, (
        f"The output doesn't match the snapshot.\n"
        f"Expected: {json.dumps(snapshot, indent=2)}\n"
        f"Actual: {json.dumps(result, indent=2)}"
    )

Snapshot with pytest-snapshot

The pytest-snapshot library automates the create-and-compare cycle:

pip install pytest-snapshot

# conftest.py — configure the snapshots directory
def pytest_configure(config):
    config.addinivalue_line(
        "markers", "snapshot: mark test as snapshot test"
    )

# pytest.ini — configure the snapshot dir
[pytest]
snapshot_default_extension = .json

# test_with_pytest_snapshot.py
import pytest
from tests.helpers import create_openai_chat_response

@pytest.mark.snapshot
def test_sentiment_output_snapshot(snapshot, mocker):
    """Snapshot of the complete sentiment analyzer output."""
    mock_create = mocker.patch("app.sentiment.client.chat.completions.create")
    mock_create.return_value = create_openai_chat_response(
        '{"sentiment": "positive", "score": 0.95, "explanation": "Positive text", "keywords": ["good", "product"]}'
    )

    result = analyze_sentiment("This product is very good")

    # First time: creates the snapshot
    # Subsequent: compares with the saved snapshot
    snapshot.assert_match(result, "sentiment_positive_case")

@pytest.mark.snapshot
def test_parser_output_snapshot(snapshot, mocker):
    """Snapshot of the parser for the standard JSON format."""
    raw_json = '{"sentiment": "negative", "score": 0.15, "explanation": "Negative text", "keywords": ["terrible", "bad"]}'

    result = parse_sentiment_response(raw_json)

    snapshot.assert_match(result, "parser_standard_json")

# First run (creates the snapshots):
pytest -m snapshot --snapshot-update

# Subsequent runs (compares with the snapshots):
pytest -m snapshot

# If the test fails, see the diff and decide:
# Intentional change → pytest -m snapshot --snapshot-update
# Bug → fix the code

Snapshot with syrupy (the most popular library)

syrupy is the most modern snapshot plugin for pytest, with better diffs and formats:

pip install syrupy

# test_with_syrupy.py
from syrupy.assertion import SnapshotAssertion

def test_sentiment_syrupy(snapshot: SnapshotAssertion, mocker):
    """Snapshot with syrupy — better diff and format."""
    mock_create = mocker.patch("app.sentiment.client.chat.completions.create")
    mock_create.return_value = create_openai_chat_response(
        '{"sentiment": "neutral", "score": 0.5, "explanation": "Neutral text", "keywords": ["normal"]}'
    )

    result = analyze_sentiment("Text with no particular sentiment")

    assert result == snapshot  # syrupy intercepts this assertion

def test_parser_multiple_formats_syrupy(snapshot: SnapshotAssertion):
    """Snapshot for multiple input formats."""
    test_cases = [
        '{"sentiment": "positive", "score": 0.9}',
        '```json\n{"sentiment": "positive", "score": 0.9}\n```',
        'The analysis: {"sentiment": "positive", "score": 0.9}'
    ]

    results = [parse_sentiment_response(tc) for tc in test_cases]

    assert results == snapshot

The correct snapshot testing workflow

This is the workflow you must follow — never skip it:

When the snapshot fails

# 1. Run the test
pytest tests/test_sentiment.py::test_sentiment_snapshot -v

# Failure output:
# FAILED tests/test_sentiment.py::test_sentiment_snapshot
# AssertionError: snapshot does not match
# --- snapshot
# +++ actual
# @@ -3,4 +3,4 @@
#  "sentiment": "positive",
# -"score": 0.95,
# +"score": 0.9500000000000001,  ← Difference due to float precision
#  "explanation": "..."

Analyze the diff

# Intentional difference: you improved the prompt
# BEFORE: {"summary": "Short summary"}
# NOW: {"summary": "Short summary", "language": "en"}  ← You added a field

# Is it intentional? Yes → update the snapshot
# pytest --snapshot-update

# Accidental difference: you refactored and changed a key by mistake
# BEFORE: {"sentiment_label": "positive"}
# NOW: {"sentiment": "positive"}  ← You renamed it by accident in the parser

# Is it intentional? No → fix the code

Anti-pattern: updating without reviewing

# ❌ NEVER do this without reviewing:
pytest --snapshot-update  # Updates ALL snapshots without reviewing the diff

# ✅ Do it test by test:
pytest tests/test_sentiment.py::test_specific_snapshot --snapshot-update
# Then review git diff tests/__snapshots__/ to see exactly what changed

Snapshot with normalization

Some outputs have fields that vary naturally and must not be included in the snapshot:

import pytest
from typing import Any

def normalize_for_snapshot(result: dict, exclude_keys: list[str] = None) -> dict:
    """
    Normalizes a result for a snapshot:
    - Excludes fields that vary (timestamps, IDs)
    - Sorts lists for consistent comparison
    - Rounds floats to avoid precision problems
    """
    if exclude_keys is None:
        exclude_keys = ["timestamp", "id", "created_at", "updated_at", "request_id"]

    normalized = {}
    for key, value in result.items():
        if key in exclude_keys:
            continue
        if isinstance(value, float):
            normalized[key] = round(value, 4)  # 4 decimals for consistency
        elif isinstance(value, list):
            normalized[key] = sorted(value) if all(isinstance(x, str) for x in value) else value
        else:
            normalized[key] = value

    return normalized

def test_with_normalization(snapshot, mocker):
    """Snapshot with normalization of variable fields."""
    mock_create = mocker.patch("app.sentiment.client.chat.completions.create")
    mock_create.return_value = create_openai_chat_response(
        '{"sentiment": "positive", "score": 0.9500000000000001, "explanation": "Positive", "keywords": ["good", "excellent"], "timestamp": "2024-01-15T10:30:00", "request_id": "abc-123"}'
    )

    result = analyze_sentiment("text")

    # Normalize before the snapshot
    normalized = normalize_for_snapshot(result, exclude_keys=["timestamp", "request_id"])

    assert normalized == snapshot
    # The snapshot doesn't include timestamp or request_id — avoids failures due to variation

Snapshot for complex outputs

When the output is complex (nested objects, long lists), the snapshot is more valuable than manual assertions:

# Complex output of a document analysis pipeline
COMPLEX_OUTPUT = {
    "document_analysis": {
        "language": "en",
        "word_count": 1250,
        "readability_score": 0.75,
        "sections": [
            {
                "title": "Introduction",
                "sentiment": "neutral",
                "key_concepts": ["Python", "API", "REST"],
                "importance": 0.8
            },
            {
                "title": "Conclusion",
                "sentiment": "positive",
                "key_concepts": ["results", "improvements"],
                "importance": 0.9
            }
        ],
        "overall_sentiment": {
            "label": "positive",
            "score": 0.72,
            "breakdown": {
                "positive": 0.72,
                "negative": 0.05,
                "neutral": 0.23
            }
        }
    }
}

def test_document_analysis_snapshot(snapshot, mocker):
    """
    For complex outputs, a snapshot is more efficient than
    writing 20 manual assertions.
    """
    mock_create = mocker.patch("app.analyzer.client.chat.completions.create")
    mock_create.return_value = create_openai_chat_response(json.dumps(COMPLEX_OUTPUT))

    result = analyze_document("Long test document...")

    # A snapshot captures everything — any change is detected
    assert normalize_for_snapshot(result) == snapshot

    # Complement with assertions about the most critical parts:
    assert result["document_analysis"]["overall_sentiment"]["label"] in ["positive", "negative", "neutral"]
    assert 0 <= result["document_analysis"]["overall_sentiment"]["score"] <= 1

Snapshot by section: strategy for very large outputs

When the output is very large, a single snapshot makes the diff hard to read:

def test_large_output_snapshot_by_section(snapshot, mocker):
    """
    For very large outputs: snapshot by section.
    That way the diff is readable when a section changes.
    """
    mock_create = mocker.patch("app.analyzer.client.chat.completions.create")
    mock_create.return_value = create_openai_chat_response(json.dumps(COMPLEX_OUTPUT))

    result = analyze_document("Document...")
    analysis = result["document_analysis"]

    # Snapshot by section
    assert analysis["sections"] == snapshot(name="sections")
    assert analysis["overall_sentiment"] == snapshot(name="overall_sentiment")

    # Basic metadata with direct assertions (clearer than a snapshot)
    assert analysis["language"] == "en"
    assert isinstance(analysis["word_count"], int)

When to use a snapshot vs a manual assert

Snapshot testing in CI/CD

# .github/workflows/tests.yml
jobs:
  test:
    steps:
      - name: Run unit tests (including snapshots)
        run: pytest -m "unit or snapshot" --tb=short

      # ❌ NEVER in CI:
      # run: pytest --snapshot-update
      # If the snapshots fail in CI, it's a real failure — don't update automatically

Rule for CI: Snapshots are NEVER updated automatically in CI. If a snapshot fails in CI, it means the output changed — which may be a bug. The developer must review the diff locally and decide whether to update.

Comparison: pytest-snapshot vs syrupy vs manual JSON file

Recommendation: For new projects, use syrupy. For existing projects with reference JSON files, the manual approach works well.

Exercises

Exercise 1: Create your first snapshot

For the following function, create a snapshot test using the manual JSON file approach:

def format_sentiment_result(raw: dict) -> dict:
    return {
        "label": raw["sentiment"].upper(),
        "confidence": round(raw["score"], 2),
        "summary": f"Sentiment: {raw['sentiment']} ({raw['score']:.0%})"
    }

# tests/snapshots/formatted_sentiment.json
{
    "label": "POSITIVE",
    "confidence": 0.95,
    "summary": "Sentiment: positive (95%)"
}

# test_format.py
import json
from pathlib import Path

def test_format_sentiment_snapshot():
    input_data = {"sentiment": "positive", "score": 0.9500}

    result = format_sentiment_result(input_data)

    snapshot_path = Path("tests/snapshots/formatted_sentiment.json")

    if not snapshot_path.exists():
        # First time: create the snapshot
        snapshot_path.parent.mkdir(exist_ok=True)
        with open(snapshot_path, "w") as f:
            json.dump(result, f, indent=2, ensure_ascii=False)
        pytest.skip("Snapshot created — run the test again to verify")

    with open(snapshot_path) as f:
        expected = json.load(f)

    assert result == expected, f"Diff:\nExpected: {expected}\nActual: {result}"

Exercise 2: Normalize the snapshot

Your function's output includes a unique request_id and a timestamp. Write the normalization function and the snapshot test:

def normalize_output(result: dict) -> dict:
    """Normalizes for a snapshot: excludes variable fields."""
    volatile_fields = {"request_id", "timestamp", "created_at", "processing_time_ms"}
    return {k: v for k, v in result.items() if k not in volatile_fields}

def test_analysis_with_volatile_fields(snapshot, mocker):
    mock_create = mocker.patch("app.analyzer.client.chat.completions.create")
    mock_create.return_value = create_openai_chat_response(
        '{"sentiment": "positive", "score": 0.9, "request_id": "abc-123", "timestamp": "2024-01-15"}'
    )

    result = analyze("text")

    # Normalize before comparing
    assert normalize_output(result) == snapshot

    # Verify the variable fields separately (without a snapshot)
    assert "request_id" in result  # Must exist even if not in the snapshot
    assert "timestamp" in result

Exercise 3: A snapshot that fails — identify whether it's a bug or an intentional change

Analyze these two scenarios and decide: update the snapshot or fix the code?

Scenario A:

- {"sentiment": "positive", "score": 0.9, "explanation": "Cheerful text"}
+ {"sentiment": "positive", "score": 0.9, "explanation": "Cheerful text", "language": "en"}

Scenario B:

- {"sentiment": "positive", "score": 0.9}
+ {"setiment": "positive", "scor": 0.9}

Exercise 4: Snapshot or manual assert?

For each case, decide whether to use a snapshot or manual assertions:

1. Output: {"ok": True, "count": 5}
2. Output: A list of 50 recommendations with 8 fields each
3. Output: {"categories": ["tech", "science"], "confidence": 0.87}
4. Output: A complete parsed document with sections, subsections, and metadata

Exercise 5: Complete workflow

Describe the complete workflow you'd follow when a snapshot fails in CI after another developer made a change to the parser:

Summary

• A snapshot captures the exact output the first time and detects any change in future runs
• Always review the diff before --snapshot-update — never update blindly
• With deterministic mocks, the snapshot is stable because the mock always returns the same thing
• Normalize variable fields (timestamp, ID) before the snapshot to avoid false failures
• For CI: never --snapshot-update automatically — if it fails, it's a signal to review
• Snapshot vs manual assert: snapshot for complex outputs; assert for simple structures

Additional resources

1. syrupy — Snapshot testing for pytest — The recommended library, excellent diffs
2. pytest-snapshot — A simpler alternative
3. Jest Snapshot Testing — The original inspiration (JavaScript), good concepts
4. Snapshot Testing Pros & Cons — Kent C. Dodds — When and when not to use snapshots
5. Approval Tests / Golden Master Testing — Related concept, with a different interface
6. Python difflib — To implement manual diffs if you need them
7. Testing file I/O in pytest — For handling temporary files in tests