4. Semantic Similarity Assertions

Description

When the LLM output varies in wording but not in meaning, string equality fails even though the output is correct. Semantic similarity compares meaning using embeddings: two texts are "equivalent" if their cosine similarity exceeds a calibrated threshold. This capsule implements robust semantic assertions, how to calibrate the thresholds, which embedding model to use (local vs API), and when semantic similarity is the right tool — and when it isn't.

The problem it solves

# The LLM can produce any of these responses for the same input.
# All are semantically equivalent and correct:

response_1 = "The capital of France is Paris."
response_2 = "Paris is the capital of France."
response_3 = "In France, the capital city is Paris."
response_4 = "Paris functions as the capital of the French country."

# Assertions that FAIL even though the output is correct:
assert result == response_1            # ❌ String exact match
assert result.startswith("The capital") # ❌ Prefix match
assert "The capital of France" in result # ❌ Substring

# Assertion that PASSES for all correct cases:
assert_semantically_similar(result, "France has Paris as its capital", threshold=0.85)
# ✅ Compares meaning, not exact words

How embeddings work

An embedding is a vector representation of a text's meaning. Texts with similar meanings have nearby vectors in the embedding space.

# Simplified visualization:
# "Paris is the capital of France"  → [0.2, -0.8, 0.5, ...]  (1536 dimensions)
# "France has its capital in Paris" → [0.21, -0.79, 0.48, ...]  (very similar)
# "Pizza is Italian"                → [0.9, 0.1, -0.3, ...]   (very different)

# Cosine similarity measures the angle between vectors:
# similarity = dot(a, b) / (|a| × |b|)
# 1.0 = identical, 0.0 = orthogonal, -1.0 = opposite

# In practice:
# 0.95+: nearly identical (same words, minimal variation)
# 0.85-0.95: same meaning, different wording
# 0.70-0.85: same topic, different perspective
# <0.70: different content or perspectives

Implementation with the OpenAI Embeddings API

# tests/semantic.py
import numpy as np
from typing import Optional

def cosine_similarity(a: list[float], b: list[float]) -> float:
    """Computes the cosine similarity between two vectors."""
    a_arr = np.array(a, dtype=float)
    b_arr = np.array(b, dtype=float)
    
    norm_a = np.linalg.norm(a_arr)
    norm_b = np.linalg.norm(b_arr)
    
    if norm_a == 0 or norm_b == 0:
        return 0.0
    
    return float(np.dot(a_arr, b_arr) / (norm_a * norm_b))

def get_embedding_openai(text: str, client) -> list[float]:
    """
    Gets an embedding using OpenAI text-embedding-3-small.
    
    Cost: ~$0.00002 per 1K tokens (very cheap)
    Dimensions: 1536
    """
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text.strip()
    )
    return response.data[0].embedding

def semantic_similarity_openai(
    text_a: str,
    text_b: str,
    client
) -> float:
    """
    Computes the semantic similarity between two texts using OpenAI embeddings.
    
    Returns:
        float between 0.0 and 1.0
    """
    emb_a = get_embedding_openai(text_a, client)
    emb_b = get_embedding_openai(text_b, client)
    return cosine_similarity(emb_a, emb_b)

def assert_semantically_similar(
    actual: str,
    expected: str,
    threshold: float = 0.85,
    client = None,
    message: str = None
) -> float:
    """
    Semantic assertion: fails if actual and expected are not similar enough.
    
    Args:
        actual: The real LLM output
        expected: The expected meaning (does not have to be exact text)
        threshold: Minimum acceptable similarity (0-1)
        client: OpenAI client (if None, uses local sentence-transformers)
        message: Custom error message
    
    Returns:
        The computed similarity (useful for debugging)
    
    Raises:
        AssertionError: If similarity < threshold
    """
    if client is not None:
        similarity = semantic_similarity_openai(actual, expected, client)
    else:
        similarity = semantic_similarity_local(actual, expected)
    
    error_msg = message or (
        f"Semantic similarity {similarity:.3f} < threshold {threshold}\n"
        f"  Actual:   '{actual[:100]}...'\n"
        f"  Expected: '{expected[:100]}...'"
    )
    
    assert similarity >= threshold, error_msg
    return similarity

Implementation with sentence-transformers (local, no cost)

For when you don't want to make calls to the embeddings API:

# pip install sentence-transformers

from functools import lru_cache
from typing import Callable

@lru_cache(maxsize=1)
def get_local_model():
    """Loads the model once and caches it."""
    from sentence_transformers import SentenceTransformer
    # Recommended models for semantic similarity:
    # - all-MiniLM-L6-v2: small, fast, good for English
    # - paraphrase-multilingual-MiniLM-L12-v2: multilingual (Spanish included)
    return SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")

def get_embedding_local(text: str) -> list[float]:
    """Gets a local embedding with sentence-transformers."""
    model = get_local_model()
    embedding = model.encode(text, normalize_embeddings=True)
    return embedding.tolist()

def semantic_similarity_local(text_a: str, text_b: str) -> float:
    """Local semantic similarity — no API calls, no cost."""
    emb_a = get_embedding_local(text_a)
    emb_b = get_embedding_local(text_b)
    return cosine_similarity(emb_a, emb_b)

Caching embeddings: save time and money

Computing embeddings twice for the same text is unnecessary:

# tests/semantic.py

_embedding_cache: dict[str, list[float]] = {}

def get_embedding_cached(
    text: str,
    client = None,
    provider: str = "auto"
) -> list[float]:
    """
    Gets an embedding with an in-memory cache.
    
    The cache persists during the test session (not across sessions).
    For most tests, the "expected" text repeats — the cache avoids
    duplicate API calls.
    """
    cache_key = f"{provider}:{text}"
    
    if cache_key not in _embedding_cache:
        if provider == "local" or (provider == "auto" and client is None):
            _embedding_cache[cache_key] = get_embedding_local(text)
        else:
            _embedding_cache[cache_key] = get_embedding_openai(text, client)
    
    return _embedding_cache[cache_key]

def assert_semantically_similar_cached(
    actual: str,
    expected: str,
    threshold: float = 0.85,
    client = None
) -> float:
    """Cached version — recommended for test suites."""
    emb_actual = get_embedding_cached(actual, client)
    emb_expected = get_embedding_cached(expected, client)
    
    similarity = cosine_similarity(emb_actual, emb_expected)
    
    assert similarity >= threshold, (
        f"Similarity {similarity:.3f} < {threshold}\n"
        f"  Actual:   '{actual[:80]}'\n"
        f"  Expected: '{expected[:80]}'"
    )
    return similarity

Calibrating the threshold: the most important part

The threshold is not a magic number — you need to calibrate it for your case:

Calibration process

# tests/calibrate_threshold.py
# Run once to calibrate — it is not part of the regular test suite

def calibrate_threshold():
    """
    Calibrates the semantic similarity threshold for your domain.
    
    How to use:
    1. Collect 10-20 pairs of texts (5 equivalent, 5 different)
    2. Compute the similarity for each pair
    3. The threshold should be between the minimum of the equivalent ones and the maximum of the different ones
    """
    # EQUIVALENT pairs (should pass the test)
    equivalent_pairs = [
        (
            "Python is a high-level programming language",
            "Python is an interpreted, high-level language"
        ),
        (
            "The sentiment of the text is positive",
            "The text expresses a positive emotion"
        ),
        (
            "The response is a JSON with the summary and confidence fields",
            "The JSON output contains summary and confidence"
        ),
    ]
    
    # DIFFERENT pairs (should fail the test)
    different_pairs = [
        (
            "The text is positive and cheerful",
            "The text is negative and sad"
        ),
        (
            "The summary talks about Python",
            "The summary talks about JavaScript"
        ),
    ]
    
    print("=== EQUIVALENT PAIRS ===")
    equivalent_scores = []
    for a, b in equivalent_pairs:
        sim = semantic_similarity_local(a, b)
        equivalent_scores.append(sim)
        print(f"  {sim:.3f}: '{a[:50]}' vs '{b[:50]}'")
    
    print("\n=== DIFFERENT PAIRS ===")
    different_scores = []
    for a, b in different_pairs:
        sim = semantic_similarity_local(a, b)
        different_scores.append(sim)
        print(f"  {sim:.3f}: '{a[:50]}' vs '{b[:50]}'")
    
    min_equivalent = min(equivalent_scores)
    max_different = max(different_scores)
    
    print(f"\n=== RECOMMENDATION ===")
    print(f"  Minimum similarity of equivalents: {min_equivalent:.3f}")
    print(f"  Maximum similarity of different:   {max_different:.3f}")
    recommended = (min_equivalent + max_different) / 2
    print(f"  Recommended threshold:             {recommended:.3f}")
    
    return recommended

if __name__ == "__main__":
    calibrate_threshold()

Thresholds by use case

Use in integration tests

# tests/integration/test_semantic.py
import pytest
import os
from tests.semantic import assert_semantically_similar_cached

@pytest.mark.integration
@pytest.mark.skipif(
    not os.getenv("OPENAI_API_KEY"),
    reason="Requires an API key for the semantic test"
)
class TestSentimentSemanticQuality:
    """Semantic quality tests for the sentiment analyzer."""
    
    def test_positive_text_recognized_as_positive(self, integration_client, e2e_budget):
        """Analyzing a positive text produces a coherent explanation."""
        text = "I love this product, it's absolutely incredible and worth every penny."
        
        result = analyze_sentiment(text, client=integration_client)
        e2e_budget.add_cost("gpt-4o-mini", 100, 50)
        
        # Structural assertion (always)
        assert result["sentiment"] == "positive"
        
        # Semantic assertion: the explanation must reflect positivity
        assert_semantically_similar_cached(
            actual=result["explanation"],
            expected="The text uses positive language and expressions of approval",
            threshold=0.70,  # Flexible — the explanation can vary a lot
            client=None  # Use local embeddings to save
        )
    
    def test_summary_captures_main_topic(self, integration_client):
        """The summary captures the main topic of the original text."""
        original = "Python was created by Guido van Rossum and released in 1991. It's a high-level language."
        
        result = summarize(original, client=integration_client)
        
        # The summary's similarity to the original must be high
        sim = assert_semantically_similar_cached(
            actual=result["summary"],
            expected=original,
            threshold=0.75  # The summary isn't equal to the original, but it must be similar
        )
        
        # Also verify the summary is shorter
        assert len(result["summary"]) < len(original)
    
    def test_different_sentiments_are_dissimilar(self, integration_client):
        """The outputs for texts with opposite sentiments must be dissimilar."""
        positive = "Incredible experience, I loved everything."
        negative = "Terrible experience, horrible in every aspect."
        
        result_positive = analyze_sentiment(positive, client=integration_client)
        result_negative = analyze_sentiment(negative, client=integration_client)
        
        # The explanations for opposite sentiments must be different
        from tests.semantic import semantic_similarity_local
        sim = semantic_similarity_local(
            result_positive["explanation"],
            result_negative["explanation"]
        )
        
        assert sim < 0.60, (
            f"The explanations for opposite sentiments are too similar: {sim:.3f}\n"
            f"Positive: {result_positive['explanation']}\n"
            f"Negative: {result_negative['explanation']}"
        )

When NOT to use semantic similarity

Semantic similarity isn't always the right tool:

# ❌ Case 1: To validate exact data (numbers, dates, names)
result = extract_data("The meeting is on January 15, 2025 at 3pm")

# Bad:
assert_semantically_similar(result["date"], "January 2025", threshold=0.8)

# Good (exact):
assert result["date"] == "2025-01-15"
assert result["time"] == "15:00"

# ❌ Case 2: To verify that something was NOT mentioned
result = analyze_sensitivity("Text that must not reveal private information")

# Bad: semantic similarity cannot verify absence
# Good:
assert result["pii_detected"] is False
assert "name" not in result["output"].lower()

# ❌ Case 3: To verify specific ordered lists
result = rank_items(items)

# Bad: the ranking can vary semantically
# Good:
assert result["top_item"] == expected_top
assert result["items"][0]["score"] >= result["items"][1]["score"]  # Correct order

Combination: semantic + properties

The best practice is to combine semantic assertions with property assertions:

def assert_quality_summary(result: dict, original: str, client=None):
    """
    Complete quality assertion for a summary:
    combines properties + semantic similarity.
    """
    # Properties (deterministic, always run)
    assert isinstance(result["summary"], str)
    assert len(result["summary"]) > 0
    assert len(result["summary"]) < len(original)  # Shorter than the original
    assert 0 <= result["confidence"] <= 1
    
    # Relevance (semantic — only if the real LLM was used)
    if client is not None:
        assert_semantically_similar_cached(
            actual=result["summary"],
            expected=original,
            threshold=0.70,  # The summary must capture the general topic
            client=client,
            message="The summary doesn't seem related to the original text"
        )

Exercises

Exercise 1: Implement with sentence-transformers

Implement assert_semantically_similar_local that uses sentence-transformers without API calls. Test it with 3 pairs of texts: two equivalent and one different.

from sentence_transformers import SentenceTransformer
import numpy as np

_model = None

def get_model():
    global _model
    if _model is None:
        _model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
    return _model

def assert_semantically_similar_local(actual, expected, threshold=0.8):
    model = get_model()
    embeddings = model.encode([actual, expected], normalize_embeddings=True)
    similarity = float(np.dot(embeddings[0], embeddings[1]))
    assert similarity >= threshold, f"Similarity {similarity:.3f} < {threshold}"
    return similarity

# Tests:
# ✅ Equivalent:
sim1 = assert_semantically_similar_local(
    "Python is an interpreted language", 
    "Python is a high-level interpreted language"
)  # ~0.90

# ✅ Equivalent:
sim2 = assert_semantically_similar_local(
    "The text is positive",
    "The text expresses positive sentiment"
)  # ~0.88

# ❌ Different (should fail):
try:
    assert_semantically_similar_local(
        "The text is positive",
        "The text is negative",
        threshold=0.8
    )
except AssertionError as e:
    print(f"Correctly failed: {e}")  # ~0.55

Exercise 2: Calibrate the threshold

For the following pairs, compute the similarity manually (using the implemented function) and determine the appropriate threshold:

1. "Positive and cheerful sentiment" vs "The text expresses joy and positivity"
2. "The result contains the summary" vs "The output has the summary field"
3. "The text is positive" vs "The text is negative"

# Computing with sentence-transformers:
pairs = [
    ("Positive and cheerful sentiment", "The text expresses joy and positivity"),
    ("The result contains the summary", "The output has the summary field"),
    ("The text is positive", "The text is negative"),
]

for a, b in pairs:
    sim = semantic_similarity_local(a, b)
    print(f"{sim:.3f}: '{a[:40]}' vs '{b[:40]}'")

# Approximate result:
# 0.89: Equivalent — use threshold 0.82
# 0.81: Technically equivalent — use threshold 0.75
# 0.52: Different — a threshold of 0.7 separates them well

# Recommended threshold for these cases: 0.75-0.80

Exercise 3: Semantic assertion for a summary

Write an integration test that uses semantic similarity to verify that the summary of a text about Python mentions the Python language as the main topic:

@pytest.mark.integration
@pytest.mark.skipif(not os.getenv("OPENAI_API_KEY"), reason="Requires an API key")
def test_summary_about_python():
    """The summary of a text about Python must be semantically about Python."""
    text = """Python is a high-level, interpreted, general-purpose programming language.
    It was created by Guido van Rossum and released in 1991. It's known for its clear, readable syntax,
    which makes it popular for beginners and experts alike."""
    
    result = summarize(text)
    
    # Semantic assertion: the summary must be about Python
    assert_semantically_similar_cached(
        actual=result["summary"],
        expected="Python is a popular programming language created by Guido van Rossum",
        threshold=0.75
    )
    
    # Also verify that "Python" or "language" is in the summary (backup assertion)
    assert "python" in result["summary"].lower() or "language" in result["summary"].lower()

Exercise 4: When not to use semantic

For each case, decide whether to use semantic similarity or a more specific assertion:

1. Verify that the response mentions the date "January 15"
2. Verify that the tone of the response is "professional and formal"
3. Verify that the output does not contain "error" or "exception"
4. Verify that the summary is coherent with the original text

Exercise 5: Debugging a failed assertion

The following assertion fails with similarity=0.62 when you expected 0.85+. How do you debug it?

assert_semantically_similar(
    actual="The model's output is a JSON with sentiment and score fields",
    expected="The result has the expected sentiment structure",
    threshold=0.85
)

Summary

• Semantic similarity compares meaning, not exact words — ideal for outputs that vary in wording
• Cosine similarity over embeddings: similarity = dot(a,b) / (|a| × |b|)
• Two options: OpenAI API (better quality, has a cost) or local sentence-transformers (no cost, good for multilingual)
• Calibrate the threshold with pairs of texts from your domain — don't use "magic" values
• Cache embeddings to avoid duplicate calls
• Combine with properties: semantic similarity + structure assertions = complete coverage
• When NOT to use: exact data (dates, IDs), verifying absence, ordered lists

Additional resources

1. OpenAI Embeddings Guide — Official API and available models
2. sentence-transformers Documentation — Local embeddings, multilingual
3. Cosine Similarity — Wikipedia — The math of the concept
4. Sentence Transformers Pretrained Models — Which model to choose
5. MTEB Benchmark — Comparison of embedding models
6. Evaluation Frameworks Guide — To evaluate quality beyond similarity
7. numpy — Linear Algebra — To compute norms and dot products