8. Module 3 Summary and Troubleshooting

Description

Closing Module 3: the 8 most common errors with their solutions, a quick diagnostic tree, a closing checklist, a summary of concepts, and preparation for Module 4 (Guardrails). If something went wrong during the integration tests project, here's the solution. If everything went well, this closing consolidates what you learned.

The 8 most common Module 3 errors

Error 1: Integration tests on every commit (no budget control)

Symptom:

# After 2 weeks of commits:
$ openai usage --month
Total: $87.50 ← This is real!
"The CI tests cost more than the infrastructure"

Cause: Integration tests configured to run on every push, without budget control and without a conditional skip.

Solution:

# .github/workflows/tests.yml
jobs:
  unit-tests:
    # ✅ Always — no API key, no cost
    steps:
      - run: pytest -m "not integration" -v

  integration-tests:
    if: github.ref == 'refs/heads/main'  # ← Only on main
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      E2E_BUDGET_USD: "0.25"               # ← Limited budget
    steps:
      - run: pytest -m integration --timeout=60

Rule: Unit tests on every push. Integration tests only on main (or nightly).

Error 2: Poorly calibrated semantic threshold

Symptom:

# Test fails even though the output is correct:
assert_semantically_similar(
    actual="The text expresses joy and satisfaction",
    expected="Positive sentiment detected",
    threshold=0.95  # ← Too high
)
# AssertionError: Similarity 0.78 < 0.95
# But 0.78 indicates the same topic — the threshold is poorly calibrated

Cause: Using a generic threshold without calibrating for the specific domain.

Solution:

# Calibrate first:
pairs = [
    ("The text expresses joy and satisfaction", "Positive sentiment detected"),
    ("The text has a positive emotional charge", "Positive and optimistic text"),
    # Add 5-10 equivalent pairs from the domain
]

for a, b in pairs:
    sim = semantic_similarity_local(a, b)
    print(f"{sim:.3f}: '{a[:40]}'")
    
# Result: ~0.72-0.80 for these pairs
# → Use threshold=0.70 for this type of comparison

Rule: For each type of semantic assertion, calibrate the threshold with 5+ example pairs from the domain before using it.

Error 3: Exact assertions in tests with the real LLM

Symptom:

# Fails randomly — the score varies on each run
@pytest.mark.integration
def test_score_exact():
    result = analyze_sentiment("I love it")
    assert result["score"] == 0.92  # ← Fails 60% of the time
    assert result["explanation"] == "The text uses positive language"  # ← Fails 80%

Cause: Applying the unit test style (deterministic mock) to integration tests with the real LLM.

Solution:

@pytest.mark.integration
def test_score_flexible(integration_client):
    result = analyze_sentiment("I love it", client=integration_client)
    
    # ✅ Range, not an exact value
    assert 0.6 <= result["score"] <= 1.0
    
    # ✅ Property, not exact text
    assert len(result.get("explanation", "")) > 5
    
    # ✅ Category (this one can be exact for clear text)
    assert result["sentiment"] == "positive"

Rule: With the real LLM, use properties and ranges. Only the category/label can be an exact assertion for unambiguous texts.

Error 4: property-based with the real LLM (no example limit)

Symptom:

# Hypothesis generates 100 examples × each one calls the real LLM
@given(text=st.text())
def test_pipeline_hypothesis():
    result = analyze_sentiment(text)  # Real LLM × 100 times
    assert result["sentiment"] in ["positive", "negative", "neutral"]

# Result: $0.10 per run × 50 runs = $5/day just for this test

Cause: Using property-based testing with the real LLM without limiting max_examples.

Solution:

# Option A: use a mock for property-based (recommended)
@given(content=st.fixed_dictionaries({
    "sentiment": st.sampled_from(["positive", "negative", "neutral"]),
    "score": st.floats(0, 1)
}))
@settings(max_examples=50)  # Enough to find edge cases
def test_pipeline_property_with_mock(content):
    mock_client = create_mock_client(content)
    result = analyze_sentiment("text", client=mock_client)
    assert result["sentiment"] in ["positive", "negative", "neutral"]

# Option B: if you need the real LLM, limit the examples
@given(text=st.sampled_from(PREDEFINED_INPUTS))  # Predefined inputs, not random
@settings(max_examples=5)  # Very few examples with the real LLM
def test_with_real_llm(text, integration_client):
    result = analyze_sentiment(text, client=integration_client)
    assert result["sentiment"] in ["positive", "negative", "neutral"]

Rule: Property-based testing + mocks = ideal. Property-based + real LLM = use a very small max_examples (≤10).

Error 5: Flaky tests without quarantine → ignored

Symptom:

CI output Monday: 2 tests failed → "It's the LLM, it's normal" → merged
CI output Tuesday: 3 tests failed → "It always happens" → merged  
CI output Wednesday: Bug in production → nobody noticed because they ignored failures

Cause: Normalizing flakiness — the team learns to ignore failing tests.

Solution:

# Step 1: Identify flaky tests
# pytest --count=10 tests/integration/ -v  ← See how many times each one fails

# Step 2: For tests flaky due to LLM variance — relax the assertion
# If it can't be relaxed → quarantine

@pytest.mark.quarantine(
    reason="Fails ~20% of the time. Score varies for ambiguous inputs. "
           "Issue: github.com/repo/issues/456 | Created: 2025-01-15"
)
@pytest.mark.integration
def test_ambiguous_text_score():
    result = analyze_sentiment("ambiguous text could be positive or neutral")
    assert result["score"] >= 0.8  # ← Too strict for ambiguous text

# CI: exclude quarantine from the normal pipeline
# pytest -m "not quarantine" -v

Rule: An uninvestigated failure = a time bomb. Quarantine is temporary, not permanent.

Error 6: Budget not tracked in multi-fixture

Symptom:

# The budget tracker has a race condition with parallel tests:
@pytest.fixture(scope="session")
def e2e_budget():
    return {"spent": 0.0, "max": 0.50}

# In test A: budget["spent"] += 0.01
# In test B: budget["spent"] += 0.01  ← Race condition in parallel
# Result: the budget checks are not reliable

Cause: The shared mutable dict is not thread-safe.

Solution:

import threading

@pytest.fixture(scope="session")
def e2e_budget():
    class SafeBudget:
        def __init__(self, max_usd=0.50):
            self.max_usd = max_usd
            self._spent = 0.0
            self._lock = threading.Lock()
        
        def add_cost(self, cost: float):
            with self._lock:
                self._spent += cost
                if self._spent >= self.max_usd:
                    pytest.skip(f"Budget exceeded: ${self._spent:.4f}")
        
        @property
        def spent(self):
            with self._lock:
                return self._spent
    
    return SafeBudget(max_usd=float(os.getenv("E2E_BUDGET_USD", "0.50")))

Rule: The budget tracker in parallel sessions needs to be thread-safe (lock).

Error 7: Skipping integration tests that blocks the CI

Symptom:

# CI output:
# SKIPPED tests/integration/test_e2e.py::test_sentiment_e2e
# SKIPPED tests/integration/test_e2e.py::test_negative_sentiment
# ...
# === 0 passed, 20 skipped in 0.5s ===
# Build status: ✅ PASSING  ← But no test actually ran

Cause: In CI, the integration tests always get skipped because OPENAI_API_KEY is not configured or RUN_INTEGRATION is not set to true.

Solution:

# Verify that the integration job has the variables:
integration-tests:
  if: github.ref == 'refs/heads/main'
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}  # ← Must exist as a secret
    RUN_INTEGRATION: "true"                           # ← Explicitly "true"
  steps:
    - run: |
        # Verify that the variables are configured before running
        if [ -z "$OPENAI_API_KEY" ]; then
          echo "ERROR: OPENAI_API_KEY not configured as a secret"
          exit 1
        fi
        pytest -m integration -v --timeout=60

Rule: Verify that the integration tests actually run in CI — not just that they don't fail.

Error 8: Property tests with vague properties

Symptom:

# A property so vague it doesn't detect any bug
@given(text=st.text())
def test_pipeline_vague_property(text):
    mock_client = make_mock(text)
    result = analyze_sentiment(text, client=mock_client)
    assert result is not None  # ← This always passes — it doesn't detect anything

Cause: Thinking in terms of "the output exists" instead of "the output has specific properties".

Solution:

@given(content=st.fixed_dictionaries({
    "sentiment": st.sampled_from(["positive", "negative", "neutral"]),
    "score": st.floats(0, 1, allow_nan=False)
}))
@settings(max_examples=50)
def test_pipeline_specific_properties(content):
    mock_client = create_mock_client(content)
    result = analyze_sentiment("text", client=mock_client)
    
    # ✅ Specific properties:
    assert result["sentiment"] in ["positive", "negative", "neutral"]  # Valid values
    assert 0.0 <= result["score"] <= 1.0   # Correct range
    assert isinstance(result["keywords"], list)  # Correct type
    assert not (result["score"] != result["score"])  # Not NaN

Rule: A good property can fail for some input. If your property always passes for any possible output, it protects nothing.

Quick diagnostic tree

Integration test fails
  ├── Is it SKIPPED? 
  │   ├── No API key → configure OPENAI_API_KEY
  │   └── No RUN_INTEGRATION → configure RUN_INTEGRATION=true
  │
  ├── Does it fail with AssertionError?
  │   ├── Exact assertion with the real LLM → relax to range/property
  │   ├── Semantic similarity < threshold → calibrate the threshold with domain pairs
  │   └── Always fails → real bug in the code
  │
  ├── Does it fail intermittently (~20-40%)?
  │   ├── Due to LLM variance → relax the assertion or add retry
  │   └── Due to timeout → add @pytest.mark.timeout + retry
  │
  ├── Budget exceeded in CI?
  │   ├── Integration runs on every push → move to main only
  │   └── Too many tests → reduce to the critical subset
  │
  └── Does a property-based test fail with a specific example?
      ├── See the example generated by Hypothesis (shrunk)
      └── The minimal example reveals the edge case to fix

Semantic test fails with similarity < threshold
  ├── Is the output semantically correct?
  │   ├── Yes → threshold too high, calibrate with domain pairs
  │   └── No → the LLM produces incoherent output → bug in the prompt
  └── Is the expected too specific?
      └── Make the expected more general, or use a property assertion

Module 3 closing checklist

Tests written

• At least 4 E2E tests with the real LLM (with conditional skip)
• At least 2 tests with semantic similarity (calibrated threshold)
• At least 4 property-based tests (with a mock, specific properties)
• Flaky management tests (at least 1 with retry, 1 with quarantine)
• Correct environment setup (unit always, integration only on main)

Test quality

• Budget tracker working and thread-safe
• Flexible assertions in integration tests (ranges, properties, semantic)
• Semantic similarity thresholds calibrated with domain pairs
• Hypothesis properties are specific and can fail for incorrect inputs

Performance and configuration

• pytest -m "not integration" < 15 seconds
• Maximum budget configured ($0.25-$0.50 per run)
• CI/CD: unit on every push, integration only on main
• pytest.ini with all markers registered

Module concept summary

Module success metrics

# Expected result upon completing the module:

$ pytest -m "not integration" -v --tb=short
=== 65+ passed in 8.2s ===  ← Fast unit tests

$ pytest -m integration -v --timeout=60
=== 12 passed in 47.3s === ← Integration passes with the real LLM

$ pytest tests/integration/test_property.py -v
=== 5 passed in 2.1s ===  ← Property-based with Hypothesis

# Final coverage:
$ pytest -m "not integration" --cov=app --cov-report=term-missing
app/parsers.py      96%
app/processors.py   97%
app/sentiment.py    82%
Total               89%

Next module: Guardrails

Module 4 (Guardrails — Input & Output Validation) directly applies what you learned in M3.

The M3 → M4 connection

Module 3: You learned to test with the real LLM using flexible assertions
          ↓
Module 4: You implement guardrails — and test them with the M3 strategies

Example:
- Guardrail: "Block prompt injection"
- M3-style test: E2E test that verifies the sanitizer blocks malicious inputs
  with the real LLM — assertion: the output doesn't contain the injection

- Guardrail: "Output can't contain PII"
- M3-style test: Property-based test that verifies that for any LLM output
  (mocked), the PII filter always removes emails and phone numbers

What you'll see in Module 4

1. Input sanitization: validate and sanitize user input before the LLM
2. Prompt injection detection: detect jailbreak/injection attacks
3. Output validation: verify that the LLM output meets security constraints
4. PII filtering: remove personally identifiable information from the output
5. Content moderation: detect toxic or inappropriate content
6. Rate limiting: protect against API abuse

And each guardrail will have its tests — using the techniques from M2 (mocks, contract tests) and M3 (property-based, semantic assertions).

Final module exercises

Exercise 1: Diagnosing a real test

The following test fails with a frequency of 35%. Diagnose the cause and propose 3 fix strategies in order of preference:

@pytest.mark.integration
def test_explanation_accuracy():
    result = analyze_sentiment("Very good purchase, totally recommendable")
    assert "positiv" in result["explanation"].lower()
    assert result["score"] > 0.85

Exercise 2: Designing a Hypothesis property

For the following processor, identify 3 invariant properties and write the tests:

def normalize_output(result: dict) -> dict:
    """Normalizes the output: sentiment to lowercase, score to 4 decimals, keywords sorted."""
    return {
        "sentiment": result.get("sentiment", "neutral").strip().lower(),
        "score": round(float(result.get("score", 0.5)), 4),
        "keywords": sorted(set(result.get("keywords", [])))
    }

from hypothesis import given, settings
import hypothesis.strategies as st

@given(sentiment=st.text(max_size=50))
def test_normalize_sentiment_always_lowercase(sentiment):
    """PROPERTY: The sentiment is always in lowercase."""
    result = normalize_output({"sentiment": sentiment, "score": 0.5})
    assert result["sentiment"] == result["sentiment"].lower()

@given(score=st.floats(allow_nan=False, allow_infinity=False))
def test_normalize_score_four_decimals(score):
    """PROPERTY: The score always has at most 4 decimals."""
    result = normalize_output({"sentiment": "neutral", "score": score})
    decimal_part = str(result["score"]).split(".")
    if len(decimal_part) > 1:
        assert len(decimal_part[1]) <= 4

@given(keywords=st.lists(st.text(min_size=1), max_size=10))
def test_normalize_keywords_unique_sorted(keywords):
    """PROPERTY: Keywords are always unique and sorted."""
    result = normalize_output({"sentiment": "neutral", "score": 0.5, "keywords": keywords})
    
    # They are unique
    assert len(result["keywords"]) == len(set(result["keywords"]))
    # They are sorted
    assert result["keywords"] == sorted(result["keywords"])

Exercise 3: Test budget

Your team has $50/month for integration tests. Design the strategy to maximize value:

With $50/month for integration tests with gpt-4o-mini:

Approximate cost per test: $0.0002 (600 tokens on average)
Possible tests per month: $50 / $0.0002 = 250,000 test runs

Recommended strategy:
  - 10 critical tests × 1 run/day × 30 days = 300 runs
  - Cost: 300 × $0.0002 = $0.06/month ← You use only 0.1% of the budget

With $50/month you can:
  - Run 200 integration tests twice a day for 30 days:
    200 × 2 × 30 = 12,000 runs × $0.0002 = $2.40/month

Conclusion: With gpt-4o-mini, the $50/month budget is
more than enough for a reasonable suite of integration tests.
The real constraint is execution time, not cost.

If you used gpt-4o (17x more expensive):
  200 × 2 × 30 × $0.0034 = $40.80/month ← Close to the limit
  → Reduce to 50 critical tests or run less frequently

Additional resources

1. pytest-rerunfailures — Automatic retry for flaky tests
2. sentence-transformers — Pretrained Models — Which model to choose for Spanish/multilingual
3. Hypothesis — Reproducing failures — Understand Hypothesis shrinking
4. Non-Determinism in Tests — Martin Fowler on flaky tests
5. GitHub Actions — Workflow syntax — To configure CI/CD
6. Module 4: Guardrails — Input/Output Validation
7. OpenAI Pricing — To calculate exact budgets
8. Testing ML Systems — Google's reference for ML testing