5. Regression Testing for Prompts

Description

Detect when a prompt change breaks cases that used to work. Implementing test suites with pytest. CI/CD integration with GitHub Actions. Handling baseline scores and regression alerts. Strategies to cut testing costs.

What Is Regression Testing for Prompts?

In traditional software, regression tests verify that a fixed bug doesn't come back. For prompts, regression is different:

Typical problem without regression testing:
1. You have a classification prompt with 94% accuracy
2. You find a new edge case, you modify the prompt
3. The edge case now works (95% accuracy on that case)
4. But without knowing it, you broke 3 cases that used to work
5. Real accuracy: 91%
6. You find out when users report problems

With regression testing:
1. When you modify the prompt, you run the golden set automatically
2. The system detects: accuracy dropped from 94% to 91%
3. It shows you exactly which cases you broke
4. Don't deploy until the regression is fixed

Difference from Traditional Software Testing

Base Implementation

The Regression Testing Engine

import json
import time
from pathlib import Path
from dataclasses import dataclass
from typing import Callable
from openai import OpenAI

client = OpenAI()

@dataclass
class TestResult:
    example_id: str
    input: str
    expected: str
    actual: str
    passed: bool
    score: float
    latency_ms: float
    tokens_used: int


def run_prompt(prompt_template: str, input_text: str) -> tuple[str, dict]:
    """
    Runs the prompt and returns (output, metadata).
    metadata includes latency and tokens.
    """
    start = time.time()
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user", "content": prompt_template.format(input=input_text)}
        ],
        temperature=0
    )
    
    latency_ms = (time.time() - start) * 1000
    output = response.choices[0].message.content.strip()
    
    metadata = {
        "latency_ms": latency_ms,
        "prompt_tokens": response.usage.prompt_tokens,
        "completion_tokens": response.usage.completion_tokens,
        "total_tokens": response.usage.total_tokens
    }
    
    return output, metadata


def evaluate_example(
    prompt_template: str,
    example: dict,
    evaluator: Callable | None = None
) -> TestResult:
    """
    Evaluates a single example from the golden set.
    
    evaluator: custom function. If not provided, uses normalized exact match.
    """
    output, metadata = run_prompt(prompt_template, example["input"])
    
    if evaluator:
        passed, score = evaluator(example["expected_output"], output)
    else:
        # Default: normalized exact match
        expected_norm = str(example["expected_output"]).strip().lower()
        output_norm = output.strip().lower()
        passed = expected_norm == output_norm
        score = 1.0 if passed else 0.0
    
    return TestResult(
        example_id=example["id"],
        input=example["input"],
        expected=str(example["expected_output"]),
        actual=output,
        passed=passed,
        score=score,
        latency_ms=metadata["latency_ms"],
        tokens_used=metadata["total_tokens"]
    )

Baseline Management

The baseline is the set of metrics from the current prompt, against which new versions are compared.

class BaselineManager:
    """Manages evaluation baselines for regression comparison."""
    
    def __init__(self, baseline_path: str = "baseline.json"):
        self.baseline_path = Path(baseline_path)
        self._baselines: dict = {}
        
        if self.baseline_path.exists():
            self._load()
    
    def _load(self) -> None:
        with open(self.baseline_path) as f:
            self._baselines = json.load(f)
        print(f"Baseline loaded: {len(self._baselines)} prompts")
    
    def save(self) -> None:
        with open(self.baseline_path, "w") as f:
            json.dump(self._baselines, f, indent=2)
        print(f"Baseline saved to {self.baseline_path}")
    
    def log(self, prompt_name: str, metrics: dict, version: str) -> None:
        """Records the current metrics as the new baseline."""
        self._baselines[prompt_name] = {
            "version": version,
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
            "metrics": metrics
        }
        self.save()
        print(f"New baseline recorded for '{prompt_name}' ({version})")
    
    def get(self, prompt_name: str) -> dict | None:
        """Gets a prompt's baseline. None if it doesn't exist."""
        return self._baselines.get(prompt_name)
    
    def compare(
        self,
        prompt_name: str,
        new_metrics: dict,
        tolerance: float = 0.02
    ) -> dict:
        """
        Compares new metrics against the baseline.
        
        tolerance: Maximum variation allowed before reporting a regression.
                   0.02 = 2% of margin — small variations are normal.
        """
        baseline = self.get(prompt_name)
        
        if baseline is None:
            return {
                "status": "NO_BASELINE",
                "message": f"No baseline for '{prompt_name}'. Record one with baseline_manager.log()",
                "regressions": [],
                "improvements": []
            }
        
        baseline_metrics = baseline["metrics"]
        regressions = []
        improvements = []
        unchanged = []
        
        for metric, new_value in new_metrics.items():
            if metric not in baseline_metrics:
                continue
            
            previous_value = baseline_metrics[metric]
            delta = new_value - previous_value
            
            if delta < -tolerance:
                regressions.append({
                    "metric": metric,
                    "previous": previous_value,
                    "new": new_value,
                    "delta": delta,
                    "severity": "CRITICAL" if delta < -0.05 else "MINOR"
                })
            elif delta > tolerance:
                improvements.append({
                    "metric": metric,
                    "previous": previous_value,
                    "new": new_value,
                    "delta": delta
                })
            else:
                unchanged.append(metric)
        
        return {
            "status": "FAIL" if regressions else "PASS",
            "regressions": regressions,
            "improvements": improvements,
            "unchanged": unchanged,
            "baseline_version": baseline["version"],
            "baseline_date": baseline["timestamp"]
        }


# Usage:
baseline_mgr = BaselineManager()

# First time: record the baseline of the current prompt
current_metrics = {"accuracy": 0.94, "faithfulness": 0.88, "format": 1.0}
baseline_mgr.log("sentiment_classifier", current_metrics, "v1.0")

# When you change the prompt: compare
new_metrics = {"accuracy": 0.91, "faithfulness": 0.89, "format": 1.0}
result = baseline_mgr.compare("sentiment_classifier", new_metrics)

if result["status"] == "FAIL":
    print("❌ REGRESSION DETECTED:")
    for r in result["regressions"]:
        print(f"  {r['metric']}: {r['previous']:.2%} → {r['new']:.2%} ({r['delta']:+.2%}) [{r['severity']}]")
else:
    print("✅ No regressions")
    for m in result["improvements"]:
        print(f"  📈 {m['metric']}: {m['previous']:.2%} → {m['new']:.2%} ({m['delta']:+.2%})")

Test Suite with pytest

To integrate with CI/CD, organize the tests with pytest:

# tests/test_classifier.py
import pytest
import json
from pathlib import Path
from openai import OpenAI

client = OpenAI()

# ===== FIXTURES =====

@pytest.fixture(scope="session")
def golden_set():
    """Loads the golden set once per session."""
    path = Path("datasets/sentiment_classifier.json")
    with open(path) as f:
        return json.load(f)


@pytest.fixture(scope="session")
def baseline():
    """Loads the metrics baseline."""
    path = Path("baseline.json")
    if not path.exists():
        return {}
    with open(path) as f:
        return json.load(f)


@pytest.fixture(scope="module")
def current_prompt():
    """Loads the current prompt from a file."""
    with open("prompts/classifier_v1.txt") as f:
        return f.read()


# ===== INDIVIDUAL TESTS =====

class TestSentimentClassifier:
    
    @pytest.mark.parametrize("example_id,expected", [
        ("001", "POSITIVE"),
        ("002", "NEGATIVE"),
        ("003", "NEUTRAL"),
    ])
    def test_critical_cases(self, current_prompt, example_id, expected, golden_set):
        """
        Tests that must NEVER fail — the most basic cases.
        If these fail, something is seriously wrong.
        """
        example = next(e for e in golden_set if e["id"] == example_id)
        output, _ = run_prompt(current_prompt, example["input"])
        
        assert output.strip().upper() == expected, \
            f"Critical case {example_id} failed: expected={expected}, got={output}"
    
    def test_min_accuracy(self, current_prompt, golden_set):
        """Accuracy across the full golden set can't drop below 85%."""
        outputs = []
        for ex in golden_set:
            output, _ = run_prompt(current_prompt, ex["input"])
            is_correct = output.strip().lower() == str(ex["expected_output"]).strip().lower()
            outputs.append(is_correct)
        
        accuracy = sum(outputs) / len(outputs)
        
        assert accuracy >= 0.85, \
            f"Accuracy {accuracy:.2%} below the acceptable minimum (85%)"
    
    def test_no_regression_vs_baseline(self, current_prompt, golden_set, baseline):
        """Accuracy can't fall more than 2% relative to the baseline."""
        if "sentiment_classifier" not in baseline:
            pytest.skip("No baseline recorded — skipping regression test")
        
        baseline_accuracy = baseline["sentiment_classifier"]["metrics"]["accuracy"]
        
        outputs = []
        for ex in golden_set:
            output, _ = run_prompt(current_prompt, ex["input"])
            outputs.append(output.strip().lower() == str(ex["expected_output"]).strip().lower())
        
        current_accuracy = sum(outputs) / len(outputs)
        
        assert current_accuracy >= baseline_accuracy - 0.02, \
            f"REGRESSION: accuracy dropped from {baseline_accuracy:.2%} to {current_accuracy:.2%} ({current_accuracy - baseline_accuracy:+.2%})"
    
    def test_format_compliance(self, current_prompt, golden_set):
        """The output must always be: POSITIVE, NEGATIVE, or NEUTRAL."""
        invalid_outputs = []
        
        for ex in golden_set[:20]:  # Only the first 20 for speed
            output, _ = run_prompt(current_prompt, ex["input"])
            if output.strip().upper() not in ["POSITIVE", "NEGATIVE", "NEUTRAL"]:
                invalid_outputs.append({"id": ex["id"], "output": output})
        
        assert len(invalid_outputs) == 0, \
            f"Outputs with invalid format: {invalid_outputs}"
    
    def test_edge_cases(self, current_prompt, golden_set):
        """Edge cases must have accuracy >= 70%."""
        edge_cases = [e for e in golden_set if e.get("difficulty") in ["hard", "very_hard"]]
        
        if not edge_cases:
            pytest.skip("No edge cases in the golden set")
        
        outputs = []
        for ex in edge_cases:
            output, _ = run_prompt(current_prompt, ex["input"])
            outputs.append(output.strip().lower() == str(ex["expected_output"]).strip().lower())
        
        accuracy = sum(outputs) / len(outputs)
        
        assert accuracy >= 0.70, \
            f"Accuracy on edge cases ({accuracy:.2%}) below 70%"
    
    def test_acceptable_latency(self, current_prompt):
        """Average latency must be < 3 seconds."""
        import time
        
        test_input = "This is an excellent product"
        latencies = []
        
        for _ in range(3):
            start = time.time()
            run_prompt(current_prompt, test_input)
            latencies.append(time.time() - start)
        
        avg_latency = sum(latencies) / len(latencies)
        
        assert avg_latency < 3.0, \
            f"Average latency {avg_latency:.1f}s exceeds the 3s limit"

CI/CD Integration (GitHub Actions)

# .github/workflows/prompt-regression-tests.yml
name: Prompt Regression Tests

on:
  pull_request:
    paths:
      - 'prompts/**'      # Only when the prompts change
      - 'datasets/**'     # Or when the golden set changes
  push:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # Daily at 2am (to catch model drift)

jobs:
  regression-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Cache dependencies
        uses: actions/cache@v3
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
      
      - name: Install dependencies
        run: pip install -r requirements.txt
      
      - name: Run critical tests (fast)
        run: pytest tests/test_classifier.py::TestSentimentClassifier::test_critical_cases -v
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      
      - name: Run regression tests
        run: |
          pytest tests/test_classifier.py -v \
            --tb=short \
            --junitxml=test-results.xml \
            -k "not latency"  # Exclude latency tests in CI (flaky)
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      
      - name: Upload test results
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: test-results
          path: test-results.xml
      
      - name: Comment PR with results
        if: github.event_name == 'pull_request'
        uses: EnricoMi/publish-unit-test-result-action@v2
        with:
          files: test-results.xml

Strategies to Cut Testing Costs

The cost of running regression tests in CI can be significant. Strategies to bring it down:

Strategy 1: Test Tiers

# conftest.py — pytest configuration
import pytest

def pytest_addoption(parser):
    parser.addoption("--test-tier", action="store", default="smoke",
                     help="Test tier: smoke | standard | full")

@pytest.fixture(scope="session")
def test_tier(request):
    return request.config.getoption("--test-tier")


# In tests:
@pytest.fixture
def golden_set_by_tier(test_tier):
    """Returns a subset of the golden set based on the tier."""
    with open("datasets/golden_set.json") as f:
        all_examples = json.load(f)
    
    if test_tier == "smoke":
        # Critical cases only: fast and cheap
        return [e for e in all_examples if e.get("critical", False)][:10]
    elif test_tier == "standard":
        # Happy path + edge cases: cost/coverage balance
        return [e for e in all_examples if e.get("difficulty") in ["easy", "hard"]][:50]
    else:  # full
        return all_examples

# In CI (PRs): smoke tests only
pytest tests/ --test-tier=smoke

# In CI (merge to main): standard tests
pytest tests/ --test-tier=standard

# Overnight: full suite
pytest tests/ --test-tier=full

Strategy 2: Result Caching

import hashlib
import json
from pathlib import Path

class TestCache:
    """Cache of test results to avoid re-running tests with no changes."""
    
    def __init__(self, cache_dir: str = ".test_cache"):
        self.cache_dir = Path(cache_dir)
        self.cache_dir.mkdir(exist_ok=True)
    
    def cache_key(self, prompt: str, input_text: str, model: str = "gpt-4o-mini") -> str:
        content = f"{prompt}|{input_text}|{model}"
        return hashlib.sha256(content.encode()).hexdigest()[:16]
    
    def get(self, prompt: str, input_text: str) -> str | None:
        key = self.cache_key(prompt, input_text)
        path = self.cache_dir / f"{key}.json"
        
        if path.exists():
            with open(path) as f:
                return json.load(f)["output"]
        return None
    
    def set(self, prompt: str, input_text: str, output: str) -> None:
        key = self.cache_key(prompt, input_text)
        path = self.cache_dir / f"{key}.json"
        
        with open(path, "w") as f:
            json.dump({"output": output, "timestamp": time.time()}, f)
    
    def stats(self) -> dict:
        files = list(self.cache_dir.glob("*.json"))
        return {"cached_results": len(files), "cache_dir": str(self.cache_dir)}


cache = TestCache()

def run_prompt_cached(prompt: str, input_text: str) -> str:
    """Cached version of run_prompt for tests."""
    cached = cache.get(prompt, input_text)
    if cached:
        return cached
    
    output, _ = run_prompt(prompt, input_text)
    cache.set(prompt, input_text, output)
    return output

Strategy 3: Parallelization

# pytest-xdist to parallelize tests
# pip install pytest-xdist

# In CI:
# pytest tests/ -n 4  # 4 workers in parallel

# In the test code, use asyncio to parallelize the API calls:
import asyncio
from openai import AsyncOpenAI

async_client = AsyncOpenAI()

async def run_prompt_async(prompt_template: str, input_text: str) -> tuple[str, dict]:
    """Async version of run_prompt."""
    response = await async_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt_template.format(input=input_text)}],
        temperature=0
    )
    return response.choices[0].message.content.strip(), {}


async def evaluate_golden_set_async(
    prompt_template: str,
    golden_set: list[dict],
    max_concurrent: int = 10
) -> list[dict]:
    """Evaluates the golden set in parallel with a concurrency limit."""
    semaphore = asyncio.Semaphore(max_concurrent)
    
    async def evaluate_with_limit(example):
        async with semaphore:
            output, _ = await run_prompt_async(prompt_template, example["input"])
            return {
                "id": example["id"],
                "expected": str(example["expected_output"]),
                "actual": output,
                "passed": output.strip().lower() == str(example["expected_output"]).strip().lower()
            }
    
    tasks = [evaluate_with_limit(ex) for ex in golden_set]
    return await asyncio.gather(*tasks)

# Run:
# results = asyncio.run(evaluate_golden_set_async(prompt, golden_set))

Regression Report

def generate_regression_report(
    prompt_name: str,
    results: list[dict],
    baseline_metrics: dict | None = None
) -> str:
    """
    Generates a markdown report of the regression test results.
    Useful for commenting on PRs or sending over Slack.
    """
    passed = sum(1 for r in results if r["passed"])
    failed = len(results) - passed
    accuracy = passed / len(results)
    
    lines = [
        f"# Regression Test: {prompt_name}",
        f"",
        f"**Date:** {time.strftime('%Y-%m-%d %H:%M:%S')}",
        f"",
        f"## Summary",
        f"| Metric | Value |",
        f"|---------|-------|",
        f"| Total examples | {len(results)} |",
        f"| Passed | {passed} ✅ |",
        f"| Failed | {failed} {'❌' if failed > 0 else '✅'} |",
        f"| Accuracy | {accuracy:.2%} |",
    ]
    
    # Comparison against the baseline
    if baseline_metrics and "accuracy" in baseline_metrics:
        baseline_acc = baseline_metrics["accuracy"]
        delta = accuracy - baseline_acc
        status = "✅ PASS" if delta >= -0.02 else "❌ REGRESSION"
        lines.extend([
            f"| Baseline accuracy | {baseline_acc:.2%} |",
            f"| Delta vs baseline | {delta:+.2%} |",
            f"| Status | {status} |",
        ])
    
    # Failed cases
    if failed > 0:
        lines.extend([
            f"",
            f"## Failed Cases ({failed})",
            f"",
        ])
        
        for r in results:
            if not r["passed"]:
                short_input = str(r.get("input", ""))[:60]
                lines.extend([
                    f"### ❌ ID: {r['id']}",
                    f"- **Input:** `{short_input}...`",
                    f"- **Expected:** `{r['expected']}`",
                    f"- **Got:** `{r['actual']}`",
                    f"",
                ])
    
    return "\n".join(lines)

Troubleshooting

Problem 1: False positives in regression tests

Symptom: The tests fail even though the prompt didn't change.

Cause: Natural model variability, even with temperature=0.

Solution:

# Raise the tolerance of the baseline comparison
result = baseline_mgr.compare(
    "classifier",
    new_metrics,
    tolerance=0.03  # 3% instead of 2%
)

# Or run several times and average
def avg_accuracy(prompt, golden_set, n_runs=3):
    accuracies = []
    for _ in range(n_runs):
        outputs = [run_prompt(prompt, e["input"])[0] for e in golden_set]
        acc = sum(o.strip().lower() == str(e["expected_output"]).strip().lower()
                  for o, e in zip(outputs, golden_set)) / len(golden_set)
        accuracies.append(acc)
    return sum(accuracies) / len(accuracies)

Problem 2: Golden set too small

Symptom: With 20 examples, a single failure moves accuracy by 5%.

Cause: 20 examples = too much variance to tell real changes from noise.

Solution:

# Calculate the minimum size you need
def current_error_margin(n: int, accuracy: float = 0.9, confidence: float = 0.95) -> float:
    """Calculates the margin of error with n examples."""
    from scipy.stats import norm
    z = norm.ppf((1 + confidence) / 2)
    return z * (accuracy * (1 - accuracy) / n) ** 0.5

# n=20:  margin ≈ ±13%  — way too wide
# n=100: margin ≈ ±6%   — acceptable
# n=400: margin ≈ ±3%   — good

print(f"n=20:  ±{current_error_margin(20):.0%}")
print(f"n=100: ±{current_error_margin(100):.0%}")
print(f"n=400: ±{current_error_margin(400):.0%}")

Problem 3: Excessive cost in CI

Symptom: Each PR costs $5-10 in API calls.

Solution:

# 1. Use small smoke tests on PRs (10-20 critical examples)
# 2. Full suite only on merges to main
# 3. Cache results for prompts that didn't change
# 4. Rate limiting to reduce cost

import time

def run_with_rate_limit(
    prompt_template: str,
    golden_set: list[dict],
    requests_per_minute: int = 30
) -> list[dict]:
    """Runs the golden set with rate limiting."""
    delay = 60.0 / requests_per_minute  # seconds between requests
    results = []
    
    for i, example in enumerate(golden_set):
        if i > 0:
            time.sleep(delay)
        
        output, metadata = run_prompt(prompt_template, example["input"])
        results.append({
            "id": example["id"],
            "output": output,
            "tokens": metadata["total_tokens"]
        })
    
    return results

Exercises

Exercise 1: Write your first regression test

Write a test that verifies the following classification prompt keeps accuracy >= 88%:

CLASSIFIER_PROMPT = """Classify the following text as POSITIVE, NEGATIVE, or NEUTRAL.
Respond only with the category.

Text: {input}
Category:"""

import pytest
import json
from openai import OpenAI

client = OpenAI()

CLASSIFIER_PROMPT = """Classify the following text as POSITIVE, NEGATIVE, or NEUTRAL.
Respond only with the category.

Text: {input}
Category:"""

@pytest.fixture
def sentiment_golden_set():
    return [
        {"id": "001", "input": "Excellent product, highly recommended", "expected_output": "POSITIVE"},
        {"id": "002", "input": "Terrible, I would never buy again", "expected_output": "NEGATIVE"},
        {"id": "003", "input": "The package arrived on time", "expected_output": "NEUTRAL"},
        {"id": "004", "input": "Incredible quality, it exceeded my expectations", "expected_output": "POSITIVE"},
        {"id": "005", "input": "Very bad shopping experience", "expected_output": "NEGATIVE"},
        {"id": "006", "input": "The product comes in a blue box", "expected_output": "NEUTRAL"},
        {"id": "007", "input": "I love it!", "expected_output": "POSITIVE"},
        {"id": "008", "input": "Total fraud, completely useless", "expected_output": "NEGATIVE"},
    ]

def test_min_accuracy(sentiment_golden_set):
    """Accuracy can't drop below 88%."""
    correct = 0
    
    for example in sentiment_golden_set:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": CLASSIFIER_PROMPT.format(input=example["input"])}],
            temperature=0
        )
        output = response.choices[0].message.content.strip().upper()
        if output == example["expected_output"].upper():
            correct += 1
    
    accuracy = correct / len(sentiment_golden_set)
    assert accuracy >= 0.88, f"Accuracy {accuracy:.2%} below the minimum"
    print(f"✅ Accuracy: {accuracy:.2%}")

def test_valid_format(sentiment_golden_set):
    """The output must always be POSITIVE, NEGATIVE, or NEUTRAL."""
    valid_categories = {"POSITIVE", "NEGATIVE", "NEUTRAL"}
    invalid = []
    
    for example in sentiment_golden_set[:5]:  # Only 5 for speed
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": CLASSIFIER_PROMPT.format(input=example["input"])}],
            temperature=0
        )
        output = response.choices[0].message.content.strip().upper()
        if output not in valid_categories:
            invalid.append({"id": example["id"], "output": output})
    
    assert len(invalid) == 0, f"Outputs with invalid format: {invalid}"

Exercise 2: Implement baseline comparison

Write a function that compares current metrics against a baseline and returns whether there's a regression:

def detect_regression(
    new_metrics: dict[str, float],
    baseline_metrics: dict[str, float],
    tolerance: float = 0.02
) -> dict:
    """
    Detects regressions by comparing current metrics against the baseline.
    Returns: {"has_regression": bool, "regressions": list, "improvements": list}
    """
    regressions = []
    improvements = []
    
    for metric, new_value in new_metrics.items():
        if metric not in baseline_metrics:
            continue
        
        previous_value = baseline_metrics[metric]
        delta = new_value - previous_value
        
        if delta < -tolerance:
            regressions.append({
                "metric": metric,
                "previous": f"{previous_value:.2%}",
                "new": f"{new_value:.2%}",
                "delta": f"{delta:+.2%}",
                "severity": "CRITICAL" if delta < -0.05 else "MINOR"
            })
        elif delta > tolerance:
            improvements.append({
                "metric": metric,
                "delta": f"{delta:+.2%}"
            })
    
    return {
        "has_regression": len(regressions) > 0,
        "regressions": regressions,
        "improvements": improvements,
        "summary": f"{'❌ REGRESSION' if regressions else '✅ OK'}: {len(regressions)} regressions, {len(improvements)} improvements"
    }

# Test
baseline = {"accuracy": 0.94, "format": 1.0, "faithfulness": 0.88}
new_metrics = {"accuracy": 0.91, "format": 1.0, "faithfulness": 0.91}

result = detect_regression(new_metrics, baseline)
print(result["summary"])
# ❌ REGRESSION: 1 regressions, 1 improvements

Summary

• Regression testing: Detect when a prompt change breaks cases that used to work
• Baseline: Save the current prompt's metrics; compare after every change
• pytest: The natural framework for organizing prompt tests; with fixtures and parametrize
• CI/CD: GitHub Actions runs the tests on every PR or push
• Tolerance: 2-3% of margin for the model's natural variation
• Tiers: Smoke (cheap, fast) → Standard → Full (expensive, complete)
• Cache: Reuse test results when the prompt didn't change
• Parallelization: Async to speed up the golden set run

Additional resources

1. pytest Documentation — Testing framework
2. GitHub Actions — CI/CD
3. pytest-xdist — Test parallelization
4. OpenAI Rate Limits — API limits for planning tests
5. LangSmith Regression Testing — Managed alternative