2. Prompt Regression Testing in CI

Overview

In traditional software, regression testing verifies that a change doesn't break existing functionality — the tests pass or fail, binary. In AI systems, the concept is subtler: a change in a prompt can produce responses that are still "correct" but of lower quality. The test doesn't fail, but the system got worse.

Prompt regression testing solves exactly that. You define test cases with inputs and expected responses (baselines), you run the current prompts, and you compare the results. If quality drops beyond a threshold, CI fails and blocks the merge.

What you're going to build:

1. A test case file (JSON) with inputs and baselines
2. An evaluate_prompts.py script with three evaluation methods
3. A GitHub Actions workflow that runs the evaluations and fails if quality drops
4. A mechanism for comparing results against baselines stored as artifacts

What prompt regression is

The problem

Prompt v1: "You are a technical assistant. Answer clearly and concisely.
            Include code examples when relevant."
→ Quality: good. Clear responses, with code, concise.

Prompt v2: "Answer technical questions."
→ Quality: degraded. Generic responses, no code, no structure.

Every unit test passes — the function returns a string, the JSON format is correct, there are no exceptions. But the quality of the output dropped significantly.

The solution: baselines and evaluation

Test Case:
  Input: "How do I make a GET request in Python?"
  Baseline: A response that mentions 'requests', includes code, is < 200 words

Evaluation:
  Prompt v1 → "Use the requests library: `import requests`..." → ✅ Meets the baseline
  Prompt v2 → "You can make HTTP requests with Python."         → ❌ Includes no code

Three levels of evaluation

Step 1: Define the test cases

Each test case has an input, expected keywords, and a reference response:

{
  "test_cases": [
    {
      "id": "tc-001",
      "input": "How do I make a GET request in Python?",
      "expected_keywords": ["requests", "import", "get"],
      "expected_min_length": 50,
      "expected_max_length": 500,
      "reference_response": "To make a GET request in Python, use the requests library. First install it with pip install requests, then use requests.get(url) to make the call.",
      "category": "code_explanation"
    },
    {
      "id": "tc-002",
      "input": "Explain what a REST API is in one sentence.",
      "expected_keywords": ["HTTP", "resources", "endpoints"],
      "expected_min_length": 20,
      "expected_max_length": 150,
      "reference_response": "A REST API is an interface that lets applications communicate using standard HTTP methods to access and manipulate resources through endpoints.",
      "category": "concept_explanation"
    },
    {
      "id": "tc-003",
      "input": "Write a Python function that computes the factorial of a number.",
      "expected_keywords": ["def", "factorial", "return"],
      "expected_min_length": 30,
      "expected_max_length": 600,
      "reference_response": "def factorial(n):\n    if n <= 1:\n        return 1\n    return n * factorial(n - 1)",
      "category": "code_generation"
    }
  ],
  "metadata": {
    "version": "1.0",
    "prompt_version": "v1",
    "description": "Test cases for the technical assistant"
  }
}

Save this file as tests/prompt_test_cases.json. The recommendation: 5 test cases minimum for a viable project, 10-20 for production.

Step 2: The evaluation script

This is the central script. It reads the test cases, runs each one with the current prompt, and evaluates the results:

# scripts/evaluate_prompts.py
"""
Prompt regression testing script.
Runs test cases against the current prompt and evaluates quality.
"""

import json
import os
import sys
from datetime import datetime, timezone
from pathlib import Path

from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()


SYSTEM_PROMPT = """You are a technical assistant specialized in Python and software development.
Answer clearly and concisely. Include code examples when relevant.
Use plain English for explanations and idiomatic Python for code."""


def load_test_cases(path: str = "tests/prompt_test_cases.json") -> list[dict]:
    with open(path) as f:
        data = json.load(f)
    return data["test_cases"]


def get_llm_response(client: OpenAI, user_input: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_input},
        ],
        max_tokens=500,
        temperature=0.0,
    )
    return response.choices[0].message.content


def evaluate_keywords(response: str, expected_keywords: list[str]) -> dict:
    response_lower = response.lower()
    found = [k for k in expected_keywords if k.lower() in response_lower]
    missing = [k for k in expected_keywords if k.lower() not in response_lower]
    score = len(found) / len(expected_keywords) if expected_keywords else 1.0

    return {
        "method": "keyword_matching",
        "score": score,
        "found_keywords": found,
        "missing_keywords": missing,
        "passed": score >= 0.6,
    }


def evaluate_length(response: str, min_length: int, max_length: int) -> dict:
    length = len(response)
    within_range = min_length <= length <= max_length

    return {
        "method": "length_check",
        "response_length": length,
        "expected_range": f"{min_length}-{max_length}",
        "passed": within_range,
        "score": 1.0 if within_range else 0.0,
    }


JUDGE_PROMPT = """You are a response quality evaluator. Compare the generated response
with the reference response and evaluate the quality.

Criteria: Accuracy, Completeness, Clarity, Code (if applicable).

Reply ONLY with JSON: {"score": <float 0.0-1.0>, "reasoning": "<brief explanation>"}"""


def evaluate_with_llm_judge(
    client: OpenAI, user_input: str, response: str, reference: str,
) -> dict:
    judge_input = f"""Question: {user_input}
Generated response: {response}
Reference response: {reference}
Evaluate the quality of the generated response compared to the reference."""

    judge_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": JUDGE_PROMPT},
            {"role": "user", "content": judge_input},
        ],
        max_tokens=200,
        temperature=0.0,
    )

    try:
        result = json.loads(judge_response.choices[0].message.content)
        return {
            "method": "llm_judge",
            "score": float(result["score"]),
            "reasoning": result["reasoning"],
            "passed": float(result["score"]) >= 0.7,
        }
    except (json.JSONDecodeError, KeyError):
        return {
            "method": "llm_judge",
            "score": 0.0,
            "reasoning": "Error parsing judge response",
            "passed": False,
        }


def run_evaluation(
    use_llm_judge: bool = False,
    test_cases_path: str = "tests/prompt_test_cases.json",
) -> dict:
    client = OpenAI()
    test_cases = load_test_cases(test_cases_path)
    results = []
    total_score = 0.0
    passed_count = 0

    print(f"Running prompt regression tests ({len(test_cases)} cases)...")
    print("=" * 60)

    for tc in test_cases:
        print(f"\n[{tc['id']}] {tc['input'][:50]}...")
        response = get_llm_response(client, tc["input"])

        keyword_eval = evaluate_keywords(response, tc["expected_keywords"])
        length_eval = evaluate_length(
            response, tc["expected_min_length"], tc["expected_max_length"]
        )
        evaluations = [keyword_eval, length_eval]

        if use_llm_judge:
            judge_eval = evaluate_with_llm_judge(
                client, tc["input"], response, tc["reference_response"]
            )
            evaluations.append(judge_eval)

        case_score = sum(e["score"] for e in evaluations) / len(evaluations)
        case_passed = all(e["passed"] for e in evaluations)

        if case_passed:
            passed_count += 1
        total_score += case_score

        status = "✅ PASS" if case_passed else "❌ FAIL"
        print(f"  {status} (score: {case_score:.2f})")
        for ev in evaluations:
            print(f"    {ev['method']}: {ev['score']:.2f}", end="")
            if not ev["passed"] and ev.get("missing_keywords"):
                print(f" (missing: {ev['missing_keywords']})", end="")
            print()

        results.append({
            "test_case_id": tc["id"],
            "input": tc["input"],
            "response": response,
            "evaluations": evaluations,
            "score": case_score,
            "passed": case_passed,
        })

    avg_score = total_score / len(test_cases) if test_cases else 0.0
    all_passed = passed_count == len(test_cases)

    print("\n" + "=" * 60)
    print(f"Results: {passed_count}/{len(test_cases)} passed")
    print(f"Average score: {avg_score:.2f}")
    print(f"Overall: {'✅ PASS' if all_passed else '❌ FAIL'}")

    return {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "prompt_version": "current",
        "total_cases": len(test_cases),
        "passed_cases": passed_count,
        "failed_cases": len(test_cases) - passed_count,
        "average_score": avg_score,
        "overall_passed": all_passed,
        "results": results,
    }


def save_report(report: dict, path: str = "prompt_eval_report.json") -> None:
    with open(path, "w") as f:
        json.dump(report, f, indent=2, ensure_ascii=False)
    print(f"\nReport saved to {path}")


def compare_with_baseline(
    current_report: dict,
    baseline_path: str = "prompt_eval_baseline.json",
    threshold: float = 0.1,
) -> bool:
    if not Path(baseline_path).exists():
        print(f"No baseline found at {baseline_path} — skipping comparison")
        return True

    with open(baseline_path) as f:
        baseline = json.load(f)

    current_score = current_report["average_score"]
    baseline_score = baseline["average_score"]
    delta = baseline_score - current_score

    print(f"\nBaseline comparison:")
    print(f"  Baseline score: {baseline_score:.2f}")
    print(f"  Current score:  {current_score:.2f}")
    print(f"  Delta:          {delta:+.2f}")
    print(f"  Threshold:      {threshold:.2f}")

    if delta > threshold:
        print(f"  ❌ REGRESSION DETECTED: score dropped by {delta:.2f}")
        return False

    print(f"  ✅ No regression: delta within threshold")
    return True


if __name__ == "__main__":
    use_judge = "--llm-judge" in sys.argv
    report = run_evaluation(use_llm_judge=use_judge)
    save_report(report)

    passed = compare_with_baseline(report)

    if not report["overall_passed"] or not passed:
        print("\n❌ Prompt regression test FAILED")
        sys.exit(1)

    print("\n✅ Prompt regression test PASSED")
    sys.exit(0)

Step 3: A real scenario — Prompt v1 vs Prompt v2

Prompt v1 (good): results 5/5 passed, score: 0.97 → you save it as the baseline.

A developer "simplifies" the prompt to "Answer technical questions briefly.":

Results: 2/5 passed, Average score: 0.62

Baseline comparison:
  Baseline score: 0.97
  Current score:  0.62
  Delta:          -0.35
  Threshold:      0.10
  ❌ REGRESSION DETECTED: score dropped by 0.35 (threshold: 0.10)

CI detected that quality dropped by 35%. The merge is blocked.

Step 4: Integrate it into GitHub Actions

# .github/workflows/ai-checks.yml
name: AI Quality Checks
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  prompt-regression:
    runs-on: ubuntu-latest
    # LLM API calls can hang — 10 min cap prevents zombie jobs from burning runner minutes
    timeout-minutes: 10

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python 3.12
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Download baseline (if exists)
        uses: actions/download-artifact@v4
        with:
          name: prompt-eval-baseline
          path: .
        continue-on-error: true

      - name: Run prompt regression tests
        run: python scripts/evaluate_prompts.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          # Without PYTHONPATH, Python can't resolve imports like "from src.config import ..."
          PYTHONPATH: ${{ github.workspace }}

      - name: Upload evaluation report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: prompt-eval-report
          path: prompt_eval_report.json
          retention-days: 30

      - name: Update baseline on main
        if: github.ref == 'refs/heads/main' && success()
        uses: actions/upload-artifact@v4
        with:
          name: prompt-eval-baseline
          path: prompt_eval_report.json
          # Baseline needs longer retention than reports — must survive across multiple sprint cycles
          retention-days: 90

continue-on-error: true on the baseline download: The first time, there is no baseline. Without this, the workflow would fail before running the evaluations.

if: always() on the report upload: It saves the report for debugging even if the test fails.

if: github.ref == 'refs/heads/main': You only update the baseline when merging to main.

Step 5: Baseline management

First run on main:
  → No baseline → skip the comparison → save the report as the first baseline

Runs on PRs:
  → Download main's baseline → compare the score → fail if delta > threshold

Successful merge to main:
  → The current report becomes the new baseline

Advanced evaluation: LLM-as-Judge

python scripts/evaluate_prompts.py              # Keyword + length only
python scripts/evaluate_prompts.py --llm-judge  # With LLM-as-Judge

[tc-001] How do I make a GET request in Python?...
  ✅ PASS (score: 0.93)
    keyword_matching: 1.00
    length_check: 1.00
    llm_judge: 0.80 (reasoning: "Correct and with code,
    but it doesn't mention error handling")

The judge detects nuances that keyword matching can't. The cost: ~$0.01 per evaluation with gpt-4o-mini. With 10 test cases and 20 PRs/week, the monthly cost is ~$10.

Troubleshooting

"The LLM gives different responses each time and the tests are flaky"

Cause: temperature > 0 introduces randomness.

Solution: Use temperature=0.0 in the evaluation script and in the judge.

"Keyword matching is too strict"

Cause: Exact keywords that don't appear in valid rephrasings.

Solution: Lower the threshold to 0.5 and use generic keywords ("request", "http") instead of exact ones ("requests.get(url)").

"The baseline goes stale quickly"

Cause: Frequent changes to the prompt without updating the baseline.

Solution: The workflow already updates the baseline automatically when merging to main. If it goes stale because of changes in the provider's model, adjust the threshold or regenerate the baseline.

Exercises

Exercise 1: Create test cases for your project

Create a tests/prompt_test_cases.json file with 5 test cases for a cooking assistant that answers questions about recipes with ingredients and steps.

{
  "test_cases": [
    {
      "id": "cook-001",
      "input": "How do I make a Spanish omelette?",
      "expected_keywords": ["eggs", "potatoes", "oil", "pan"],
      "expected_min_length": 80,
      "expected_max_length": 600,
      "reference_response": "To make a Spanish omelette you need eggs, potatoes, olive oil and salt. Peel and cut the potatoes, fry them in oil, beat the eggs, mix everything and cook in a pan over medium heat.",
      "category": "recipe"
    },
    {
      "id": "cook-002",
      "input": "What ingredients do I need for guacamole?",
      "expected_keywords": ["avocado", "lime", "salt"],
      "expected_min_length": 30,
      "expected_max_length": 300,
      "reference_response": "For guacamole you need ripe avocados, lime juice, salt, cilantro, onion and tomato.",
      "category": "ingredients"
    },
    {
      "id": "cook-003",
      "input": "What's the difference between baking and roasting?",
      "expected_keywords": ["temperature", "heat", "oven"],
      "expected_min_length": 40,
      "expected_max_length": 400,
      "reference_response": "Baking uses indirect heat at a moderate temperature. Roasting uses direct heat at a high temperature to brown the surface.",
      "category": "technique"
    },
    {
      "id": "cook-004",
      "input": "Give me a quick pasta recipe (under 15 minutes)",
      "expected_keywords": ["pasta", "water", "minutes"],
      "expected_min_length": 60,
      "expected_max_length": 500,
      "reference_response": "Boil salted water, cook the pasta per the instructions. Sauté garlic in olive oil, add cherry tomatoes. Toss the pasta with the sauce.",
      "category": "recipe"
    },
    {
      "id": "cook-005",
      "input": "How do I know if an avocado is ripe?",
      "expected_keywords": ["color", "texture", "pressure"],
      "expected_min_length": 30,
      "expected_max_length": 300,
      "reference_response": "A ripe avocado gives when you press it, has a dark green to black color, and the stem pulls off showing green underneath.",
      "category": "tip"
    }
  ],
  "metadata": {
    "version": "1.0",
    "prompt_version": "v1",
    "description": "Test cases for the cooking assistant"
  }
}

Exercise 2: Implement a format consistency evaluation

Write a function that evaluates whether the LLM's responses are consistent in format (bullet points vs paragraphs). It takes a list of responses and returns a score.

def evaluate_format_consistency(responses: list[str]) -> dict:
    formats = []
    for response in responses:
        has_bullets = any(
            line.strip().startswith(("-", "•", "*", "1."))
            for line in response.split("\n") if line.strip()
        )
        has_code = "```" in response
        formats.append({"bullets": has_bullets, "code": has_code})

    bullet_ratio = sum(f["bullets"] for f in formats) / len(formats)
    code_ratio = sum(f["code"] for f in formats) / len(formats)
    consistency_score = max(bullet_ratio, 1 - bullet_ratio) * 0.6 + \
                        max(code_ratio, 1 - code_ratio) * 0.4

    return {
        "method": "format_consistency",
        "score": round(consistency_score, 2),
        "passed": consistency_score >= 0.7,
    }

Exercise 3: A workflow with prompt regression as a quality gate

Create a workflow that: (1) runs lint, (2) runs unit tests, and (3) runs prompt regression. If any step fails, the workflow stops.

# .github/workflows/ai-quality-gate.yml
name: AI Quality Gate
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  quality-checks:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install -r requirements.txt

      - name: Lint with ruff
        run: ruff check src/ scripts/ tests/

      - name: Run unit tests
        run: pytest tests/ -v --tb=short
        env:
          PYTHONPATH: ${{ github.workspace }}

      - name: Download prompt baseline
        uses: actions/download-artifact@v4
        with:
          name: prompt-eval-baseline
          path: .
        continue-on-error: true

      - name: Run prompt regression tests
        run: python scripts/evaluate_prompts.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Upload prompt eval report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: prompt-eval-report
          path: prompt_eval_report.json

      - name: Update baseline on main
        if: github.ref == 'refs/heads/main' && success()
        uses: actions/upload-artifact@v4
        with:
          name: prompt-eval-baseline
          path: prompt_eval_report.json

Exercise 4: A threshold configurable by environment variable

Modify compare_with_baseline so the threshold is configurable via PROMPT_REGRESSION_THRESHOLD. Default: 0.10.

import os

def compare_with_baseline(
    current_report: dict,
    baseline_path: str = "prompt_eval_baseline.json",
) -> bool:
    threshold = float(os.environ.get("PROMPT_REGRESSION_THRESHOLD", "0.10"))

    if not Path(baseline_path).exists():
        print(f"No baseline found — skipping comparison")
        return True

    with open(baseline_path) as f:
        baseline = json.load(f)

    current_score = current_report["average_score"]
    baseline_score = baseline["average_score"]
    delta = baseline_score - current_score

    print(f"\nBaseline comparison:")
    print(f"  Baseline: {baseline_score:.2f} | Current: {current_score:.2f}")
    print(f"  Delta: {delta:+.2f} | Threshold: {threshold:.2f}")

    if delta > threshold:
        print(f"  ❌ REGRESSION DETECTED")
        return False
    print(f"  ✅ No regression")
    return True

      - name: Run prompt regression tests
        run: python scripts/evaluate_prompts.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PROMPT_REGRESSION_THRESHOLD: "0.15"

Summary

• ✅ Prompt regression testing detects quality degradation that unit tests don't capture
• ✅ Test cases are a JSON file with inputs, expected keywords, and reference responses
• ✅ Three levels of evaluation: keyword matching (free), length check (basic), and LLM-as-judge (precise, with a cost)
• ✅ The evaluate_prompts.py script runs the evaluations, generates reports, and compares against baselines
• ✅ Baselines are only updated when the code reaches main — on PRs, they're only compared
• ✅ Temperature 0.0 is mandatory for deterministic evaluations
• ✅ The workflow runs the cheap checks first, before the ones that cost API money

Additional resources

1. OpenAI Evals — OpenAI's evaluation framework
2. LangSmith Evaluation — Prompt evaluation with LangChain
3. DeepEval — Open-source evaluation framework
4. GitHub Actions Artifacts — Artifacts documentation
5. Prompt Engineering Guide — Evaluations — Prompt evaluation methods