Regression Testing and Quality Thresholds

Capsule description

Having an evaluation pipeline that produces metrics isn't enough. Metrics are information — you need to turn them into automatic decisions. That conversion happens through two mechanisms: absolute thresholds ("faithfulness must be above 0.85") and regression testing ("faithfulness can't drop more than 2% from the last release").

Without these mechanisms, someone has to look at the reports after every PR and manually decide whether quality went down. That doesn't scale, it misses subtlety, and eventually someone accepts a degradation because "it looks about the same". With thresholds and regression tests, the system automatically blocks any change that crosses predefined lines — and the conversation moves from "did quality drop?" (subjective) to "do we want to move the threshold?" (an explicit, documented decision).

In this capsule you'll calibrate defensible thresholds for your case, implement regression testing against a versioned baseline, handle the "tolerance" question (how much noise is acceptable), and design error messages so clear that a developer can act without opening the pipeline's code.

By the end you'll have the quality gate system ready to integrate into CI/CD (capsule 07), where every PR passes or fails automatically with immediate feedback.

Two complementary mechanisms: absolute vs relative

There are two kinds of check your system should run:

You need both:

• Absolute only: you accept gradual degradations (from 0.95 to 0.86 with no alert because both are > 0.85)
• Regression only: you accept a bad baseline (if the baseline is 0.50 and you hold at 0.49, "there's no regression" but the system is bad)

def quality_gate(current: dict, baseline: dict, thresholds: dict) -> dict:
    absolute_failures = check_absolute_thresholds(current, thresholds)
    regression_failures = check_no_regression(current, baseline)
    return {
        "passed": not (absolute_failures or regression_failures),
        "absolute_failures": absolute_failures,
        "regression_failures": regression_failures,
    }

Calibrating the absolute thresholds

Made-up thresholds are useless. There are three valid ways to pick thresholds:

Method 1: An empirical baseline

Run the evaluation 10 times over the current, stable system. Compute the mean and the standard deviation. Threshold = mean - 2 standard deviations.

import statistics

def empirical_thresholds(historical_runs: list[dict], confidence: float = 0.95) -> dict:
    """historical_runs: a list of scores from consecutive stable runs"""
    z_score = 1.96 if confidence == 0.95 else 2.58  # 99% confidence
    thresholds = {}
    for metric in ["faithfulness", "answer_relevancy", "context_precision", "context_recall"]:
        values = [r[metric] for r in historical_runs]
        mean = statistics.mean(values)
        stdev = statistics.stdev(values) if len(values) > 1 else 0
        thresholds[metric] = round(mean - z_score * stdev, 3)
    return thresholds

The advantage: the thresholds reflect your system's real variability, not opinions.

Method 2: A business requirement

"For technical support, faithfulness below 0.85 means >15% of the answers hallucinate, which generates false tickets." Define the threshold by the business consequence.

Method 3: Comparison with a human

Take 50 queries and have a human produce the answers. Measure RAGAS over the human answers. Your threshold is some percentage of the human score (typically 90-95%).

The anti-method: copying thresholds from a blog post. Every system has a different baseline. A faithfulness threshold of 0.90 can be trivial for a system over Wikipedia and aggressive for technical support with incomplete docs.

The implementation: thresholds and comparison

# eval/thresholds.py
from dataclasses import dataclass

@dataclass(frozen=True)
class Threshold:
    metric: str
    minimum: float
    severity: str  # "blocking", "warning"

DEFAULT_THRESHOLDS = [
    Threshold("faithfulness", 0.85, "blocking"),
    Threshold("answer_relevancy", 0.80, "blocking"),
    Threshold("context_precision", 0.75, "warning"),
    Threshold("context_recall", 0.80, "blocking"),
]

def check_absolute_thresholds(scores: dict, thresholds: list[Threshold]) -> list[dict]:
    failures = []
    for t in thresholds:
        category, metric_name = ("ragas", t.metric) if t.metric in scores.get("ragas", {}) else ("retrieval", t.metric)
        actual = scores.get(category, {}).get(metric_name)
        if actual is None:
            continue
        if actual < t.minimum:
            failures.append({
                "metric": t.metric,
                "actual": actual,
                "minimum": t.minimum,
                "severity": t.severity,
                "delta": actual - t.minimum,
            })
    return failures

def check_no_regression(current: dict, baseline: dict, tolerance: float = 0.02) -> list[dict]:
    failures = []
    for category in ["ragas", "retrieval"]:
        baseline_cat = baseline.get(category, {})
        current_cat = current.get(category, {})
        for metric, base_value in baseline_cat.items():
            curr_value = current_cat.get(metric, 0)
            if curr_value < base_value - tolerance:
                failures.append({
                    "metric": metric,
                    "category": category,
                    "baseline": base_value,
                    "current": curr_value,
                    "delta": curr_value - base_value,
                    "tolerance": tolerance,
                })
    return failures

A key design point: severity distinguishes "blocking" (blocks the PR) from "warning" (lets it through but raises an alert). Context precision tends to be noisier, so treating it as a warning avoids false positives without losing visibility.

Handling tolerance: how much noise is noise

An LLM-as-judge has inherent variability. Even with temperature=0, scores can vary ±0.5-1.5% between identical runs. If your tolerance is 0.5%, you'll get constant false positives.

How to choose the tolerance:

def measure_judge_noise(rag_app, golden_records, n_runs: int = 5) -> dict:
    """Runs the evaluation N times over the same dataset to measure the variance"""
    runs = []
    for _ in range(n_runs):
        samples = await execute_batch(golden_records, rag_app)
        scores = compute_ragas_metrics(samples)
        runs.append(scores)
    noise = {}
    for metric in runs[0].keys():
        values = [r[metric] for r in runs]
        noise[metric] = {
            "stdev": statistics.stdev(values),
            "range": max(values) - min(values),
        }
    return noise

If the stdev is 0.015, your tolerance should be at least 2× stdev = 0.03 to avoid false positives. For production systems, typically:

A versioned baseline and its rotation

The baseline is the "currently acceptable state". It lives versioned in git as eval/baseline.json:

{
  "version": "v1.4.0",
  "commit_sha": "a3f29c1",
  "captured_at": "2026-04-15",
  "dataset_version": "v1.2.0",
  "ragas": {
    "faithfulness": 0.91,
    "answer_relevancy": 0.87,
    "context_precision": 0.82,
    "context_recall": 0.85
  },
  "retrieval": {
    "precision_at_5": 0.78,
    "recall_at_5": 0.81,
    "mrr": 0.74
  }
}

When to update the baseline:

1. After an intentional improvement that raises the metrics → the new baseline = the new release. Document it in git: "baseline updated post-rerank improvement, faithfulness 0.87 → 0.91"
2. After a dataset change → you can't compare against the old baseline. Create a new baseline.
3. NEVER "so the test passes". If a change degrades things and you update the baseline, you've lost the system.

# scripts/update_baseline.py
def update_baseline(latest_report_path: Path, baseline_path: Path, justification: str):
    latest = json.loads(latest_report_path.read_text())
    new_baseline = {
        "version": latest["version"],
        "commit_sha": latest["commit_sha"],
        "captured_at": datetime.now().isoformat(),
        "dataset_version": latest["dataset_version"],
        "justification": justification,
        "ragas": latest["ragas"],
        "retrieval": latest["retrieval"],
    }
    baseline_path.write_text(json.dumps(new_baseline, indent=2))
    print(f"Baseline updated. Justification: {justification}")

The justification field is mandatory and must be present in the commit that updates the baseline. Without it, you get mysterious updates nobody can defend later.

Actionable error messages

A test that fails with "AssertionError: Quality gate failed" is useless. The developer doesn't know what to do. Compare:

A bad message:

AssertionError: Quality gate failed

An actionable message:

❌ Quality gate FAILED

Blocking failures (must fix to merge):
  • faithfulness: 0.78 (minimum 0.85, deficit -0.07)
    → Likely cause: prompt change accepted ungrounded inferences
    → Action: review changes to app/pipeline.py prompt template
  
Regression vs baseline v1.4.0:
  • context_precision: 0.71 (baseline 0.82, delta -0.11, tolerance ±0.04)
    → Likely cause: re-ranker change degraded ranking quality
    → Action: compare reranker config in eval_reports/

Warnings (non-blocking):
  • answer_relevancy: 0.79 (minimum 0.80, deficit -0.01)

def format_failure_report(absolute: list[dict], regression: list[dict]) -> str:
    if not absolute and not regression:
        return "✅ Quality gate PASSED"

    lines = ["❌ Quality gate FAILED", ""]

    blocking = [f for f in absolute if f["severity"] == "blocking"]
    warnings = [f for f in absolute if f["severity"] == "warning"]

    if blocking:
        lines.append("Blocking failures (must fix to merge):")
        for f in blocking:
            lines.append(f"  • {f['metric']}: {f['actual']:.3f} (minimum {f['minimum']:.3f}, deficit {f['delta']:.3f})")
        lines.append("")

    if regression:
        lines.append("Regression vs baseline:")
        for r in regression:
            lines.append(f"  • {r['metric']}: {r['current']:.3f} (baseline {r['baseline']:.3f}, delta {r['delta']:.3f}, tolerance ±{r['tolerance']:.3f})")
        lines.append("")

    if warnings:
        lines.append("Warnings (non-blocking):")
        for w in warnings:
            lines.append(f"  • {w['metric']}: {w['actual']:.3f} (minimum {w['minimum']:.3f}, deficit {w['delta']:.3f})")

    return "\n".join(lines)

Comparison: evaluating without gates vs with gates

Connection with the final project

Your Advanced RAG System must deliver:

eval/
├── thresholds.py              # DEFAULT_THRESHOLDS
├── baseline.json              # versioned in git
├── BASELINE_HISTORY.md        # a log of the updates, with justifications
└── tests/
    └── test_quality_gates.py  # a pytest that fails if the quality gate doesn't pass

The integrated test:

# tests/test_quality_gates.py
import json
import subprocess
from pathlib import Path

def test_quality_gates_pass():
    """Run smoke evaluation and assert all gates pass"""
    result = subprocess.run(
        ["python", "scripts/evaluate.py", "--mode", "smoke", "--enforce-thresholds"],
        capture_output=True, text=True,
    )
    assert result.returncode == 0, f"Quality gate failed:\n{result.stdout}\n{result.stderr}"

This test runs on every PR. If it fails, no merge.

Troubleshooting

Problem 1: "The tests fail too often and the devs ignore them"

The cause: unrealistically high thresholds, or a tolerance that's too strict.
The fix: measure the empirical noise (the "Handling tolerance" section). Adjust the thresholds and the tolerance to 2× stdev. If they're still failing after a week, calibrate the system, not the thresholds.

Problem 2: "We're not catching real degradations"

The cause: thresholds that are too low, or a tolerance that's too permissive.
The fix: review the last 3 months of reports. Identify the run with the worst quality that still passed. Raise the threshold slightly above that point.

Problem 3: "A legitimate architecture change breaks the regression test"

The cause: the baseline corresponds to the old system, and the metrics changed for a good reason.
The fix: update the baseline with a documented justification. If the metric dropped but for a good reason (more coverage of hard queries in the new dataset), document it and move on.

Problem 4: "The test passes locally and fails in CI"

The cause: a different seed or different environment variables.
The fix: set OPENAI_SEED=42 in CI. Make sure the RAGAS version is pinned. Verify the dataset_version is the same.

Problem 5: "Too many warnings, noise on every PR"

The cause: too many metrics as warnings, none of them actionable.
The fix: consolidate the warnings. Ideally 1-2 metrics as warnings, maximum. If you have 6 warnings, nobody reads them.

Exercises

Exercise 1: Computing empirical thresholds

Given a set of historical runs, compute the thresholds using a 95% z-score.

import statistics

def calculate_empirical_thresholds(runs: list[dict], z: float = 1.96) -> dict:
    if len(runs) < 5:
        raise ValueError("Need at least 5 runs for stable thresholds")
    thresholds = {}
    for category in ["ragas", "retrieval"]:
        for metric in runs[0][category].keys():
            values = [r[category][metric] for r in runs]
            mean = statistics.mean(values)
            stdev = statistics.stdev(values)
            thresholds[f"{category}.{metric}"] = round(mean - z * stdev, 3)
    return thresholds

# Usage:
historical = [json.loads(p.read_text()) for p in Path("eval_reports").glob("*.json")]
thresholds = calculate_empirical_thresholds(historical[-10:])
print(thresholds)

Exercise 2: Implementing a regression check with per-metric tolerance

Every metric gets a different tolerance based on its inherent variability.

TOLERANCES = {
    "faithfulness": 0.02,
    "answer_relevancy": 0.03,
    "context_precision": 0.04,
    "context_recall": 0.02,
    "precision_at_5": 0.03,
    "recall_at_5": 0.03,
    "mrr": 0.04,
}

def check_no_regression_per_metric(current: dict, baseline: dict, tolerances: dict = TOLERANCES) -> list[dict]:
    failures = []
    for category in ["ragas", "retrieval"]:
        for metric, base_value in baseline.get(category, {}).items():
            curr_value = current.get(category, {}).get(metric, 0)
            tol = tolerances.get(metric, 0.02)
            if curr_value < base_value - tol:
                failures.append({
                    "metric": metric,
                    "current": curr_value,
                    "baseline": base_value,
                    "tolerance": tol,
                    "deficit": base_value - tol - curr_value,
                })
    return failures

Exercise 3: An error message with suggestions

Implement an error format that suggests the likely cause based on the metric that failed.

SUGGESTIONS = {
    "faithfulness": [
        "Review prompt: enforce 'answer only from context'",
        "Check if model was downgraded (gpt-4o → gpt-4o-mini?)",
        "Verify context is not being truncated by token limit",
    ],
    "answer_relevancy": [
        "Check query understanding (M03 query expansion)",
        "Review prompt: be more directive about answer format",
    ],
    "context_precision": [
        "Re-ranker may be misconfigured (M04)",
        "Verify top_k after rerank is reasonable",
    ],
    "context_recall": [
        "Retriever missing relevant docs (chunking, embeddings)",
        "Hybrid search may be needed (M05)",
        "Check metadata filters not over-restrictive (M06)",
    ],
}

def format_failure_with_suggestions(failure: dict) -> str:
    suggestions = SUGGESTIONS.get(failure["metric"], [])
    lines = [
        f"  ❌ {failure['metric']}: {failure.get('current', failure.get('actual')):.3f}",
        f"     Possible causes:",
    ]
    for s in suggestions:
        lines.append(f"       • {s}")
    return "\n".join(lines)

Exercise 4: A baseline update workflow

Implement a CLI to update the baseline only if the metrics improved, and with a mandatory justification.

import argparse
import json
from datetime import datetime
from pathlib import Path

def update_baseline_safely(report_path: Path, baseline_path: Path, justification: str) -> bool:
    if not justification or len(justification) < 20:
        raise ValueError("Justification required (min 20 chars)")

    new_report = json.loads(report_path.read_text())
    if not baseline_path.exists():
        baseline_path.write_text(json.dumps({**new_report, "justification": justification}, indent=2))
        return True

    old_baseline = json.loads(baseline_path.read_text())
    degraded = []
    for category in ["ragas", "retrieval"]:
        for metric, old_val in old_baseline[category].items():
            new_val = new_report[category].get(metric, 0)
            if new_val < old_val - 0.01:
                degraded.append(f"{metric}: {old_val:.3f} → {new_val:.3f}")

    if degraded and "FORCE" not in justification:
        print("Refusing to update: metrics degraded:")
        for d in degraded:
            print(f"  • {d}")
        print("To force, prefix justification with 'FORCE: <reason>'")
        return False

    new_baseline = {
        **new_report,
        "previous_baseline_commit": old_baseline.get("commit_sha"),
        "captured_at": datetime.now().isoformat(),
        "justification": justification,
    }
    baseline_path.write_text(json.dumps(new_baseline, indent=2))
    return True

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--report", required=True)
    parser.add_argument("--baseline", default="eval/baseline.json")
    parser.add_argument("--justification", required=True)
    args = parser.parse_args()
    update_baseline_safely(Path(args.report), Path(args.baseline), args.justification)

Summary

• An absolute threshold + a regression check: two complementary mechanisms, and you need both
• Calibrate the thresholds empirically (a z-score over historical runs), by business requirement, or against a human
• The tolerance must be ≥2× the metric's stdev to avoid false positives
• A baseline versioned in git, with a mandatory justification when you update it
• Severity (blocking vs warning) modulates which failure blocks the PR vs merely alerts
• Actionable error messages with per-metric suggestions speed up debugging
• Quality gates turn subjective conversations into explicit decisions

Additional resources

1. Pytest Assertions - The test pattern for Python.
2. Continuous Integration - Martin Fowler - CI as a quality control.
3. Statistical Process Control - The origin of z-scores in thresholds.
4. ML Model Monitoring - Patterns for detecting degradation.
5. OpenAI Evals - An alternative framework with quality gates.

Created: March 13, 2026
Version: 2.0