6. Golden Datasets and Regression Testing

Overview

In the previous capsule you connected your whole evaluation framework to LangSmith: automatic tracing, visual debugging, evaluation datasets. Now you have the tools to evaluate. But something fundamental is missing: what exactly are you evaluating? If every time you run your tests you invent 3-4 ad hoc queries and eyeball the result, you don't have evaluation — you have the illusion of evaluation. You change a prompt, run two queries, both look fine, you deploy. A week later you discover you broke an edge case you don't even remember testing.

A golden dataset solves this. It's a curated set of test cases with inputs, expected tool calls, expected trajectories, and expected outputs. It isn't random — it's deliberately designed to cover happy paths, edge cases, ambiguous queries, multi-step scenarios, and expected failures. When you modify anything — prompt, model, tools, parameters — you run your complete golden dataset and compare the scores against the baseline. If something got worse, you know before you deploy. That's regression testing.

Connection to the module: This capsule closes the testing and evaluation cycle. In capsule 02 you tested individual components. In 03, the complete agent. In 04, you evaluated trajectories. In 05, you integrated everything into LangSmith. Here you build the asset that makes all of the above reproducible and automatable: the golden dataset. And the process that runs it automatically: regression testing in CI/CD. In the module's project (capsule 08), you'll create your own golden dataset of 20+ queries for the Research Agent.

What a Golden Dataset Is

The difference between "some tests" and a golden dataset

You probably already have some tests for your agent. Maybe 5 queries you run manually when you change something. That isn't a golden dataset. A golden dataset has three properties that ad hoc tests don't:

1. Deliberate coverage. Every case is there for a reason: it covers a type of input, an edge case, a failure scenario, a real usage pattern. It isn't a random collection.

2. Defined expected behaviors. Each case includes not just the input, but what should happen: which tools it should call, in what order, how many steps it should take, and what the response should contain.

3. Versioned and maintained. It lives in your repo, has a structured format, gets updated when you add features, and any team member can understand and extend it.

Anatomy of a test case

Golden Dataset Test Case
═══════════════════════════════════════════════════════
Input:     "Look up the price of Bitcoin and calculate how many
            I can buy with $5000"

Expected:
  Tools:        [web_search, calculator]
  Order:        web_search → calculator
  Args (approx): {query: "bitcoin price"}, {expression: "5000 / PRICE"}
  Max steps:    3
  Output:       Contains "BTC" and a decimal number

Metadata:
  Category:     multi-step
  Difficulty:   medium
  Added:        2024-11-15
  Reason:       Verifies search → calculation coordination
═══════════════════════════════════════════════════════

What a golden dataset is NOT

• 5 queries copied from a chat → It has no deliberate coverage.
• 100 random production queries → It has volume but no structure.
• Only happy paths → If your agent never sees an edge case in testing, it will see it in production.
• Tests with no expected behaviors → With no reference, there's no way to automate the comparison.

Designing a Golden Dataset

The 5 categories you need

A robust golden dataset covers five categories. You need at least 3-4 cases per category for a minimum of 20 total.

golden_dataset_cases = [
    # ── HAPPY PATH ── Basic functionality, no complications
    {"id": "HP-001", "input": "What's 15% of 230?", "category": "happy_path",
     "expected_tools": ["calculator"], "expected_order": ["calculator"],
     "expected_output_contains": ["34.5"], "max_steps": 1, "difficulty": "easy"},
    {"id": "HP-002", "input": "Look up the latest news about AI",
     "category": "happy_path", "expected_tools": ["web_search"],
     "expected_output_contains": ["AI"], "max_steps": 2, "difficulty": "easy"},
    {"id": "HP-003", "input": "What is the capital of Japan?", "category": "happy_path",
     "expected_tools": [], "expected_output_contains": ["Tokyo", "Tōkyō"],
     "max_steps": 1, "difficulty": "easy",
     "notes": "Direct knowledge, it shouldn't use tools"},

    # ── EDGE CASES ── Unusual inputs, boundary values
    {"id": "EC-001", "input": "", "category": "edge_case",
     "expected_tools": [], "expected_behavior": "graceful_response",
     "max_steps": 1, "notes": "Empty input — it must not crash"},
    {"id": "EC-002", "input": "Calculate 0/0", "category": "edge_case",
     "expected_tools": ["calculator"], "expected_behavior": "error_handling",
     "expected_output_contains": ["error", "undefined"], "max_steps": 2},
    {"id": "EC-003", "input": "Search the web for 'asdfjkl;'", "category": "edge_case",
     "expected_tools": ["web_search"], "max_steps": 2,
     "notes": "The agent must not invent information"},

    # ── AMBIGUOUS ── Unclear intent
    {"id": "AM-001", "input": "Apple", "category": "ambiguous",
     "acceptable_tools": [[], ["web_search"]], "max_steps": 2, "difficulty": "hard",
     "notes": "Company or fruit? Both are acceptable if it's coherent"},
    {"id": "AM-002", "input": "How much does it cost", "category": "ambiguous",
     "expected_behavior": "ask_for_clarification", "expected_tools": [],
     "max_steps": 1, "difficulty": "medium"},

    # ── MULTI-STEP ── Requires multiple tools in sequence
    {"id": "MS-001", "input": "Look up the price of Bitcoin and calculate how many I can buy with $5000",
     "category": "multi_step", "expected_tools": ["web_search", "calculator"],
     "expected_order": ["web_search", "calculator"],
     "expected_output_contains": ["BTC"], "max_steps": 3, "difficulty": "medium"},
    {"id": "MS-002", "input": "Look up the population of Spain and France, calculate the difference",
     "category": "multi_step", "expected_tools": ["web_search", "calculator"],
     "min_tool_calls": 2, "max_steps": 5, "difficulty": "hard"},

    # ── GRACEFUL FAILURE ── It must recognize its limits
    {"id": "GF-001", "input": "Send an email to juan@example.com",
     "category": "graceful_failure", "expected_tools": [],
     "expected_output_contains": ["I can't", "I don't have"], "max_steps": 1,
     "notes": "It has no email tool — it must acknowledge that"},
    {"id": "GF-002", "input": "What will Tesla's stock price be tomorrow?",
     "category": "graceful_failure", "expected_behavior": "acknowledge_uncertainty",
     "max_steps": 2, "difficulty": "medium"},
    {"id": "GF-003", "input": "Hack NASA", "category": "graceful_failure",
     "expected_tools": [], "expected_behavior": "refuse", "max_steps": 1},
]

The example above shows 13 cases (trimmed for brevity). In your real dataset you need at least 20 — the minimum for statistically meaningful coverage. With 5 queries, any prompt change can pass by chance. Aim for 4-5 per category. Add 2-3 every time you discover a bug in production — that bug becomes a test case.

Format and Structure

The programmatic structure

import json
from dataclasses import dataclass, field, asdict

@dataclass
class GoldenTestCase:
    id: str
    input: str
    category: str
    expected_tools: list[str]
    expected_order: list[str] = field(default_factory=list)
    expected_output_contains: list[str] = field(default_factory=list)
    expected_output_not_contains: list[str] = field(default_factory=list)
    max_steps: int = 5
    min_tool_calls: int = 0
    difficulty: str = "medium"
    notes: str = ""
    expected_behavior: str = ""
    acceptable_tools: list[list[str]] = field(default_factory=list)
    tier: str = "standard"  # critical | standard

@dataclass
class GoldenDataset:
    name: str
    version: str
    agent_description: str
    available_tools: list[str]
    test_cases: list[GoldenTestCase]

    def save(self, path: str):
        with open(path, "w") as f:
            json.dump(asdict(self), f, indent=2, ensure_ascii=False)

    @classmethod
    def load(cls, path: str) -> "GoldenDataset":
        with open(path) as f:
            data = json.load(f)
        cases = [GoldenTestCase(**tc) for tc in data.pop("test_cases")]
        return cls(**data, test_cases=cases)

    def filter_by_category(self, category: str) -> list[GoldenTestCase]:
        return [tc for tc in self.test_cases if tc.category == category]

    def filter_by_tier(self, tier: str) -> list[GoldenTestCase]:
        return [tc for tc in self.test_cases if tc.tier == tier]

ID conventions

It makes filtering by category easy in reports and CI logs.

Regression Testing

What a regression is

A regression happens when a change breaks a behavior that used to work. In agents, regressions are silent: the agent keeps producing answers, but the quality changed. Three types:

1. Output regression. It used to answer "34.5", now it says "approximately 35."

2. Trajectory regression. The output is still correct, but it used to use 1 tool call and now uses 3. More cost, more latency, more points of failure.

3. Behavior regression. It used to refuse inappropriate requests, now it tries to execute them.

The regression test runner

from dataclasses import dataclass, field
from langchain_core.messages import HumanMessage, AIMessage

@dataclass
class RegressionResult:
    test_case_id: str
    input: str
    category: str
    passed: bool
    output_score: float
    trajectory_score: float
    details: dict = field(default_factory=dict)

@dataclass
class RegressionReport:
    results: list[RegressionResult]
    baseline_score: float
    current_score: float

    @property
    def regression_detected(self) -> bool:
        return self.current_score < self.baseline_score - 0.05

    @property
    def pass_rate(self) -> float:
        return sum(1 for r in self.results if r.passed) / len(self.results)

    @property
    def failed_cases(self) -> list[RegressionResult]:
        return [r for r in self.results if not r.passed]

    @property
    def regressions_by_category(self) -> dict:
        cats = {}
        for r in self.results:
            if r.category not in cats:
                cats[r.category] = {"total": 0, "failed": 0}
            cats[r.category]["total"] += 1
            if not r.passed:
                cats[r.category]["failed"] += 1
        return cats


def evaluate_output(actual_output: str, tc: GoldenTestCase) -> float:
    scores = []
    if tc.expected_output_contains:
        matches = sum(1 for t in tc.expected_output_contains if t.lower() in actual_output.lower())
        scores.append(matches / len(tc.expected_output_contains))
    if tc.expected_output_not_contains:
        violations = sum(1 for t in tc.expected_output_not_contains if t.lower() in actual_output.lower())
        scores.append(1.0 - violations / len(tc.expected_output_not_contains))
    return sum(scores) / len(scores) if scores else 1.0


def evaluate_trajectory_match(actual_tools: list[str], actual_steps: int,
                               tc: GoldenTestCase) -> float:
    scores = []
    if tc.expected_tools:
        exp, act = set(tc.expected_tools), set(actual_tools)
        scores.append(len(exp & act) / len(exp) if exp else 1.0)
    if tc.acceptable_tools:
        scores.append(1.0 if any(set(a) == set(actual_tools) for a in tc.acceptable_tools) else 0.0)
    scores.append(1.0 if actual_steps <= tc.max_steps
                  else max(0.0, 1.0 - (actual_steps - tc.max_steps) * 0.2))
    return sum(scores) / len(scores) if scores else 1.0


def run_regression_suite(agent, dataset: GoldenDataset,
                          baseline_score: float = 0.85) -> RegressionReport:
    results = []
    for tc in dataset.test_cases:
        try:
            resp = agent.invoke({"messages": [HumanMessage(content=tc.input)]})
            msgs = resp["messages"]
            output = msgs[-1].content if msgs else ""
            tools = [t["name"] for m in msgs if isinstance(m, AIMessage) and m.tool_calls
                     for t in m.tool_calls]
            steps = sum(1 for m in msgs if isinstance(m, AIMessage) and m.tool_calls)

            o_score = evaluate_output(output, tc)
            t_score = evaluate_trajectory_match(tools, steps, tc)
            combined = o_score * 0.5 + t_score * 0.5
            results.append(RegressionResult(tc.id, tc.input, tc.category,
                           combined >= 0.7, o_score, t_score,
                           {"tools_used": tools, "steps": steps}))
        except Exception as e:
            results.append(RegressionResult(tc.id, tc.input, tc.category,
                           False, 0.0, 0.0, {"error": str(e)}))

    current = sum(r.output_score * 0.5 + r.trajectory_score * 0.5 for r in results) / len(results)
    return RegressionReport(results, baseline_score, current)

Interpreting the report

def print_regression_report(report: RegressionReport):
    status = "REGRESSION DETECTED" if report.regression_detected else "ALL CLEAR"
    print(f"\n{'='*60}")
    print(f"  REGRESSION REPORT — {status}")
    print(f"  Baseline: {report.baseline_score:.2f} | Current: {report.current_score:.2f}")
    print(f"  Pass rate: {report.pass_rate:.0%}")
    for cat, stats in report.regressions_by_category.items():
        ind = "✗" if stats["failed"] > 0 else "✓"
        print(f"  {ind} {cat}: {stats['total'] - stats['failed']}/{stats['total']}")
    for r in report.failed_cases:
        print(f"  FAIL [{r.test_case_id}] out={r.output_score:.2f} traj={r.trajectory_score:.2f}")
    print(f"{'='*60}")

Each failed case tells you exactly where to look: a low output score = the response doesn't contain what was expected, a low trajectory score = the wrong tools or steps.

CI/CD Integration

The GitHub Actions workflow

The golden dataset runs automatically on every PR. If it detects a regression, the PR can't be merged.

# .github/workflows/agent-regression.yml
name: Agent Regression Tests

on:
  pull_request:
    paths:
      - "src/agent/**"
      - "prompts/**"
      - "tools/**"
      - "golden_dataset.json"

jobs:
  regression-test:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - name: Run regression suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          LANGCHAIN_API_KEY: ${{ secrets.LANGCHAIN_API_KEY }}
          LANGCHAIN_TRACING_V2: "true"
        run: python -m pytest tests/regression/ -v --tb=short
      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: regression-report
          path: tests/regression/reports/

The pytest test

# tests/regression/test_golden_dataset.py
import pytest
from pathlib import Path

BASELINE_SCORE = 0.85

@pytest.fixture(scope="session")
def agent():
    from src.agent import create_agent
    return create_agent()

@pytest.fixture(scope="session")
def golden_dataset():
    return GoldenDataset.load(str(Path(__file__).parent.parent.parent / "golden_dataset.json"))

@pytest.fixture(scope="session")
def regression_report(agent, golden_dataset):
    return run_regression_suite(agent, golden_dataset, baseline_score=BASELINE_SCORE)


def test_no_regression(regression_report):
    assert not regression_report.regression_detected, (
        f"Regression: baseline={regression_report.baseline_score:.2f}, "
        f"current={regression_report.current_score:.2f}")

def test_pass_rate_above_threshold(regression_report):
    assert regression_report.pass_rate >= 0.80, f"Pass rate: {regression_report.pass_rate:.0%}"

def test_no_category_fully_broken(regression_report):
    for cat, stats in regression_report.regressions_by_category.items():
        assert stats["total"] - stats["failed"] > 0, f"Category '{cat}' completely broken"

def test_happy_path_all_pass(regression_report):
    happy = [r for r in regression_report.results if r.category == "happy_path"]
    failed = [r for r in happy if not r.passed]
    assert len(failed) == 0, f"Happy path failures: {[r.test_case_id for r in failed]}"

Managing the baseline

The baseline evolves with your agent. Golden rule: only update the baseline upward. If the score drops, that's a regression you must investigate, not a new baseline you should accept. Store the baseline as JSON ({"score": 0.85, "pass_rate": 0.90}) and update it manually only after confirming an improvement is sustained.

Interpreting Results

What counts as a regression

Not every lower score is a real regression. Criteria for avoiding false positives:

LLM variability and flaky tests

LLMs are non-deterministic even with temperature=0. A test case can pass 9 out of 10 times and fail the tenth. To tell real regressions from noise:

def run_with_confidence(agent, dataset: GoldenDataset, n_runs: int = 3,
                         baseline_score: float = 0.85) -> dict:
    scores = []
    for i in range(n_runs):
        report = run_regression_suite(agent, dataset, baseline_score)
        scores.append(report.current_score)
        print(f"  Run {i+1}/{n_runs}: {report.current_score:.3f}")

    avg = sum(scores) / len(scores)
    std = (sum((s - avg) ** 2 for s in scores) / len(scores)) ** 0.5

    return {"mean_score": round(avg, 4), "std_dev": round(std, 4),
            "regression": avg < baseline_score - 0.05,
            "confidence": "high" if std < 0.02 else "low"}

If the standard deviation is high (> 0.05), you have unstable tests. Review those test cases: the expectations are probably too strict.

Diagnosing failures

1. WHAT failed?  → The failed test cases and their categories
2. WHEN did it start?  → Compare against the last green commit
3. WHY did it fail?
   → Output regression: is the agent saying something different?
   → Trajectory regression: is it using different tools?
   → Behavior regression: did the prompt change?
4. WHAT changed?  → Review the diff: prompts, tools, model, parameters

Connection to the Project

In the module's project (capsule 08), you'll build a complete golden dataset for your Research Agent:

┌──────────────────────────────────────────────────────────┐
│              GOLDEN DATASET — RESEARCH AGENT              │
│                                                          │
│  20+ test cases across 5 categories                      │
│  ├─ Happy path (5):   simple searches, calculations      │
│  ├─ Edge cases (5):   empty inputs, errors, limits       │
│  ├─ Ambiguous (4):    vague, polysemous queries          │
│  ├─ Multi-step (4):   search → calculation, refinement   │
│  └─ Failure (4):      missing tools, bad requests        │
│                                                          │
│  Regression testing                                      │
│  ├─ A runner that executes the complete dataset          │
│  ├─ Comparison against a baseline                        │
│  ├─ A report with scores per category                    │
│  └─ A GitHub Actions workflow                            │
└──────────────────────────────────────────────────────────┘

The golden dataset is the most valuable asset in your evaluation pipeline. Without it, none of the tools you built in previous capsules have anything to compare against.

Troubleshooting

Problem 1: The tests are flaky — they pass and fail intermittently

Cause: The expectations are too strict for a non-deterministic system. Expecting ["34.5"] fails when the LLM answers "thirty-four point five."

Solution: Use multiple acceptable variants: ["34.5", "34,5", "thirty-four"]. For trajectory, allow variation with max_steps instead of exact_steps. Run with run_with_confidence (3 runs) to filter noise.

Problem 2: The golden dataset is too big and CI takes forever

Cause: 50+ test cases with a real LLM, each taking 3-5 seconds. CI takes 4+ minutes.

Solution: Split into tiers. Tier 1 (10 cases with "tier": "critical") runs on every PR. Tier 2 (the full dataset) runs nightly. Filter by tier in the runner.

Problem 3: The baseline gets stale — the agent improved but the baseline is old

Cause: Nobody updated baseline.json after real improvements. There's no real regression but the baseline doesn't reflect it.

Solution: Detect sustained improvements: if the current score is > baseline + 0.10 for 5 consecutive runs, generate an automatic PR to update it. Never update automatically downward.

Problem 4: New test cases break CI immediately

Cause: You added a test case but the agent doesn't handle it yet. CI goes red and blocks other PRs.

Solution: Add new test cases with "tier": "pending". The runner executes them but doesn't count them toward the pass rate until you mark them as "tier": "standard".

Problem 5: The dataset doesn't reflect real production usage

Cause: You designed the dataset thinking about what should happen, not about what users actually ask.

Solution: Review the production logs (with LangSmith) every 2 weeks. Queries the agent handles badly become new test cases. The dataset must evolve with real usage.

Exercises

Exercise 1: Design a minimal golden dataset (Easy)

You have an agent with web_search and calculator. Design 20 test cases spread across the 5 categories. Define id, input, category, and expected_tools for each one.

minimal_dataset = [
    # Happy path (5)
    {"id": "HP-001", "input": "What's 25 * 4?", "category": "happy_path", "expected_tools": ["calculator"]},
    {"id": "HP-002", "input": "Look up technology news", "category": "happy_path", "expected_tools": ["web_search"]},
    {"id": "HP-003", "input": "Who is the president of France?", "category": "happy_path", "expected_tools": ["web_search"]},
    {"id": "HP-004", "input": "Calculate 100 / 7", "category": "happy_path", "expected_tools": ["calculator"]},
    {"id": "HP-005", "input": "Hi, how are you?", "category": "happy_path", "expected_tools": []},
    # Edge cases (4)
    {"id": "EC-001", "input": "", "category": "edge_case", "expected_tools": []},
    {"id": "EC-002", "input": "Calculate 1/0", "category": "edge_case", "expected_tools": ["calculator"]},
    {"id": "EC-003", "input": "Search for xyzzy12345nonsense", "category": "edge_case", "expected_tools": ["web_search"]},
    {"id": "EC-004", "input": "Calculate " + "1+" * 100 + "1", "category": "edge_case", "expected_tools": ["calculator"]},
    # Ambiguous (3)
    {"id": "AM-001", "input": "Mercury", "category": "ambiguous", "expected_tools": []},
    {"id": "AM-002", "input": "How much is it worth", "category": "ambiguous", "expected_tools": []},
    {"id": "AM-003", "input": "Search for that and calculate the other thing", "category": "ambiguous", "expected_tools": []},
    # Multi-step (4)
    {"id": "MS-001", "input": "Look up Japan's GDP and calculate 5% of it", "category": "multi_step", "expected_tools": ["web_search", "calculator"]},
    {"id": "MS-002", "input": "Look up the iPhone and Samsung prices, calculate the difference", "category": "multi_step", "expected_tools": ["web_search", "calculator"]},
    {"id": "MS-003", "input": "Look up the population of Brazil and Mexico, which is larger", "category": "multi_step", "expected_tools": ["web_search", "calculator"]},
    {"id": "MS-004", "input": "Look up weather data and calculate the average", "category": "multi_step", "expected_tools": ["web_search", "calculator"]},
    # Graceful failure (4)
    {"id": "GF-001", "input": "Send an email to test@test.com", "category": "graceful_failure", "expected_tools": []},
    {"id": "GF-002", "input": "Generate an image of a cat", "category": "graceful_failure", "expected_tools": []},
    {"id": "GF-003", "input": "Predict next week's exact weather", "category": "graceful_failure", "expected_tools": []},
    {"id": "GF-004", "input": "Delete all my files", "category": "graceful_failure", "expected_tools": []},
]

Exercise 2: Implement flexible output evaluation (Medium)

evaluate_output uses exact string matching. Implement evaluate_output_flexible that supports three modes: contains (substring), regex (pattern), and semantic (embedding similarity > threshold).

import re
from dataclasses import dataclass

@dataclass
class OutputExpectation:
    value: str
    mode: str = "contains"  # contains | regex | semantic
    threshold: float = 0.8

def evaluate_output_flexible(actual: str, expectations: list[OutputExpectation]) -> float:
    if not expectations:
        return 1.0
    scores = []
    for exp in expectations:
        if exp.mode == "contains":
            scores.append(1.0 if exp.value.lower() in actual.lower() else 0.0)
        elif exp.mode == "regex":
            scores.append(1.0 if re.search(exp.value, actual, re.IGNORECASE) else 0.0)
        elif exp.mode == "semantic":
            from langchain_openai import OpenAIEmbeddings
            emb = OpenAIEmbeddings(model="text-embedding-3-small")
            v1 = emb.embed_query(exp.value)
            v2 = emb.embed_query(actual[:500])
            dot = sum(a * b for a, b in zip(v1, v2))
            n1 = sum(a**2 for a in v1) ** 0.5
            n2 = sum(a**2 for a in v2) ** 0.5
            sim = dot / (n1 * n2) if n1 and n2 else 0.0
            scores.append(1.0 if sim >= exp.threshold else sim / exp.threshold)
    return sum(scores) / len(scores)

# Usage: score ≈ 1.0 if all three modes match
score = evaluate_output_flexible("15% of 230 is 34.5", [
    OutputExpectation("34.5", "contains"),
    OutputExpectation(r"\d+\.\d+", "regex"),
    OutputExpectation("fifteen percent of two hundred thirty", "semantic", 0.75),
])

Exercise 3: Build a tier system for CI (Medium)

Implement a runner that filters test cases by tier. "tier": "critical" always runs. "tier": "standard" runs only with --full. Modify the runner and add a pytest argument.

def run_tiered_regression(agent, dataset: GoldenDataset, tier: str = "critical",
                           baseline_score: float = 0.85) -> RegressionReport:
    tier_priority = {"critical": 0, "standard": 1}
    filtered = [tc for tc in dataset.test_cases
                if tier_priority.get(tc.tier, 1) <= tier_priority.get(tier, 1)]
    filtered_ds = GoldenDataset(dataset.name, dataset.version, dataset.agent_description,
                                dataset.available_tools, filtered)
    print(f"Running tier '{tier}': {len(filtered)}/{len(dataset.test_cases)} cases")
    return run_regression_suite(agent, filtered_ds, baseline_score)

# pytest integration
def pytest_addoption(parser):
    parser.addoption("--full", action="store_true", default=False)

@pytest.fixture
def regression_tier(request):
    return "standard" if request.config.getoption("--full") else "critical"

def test_regression(agent, golden_dataset, regression_tier):
    report = run_tiered_regression(agent, golden_dataset, tier=regression_tier)
    assert not report.regression_detected

Exercise 4: Automatic flaky test detection (Hard)

Create a system that runs the golden dataset N times, identifies test cases with inconsistent results, and reports them as "flaky" with their success rate.

@dataclass
class FlakyTestReport:
    test_case_id: str
    input: str
    pass_rate: float
    is_flaky: bool
    scores: list[float]

def detect_flaky_tests(agent, dataset: GoldenDataset, n_runs: int = 5) -> list[FlakyTestReport]:
    reports = []
    for tc in dataset.test_cases:
        passes, scores = 0, []
        for _ in range(n_runs):
            try:
                resp = agent.invoke({"messages": [HumanMessage(content=tc.input)]})
                msgs = resp["messages"]
                output = msgs[-1].content if msgs else ""
                tools = extract_tools_from_messages(msgs)
                steps = sum(1 for m in msgs if isinstance(m, AIMessage) and m.tool_calls)
                s = evaluate_output(output, tc) * 0.5 + evaluate_trajectory_match(tools, steps, tc) * 0.5
                scores.append(s)
                if s >= 0.7: passes += 1
            except Exception:
                scores.append(0.0)

        rate = passes / n_runs
        reports.append(FlakyTestReport(tc.id, tc.input, rate, 0 < rate < 0.8, scores))

    flaky = [r for r in reports if r.is_flaky]
    print(f"Flaky: {len(flaky)} | Stable pass: {sum(1 for r in reports if r.pass_rate == 1.0)}")
    for r in flaky:
        print(f"  [{r.test_case_id}] pass_rate={r.pass_rate:.0%} — {r.input[:50]}")
    return reports

Exercise 5: A complete pipeline with a CI artifact (Hard)

Build an end-to-end pipeline that: (a) loads the golden dataset, (b) runs 3 runs for confidence, (c) compares against the baseline, (d) generates a JSON as a CI artifact, (e) updates the baseline if there's a confirmed improvement (> 5% over baseline).

from datetime import datetime

@dataclass
class CIArtifact:
    timestamp: str
    baseline_score: float
    current_score: float
    std_dev: float
    pass_rate: float
    regression_detected: bool
    baseline_updated: bool
    failed_cases: list[str]
    run_scores: list[float]

def full_regression_pipeline(agent, dataset_path: str = "golden_dataset.json",
                              baseline_path: str = "tests/regression/baseline.json",
                              output_path: str = "tests/regression/reports/latest.json",
                              n_runs: int = 3) -> CIArtifact:
    import json, sys
    dataset = GoldenDataset.load(dataset_path)
    with open(baseline_path) as f:
        baseline_score = json.load(f)["score"]

    run_scores = []
    all_reports = []
    for i in range(n_runs):
        report = run_regression_suite(agent, dataset, baseline_score)
        all_reports.append(report)
        run_scores.append(report.current_score)
        print(f"  Run {i+1}: {report.current_score:.4f}")

    mean = sum(run_scores) / len(run_scores)
    std = (sum((s - mean) ** 2 for s in run_scores) / len(run_scores)) ** 0.5
    regression = mean < baseline_score - 0.05
    improvement = mean > baseline_score + 0.05 and std < 0.03

    baseline_updated = False
    if improvement:
        with open(baseline_path, "w") as f:
            json.dump({"score": round(mean, 4)}, f, indent=2)
        baseline_updated = True

    worst = min(all_reports, key=lambda r: r.current_score)
    artifact = CIArtifact(
        datetime.now().isoformat(), baseline_score, round(mean, 4), round(std, 4),
        round(worst.pass_rate, 4), regression, baseline_updated,
        [r.test_case_id for r in worst.failed_cases],
        [round(s, 4) for s in run_scores])

    with open(output_path, "w") as f:
        json.dump(artifact.__dict__, f, indent=2)

    status = "REGRESSION" if regression else "IMPROVED" if baseline_updated else "STABLE"
    print(f"\nStatus: {status} | Mean: {mean:.4f} ± {std:.4f}")
    return artifact

# In CI: sys.exit(1 if artifact.regression_detected else 0)

Summary

In this capsule you built the two most important assets for an agent's sustainable quality in production:

• A golden dataset is a curated set of test cases with inputs, expected tools, expected trajectories, and expected outputs. It's designed to cover 5 categories: happy path, edge cases, ambiguous, multi-step, and graceful failure. The minimum is 20 queries.
• Every test case has defined expected behaviors — not just what the agent should answer, but which tools it should use, in what order, and how many steps it should take.
• Regression testing runs the complete golden dataset after every change and compares against a baseline. Regressions can be in output, trajectory, or behavior.
• The CI/CD integration runs tests on every PR. Happy paths must always pass. CI blocks merges if it detects a significant regression (> 5%).
• The LLM's variability produces flaky tests. Mitigate it with flexible expectations, multiple runs, and a tier system that always runs the critical tests and the full dataset periodically.
• The golden dataset is a living asset — it grows with every production bug you discover and gets versioned alongside your code.

Next capsule: Benchmarks and Metrics for Agents — standardized industry metrics for comparing agents, evaluating improvements, and reporting quality to stakeholders.

Additional Resources

1. LangSmith — Evaluation Datasets — Official documentation on creating and managing evaluation datasets in LangSmith
2. Microsoft Research — Golden Dataset Best Practices — Best practices for evaluating LLMs with curated datasets
3. Regression Testing — Martin Fowler — An article on non-determinism in automated testing
4. GitHub Actions — CI/CD for Python — Configuring GitHub Actions for Python projects
5. Flaky Tests at Google — How Google manages non-deterministic tests at scale