Mini-Project: A Regression Gate for Reservo

Description

Seven lessons built, separately, each piece: why a gate is needed and what boundary it respects (01), the three questions it can answer (02), the fixed CASE_SET and its isolation discipline (03), the result's form and the boundary with semantic quality (04), the literal tool choice (05), the cost and latency thresholds (06), and the complete assembly in run_case/run_regression_gate (07). This mini-project brings them all together into a single working directory — regression/harness.py and regression/golden_cases.json — and runs them over a realistic scenario end to end: Reservo's current state, PASS; a proposed prompt change that introduces a real regression, FAIL, caught before reaching production; the prompt fix, PASS again, ready to ship.

By the end of this lesson you're going to have a real regression_report.json, written to disk, with the complete verdict for the five cases — the artifact Module 7 is going to reuse, unmodified, to decide GO/NO-GO in a version comparison.

Connection to the module

This is the complete module's synthesis. There's no new piece of regression/harness.py — the mini-project reuses run_case, run_regression_gate, CaseResult, and GateReport exactly as they stood at the end of lesson 07, and adds a single new function, print_gate_summary, which builds a readable console report from a GateReport — the missing piece that makes the gate useful at a glance, not just programmatically.

Worked example: the complete flow, end to end

print_gate_summary: a readable report, with the exact reason for every FAIL

def print_gate_summary(report):
    """Un reporte legible en consola: PASS/FAIL del lote, y para cada caso
    roto, EXACTAMENTE cuál de los cuatro chequeos falló y con qué valores."""
    passed_n = sum(c.passed for c in report.cases)
    total_n = len(report.cases)
    print(f"=== GATE: {'PASS' if report.passed else 'FAIL'} ({passed_n}/{total_n}) ===")
    for c in report.cases:
        status = "PASS" if c.passed else "FAIL"
        line = f"  {c.name:38} {status}"
        if not c.passed:
            reasons = []
            if not c.tool_choice_ok:
                reasons.append(f"tool_choice(esperaba distinto, obtuvo {c.actual_tools})")
            if c.schema_errors:
                reasons.append(f"schema{c.schema_errors}")
            if c.output_errors:
                reasons.append(f"output{c.output_errors}")
            if not c.cost_ok:
                reasons.append(f"cost({c.cost_cents}c)")
            if not c.latency_ok:
                reasons.append(f"latency({c.latency_ms}ms)")
            line += "  -- " + "; ".join(reasons)
        print(line)

print_gate_summary doesn't add any new check — it only reads the fields CaseResult already carries and builds one line per case, with the complete detail only for the ones that failed. A case that passed needs no explanation; one that failed needs to say, without anyone having to inspect the object by hand, exactly what broke.

Step 1: Reservo's current state — PASS, ready for production

CASE_SET = load_case_set("golden_cases.json")

print("--- 1. estado actual: PASS, listo para producción ---")
report = run_regression_gate(CASE_SET)
print_gate_summary(report)

What to expect:

--- 1. estado actual: PASS, listo para producción ---
=== GATE: PASS (5/5) ===
  quote_focus_pro_3h                     PASS
  quote_focus_basic_3h                   PASS
  book_focus_pro_3h_ana                  PASS
  book_boardroom_pro_1h_sofia            PASS
  book_and_cancel_studio_basic_1h_diego  PASS

This is the starting point: the Reservo agent, exactly as agent-fundamentals M8 delivered it, passes all five cases. This verdict is the baseline any future change gets compared against.

Step 2: a prompt change gets proposed — the gate runs against it before shipping

Imagine someone on the team proposes a tweak to Reservo's system prompt, intending the agent to be "more direct" with users who've already booked before. The change, without anyone noticing it in a quick code review (the prompt is free text, not code a linter can analyze), makes the model skip the quoting step for the set's simplest case:

print("--- 2. antes de desplegar: se propone un cambio de prompt, se corre el gate contra él ---")
proposed_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro 3h para Ana."}]},
]
report_proposed = run_regression_gate(CASE_SET, overrides={"quote_focus_pro_3h": proposed_script})
print_gate_summary(report_proposed)
print()
print("decisión: NO-GO -- el cambio propuesto no se despliega hasta corregirlo.")

What to expect:

--- 2. antes de desplegar: se propone un cambio de prompt, se corre el gate contra él ---
=== GATE: FAIL (4/5) ===
  quote_focus_pro_3h                     FAIL  -- tool_choice(esperaba distinto, obtuvo ['book_room']); latency(120ms)
  quote_focus_basic_3h                   PASS
  book_focus_pro_3h_ana                  PASS
  book_boardroom_pro_1h_sofia            PASS
  book_and_cancel_studio_basic_1h_diego  PASS

decisión: NO-GO -- el cambio propuesto no se despliega hasta corregirlo.

Two things worth noticing in this FAIL. First, the broken case is exactly the one that got touched — quote_focus_pro_3h — and the other four, with no change at all, stay PASS. Second, and this is a real consequence of run_case's design: this case fails for two reasons at once, not just one. book_room (120 ms of modeled latency) is a slower tool than get_quote (25 ms), and this specific case's latency threshold — 100 ms, designed for a simple quote, from a single fast tool — never anticipated a different, more expensive tool getting called. The tool-choice regression brought a second break with it, a threshold break, as a side effect — exactly the kind of cascading damage a complete gate, with the four questions evaluated together, can expose, and that an isolated check (only check_tool_choice, say) would have left partially hidden.

Step 3: the prompt gets fixed — the gate goes back to PASS, ready to ship

With the problem precisely identified — the prompt needs to keep quoting before booking — the team reverts the change. The gate runs again, over the CASE_SET with no overrides:

print("--- 3. tras corregir el prompt, se vuelve a correr el gate: PASS, ahora sí se despliega ---")
report_fixed = run_regression_gate(CASE_SET)
print_gate_summary(report_fixed)

What to expect:

--- 3. tras corregir el prompt, se vuelve a correr el gate: PASS, ahora sí se despliega ---
=== GATE: PASS (5/5) ===
  quote_focus_pro_3h                     PASS
  quote_focus_basic_3h                   PASS
  book_focus_pro_3h_ana                  PASS
  book_boardroom_pro_1h_sofia            PASS
  book_and_cancel_studio_basic_1h_diego  PASS

This is, start to finish, the complete cycle this module exists to make possible: propose a change → run the gate → decide with evidence, not with a vibe → fix if needed → confirm again before shipping. No step of this cycle needed a human to run the agent by hand and "check if it looked fine" — every decision rested on a deterministic, reproducible verdict, with the exact reason for every FAIL.

The deliverable: regression_report.json

With the final state (Step 3) confirmed at PASS, persist the complete GateReport to disk — the same asdict + json.dump pattern you've already used for every structured deliverable in this guide:

import json
from dataclasses import asdict

with open("regression_report.json", "w", encoding="utf-8") as fh:
    json.dump(asdict(report_fixed), fh, ensure_ascii=False, indent=2)

raw = open("regression_report.json", encoding="utf-8").read()
print("regression_report.json escrito:", len(raw), "bytes")

What to expect:

regression_report.json escrito: 1700 bytes

The complete file — the five CaseResults, with every field, plus the global passed — is available to anyone needing to audit the decision without running anything again:

{
  "cases": [
    {
      "name": "quote_focus_pro_3h",
      "passed": true,
      "actual_tools": ["get_quote"],
      "tool_choice_ok": true,
      "schema_errors": [],
      "output_errors": [],
      "cost_cents": 0,
      "cost_ok": true,
      "latency_ms": 25,
      "latency_ok": true
    }
  ],
  "passed": true
}

(the real file contains all five complete cases; the first one is shown as a format reference). This is, precisely, the same kind of artifact as RUN_LOG.jsonl (Module 2) or a serialized CostReport (Module 3): plain, parseable text, surviving the process that generated it — anyone can open it, without running a single line of Python again, and confirm exactly what verdict the gate produced and why.

🛑 What this mini-project demonstrated, and what it deliberately never did

It's worth closing with the same precision the module opened with. Across this lesson's three runs — the initial PASS, the proposal's FAIL, the fix's PASS — at no point did anything call a model to grade anything. Every verdict came from four deterministic comparisons: does the result have the right shape? does the chosen tool literally match the expected one? does the output value match the case's fixed anchor? are the modeled cost and latency under the threshold? Step 2's FAIL didn't say "the agent's response sounds bad" — in fact, "Reservé Focus pro 3h para Ana" is a perfectly clear response as text. It said, with mechanical precision, that the action taken — booking instead of quoting — wasn't the expected one, and that the tool used to take it exceeded that case's time budget.

If the question, at any point in this mini-project, had been "is the confirmation the agent showed Ana clear and professional?" or "did the agent correctly understand the user's intent?", no function in this module could answer it. Those questions — semantic quality, not form — belong to evaluation-frameworks-guide: its evaluating-agents module (trajectory evaluation, judged tool-call accuracy, reasoning-quality) and its evaluation-pipelines-in-production module (evaluation CI/CD with golden datasets, LLM-as-judge, quality A/B testing) cover, with their own tools and their own stack, exactly that territory. A real, mature system runs both kinds of gate — this module's, deterministic and fast, on every change; that guide's, more expensive and with semantic judgment, less frequently or over a sample — never confusing which one answers which question.

Common mistakes

1. Thinking this mini-project "already solved" the Reservo agent's complete reliability. It solved one layer — form regressions, catchable before production. It's still missing resilience against repeated failures across runs (Module 6) and an explicit versioning discipline for the prompt and the tools (Module 7). Each of those modules rests on the gate this mini-project just built — it doesn't replace it.

2. Forgetting reset_reservo_state when reproducing this flow with your own, bigger CASE_SET. As lesson 03 warned, without that reset, the order cases run in can produce a FAIL with nothing to do with any real regression — a risk that grows, not shrinks, as the CASE_SET grows.

3. Running the gate once and not running it again after "fixing" the problem. This lesson's Step 3 isn't optional — it's the confirmation that the fix actually worked, with the same deterministic evidence that caught the problem in the first place. "I think I fixed it" isn't a verdict; a gate at PASS is.

4. Saving regression_report.json only when the gate fails, "to avoid filling the disk with success reports." This file's value isn't just alerting about a FAIL — it's leaving an auditable trail of every decision, including the ones that confirmed everything was fine. A history of PASS reports is what lets you, later, confirm exactly when something that works today started failing.

5. Confusing Step 2's NO-GO with a judgment about whether the prompt-change idea was bad. The idea — "be more direct with users who've already booked before" — could be perfectly reasonable; what the gate detected was that this specific implementation of that idea, in the proposed script, broke a behavior the CASE_SET protects (quoting before booking in the simplest case). The gate doesn't evaluate ideas — it evaluates observed behavior, against a fixed expectation.

Exercises

Exercise 1: Reproduce the complete cycle with a different regression (Easy)

Repeat this lesson's three-part flow — initial PASS, proposed FAIL, fixed PASS — but using lesson 05's "by order" regression (list_rooms and get_quote swapped) over book_boardroom_pro_1h_sofia, instead of the quote_focus_pro_3h regression used in the worked example.

out_of_order_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "get_quote",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "book_room",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1, "member": "Sofía"}}]},
    {"stop_reason": "end_turn", "content": [{"type": "text", "text": "Reservé Boardroom pro 1h para Sofía."}]},
]

print("--- 1. estado actual ---")
print_gate_summary(run_regression_gate(CASE_SET))

print("--- 2. propuesta con orden alterado ---")
print_gate_summary(run_regression_gate(CASE_SET, overrides={"book_boardroom_pro_1h_sofia": out_of_order_script}))

print("--- 3. tras corregir ---")
print_gate_summary(run_regression_gate(CASE_SET))

--- 1. estado actual ---
=== GATE: PASS (5/5) ===
...
--- 2. propuesta con orden alterado ---
=== GATE: FAIL (4/5) ===
  book_boardroom_pro_1h_sofia            FAIL  -- tool_choice(esperaba distinto, obtuvo ['get_quote', 'list_rooms', 'book_room'])
...
--- 3. tras corregir ---
=== GATE: PASS (5/5) ===
...

Exercise 2: Compare two regression_report.json files and spot the difference (Medium)

Save Step 2's GateReport (the FAIL) as regression_report_before.json, and Step 3's (the PASS) as regression_report_after.json. Write code that loads both files and reports, by case name, which ones changed passed between the two.

with open("regression_report_before.json", "w", encoding="utf-8") as fh:
    json.dump(asdict(report_proposed), fh, ensure_ascii=False, indent=2)
with open("regression_report_after.json", "w", encoding="utf-8") as fh:
    json.dump(asdict(report_fixed), fh, ensure_ascii=False, indent=2)

before = json.load(open("regression_report_before.json", encoding="utf-8"))
after = json.load(open("regression_report_after.json", encoding="utf-8"))

before_by_name = {c["name"]: c["passed"] for c in before["cases"]}
after_by_name = {c["name"]: c["passed"] for c in after["cases"]}

for name in before_by_name:
    if before_by_name[name] != after_by_name[name]:
        print(f"{name}: {before_by_name[name]} -> {after_by_name[name]}")

quote_focus_pro_3h: False -> True

Exercise 3: Design a sixth case covering a scenario not covered yet (Hard)

The current CASE_SET has no case exercising a tier="basic" in a complete booking (the two booking cases use pro). Design a sixth case, book_focus_basic_2h_luis, that books Focus, basic, 2 hours, for "Luis" — with its question, complete model_script (list_rooms → get_quote → book_room → end_turn), expected_tools, expected_output (calculate price_cents by hand first), and reasonable thresholds. Add it to the CASE_SET in memory (without modifying the file) and confirm the six-case gate stays PASS.

new_case = {
    "name": "book_focus_basic_2h_luis",
    "question": "Reserva Focus basic 2h para Luis",
    "model_script": [
        {"stop_reason": "tool_use", "content": [
            {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
        {"stop_reason": "tool_use", "content": [
            {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
             "input": {"room": "Focus", "tier": "basic", "hours": 2}}]},
        {"stop_reason": "tool_use", "content": [
            {"type": "tool_use", "id": "toolu_03", "name": "book_room",
             "input": {"room": "Focus", "tier": "basic", "hours": 2, "member": "Luis"}}]},
        {"stop_reason": "end_turn", "content": [
            {"type": "text", "text": "Reservé Focus basic por 2 horas para Luis. Total $50.00."}]},
    ],
    "expected_tools": ["list_rooms", "get_quote", "book_room"],
    "expected_output": {"booking_id": 1, "confirmed": True},
    "cost_threshold_cents": 5,
    "latency_threshold_ms": 250,
}

extended_case_set = CASE_SET + [new_case]
print_gate_summary(run_regression_gate(extended_case_set))

=== GATE: PASS (6/6) ===
  quote_focus_pro_3h                     PASS
  quote_focus_basic_3h                   PASS
  book_focus_pro_3h_ana                  PASS
  book_boardroom_pro_1h_sofia            PASS
  book_and_cancel_studio_basic_1h_diego  PASS
  book_focus_basic_2h_luis               PASS

Summary and next step

• We assembled a complete regression/harness.py and regression/golden_cases.json, and added print_gate_summary, the final piece that makes any GateReport readable at a glance.
• We ran a real change's complete cycle: PASS (current state) → FAIL (a proposal introducing a tool-choice regression, with a real latency-threshold side effect) → PASS (after fixing) — with GO/NO-GO decisions backed by deterministic evidence at every step.
• We produced regression_report.json, this module's deliverable: a real, parseable file documenting the five cases' complete verdict without depending on any process staying alive.
• We closed with the statement that accompanied every lesson in this module: this gate checks form — schema, correct tool, threshold — and exact match against fixed anchors, never semantic quality. That evaluation — is the response clear? was the reasoning sound? — belongs, with its own entire discipline and tools, to evaluation-frameworks-guide.

This closes Module 5. You have a complete, run regression gate — regression/harness.py, regression/golden_cases.json, regression_report.json — the boundary with semantic evaluation stated precisely at every point where it mattered, and evidence that a real behavior change can be detected, diagnosed, and fixed with no subjective judgment involved at all.

Next module: Module 6 — Failures at Scale: Backoff, Circuit Breakers, and Rate Limits. With this module's gate already protecting against form regressions on every change, that module solves a different question: what does the system do when a specific tool has genuinely been failing several runs in a row? A per-tool CircuitBreaker, with state that persists across runs — never within a single one, agent-fundamentals M7 already solved that — is going to reuse resilience-and-reliability-patterns-guide's same vocabulary, applied, for the first time, to an agent's tool-call layer.

Additional resources

1. Anthropic — Tool use (function calling) overview — The complete protocol every CASE_SET case exercises, with no modification relative to agent-fundamentals.
2. Python — json — The complete foundation of regression_report.json, and of Exercise 2's comparison between two reports.
3. Python — dataclasses.asdict — The function that converts GateReport (and every nested CaseResult) into a serializable dict, the same technique used for every structured deliverable in this guide.
4. Anthropic — Building effective agents — On why a repeatable verification procedure, run before every change, is one of the practices that separates a reliable agentic system from a fragile one.
5. Python 3.14 — What's New — The version every line of code in this module ran on, including this lesson's real evidence of the complete cycle.