The Regression Gate in the Capstone

Description

With cost and latency already measured, this lesson sets the third discipline in motion: gating. This lesson runs the same mechanism twice, with two different questions — and it's worth naming, precisely, that it's the same mechanism, not two separate fixtures: the five-case CASE_SET M5 (Lesson 3) fixed, and the run_regression_gate(case_set, overrides=None) function M5 (Lesson 7) built. The first question, running the CASE_SET with no overrides, is "is the Reservo agent, exactly as it stands today, still behaving exactly as expected?" The second, picking that same function back up with an overrides substituting a specific case's script, is the one M7 (Lessons 3 through 7) built on top of that foundation: "is this candidate prompt version safe to replace today's with?" This lesson runs both, with real numbers, and confirms something worth keeping in mind before starting: neither question needed a new fixture — the second reuses, without changing a single line, exactly what the first already left ready.

Connection to the module

This lesson rebuilds no check — it reuses, without changing a single line, CASE_SET/run_case/run_regression_gate from M5 (Lesson 7) for the first question, and that same run_regression_gate (through overrides) plus PROMPT_REGISTRY/rollout_decision/rollback from M7 (Lessons 3, 4, 6, 7) for the second. This capstone's third real artifact, regression_report.json, comes directly out of the first part.

Analogy: the daily inspection, and the same yardstick applied to a candidate

This module's introduction's restaurant has two different uses for the same five-dish inspection. Every morning, before opening, someone runs the daily inspection: the same five dishes as always, prepared by today's kitchen exactly as it stands, to confirm nothing changed since yesterday. Separately, when someone proposes a candidate chef with a new recipe — "I want the main dish served faster, skipping one step" — the restaurant doesn't invent a different inspection or a bigger test menu: it applies the same five-dish inspection, substituting only the recipe for the dish the candidate wants to change, and compares the result against the same standard as always. Both are the same measuring stick — neither one "judges whether the food tastes good" (a food critic would do that, not an inspector) — but one confirms today's kitchen is still healthy, and the other decides whether a candidate can replace the head chef, with the advantage of comparing against exactly the same five dishes the head chef is already known to prepare well. This capstone runs both, over the same restaurant, the same morning.

Part 1: the gate, run against the agent exactly as it stands today

run_regression_gate(CASE_SET): M5's five-case CASE_SET

Reuse, without changing anything, regression/harness.py and regression/golden_cases.json exactly as they stood at the end of M5 (Lesson 8) — the CASE_SET that runs the complete run_reservo_agent, with one turn script per case:

import harness as hn

report = hn.run_regression_gate(hn.CASE_SET)
hn.print_gate_summary(report)

What to expect:

=== GATE: PASS (5/5) ===
  quote_focus_pro_3h                     PASS
  quote_focus_basic_3h                   PASS
  book_focus_pro_3h_ana                  PASS
  book_boardroom_pro_1h_sofia            PASS
  book_and_cancel_studio_basic_1h_diego  PASS

Five for five. This is the first question's answer: the Reservo agent, exactly as agent-fundamentals M8 delivered it and as this capstone has been operating it since Lesson 3, still chooses the correct tool, produces correctly shaped results, and anchors get_quote(Focus, pro, 3h)'s price at 6000 cents — the same anchor accompanying this entire guide since Module 1.

Persisting the verdict: regression_report.json

import json
from dataclasses import asdict

with open("regression_report.json", "w", encoding="utf-8") as fh:
    json.dump(asdict(report), fh, ensure_ascii=False, indent=2)

print("regression_report.json escrito:", len(open("regression_report.json", encoding="utf-8").read()), "bytes")

What to expect:

regression_report.json escrito: 1700 bytes

This is this capstone's third real artifact — the same kind of plain, auditable file as RUN_LOG.jsonl (Lesson 3): anyone can open it, without running a single line of Python again, and confirm exactly what verdict the gate produced and why.

Part 2: the same discipline, applied to comparing two complete versions

The first part confirmed today's agent still works. The second picks back up exactly the same CASE_SET and the same run_regression_gate for a different question: comparing a candidate version of the system prompt against the one running today, before that candidate ever talks to a single real user. M7 (Lessons 3 through 7) built this piece without declaring a single new case — run_regression_gate(case_set, overrides=None)'s overrides parameter (M5, Lesson 7) is, precisely, "the piece Module 7 is going to reuse to compare an old agent version against a new one," exactly as M5's Lesson 7 previewed. Every registry version gets modeled as an overrides: it substitutes the script of the specific cases that version would change, and leaves the rest of the CASE_SET untouched.

The registry: v1 and v2, each with its hash

from prompt_registry import PROMPT_REGISTRY

v1 = PROMPT_REGISTRY["v1"]
v2 = PROMPT_REGISTRY["v2"]
print(f"v1: hash={v1.prompt_hash}  nota={v1.note!r}")
print(f"v2: hash={v2.prompt_hash}  nota={v2.note!r}")

What to expect:

v1: hash=c5757b6d6264  nota='System prompt original del capstone de agent-fundamentals M8.'
v2: hash=c364e85e5649  nota='Agrega una instruccion de proactividad para reducir turnos.'

v2 isn't an arbitrary change — it's exactly the kind of change that looks reasonable in a quick review: "if the agent already has all the information needed to complete a booking, let it complete it directly, to save the user a step." Tested by hand, with a single question like "Book Focus pro 3h for Ana", the change looks like an improvement — the agent books, exactly as expected. The problem only shows up with questions that should never have ended in a booking.

The gate: v1 against v2, over the same five-case CASE_SET

from rollout import rollout_decision

V2_REGRESSED_SCRIPT = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro 3h para Ana."}]},
]
VERSION_OVERRIDES = {
    "v1": {},
    "v2": {"quote_focus_pro_3h": V2_REGRESSED_SCRIPT},
}

report_v1 = hn.run_regression_gate(hn.CASE_SET, overrides=VERSION_OVERRIDES["v1"])
report_v2 = hn.run_regression_gate(hn.CASE_SET, overrides=VERSION_OVERRIDES["v2"])

print("GATE v1:", "PASS" if report_v1.passed else "FAIL",
      f"({sum(c.passed for c in report_v1.cases)}/{len(report_v1.cases)})")
print("GATE v2:", "PASS" if report_v2.passed else "FAIL",
      f"({sum(c.passed for c in report_v2.cases)}/{len(report_v2.cases)})")
for c in report_v2.cases:
    if not c.passed:
        print(f"  {c.name:38} FAIL -- tool_choice_ok={c.tool_choice_ok} actual_tools={c.actual_tools}")

decision, broken = rollout_decision(report_v1, report_v2)
print(f"rollout_decision(v1, v2) -> {decision}, casos_rotos={broken}")

What to expect:

GATE v1: PASS (5/5)
GATE v2: FAIL (4/5)
  quote_focus_pro_3h                     FAIL -- tool_choice_ok=False actual_tools=['book_room']
rollout_decision(v1, v2) -> NO-GO, casos_rotos=['quote_focus_pro_3h']

v1 passes all five cases — the baseline. v2 passes four, and fails exactly on quote_focus_pro_3h, the same question — "How much does Focus pro 3h cost?" — anchored to the same 6000 cents accompanying this guide since Module 1: instead of quoting, v2 decides to book directly, with nobody having explicitly asked for it. rollout_decision — M7's (Lesson 6) hard rule: it can never break a case the old version already passed — needs no additional judgment to decide NO-GO; the evidence, case by case, already says it all.

The rollback: back to v1, a pointer change

from rollout import rollback

ACTIVE_VERSION = "v2"  # alguien la promovió antes de correr el gate -- la trampa que M7 (Lección 2) advirtió
print("version activa (antes del gate):", ACTIVE_VERSION)

if decision == "NO-GO":
    ACTIVE_VERSION = rollback(ACTIVE_VERSION, "v1", PROMPT_REGISTRY)

print("version activa (después del rollback):", ACTIVE_VERSION)

What to expect:

version activa (antes del gate): v2
version activa (después del rollback): v1

rollback rebuilds nothing — AgentVersion is immutable (frozen=True, M7 Lesson 3) and v1 never stopped existing, complete, inside PROMPT_REGISTRY. "Going back to v1" is, precisely, a pointer change: which version_id is active right now, nothing more.

The two questions, over the same CASE_SET

It's worth saying it once more, with this lesson's numbers already on the table, because it's this capstone's integration central finding:

Neither row contradicts the other — they are, precisely, two different questions, over the same five-case CASE_SET, with the same mechanism (run_regression_gate) applied twice with a different overrides. And both share the same underlying discipline, with no exception: literal comparison against a fixed value, never a judge, never a semantic-quality score. quote_focus_pro_3h doesn't fail because "booking instead of quoting sounds bad" — it fails because check_tool_choice compares, with ==, the list ['book_room'] against the ['get_quote'] list the case requires. If at any point the real question were "is the confirmation v2 showed Ana clear and professional?", this mechanism has no way to answer it — that question belongs, unambiguously, to evaluation-frameworks-guide.

Common mistakes

1. Thinking M7 runs a different check than M5's, with its own case set. It doesn't — it reuses, without changing a single line, M5's five-case CASE_SET and run_regression_gate. The only thing changing between this lesson's two questions is the overrides passed to the same function.

2. Thinking a NO-GO on quote_focus_pro_3h means v2 is "worse across the board." rollout_decision doesn't evaluate "better overall" — it evaluates, precisely, whether v2 broke something v1 already handled well. v2 could, in theory, improve other interactions no CASE_SET case covers; the rule is still NO-GO because it broke a case that mattered, with no courtesy exception.

3. Forgetting reset_reservo_state() before every gate case. Without that isolation (M5, Lesson 3), the order the booking cases run in could produce a FAIL with nothing to do with any real regression — M5's Lesson 3 demonstrated it with a contaminated booking_id between two cases, and run_case (M5, Lesson 7) calls it at the start of every run, with no exception.

4. Running rollback without having confirmed the NO-GO decision first. rollback on its own decides nothing — it just moves the pointer to whichever version_id it's told, and validates it exists in the registry. The decision of when to call it always lives in rollout_decision, never in a last-minute manual judgment.

5. Confusing Part 1's GateReport (no overrides, verdict on today's system) with Part 2's (with overrides, verdict on a candidate version). Both have exactly the same shape — cases/passed — so the only thing telling one apart from the other is which overrides was used to produce it. AGENT_CHANGELOG.md, in Lesson 8, documents both, precisely noting which one answers which question.

Exercises

Exercise 1: Confirm book_focus_pro_3h_ana doesn't get contaminated under v2 (Easy)

book_focus_pro_3h_ana also calls book_room — as part of its list_rooms → get_quote → book_room sequence — just like quote_focus_pro_3h's regressed version. Without looking at the output above again: confirm, with report_v2.cases, that this case stays PASS under v2.

ana_case = next(c for c in report_v2.cases if c.name == "book_focus_pro_3h_ana")
print("passed:", ana_case.passed, " actual_tools:", ana_case.actual_tools)

passed: True  actual_tools: ['list_rooms', 'get_quote', 'book_room']

Exercise 2: Simulate a v4 that breaks two cases at once (Medium)

Build VERSION_OVERRIDES["v4"] combining quote_focus_pro_3h's regressed script with a second script that books book_and_cancel_studio_basic_1h_diego but never cancels. Run the gate and confirm the count and both broken cases.

no_cancel_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Studio", "tier": "basic", "hours": 1, "member": "Diego"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Studio basic 1h para Diego."}]},
]
VERSION_OVERRIDES["v4"] = {
    "quote_focus_pro_3h": V2_REGRESSED_SCRIPT,
    "book_and_cancel_studio_basic_1h_diego": no_cancel_script,
}
report_v4 = hn.run_regression_gate(hn.CASE_SET, overrides=VERSION_OVERRIDES["v4"])
print("GATE v4:", "PASS" if report_v4.passed else "FAIL",
      f"({sum(c.passed for c in report_v4.cases)}/{len(report_v4.cases)})")
for c in report_v4.cases:
    if not c.passed:
        print("  roto:", c.name, c.actual_tools)

GATE v4: FAIL (3/5)
  roto: quote_focus_pro_3h ['book_room']
  roto: book_and_cancel_studio_basic_1h_diego ['book_room']

Exercise 3: Design a v3 that fixes the regression, and confirm GO (Hard)

Pick back up SYSTEM_PROMPT_V3 — the fixed version M7 (Lesson 8, mini-project) already registered and calculated, with its own real hash — the same instruction as v1, but explicit that v2's proactivity should never apply to a quote-only question. Register it in PROMPT_REGISTRY, run the gate against v1 as a baseline — with no overrides at all, because v3's behavior over the CASE_SET's five cases matches v1's again — and confirm the decision is GO.

from prompt_registry import hash_prompt, AgentVersion

SYSTEM_PROMPT_V3 = (
    "Eres el asistente de reservas de Reservo, un sistema de coworking. "
    "Ayudas a los usuarios a consultar salas, cotizar precios, reservar y "
    "cancelar reservas. Usa siempre las tools disponibles para cotizar y "
    "reservar -- nunca inventes un precio de memoria. Cuando el usuario "
    "pregunta un precio, usa get_quote y NO reserves, incluso si podrias "
    "inferir todos los datos necesarios para reservar. Usa book_room "
    "unicamente cuando el usuario pide reservar de forma explicita."
)
v3_hash = hash_prompt(SYSTEM_PROMPT_V3)
PROMPT_REGISTRY["v3"] = AgentVersion(
    version_id="v3", prompt_text=SYSTEM_PROMPT_V3, prompt_hash=v3_hash,
    tools_version="tools-v1", model="claude-sonnet-5",
    note="Corrige la regresion de v2 en quote_focus_pro_3h.",
)

VERSION_OVERRIDES["v3"] = {}  # v3 vuelve a decidir get_quote en quote_focus_pro_3h, igual que v1
report_v3 = hn.run_regression_gate(hn.CASE_SET, overrides=VERSION_OVERRIDES["v3"])
decision_v3, broken_v3 = rollout_decision(report_v1, report_v3)
print(f"v3: hash={v3_hash}")
print("GATE v3:", "PASS" if report_v3.passed else "FAIL",
      f"({sum(c.passed for c in report_v3.cases)}/{len(report_v3.cases)})")
print(f"rollout_decision(v1, v3) -> {decision_v3}, casos_rotos={broken_v3}")

v3: hash=c5c4c4631f7e
GATE v3: PASS (5/5)
rollout_decision(v1, v3) -> GO, casos_rotos=[]

Summary and next step

• We ran the gate against the agent exactly as it stands today — the five-case CASE_SET, no overrides at all: PASS (5/5), persisted to regression_report.json, this capstone's third real artifact.
• We ran, without changing a single line, the same mechanism with overrides comparing v1 against v2: PASS (5/5) for v1, FAIL (4/5) for v2 — exactly on quote_focus_pro_3h — and rollout_decision(v1, v2) -> NO-GO, casos_rotos=['quote_focus_pro_3h'].
• We ran rollback("v2", "v1", PROMPT_REGISTRY): the active version returns to v1, a deterministic pointer change, with no real deployment mechanism at all.
• We confirmed this lesson's two questions — does today's system still work? is the candidate version safe? — share the same CASE_SET and the same function, run_regression_gate, applied with a different overrides — never two separate fixtures, and in both cases under the same form discipline, never semantic judgment.

Next lesson: 06 — The Resilience Layer. With the gate and the rollout already run, we put book_room to genuinely fail and watch M6's CircuitBreaker open, integrated with the rest of this capstone's operations layer.

Additional resources

1. Anthropic — Tool use (function calling) overview — The protocol this mechanism validates end to end, applied twice with a different overrides.
2. evaluation-frameworks-guide — for when the question stops being "is the form correct?" and becomes "is the response semantically good?" — its evaluating-agents module covers exactly that territory, with its own tools.
3. Python — hashlib — hash_prompt's foundation, reused unchanged from M7 in this lesson.
4. Python — dataclasses.frozen — Why rollback never "rebuilds" v1: AgentVersion(frozen=True) guarantees it never stopped existing, complete, in the registry.