Comparing a New Version Against the Old

Description

With PROMPT_REGISTRY built in the previous lesson, this lesson does the real comparison: run the same regression gate — the one Module 5 built as a deterministic form check, no judge — against v1 and against v2, and see, with evidence, what differs. This lesson doesn't rebuild that gate: it reuses it, exactly as it stood in regression/harness.py and regression/golden_cases.json, without touching a single line. The only genuinely new thing here is how it gets used to compare two versions — the piece Module 5 itself left ready, on purpose, for this exact moment.

By the end of this lesson you're going to have two numbers — PASS (5/5) and FAIL (4/5) — and, more importantly, you're going to know exactly on which case they diverge, with the exact message this guide's DISEÑO demands citing.

Connection to the module

This is the module's hinge lesson: Lesson 02 showed the danger in the abstract (one question, two different decisions); Lesson 03 gave each version an identity (the hash-based registry); this lesson runs the real check, end to end, over the CASE_SET's five cases, not just the isolated question you already saw. The result — v1 passes, v2 doesn't — is the evidence Lessons 05, 06, and 07 are going to use to explain, decide, and act.

Analogy: the same test battery, on two pilots

Going back to the introduction's pilot medical-clearance analogy: the exam doesn't change depending on who takes it. The same five checks, in the same order, with the same thresholds, apply to the pilot who's been flying for twenty years and to the one getting certified for the first time. This lesson is, precisely, that exam applied twice: once to v1, once to v2, without changing a single check between one run and the other.

CASE_SET: picked back up from Module 5, unchanged

regression/golden_cases.json carries five fixed cases, covering Reservo's four tools:

from regression.harness import load_case_set

CASE_SET = load_case_set("regression/golden_cases.json")
print("casos cargados:", len(CASE_SET))
for c in CASE_SET:
    print(f"{c['name']:38} tools={c['expected_tools']}")

What to expect:

casos cargados: 5
quote_focus_pro_3h                     tools=['get_quote']
quote_focus_basic_3h                   tools=['get_quote']
book_focus_pro_3h_ana                  tools=['list_rooms', 'get_quote', 'book_room']
book_boardroom_pro_1h_sofia            tools=['list_rooms', 'get_quote', 'book_room']
book_and_cancel_studio_basic_1h_diego  tools=['book_room', 'cancel_booking']

This module doesn't declare a single new case — it uses exactly the CASE_SET Module 5 fixed. The first entry, quote_focus_pro_3h, is the same question — "How much does Focus pro 3h cost?" — you already saw in Lesson 02, with the same anchor as always: get_quote(Focus, pro, 3h) = 6000 cents.

Modeling each version as an overrides: the piece Module 5 left ready

run_regression_gate(case_set, overrides=None) accepts an optional {case_name: substituted_script} dictionary: for any case that doesn't appear there, it runs the CASE_SET's original model_script; for the one that does appear, it runs the substituted script, compared against the same expected_tools as always. Module 5's Lesson 05 documented this technique, explicitly, as "the piece Module 7 is going to reuse to compare an old agent version against a new one" — this is, precisely, that moment.

Each registry version gets modeled as an overrides: v1 is the CASE_SET with no change at all ({}); v2 substitutes quote_focus_pro_3h's script for the one the "proactive" system prompt would produce — the same regressed script this module's Lesson 02 already previewed, and that Module 5 used as its canonical FAIL example:

# El guion que v2 produciría para "¿Cuánto cuesta Focus pro 3h?": salta la
# cotización y reserva directo -- el mismo guion regresivo del Módulo 5.
V2_REGRESSED_SCRIPT = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro 3h para Ana."}]},
]

VERSION_OVERRIDES = {
    "v1": {},
    "v2": {"quote_focus_pro_3h": V2_REGRESSED_SCRIPT},
}

VERSION_OVERRIDES is the bridge between Lesson 03's registry (which version exists, with which hash) and Module 5's gate (which behavior to check): for every version_id, it declares which CASE_SET cases run differently under that version. v1, the original version, substitutes nothing — it runs the CASE_SET exactly as Module 5 left it. v2 substitutes only quote_focus_pro_3h — the other four questions never get touched, because the proactivity line added to the prompt doesn't affect any of them.

compare_versions: running the same gate against two configurations

With VERSION_OVERRIDES in place, the complete comparison fits into a single function: it runs run_regression_gate twice — once per version — and returns both GateReports, ready for Lesson 06 to compare with rollout_decision.

from regression.harness import run_regression_gate


def compare_versions(case_set, overrides_old, overrides_new):
    """Corre el MISMO gate del Módulo 5 contra dos configuraciones de
    overrides -- 'la version vieja' y 'la version nueva' -- y devuelve
    ambos GateReport, sin decidir nada todavía (eso es rollout_decision,
    lección 06)."""
    gate_old = run_regression_gate(case_set, overrides=overrides_old)
    gate_new = run_regression_gate(case_set, overrides=overrides_new)
    return gate_old, gate_new

compare_versions adds no new check at all — it's, deliberately, a thin wrapper over run_regression_gate, the same "wrap, don't rebuild" discipline you already saw in Module 1 with run_and_observe. Its only job is to make explicit, with a name, the pattern the rest of this module repeats: two runs of the same gate, one per version, never a mixed run.

Running the comparison: v1 against v2

report_v1, report_v2 = compare_versions(CASE_SET, VERSION_OVERRIDES["v1"], VERSION_OVERRIDES["v2"])

print("GATE v1:", "PASS" if report_v1.passed else "FAIL",
      f"({sum(c.passed for c in report_v1.cases)}/{len(report_v1.cases)})")
for c in report_v1.cases:
    print(f"  {c.name:38} {'PASS' if c.passed else 'FAIL'}")

What to expect:

GATE v1: PASS (5/5)
  quote_focus_pro_3h                     PASS
  quote_focus_basic_3h                   PASS
  book_focus_pro_3h_ana                  PASS
  book_boardroom_pro_1h_sofia            PASS
  book_and_cancel_studio_basic_1h_diego  PASS

Now v2's report, calculated in the same compare_versions call:

print("GATE v2:", "PASS" if report_v2.passed else "FAIL",
      f"({sum(c.passed for c in report_v2.cases)}/{len(report_v2.cases)})")
for c in report_v2.cases:
    status = "PASS" if c.passed else "FAIL"
    line = f"  {c.name:38} {status}"
    if not c.passed:
        line += f"  -- tool_choice_ok={c.tool_choice_ok} actual_tools={c.actual_tools}"
    print(line)

What to expect:

GATE v2: FAIL (4/5)
  quote_focus_pro_3h                     FAIL  -- tool_choice_ok=False actual_tools=['book_room']
  quote_focus_basic_3h                   PASS
  book_focus_pro_3h_ana                  PASS
  book_boardroom_pro_1h_sofia            PASS
  book_and_cancel_studio_basic_1h_diego  PASS

There's the complete evidence: v1 solves all five cases exactly as expected; v2 solves four, and fails on quote_focus_pro_3h — with the exact message this guide's DISEÑO demands citing: expected tool get_quote, got book_room. v2 didn't just choose the wrong tool: it genuinely ran it, and book_focus_pro_3h_ana — the case that genuinely should book — still passes with no side effect at all, because run_case resets Reservo's state before running every case.

Confirming overrides doesn't contaminate the rest of the CASE_SET

It's worth confirming, with code, something the output above already suggests: book_focus_pro_3h_ana also calls book_room — just like quote_focus_pro_3h's regressed version — and it still stays PASS under v2. This isn't a coincidence:

ana_case = next(c for c in report_v2.cases if c.name == "book_focus_pro_3h_ana")
print("book_focus_pro_3h_ana bajo v2 -- passed:", ana_case.passed)
print("actual_tools:", ana_case.actual_tools)

What to expect:

book_focus_pro_3h_ana bajo v2 -- passed: True
actual_tools: ['list_rooms', 'get_quote', 'book_room']

run_case calls reset_reservo_state() at the start of every case — before quote_focus_pro_3h, and again before book_focus_pro_3h_ana — so the booking quote_focus_pro_3h's regression mistakenly creates never leaks into the booking_id Ana's case expects to find. Without that isolation — the discipline Module 5's Lesson 03 established precisely — a real FAIL could hide behind a contaminated booking_id, or a healthy case could report a FAIL with nothing to do with any genuine regression.

Common mistakes

1. Running the gate against v2 without having run it against v1 first. Without v1's baseline (PASS 5/5), a FAIL 4/5 result on v2 says nothing on its own — it could be the CASE_SET has a problem, not the new version. Always comparing against a known baseline is what gives the 4/5 its meaning.

2. Thinking overrides "rewrites" the CASE_SET. It doesn't — golden_cases.json on disk never changes. overrides is a dictionary living only in memory, during a specific run_regression_gate call; the next run with no overrides goes back to using every case's original model_script, with no trace of the previous substitution.

3. Building VERSION_OVERRIDES["v2"] with the case's complete model_script, instead of just the substituted script. overrides expects {case_name: complete_alternative_script} — a complete turn script, with its own stop_reason and content, not a fragment or a dictionary of differences. The substituted script replaces the entire original for that case.

4. Assuming v2's regression affects all five cases equally. This lesson's overrides only touches quote_focus_pro_3h — the other four CASE_SET questions (quote_focus_basic_3h, the two complete bookings, the cancellation) never appear in VERSION_OVERRIDES["v2"], so they run with their original script and pass with no change. A prompt regression doesn't have to break everything the agent does — it often breaks a specific slice of behavior, and the gate, run over the complete CASE_SET, is what reveals exactly which one.

5. Interpreting FAIL (4/5) as "the agent works 80% fine." It isn't a percentage-of-quality measure — it's an exact count of cases that passed a literal comparison. One single broken case can be, as in this example, the difference between a safe agent and one that books without permission; the number doesn't capture severity, only quantity.

Exercises

Exercise 1: Confirm book_focus_pro_3h_ana doesn't get contaminated, running it in isolation (Easy)

Using run_case directly (not the complete gate), run book_focus_pro_3h_ana with no overrides, twice in a row in the same process. Confirm both runs produce exactly the same CaseResult.

from regression.harness import run_case

case_ana = CASE_SET[2]  # book_focus_pro_3h_ana
r1 = run_case(case_ana, 1)
r2 = run_case(case_ana, 2)
print("primera corrida :", r1.passed, r1.actual_tools)
print("segunda corrida :", r2.passed, r2.actual_tools)
print("resultados identicos:", r1.passed == r2.passed and r1.actual_tools == r2.actual_tools)

primera corrida : True ['list_rooms', 'get_quote', 'book_room']
segunda corrida : True ['list_rooms', 'get_quote', 'book_room']
resultados identicos: True

Exercise 2: Simulate a v4 that breaks TWO cases at once (Medium)

Build VERSION_OVERRIDES["v4"] combining quote_focus_pro_3h's regressed script with a second script that books book_and_cancel_studio_basic_1h_diego but never cancels (the same "stops before completing the sequence" pattern from Module 5). Run the gate and confirm the count and both broken cases.

no_cancel_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Studio", "tier": "basic", "hours": 1, "member": "Diego"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Studio basic 1h para Diego."}]},
]
VERSION_OVERRIDES["v4"] = {
    "quote_focus_pro_3h": V2_REGRESSED_SCRIPT,
    "book_and_cancel_studio_basic_1h_diego": no_cancel_script,
}
report_v4 = run_regression_gate(CASE_SET, overrides=VERSION_OVERRIDES["v4"])
print("GATE v4:", "PASS" if report_v4.passed else "FAIL",
      f"({sum(c.passed for c in report_v4.cases)}/{len(report_v4.cases)})")
for c in report_v4.cases:
    if not c.passed:
        print("  roto:", c.name, c.actual_tools)

GATE v4: FAIL (3/5)
  roto: quote_focus_pro_3h ['book_room']
  roto: book_and_cancel_studio_basic_1h_diego ['book_room']

Exercise 3: Trigger a FAIL without changing the final tool — just the path to get there (Hard)

Build VERSION_OVERRIDES["v5"] substituting book_boardroom_pro_1h_sofia for a script that calls get_quote twice in a row, with the exact same arguments, before booking (the agent "changes its mind" and quotes again). Run the gate and confirm that, even though the final booking would be identical, the case fails all the same.

double_quote_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "get_quote",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "book_room",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1, "member": "Sofía"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Boardroom pro 1h para Sofía."}]},
]
VERSION_OVERRIDES["v5"] = {"book_boardroom_pro_1h_sofia": double_quote_script}
report_v5 = run_regression_gate(CASE_SET, overrides=VERSION_OVERRIDES["v5"])
print("GATE v5:", "PASS" if report_v5.passed else "FAIL",
      f"({sum(c.passed for c in report_v5.cases)}/{len(report_v5.cases)})")
broken = next(c for c in report_v5.cases if not c.passed)
print("roto:", broken.name, "obtenido:", broken.actual_tools, "tool_choice_ok:", broken.tool_choice_ok)

GATE v5: FAIL (4/5)
roto: book_boardroom_pro_1h_sofia obtenido: ['list_rooms', 'get_quote', 'get_quote', 'book_room'] tool_choice_ok: False

Summary and next step

• We picked Module 5's CASE_SET back up, unchanged: five fixed cases, covering Reservo's four tools.
• We modeled every registry version as an overrides over that CASE_SET: v1 substitutes nothing; v2 substitutes only quote_focus_pro_3h for the script the proactivity instruction would produce.
• We built compare_versions, the thin wrapper that runs run_regression_gate twice — once per version — and returns both GateReports together.
• We ran the comparison: v1 gives PASS (5/5) — the baseline. v2, with not a single gate check changed: FAIL (4/5), with the DISEÑO's exact message: expected tool get_quote, got book_room.
• We confirmed, running it, that per-case isolation (reset_reservo_state() inside run_case) stops one case's regression from contaminating another that also uses book_room.

Next lesson: 05 — The Gate as a Rollout Check. We open quote_focus_pro_3h's complete CaseResult under v2 and confirm, with evidence, which of its checks genuinely detected the regression — and why one of them fails as a side effect of the other.

Additional resources

1. Anthropic — Tool use (function calling) overview — The tool_use/tool_result contract check_tool_choice compares, in its most literal form.
2. Python — list comparison with == — check_tool_choice's foundation: element by element, in order, sensitive to length.
3. Python — dict.get with a default value — The foundation of overrides.get(case["name"]) inside run_regression_gate, which returns None (and therefore "uses the original script") for any non-substituted case.
4. Python 3.14 — What's New — The version every line of code in this lesson ran on.