The Gate as a Rollout Check

Description

The previous lesson showed the result: v2 gets FAIL (4/5), exactly on quote_focus_pro_3h. This lesson opens that CaseResult under a magnifying glass. run_case (Module 5) doesn't apply a single check — it applies five, grouped into three questions: does the chosen tool match? (tool_choice_ok), does the result have the right shape? (schema_errors) and does it match the case's fixed anchor? (output_errors), did cost and latency stay within budget? (cost_ok, latency_ok). A case passes only if all five are favorable. This lesson's question is precise: of those five, which ones genuinely caught v2's regression — and which ones didn't even get evaluated?

Connection to the module

This lesson connects Lesson 04's run result with the boundary Module 5 traces precisely starting in its Lesson 04: a form gate never replaces a semantic judgment on whether the agent's response is "good." Here you see, with numeric evidence, exactly how far that form gate reaches — and a real side effect that only shows up when you look at the complete CaseResult, not just the final verdict.

quote_focus_pro_3h's complete CaseResult under v2

from regression.harness import run_case

case = next(c for c in CASE_SET if c["name"] == "quote_focus_pro_3h")
result = run_case(case, 1, model_script=VERSION_OVERRIDES["v2"]["quote_focus_pro_3h"])

print("name          :", result.name)
print("passed        :", result.passed)
print()
print("tool_choice_ok:", result.tool_choice_ok, " actual_tools:", result.actual_tools)
print("schema_errors :", result.schema_errors)
print("output_errors :", result.output_errors)
print("cost_ok       :", result.cost_ok, " cost_cents:", result.cost_cents)
print("latency_ok    :", result.latency_ok, " latency_ms:", result.latency_ms)

What to expect:

name          : quote_focus_pro_3h
passed        : False

tool_choice_ok: False  actual_tools: ['book_room']
schema_errors : []
output_errors : []
cost_ok       : True  cost_cents: 0
latency_ok    : False  latency_ms: 120

Two findings, neither obvious at first glance. First: schema_errors and output_errors are empty — but that doesn't mean those two checks "passed." run_case evaluates them inside an if tool_choice_ok: (Module 5, Lesson 07): since the chosen tool is already wrong, there's no get_quote result to look for inside history to validate against its schema or its anchor — those two checks don't even get to run. An empty list, in this context, means "never evaluated," not "passed."

Second, and subtler: latency_ok is False. quote_focus_pro_3h has a 100 ms threshold — generous for a simple quote, a single call to get_quote (25 modeled ms). But v2 didn't call get_quote: it called book_room, which models 120 ms — slower, and over the threshold this specific case declares. The tool-choice regression brought a second break with it, a threshold break, as a side effect: nobody designed the CASE_SET thinking this case might call a different, more expensive tool, because it never should.

cost_ok does pass — and that's also information

It's worth noting what didn't fail: cost_ok is True. The final text v2 produces ("Reservé Focus pro 3h para Ana.") isn't any longer than what v1 would have produced, so the estimated cost stays just as low (0 cents, the same honest scale as always). This confirms something important about the gate's design: not every check automatically fails together when something breaks. Each one measures a different dimension, and a specific regression can affect some without affecting the others. If the gate only had cost_ok and latency_ok, and not tool_choice_ok, this case would have failed anyway — on latency — but the message would have been much less useful: "latency over threshold" tells nobody the real problem is the agent booked without being asked.

Why tool_choice_ok is the check that makes the problem legible

tool_choice_ok is the only one of the five that compares against what was expected for this specific case, not against a generic property of the result. The others — schema, anchor, latency — evaluate whether what book_room returned is valid for book_room; tool_choice_ok is the only one that asks whether book_room was, to begin with, the tool that should have run. That's why Lesson 04's FAIL message — expected tool get_quote, got book_room — is the one a real team needs to read first: it names the root cause, not a derived symptom like the crossed latency threshold.

What this gate CANNOT answer

It's worth being explicit about the limit, because it's the exact boundary drawn with the sister guide. Imagine a hypothetical version whose prompt does call get_quote correctly on quote_focus_pro_3h — passes all five of this gate's checks with no problem — but whose final text response, the one the agent shows the user after the tool_result, says something confusing, aggressive, or just poorly written. This guide's gate has no way of detecting that. It doesn't read the agent's final text with any quality criterion; it doesn't ask any model to evaluate it; it has no concept of "tone" or "clarity" at all.

# Un output que PASARIA los 5 chequeos de este gate, con una respuesta
# final pesima -- ninguno de los cinco inspecciona este string.
respuesta_final_hipotetica = "6000. eso es todo, no tengo mas para decir."
print("El gate de esta guia nunca inspecciona:", repr(respuesta_final_hipotetica))

What to expect:

El gate de esta guia nunca inspecciona: '6000. eso es todo, no tengo mas para decir.'

That's, precisely, the boundary Module 5 draws in its Lesson 04, and this module inherits unmodified: evaluating whether a response is semantically good — clear, useful, with the right tone, faithful to the user's intent beyond just the correct technical action — is evaluation-frameworks-guide's job, with its trajectory evaluation and its model-judged tool-call accuracy. That other kind of evaluation needs an LLM-as-judge or a golden dataset with "roughly correct" ground truth — exactly what this gate deliberately forbids itself from using, because a deterministic form check has a property a semantic judge never has: the same input always produces the same result, never varying from one run to the next. When you need to answer "is the agent's response good, not just did it take the right action?", that's the guide to consult, named here precisely.

Common mistakes

1. Reading schema_errors == [] as "the schema passed." As this lesson's example shows, an empty list can mean "never got evaluated" — because tool_choice_ok was already False — instead of "evaluated and found no problem." Always check tool_choice_ok first before interpreting any of the other four fields.

2. Thinking a gate that passed cost_ok is already evidence that "nothing serious happened." This lesson's example disproves that with numbers: cost_ok=True and, even so, the complete case has to fail. A gate with several independent checks is only reliable if it requires all of them to pass, not a majority, and a single field being green should never be read as "the main signal."

3. Confusing "the gate is about form" with "the gate is weak." This guide's gate detected, with total precision, a behavior regression no error log, no isolated latency check, and no superficial inspection ("did the agent respond with something coherent?") would have caught with the same level of detail. Being "about form" doesn't mean being unrigorous — it means its rigor is bounded to literal comparisons, not quality judgments.

4. Not realizing latency_ok=False here is an EFFECT, not an independent cause. If someone only looked at latency_ok without checking tool_choice_ok, they could mistakenly conclude "Reservo got slow" — when the real problem is a different tool got called, one that's slower by design. Diagnosing from the wrong field leads to fixing the symptom (raising the latency threshold) instead of the cause (fixing the prompt).

5. Thinking this gate can replace a review of the agent's final response. As the previous section shows, the gate never inspects the text the agent shows the user. A real production system needs both: this gate for structural behavior regressions, and something like evaluation-frameworks-guide for the response's semantic quality.

Exercises

Exercise 1: Repeat the breakdown on a case that DOES pass (Easy)

Run run_case over book_focus_pro_3h_ana under v2 (with no overrides for that case) and confirm all five relevant fields (tool_choice_ok, schema_errors, output_errors, cost_ok, latency_ok) indicate the case passed clean.

case_ana = next(c for c in CASE_SET if c["name"] == "book_focus_pro_3h_ana")
result_ana = run_case(case_ana, 1)

print("tool_choice_ok:", result_ana.tool_choice_ok)
print("schema_errors :", result_ana.schema_errors)
print("output_errors :", result_ana.output_errors)
print("cost_ok       :", result_ana.cost_ok)
print("latency_ok    :", result_ana.latency_ok)

tool_choice_ok: True
schema_errors : []
output_errors : []
cost_ok       : True
latency_ok    : True

Exercise 2: Design a latency threshold that ignores the side effect (Medium)

Without touching golden_cases.json, build a copy of quote_focus_pro_3h with latency_threshold_ms raised to 150 (instead of 100) — a threshold that would tolerate book_room's latency (120 ms). Run run_case with v2's regressed script over that copy, and confirm latency_ok is now True, but passed is still False.

case_lenient = {**case, "latency_threshold_ms": 150}
result_lenient = run_case(case_lenient, 1, model_script=VERSION_OVERRIDES["v2"]["quote_focus_pro_3h"])

print("latency_ok:", result_lenient.latency_ok, " latency_ms:", result_lenient.latency_ms)
print("tool_choice_ok:", result_lenient.tool_choice_ok)
print("passed:", result_lenient.passed)

latency_ok: True  latency_ms: 120
tool_choice_ok: False
passed: False

Exercise 3: Argue whether a sixth check ("does the final text mention the correct price?") would still be "about form" (Hard)

Someone on the team proposes adding a sixth check to the gate: check_final_text_contains_price(text, expected_price_cents), which confirms — with a substring search, not a model — that the agent's final text contains the expected price number in dollars (for example, "$60.00" for price_cents=6000). Argue, in one paragraph, whether this check respects the gate's hard rule ("about form, deterministic, no judge") or crosses the boundary into evaluation-frameworks-guide's semantic territory.

Summary and next step

• CaseResult breaks down into five independent checks: tool choice, result shape, value anchor, cost budget, latency budget. A case passes only if all five are favorable.
• On v2's quote_focus_pro_3h, schema_errors and output_errors stay empty because they never got evaluated — tool_choice_ok was already False — not because "they passed." cost_ok genuinely does pass. And latency_ok fails as a real side effect of the wrong tool: book_room (120 ms) exceeds the 100 ms threshold this case, designed for a simple quote, never anticipated.
• tool_choice_ok is the check that makes the root cause legible — the message expected tool get_quote, got book_room says, unambiguously, what broke, while looking only at latency_ok would have pointed at the wrong symptom.
• The gate has an explicit limit: it never inspects the final text the agent shows the user. Evaluating whether that response is semantically good is evaluation-frameworks-guide's job, named here precisely as this guide's boundary.

Next lesson: 06 — Go or No-Go. We build rollout_decision: the explicit rule that turns v2's FAIL (4/5) into a one-word decision — and why that rule never lets a new version break a case the old one already handled well.

Additional resources

1. Anthropic — Building effective agents — On the difference between verifying an agent ran the correct action and evaluating its final response's quality.
2. Anthropic — Tool use (function calling) overview — The exact tool-contract shape check_schema validates against OUTPUT_SCHEMAS.
3. Python — dictionary equality comparison — The foundation of check_expected_output, used inside run_case only when tool_choice_ok is already True.
4. Python 3.14 — What's New — The version every line of code in this lesson ran on.