The Gate: Pass or Fail the Build

Description

Lessons 04, 05, and 06 built, one by one, the gate's three questions: does the result have the right shape? did the agent choose the correct tool? did cost and latency stay under the threshold? This lesson brings them together into the two functions that give this module its name: run_case, which runs an individual CASE_SET case against the four comparisons and returns a complete CaseResult; and run_regression_gate, which runs the entire CASE_SET and returns a GateReport with a PASS/FAIL verdict for the batch — like a continuous-integration gate, exactly like the one that reviews any pull request before letting it through.

This is the lesson where the gate runs, end to end, for the first time: first clean, with the CASE_SET exactly as it stands — five for five, PASS; then with a real regression substituted into one of the five cases — four for five, FAIL, with the exact case and the exact reason flagged.

Connection to the module

This lesson assembles CaseResult, GateReport, run_case, and run_regression_gate — regression/harness.py's last four pieces. With them complete, the module has, for the first time, an artifact that runs end to end over the complete CASE_SET, ready for lesson 08's mini-project and for Module 7's version comparison.

CaseResult and GateReport: the verdict, structured

from dataclasses import dataclass, field


@dataclass
class CaseResult:
    """El veredicto de FORMA de un solo caso -- cuatro chequeos deterministas,
    ninguno un juicio de calidad."""
    name: str
    passed: bool
    actual_tools: list = field(default_factory=list)
    tool_choice_ok: bool = True
    schema_errors: list = field(default_factory=list)
    output_errors: list = field(default_factory=list)
    cost_cents: int = 0
    cost_ok: bool = True
    latency_ms: int = 0
    latency_ok: bool = True


@dataclass
class GateReport:
    """El veredicto del CASE_SET completo -- PASS solo si los N casos
    pasan los cuatro chequeos, como un gate de CI."""
    cases: list = field(default_factory=list)
    passed: bool = True

Notice, carefully, what fields CaseResult carries: not a single opaque boolean, but the complete detail of each check — tool_choice_ok and actual_tools separately, schema_errors and output_errors as lists (empty if everything passed), cost_cents/cost_ok and latency_ms/latency_ok as value-verdict pairs. This structure is what makes it possible for a FAIL, in the next section, to flag exactly what broke — never a plain "something failed" with no clue at all.

run_case: the four questions, in a single case

def run_case(case, sequence_number, model_script=None):
    """Corre UN caso del CASE_SET: ejecuta run_reservo_agent (sin tocar su
    lógica) con `model_script` (el del caso, o uno sustituido -- la pieza
    que M7 reusa para comparar una versión vieja contra una nueva), y
    aplica los cuatro chequeos de FORMA."""
    reset_reservo_state()
    script = model_script if model_script is not None else case["model_script"]

    with rl.traced_run(case["question"], sequence_number) as trace_id:
        final, history = ra.run_reservo_agent(case["question"], script)

    tool_choice_ok, actual_tools = check_tool_choice(history, case["expected_tools"])

    schema_errors = []
    output_errors = []
    if tool_choice_ok:
        target_tool = case["expected_tools"][-1]
        raw_result = _last_result_for_tool(history, target_tool)
        target_result = json.loads(raw_result)
        schema_errors = check_schema(target_result, OUTPUT_SCHEMAS[target_tool])
        output_errors = check_expected_output(target_result, case["expected_output"])

    cost_report = cost_for_run(trace_id, case["question"], history)
    latency_ms = latency_for_run(history)
    cost_ok = check_cost_threshold(cost_report.cost_cents, case["cost_threshold_cents"])
    latency_ok = check_latency_threshold(latency_ms, case["latency_threshold_ms"])

    passed = tool_choice_ok and not schema_errors and not output_errors and cost_ok and latency_ok
    return CaseResult(
        name=case["name"], passed=passed, actual_tools=actual_tools, tool_choice_ok=tool_choice_ok,
        schema_errors=schema_errors, output_errors=output_errors,
        cost_cents=cost_report.cost_cents, cost_ok=cost_ok,
        latency_ms=latency_ms, latency_ok=latency_ok,
    )

Notice a real decision, marked with if tool_choice_ok:: the form and output-value checks only make sense if the tool that's expected to be validated actually got called. If the agent chose a different tool than expected — like in lesson 05's FAIL — there's no result of the expected tool to look for inside history (_last_result_for_tool would return None, and json.loads(None) would raise a TypeError, not a controlled FAIL). Instead of letting that programming error propagate, run_case cuts it off right there: if the chosen tool is already wrong, the case is an immediate FAIL, with no need — or point — in continuing to validate a result that was never produced. passed is the conjunction of five conditions: correct tool, no schema errors, no value errors, cost under threshold, latency under threshold — any one of the five can fail the entire case.

run_regression_gate: the complete batch, with optional overrides

def run_regression_gate(case_set, overrides=None):
    """El gate: corre cada caso del CASE_SET fijo y agrega un PASS/FAIL
    global -- como un gate de CI. `overrides` (opcional) sustituye el
    model_script de casos puntuales por nombre -- la misma técnica de
    run_case, aplicada al lote completo, para comparar 'la versión de
    antes' contra 'la versión de después' (la pieza que M7 reusa)."""
    overrides = overrides or {}
    cases = [
        run_case(case, i, model_script=overrides.get(case["name"]))
        for i, case in enumerate(case_set, start=1)
    ]
    return GateReport(cases=cases, passed=all(c.passed for c in cases))

overrides is an optional dict, {case_name: substituted_script} — for any case that doesn't appear in that dictionary, overrides.get(case["name"]) returns None, and run_case uses the case's original script, unchanged. This lets you run the complete CASE_SET with a single case substituted, without having to rebuild the other four by hand — exactly how Module 7 is going to compare an old agent version against a new one: the same CASE_SET, the same criteria, and only the behavior under test changes.

Worked example, part 1: the clean gate — the complete batch's PASS

report = run_regression_gate(CASE_SET)
print("GATE:", "PASS" if report.passed else "FAIL", f"({sum(c.passed for c in report.cases)}/{len(report.cases)})")
for c in report.cases:
    print(f"  {c.name:38} {'PASS' if c.passed else 'FAIL'}")

What to expect:

GATE: PASS (5/5)
  quote_focus_pro_3h                     PASS
  quote_focus_basic_3h                   PASS
  book_focus_pro_3h_ana                  PASS
  book_boardroom_pro_1h_sofia            PASS
  book_and_cancel_studio_basic_1h_diego  PASS

Five for five. report.passed is True because all(c.passed for c in report.cases) — the same "one broken case breaks the entire batch" discipline any real CI gate applies: there's no "partial PASS" in this design. If even one of the five cases fails, the entire build fails.

Worked example, part 2: the gate with a regression — the complete batch's FAIL

Now, the same CASE_SET, with no change at all, but with quote_focus_pro_3h's script substituted for lesson 05's "after the change" script — the one that skips quoting and books directly:

regressed_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro 3h para Ana."}]},
]
report2 = run_regression_gate(CASE_SET, overrides={"quote_focus_pro_3h": regressed_script})
print("GATE:", "PASS" if report2.passed else "FAIL", f"({sum(c.passed for c in report2.cases)}/{len(report2.cases)})")
for c in report2.cases:
    status = "PASS" if c.passed else "FAIL"
    line = f"  {c.name:38} {status}"
    if not c.passed:
        line += f"  -- tool_choice_ok={c.tool_choice_ok} actual_tools={c.actual_tools}"
    print(line)

What to expect:

GATE: FAIL (4/5)
  quote_focus_pro_3h                     FAIL  -- tool_choice_ok=False actual_tools=['book_room']
  quote_focus_basic_3h                   PASS
  book_focus_pro_3h_ana                  PASS
  book_boardroom_pro_1h_sofia            PASS
  book_and_cancel_studio_basic_1h_diego  PASS

Four for five. report2.passed is False — the entire build fails, even though only one of the five cases broke. The message flags, with no ambiguity, which one: quote_focus_pro_3h, with tool_choice_ok=False and actual_tools=['book_room'] — the exact same information any real test suite's FAIL message would need to give to be useful: which case, which check, what got obtained instead of what was expected. The other four cases, with no change at all, stay PASS — proof that overrides only affects the named case, with no side effect on the rest of the CASE_SET.

Reading a GateReport the way you read a real CI result

It's worth pausing on the parallel, because it isn't accidental: a real CI gate — the one that runs on any pull request in a serious repository — does exactly this same thing, at a bigger scale. It runs a fixed set of tests, each with a binary criterion, and aggregates a single verdict: green (everything passed, the change can be merged) or red (something failed, it needs fixing first). Nobody expects a CI to "judge" whether the code is elegant — they expect it to confirm, with mechanical certainty, that nothing that already worked stopped working. run_regression_gate is that same discipline, applied to an LLM agent's behavior instead of to a traditional piece of software's behavior. Lesson 08 is going to show how that verdict gets saved to a file, regression_report.json, exactly like any real CI leaves an artifact behind — a log, a coverage report — so someone can review it later, without having to run anything again.

And, as in every lesson in this module, one last precision before moving on: a GateReport.passed=False is, always, a FORM verdict — chosen tool, schema, threshold — never a judgment on whether the agent's response was good. A GateReport in PASS doesn't certify quality either — it only certifies that nothing this gate knows how to check broke. The question of whether the agent, beyond form, is responding well remains, throughout this entire guide, evaluation-frameworks-guide's territory.

Common mistakes

1. Thinking a GateReport.passed=False means the change has to be reverted with no further analysis. The gate flags that something changed — the human work that follows is deciding whether that change is a real regression (revert or fix the prompt) or an intentional behavior change (update the CASE_SET to reflect the new expectation, with full knowledge of the reason).

2. Forgetting overrides only affects the named case, and expecting it to "contaminate" the rest. Worked example, part 2, demonstrates this explicitly: the other four cases stay PASS, exactly like in the clean gate — overrides.get(case["name"]) only returns something other than None for the case whose name matches.

3. Running run_case directly instead of run_regression_gate when you need the complete batch's verdict. run_case gives one case's detail; run_regression_gate is the one that aggregates the global passed with all(...). Confusing the two — for example, checking only the first CaseResult and assuming it represents the whole batch — loses exactly the property that makes a gate useful: the verdict is about the whole set, not a sample.

4. Ignoring schema_errors/output_errors when tool_choice_ok is already False. As the run_case section noted, those two lists stay empty ([]) when the chosen tool is already wrong — an empty list in that context doesn't mean "the form is correct," it means "the form never got checked, because the first check already failed." Reading schema_errors == [] as "the schema passed" without also checking tool_choice_ok can lead to a wrong conclusion.

5. Thinking adding more cases to the CASE_SET automatically makes the gate "better." Every new case adds coverage over a specific behavior, but it also adds run time and maintenance surface (every case needs its expected_output reviewed if business logic legitimately changes). Deciding which cases to include is, itself, a design decision — covering every tool at least once, every known price anchor, and every critical multi-step sequence, like this module's five-case CASE_SET does, is a reasonable starting point, not a universal formula.

Exercises

Exercise 1: Trigger a FAIL with overrides on a different case (Easy)

Using lesson 05's Exercise 1 "no-cancel" script, run run_regression_gate with overrides={"book_and_cancel_studio_basic_1h_diego": no_cancel_script}. Confirm the batch's verdict and the failed case's detail.

no_cancel_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Studio", "tier": "basic", "hours": 1, "member": "Diego"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Studio basic 1h para Diego."}]},
]
report = run_regression_gate(CASE_SET, overrides={"book_and_cancel_studio_basic_1h_diego": no_cancel_script})
print("GATE:", "PASS" if report.passed else "FAIL", f"({sum(c.passed for c in report.cases)}/{len(report.cases)})")
broken = next(c for c in report.cases if not c.passed)
print("caso roto:", broken.name, "tools obtenidas:", broken.actual_tools)

GATE: FAIL (4/5)
caso roto: book_and_cancel_studio_basic_1h_diego tools obtenidas: ['book_room']

Exercise 2: Simulate two regressions at once (Medium)

Run run_regression_gate with overrides substituting two cases at once: quote_focus_pro_3h (with this lesson's regressed script) and book_and_cancel_studio_basic_1h_diego (with Exercise 1's "no-cancel" script). Confirm the gate reports exactly 3/5, with both broken cases identified.

report = run_regression_gate(CASE_SET, overrides={
    "quote_focus_pro_3h": regressed_script,
    "book_and_cancel_studio_basic_1h_diego": no_cancel_script,
})
print("GATE:", "PASS" if report.passed else "FAIL", f"({sum(c.passed for c in report.cases)}/{len(report.cases)})")
for c in report.cases:
    if not c.passed:
        print("  roto:", c.name)

GATE: FAIL (3/5)
  roto: quote_focus_pro_3h
  roto: book_and_cancel_studio_basic_1h_diego

Exercise 3: Build a "why it failed" summary grouped by check type (Hard)

Using worked example part 2's GateReport (the quote_focus_pro_3h regressed script), write a summarize_failures(report) function that, for every case with passed=False, classifies the cause into one of four categories: "tool_choice", "schema", "output", "threshold" (this last one if either cost_ok or latency_ok is False). A case can have more than one category if it failed for more than one reason.

def summarize_failures(report):
    """Clasifica, por caso, en cuáles de las cuatro categorías de chequeo
    falló -- útil para un resumen rápido de un GateReport grande."""
    summary = {}
    for c in report.cases:
        if c.passed:
            continue
        reasons = []
        if not c.tool_choice_ok:
            reasons.append("tool_choice")
        if c.schema_errors:
            reasons.append("schema")
        if c.output_errors:
            reasons.append("output")
        if not c.cost_ok or not c.latency_ok:
            reasons.append("threshold")
        summary[c.name] = reasons
    return summary

print(summarize_failures(report2))

{'quote_focus_pro_3h': ['tool_choice']}

Summary and next step

• We built CaseResult and GateReport, the structures that capture a case's and the complete batch's detailed verdict.
• We built run_case, which applies the four comparisons (tool, schema, value, thresholds) over a case, and run_regression_gate, which aggregates them into a PASS/FAIL verdict over the complete CASE_SET — with an all(...) that makes a single broken case bring down the entire build, the same discipline as any real CI gate.
• We ran the clean gate: GATE: PASS (5/5), the CASE_SET's five cases with no change at all.
• We ran the gate with a regression substituted via overrides: GATE: FAIL (4/5), with the exact case (quote_focus_pro_3h) and the exact reason (tool_choice_ok=False, actual_tools=['book_room']) flagged with no ambiguity.

Next lesson: 08 — Mini-Project: A Regression Gate for Reservo. We assemble a complete regression/harness.py and regression/golden_cases.json in a single directory, run the gate end to end, and produce regression_report.json — the artifact Module 7 is going to reuse to decide whether a new agent version is ready for production.

Additional resources

1. Python — dataclasses — CaseResult and GateReport, and field(default_factory=list), the same technique already used in StepCost/CostReport (Module 3) to avoid the classic shared-mutable-default error.
2. Python — the all() function — The exact aggregator behind GateReport.passed: True only if every element of the sequence is.
3. Python — dict.get with a default value — The foundation of overrides.get(case["name"]), which returns None (and therefore "uses the original script") for any non-substituted case.
4. Anthropic — Building effective agents — On why a repeatable verification procedure, run before every change, is a core practice of any reliable agentic system in production.
5. Python 3.14 — What's New — The version every line of code in this lesson ran on, including the gate's two complete verdicts.