Module 5: Regression Evals as a Production Gate

Description

The three previous modules built, one by one, the ability to know what happened to a Reservo agent run: Module 2 gave it a trace_id and a complete, step-by-step trail in RUN_LOG.jsonl. Module 3 put a price on it, in cents. Module 4 put a modeled time on it, tool by tool. With all three pieces in place, you can today take any Reservo agent run and answer, with real numbers: how many steps it took, what it cost, how long it took. That's observe and measure — the first two disciplines of the map Module 1 opened.

But none of those three pieces answers a different, more urgent question that anyone who's already touched a production system knows well: "I changed something — a system prompt, a tool description, a model — is the agent still working the same as yesterday?" Measuring a run, one at a time, after it already happened, doesn't answer that. What's needed is something that runs before a change reaches production, over a set of cases you already know, and tells you, with a binary verdict, whether something broke. That's a regression gate — this guide's third discipline: gate.

This module builds exactly that: regression/harness.py, a deterministic harness that runs a fixed set of cases — regression/golden_cases.json — against the Reservo agent exactly as agent-fundamentals-and-tool-calling-guide M8 left it built, and produces a PASS/FAIL verdict for each case and for the whole batch. Nothing you build here calls a model to grade anything. Every check is a literal, deterministic comparison against a fixed value: does the tool's result have the right shape? did the agent call exactly the tool that was expected, in the expected order? did the modeled cost and latency stay under a threshold? This module is, quite intentionally, the narrowest-scope piece in the entire guide — and the easiest to confuse with something it isn't. Clearing up that confusion, precisely, matters as much as the code that builds the gate.

Connection to the module

This module is the first in the guide that doesn't add a new way to measure — it reuses, without changing a single line, everything you already built: reservo_agent.run_reservo_agent (agent-fundamentals M8), run_logger.traced_run (Module 2), cost_calculator.estimate_cost_cents and cost_for_run (Module 3), and the per-tool latency model (Module 4). What's new is the layer that decides, using those already-built pieces, whether the agent's behavior is still the behavior that's expected.

Where we are in the ecosystem

Agents in production — operating the Reservo agent
├── Module 1: Why Operating Is Different From Building
├── Module 2: Structured Logging and Tracing a Run
├── Module 3: Measuring Cost and Tokens per Run
├── Module 4: Measuring Latency Honestly
├── Module 5: Regression Evals as a Production Gate  ← YOU ARE HERE
│   → A fixed set of cases, literal comparison of the chosen
│     tool, output schema, cost/latency threshold, PASS/FAIL
│     as a CI gate
├── Module 6: Failures at Scale — Backoff, Circuit Breakers, and Rate Limits
├── Module 7: Versioning and Safe Rollout
└── Module 8: Project — the Reservo Agent in Production

With the guide's first three disciplines complete (Module 1 named the problem; Modules 2-4 built how to observe and measure), this is the module that turns that measurement into a decision: PASS or FAIL, with no ambiguity, no subjective criteria, with nobody having to read an entire run by hand to decide whether something broke. Module 6 is going to use the same "something broke" criterion but applied to a specific tool's repeated failures across runs (a circuit breaker); Module 7 is going to reuse this exact gate, literally, to compare an old agent version against a new one before deciding whether a change gets shipped. Everything you build here is the foundation for both.

This module's central analogy: the vehicle inspection, before the car hits the road

Before a car can legally drive, it goes through a vehicle inspection. The inspector doesn't sit at the wheel to judge whether the car is comfortable, whether the dashboard looks elegant, or whether the color matches the owner's taste — none of that is their job, and they don't even have a criterion to judge it by. What they do is run a fixed list of checks, each with a binary criterion: do the brakes respond within the expected distance? do the lights turn on? do the seatbelts hold? Every check has an exact threshold, measured with an instrument, not an opinion. If the car passes all fifteen items on the list, it leaves with the sticker. If it fails even one — even the "most minor" of all, like a burned-out taillight — it doesn't leave, and the shop has to fix exactly that item before trying again.

A regression gate does, with an LLM agent, exactly what that inspection does with a car. It doesn't evaluate whether the agent's response is "good," "clear," or "well written" — that isn't what this module measures, and it doesn't even have a criterion to attempt it (that evaluation, semantic-quality evaluation, is a different discipline's job, which this lesson names precisely further below). What it does is run a fixed list of cases, each with a binary criterion, measured with code: does the tool's result have the right shape (the output "schema," like the sensor that confirms the lights turn on)? did the agent call the right tool, in the right order (like confirming the steering wheel turns the right wheel, not a test of whether driving it "feels good")? did the modeled cost and latency stay under the expected threshold (like braking distance, a number, not an impression)? If the agent passes all five cases in the fixed set, the build passes. If it fails even one — even the "simplest" of the five — the gate fails, and that specific case, with its exact reason, is what needs fixing before trying again. A prompt change that breaks tool choice is, quite precisely, the equivalent of brakes that stopped responding: the gate stops it before it reaches the road.

What this module builds, and what it deliberately doesn't touch

Before writing a single line of code, it's worth being precise about what this module adds to what you already have:

regression/golden_cases.json — the fixed set of cases: real questions from Reservo's domain, each with the model's turn script (concept, already handwritten, as throughout this guide) and the exact expected result. Fixed means fixed: nobody generates new cases at random, nobody updates them "by eye" — it's the same file, run over and over, every time something changes.
regression/harness.py — the functions that run each case, apply the checks, and aggregate a verdict: check_schema (is the result's shape correct?), check_tool_choice (does the chosen tool literally match the expected one?), check_cost_threshold and check_latency_threshold (are the modeled cost and latency under the case's fixed threshold?), and run_regression_gate, the function that runs the complete CASE_SET and returns a GateReport with each case's and the whole batch's PASS/FAIL.
Nothing in reservo_tools.py, reservo_contracts.py, reservo_robust.py, or reservo_agent.py changes. This module, like each one before it, wraps the agent agent-fundamentals already built — it never rewrites its logic.
Nothing in run_logger.py or cost_calculator.py changes either. Every case in the gate runs under traced_run (Module 2, unmodified) and its cost is calculated with cost_for_run (Module 3, unmodified). The per-tool latency model this module uses is the same TOOL_LATENCY_MS already established and run since Module 1.

🛑 This entire module's most important boundary: FORM, never quality

This is the point that needs to be completely clear before moving on, because it is, precisely, the most dangerous overlap in the entire agentic guide ecosystem — and a confusion here contaminates every lesson that follows.

What this module checks is FORM. Three questions, and only three, all answerable with an exact comparison against a fixed value, with no interpretation involved:

Does the output validate against its output schema? — did get_quote return a dict with an integer price_cents key, or did it return something with a different shape?
Does the agent still choose the correct tool, for an exact scripted turn? — a literal comparison: is the sequence of tools the agent called exactly equal, element by element, to the expected sequence?
Do the modeled cost and latency stay under a fixed threshold? — one number compared against another, with <=.

What this module NEVER checks is whether the response is good. Nowhere in this module's files is there a call to a language model to "opine" on a response's quality. There's no dataset with a "roughly correct" ground truth that a judge has to interpret. There's no score from 1 to 10, no "does this response sound natural?", no trajectory scoring that evaluates whether the path the agent took was reasonable beyond the literal comparison of which tool it called. That evaluation — semantic-quality evaluation, "does the agent's response actually answer what the user asked well?" — is real, important work, and completely different, and it belongs to evaluation-frameworks-guide: its evaluating-agents module covers trajectory evaluation, tool-call accuracy judged (not compared literally, but evaluated with a "was this a good decision?" criterion), and task-completion/reasoning-quality; its evaluation-pipelines-in-production module covers golden datasets with fuzzy ground truth, LLM-as-judge, and RAGAS.

Hold onto this sentence, because every lesson in this module repeats it in a different context: this gate confirms the form didn't break — schema, correct tool, threshold — never whether the response is good. That's evaluation-frameworks-guide.

The case that keeps accompanying the guide: Reservo, with a fixed case set

The four tools are the same as always — list_rooms(), get_quote(room, tier, hours), book_room(room, tier, hours, member), cancel_booking(id) — and the price anchors remain intact: Focus basic 3h = 7500 cents, Focus pro 3h = 6000 cents. This module doesn't declare a single new tool or change Reservo's business logic — it builds, around it, a set of five fixed cases covering quoting (with both anchors), booking (with two different rooms), and booking-and-canceling. Each case gets anchored, besides to the correct tool, to an exact output value: price_cents=6000 for Focus pro 3h, booking_id=1 for a freshly reset run's first booking, cancelled=True for a valid cancellation.

As in every module of this guide: identifiers and code in English; prose and comments in Spanish; money, whenever it appears, in int cents. Current models (claude-sonnet-5) whenever a lesson mentions the model as a concept.

Prerequisites

Required knowledge:

✅ Having completed Modules 1-4 of this guide. This module assumes RUN_LOG.jsonl, estimate_cost_cents/cost_for_run, and the TOOL_LATENCY_MS model already exist and work — it doesn't re-explain them from scratch.
✅ Having completed (or knowing well) agent-fundamentals-and-tool-calling-guide M8: run_reservo_agent(question, model_script, max_iterations=10, summarize=None)'s signature, history's format, and the four tool contracts with their input_schemas.
✅ Python: functions, dict/list, dataclasses, json.load/json.dumps. Nothing in this module uses any library outside the standard one.

NOT required:

❌ You don't need an API key or an internet connection: the model's decision is still concept, and this module's entire harness runs 100% locally.
❌ You don't need to know any semantic-evaluation framework (RAGAS, TruLens, LangSmith Evals). This module's patterns are deliberately simpler — and that simplicity is the point: a FORM gate doesn't need any of those tools.
❌ You don't need to know anything about real CI/CD (GitHub Actions, deployment pipelines). This module builds the PASS/FAIL criterion in pure Python — how to wire it into a real pipeline is an infrastructure decision outside this guide's scope.

Environment:

✅ Python 3.14.0 with its standard library (json, dataclasses, itertools). Nothing to install.
✅ A text editor and a terminal.

Module roadmap

Lesson 01 — Module introduction (this one)

The vehicle-inspection analogy, the FORM-vs-quality boundary declared precisely, and the map of the eight lessons.

Lesson 02 — What a Regression Eval Checks

The three exact questions this gate can answer, with run examples of each in isolation, before joining them into a complete harness.

Lesson 03 — The Fixed Case Set

regression/golden_cases.json: why fixed, how each case is structured, and the discipline of isolating Reservo's state between cases so none depends on the order the others ran in.

Lesson 04 — Form, Not Quality: the Boundary

The central lesson on the boundary with evaluation-frameworks-guide: check_schema built and thoroughly tested, with cases that pass and cases that fail, and the explicit statement, with examples, of what questions this module never answers.

Lesson 05 — Checking Tool Choice

check_tool_choice: literal comparison of the sequence of tools called, and the first complete demonstration of a real FAIL — a prompt regression that changes which tool gets chosen.

Lesson 06 — Checking Cost and Latency Thresholds

check_cost_threshold and check_latency_threshold, reusing cost_for_run (Module 3) and the latency model (Module 4) without modifying them, with a case that fails by exceeding a threshold.

Lesson 07 — The Gate: Pass or Fail the Build

A complete run_regression_gate, running the five-case CASE_SET end to end: the clean batch's PASS verdict, and the FAIL verdict when a script gets swapped for one "after a change" that breaks tool choice.

Lesson 08 — Mini-Project: A Regression Gate for Reservo

You assemble a complete regression/harness.py, run the gate over the real CASE_SET, and produce regression_report.json — the artifact Module 7 is going to reuse to compare two agent versions.

Progression map

Lesson 01 (this)  → The analogy, the FORM-vs-quality boundary, the map
Lesson 02         → The three questions this gate can answer
Lesson 03         → golden_cases.json: the fixed set, isolated between cases
Lesson 04         → check_schema, and the boundary with evaluation-frameworks
Lesson 05         → check_tool_choice, and the first real FAIL
Lesson 06         → check_cost_threshold / check_latency_threshold
Lesson 07         → run_regression_gate: PASS/FAIL for the complete batch
Lesson 08         → Mini-project: a real regression_report.json

Difficulty: ⭐⭐ ──────────────────▶ ⭐⭐⭐

What you'll achieve in this module

By completing the 8 lessons, you'll be able to:

Explain, precisely and with examples, the difference between a FORM check and a semantic-quality judgment — and say, for any new question about an agent, which of the two categories it belongs to.
Design a fixed set of regression cases, each with its expected tool, its output schema, and its cost/latency thresholds.
Build check_schema: validate a tool result's shape against an output schema, without evaluating whether the value "makes sense."
Build check_tool_choice: literally compare the sequence of tools an agent called against the expected sequence.
Build check_cost_threshold and check_latency_threshold, reusing Modules 3 and 4's cost and latency engineering without duplicating it.
Run a complete regression gate over the Reservo agent, read a GateReport, and diagnose exactly why a specific case failed.
Trace, without hesitation, the boundary with evaluation-frameworks-guide every time a question about an agent starts to sound like "is the response good?" instead of "did the form break?"

Before and after

BEFORE the module:
→ "if the agent responds with something reasonable, it's fine"
→ "testing a change means running the agent by hand a couple of
  times and seeing if it 'looks good'"
→ "an eval always needs a model to judge the response"
→ "if something breaks, it'll show up in production eventually"

AFTER the module:
→ a regression gate is a BINARY, deterministic criterion, with
  no quality judgment involved
→ a FIXED set of cases, always run the same way, replaces "by
  eye" manual testing with a repeatable procedure
→ FORM (schema, chosen tool, threshold) and QUALITY (is the
  response good?) are two distinct questions, with two distinct
  disciplines -- this guide only solves the first one
→ a change that breaks something gets caught BEFORE production,
  with an exact message of which case failed and why

Traps to avoid in this module

1. "This module is going to call `claude-sonnet-5` to judge whether the response sounds good"

No, never. Every model call in this module stays concept — a handwritten turn script, as throughout this guide. The checks that do run for real (check_schema, check_tool_choice, check_cost_threshold, check_latency_threshold) are pure Python functions that compare values against other fixed values — they never invoke any model for anything.

2. "If the case set is fixed, it never catches new problems"

It's true a fixed set doesn't catch every problem — but that isn't its job. A regression gate detects whether a change broke a behavior that was already known to be expected to work. Detecting new problems, unanticipated by any existing case, is a different job (exploratory testing, production monitoring of real cases) that this guide doesn't cover in this module.

3. "A FAIL from this gate means the agent 'is broken'"

A FAIL means, precisely, that the behavior changed relative to what was expected — not necessarily that the new behavior is worse. Lesson 07 shows this with a real case: the gate can fail for a completely legitimate reason worth reviewing (someone changed a room's base price on purpose), or for a genuine regression (the prompt started skipping a step). The gate flags that something changed; deciding whether that change is intentional or a mistake stays human work.

4. "`check_tool_choice`'s literal comparison is 'too strict'"

It's strict on purpose. The literal comparison (actual == expected, element by element) is what makes the check deterministic and reliable — any form of "approximate" comparison (is the chosen tool "similar" to the expected one?) would, again, require some kind of judgment, and that's exactly the territory this module deliberately avoids.

5. "This is already the same as `evaluation-frameworks-guide`, so I can use its techniques here"

No. That guide solves a real and different problem — semantic quality, with golden datasets and judges — with its own stack (OpenAI, RAGAS, TruLens) that doesn't even share conventions with this guide. Mixing its techniques in here would break this module's central property: that every check is deterministic and reproducible, with no probabilistic component involved.

How to work through this module

Run each check on its own before joining them. Lessons 04-06 build check_schema, check_tool_choice, check_cost_threshold, and check_latency_threshold in isolation, with examples that pass and examples that deliberately fail — before lesson 07 joins them into a single gate.
Pay attention to the FAIL messages, not just the PASS. A useful GateReport doesn't just say "something failed" — it says exactly which case, which check, and with what values. Every example in this module that produces a FAIL shows that full detail.
The mini-project (lesson 08) is the complete, real gate. There you run the CASE_SET's five cases, see the clean batch's PASS, and confirm the FAIL when a regression gets simulated — the same procedure Module 7 is going to reuse to compare two agent versions.

Estimated time:

Lesson 01 (this)  →  20 min reading
Lesson 02         →  25 min + running the example
Lesson 03         →  25 min + running the example
Lesson 04         →  30 min + running the example (the central boundary)
Lesson 05         →  30 min + running the example
Lesson 06         →  25 min + running the example
Lesson 07         →  30 min + running the complete gate
Lesson 08         →  40 min + building the complete mini-project

Total: ~3.5 hours

Evidence of success

Before moving on to Module 6 (Failures at Scale), you should be able to:

✅ Explain, without hesitation, the boundary between a FORM check and a semantic-quality judgment, with one example of each.
✅ Build check_schema, check_tool_choice, check_cost_threshold, and check_latency_threshold, each with at least one passing case and one failing case.
✅ Run run_regression_gate over a real CASE_SET and read a complete GateReport, identifying which case failed and why.
✅ Simulate a regression (an "after a change" script that breaks tool choice) and confirm the gate catches it with a precise message.
✅ Name evaluation-frameworks-guide as the right guide every time a question about an agent starts to require a quality judgment, not just a form comparison.

Summary

This module builds the guide's third discipline — gate: a deterministic harness, regression/harness.py, that runs a fixed set of cases, regression/golden_cases.json, against the Reservo agent, and produces a PASS/FAIL verdict.
The central analogy is a car's vehicle inspection: it verifies the brakes respond and the lights turn on — the form didn't break — never whether the car "looks nice" — that's a different evaluation.
This entire module's most important boundary: this gate checks FORM (output schema, tool chosen via literal comparison, cost/latency under a fixed threshold) — never semantic quality. That evaluation belongs to evaluation-frameworks-guide, named precisely in every lesson that brushes up against its territory.
Everything this module builds reuses, without modifying, what already exists: run_reservo_agent (agent-fundamentals M8), traced_run (Module 2), cost_for_run (Module 3), and the per-tool latency model (Module 4).

Next lesson: 02 — What a Regression Eval Checks. Before building the complete harness, we put the three exact questions this gate can answer to the test, one by one — with run examples of each.

Additional resources

Anthropic — Tool use (function calling) overview — The tool_use/tool_result contract every case in this gate runs on, unchanged from agent-fundamentals.
Python — json — json.load/json.dumps, the foundation of golden_cases.json and of every GateReport this module produces.
Python — dataclasses — CaseResult and GateReport, the structures this module uses to represent each case's and the whole batch's verdict.
Anthropic — Building effective agents — On why a reliable agentic system needs a repeatable procedure for catching regressions, not just occasional manual testing.
Python 3.14 — What's New — The exact version every line of code in this module runs on.

← PreviousMini-Project: A Latency Report Next →What a Regression Eval Checks