What a Regression Eval Checks

Description

Lesson 01 named three questions and promised each one can be answered with an exact comparison, with no interpretation involved. This lesson delivers on that promise: before building the complete harness — the work of lessons 03 through 07 — you're going to see, run for real and in isolation, the three functions that end up being regression/harness.py's heart: check_schema, check_tool_choice, and the check_cost_threshold/check_latency_threshold pair. Each one is, deliberately, simpler than its name suggests — and that simplicity is exactly this module's point.

By the end of the lesson, you're going to bring all three together over a real Reservo case — still without the fixed CASE_SET or the gate's complete infrastructure, that's the work of the lessons that follow — and you're going to see, with your own eyes, how an entire run produces a PASS verdict with just four deterministic comparisons.

Connection to the module

This lesson delivers the first complete, working version of the four functions regression/harness.py is going to expose: check_schema, check_tool_choice, check_cost_threshold, check_latency_threshold. Lessons 04, 05, and 06 don't rewrite them — they integrate them with the fixed CASE_SET (lesson 03) and test them thoroughly, with more cases and with the failures each one is designed to catch.

The analogy, with the inspection's three instruments

Lesson 01 compared this module to a vehicle inspection: a fixed list of checks, each with a binary criterion, measured with an instrument. This lesson names each instrument:

• The light check is check_schema: a simple sensor that confirms the signal has the right shape — it turns on, or it doesn't — with no opinion on whether the light's design looks nice.
• The brake check is check_tool_choice: you press the pedal, and the car brakes exactly where expected, or it doesn't. There's no "braked more or less well" — either it stopped within the expected distance, or it didn't.
• The speedometer and odometer check is check_cost_threshold/check_latency_threshold: a number, read off an instrument, compared against a fixed limit. The speedometer doesn't "opine" on whether you're going fast — it simply reports a number, and the limit decides.

No inspector needs to drive the car to do these three checks. In the same way, none of the three functions you build in this lesson needs to understand what the question the user asked the Reservo agent was about — each one compares a shape, a sequence, or a number, against a fixed value.

Question 1: does the tool result have the right shape?

check_schema receives a result — the dict (or list) a Reservo tool returned — and an output schema — a description, with the same type/properties/required vocabulary you already used for agent-fundamentals's input_schemas, but now applied to what the tool returns, not what it receives. The function walks the schema, field by field, and collects a list of errors; an empty list means the result passed.

_PY_TYPE = {"string": str, "integer": int, "number": (int, float), "boolean": bool, "object": dict, "array": list}


def check_schema(result, schema):
    """Valida un resultado de tool contra su schema de SALIDA: ¿el tipo
    raíz coincide?, ¿los campos requeridos están?, ¿cada tipo coincide?
    FORMA, nunca contenido -- no evalúa si el valor es 'bueno'."""
    errors = []
    root_type = _PY_TYPE.get(schema.get("type"))
    if root_type and not isinstance(result, root_type):
        return [f"tipo raíz debe ser {schema['type']}, llegó {type(result).__name__}"]
    props = schema.get("properties", {})
    for name in schema.get("required", []):
        if name not in result:
            errors.append(f"falta el campo requerido '{name}'")
    for name, value in result.items():
        if name in props:
            expected = _PY_TYPE.get(props[name].get("type"))
            if expected and not isinstance(value, expected):
                errors.append(f"'{name}' debe ser {props[name]['type']}, llegó {type(value).__name__}")
    return errors

With get_quote's output schema — a dict with a single required field, price_cents, of integer type — three cases: one valid, one missing the field, one with the wrong type.

get_quote_schema = {"type": "object", "properties": {"price_cents": {"type": "integer"}}, "required": ["price_cents"]}

print("resultado válido    :", check_schema({"price_cents": 6000}, get_quote_schema))
print("resultado sin campo :", check_schema({}, get_quote_schema))
print("resultado tipo roto :", check_schema({"price_cents": "6000.00"}, get_quote_schema))

What to expect:

resultado válido    : []
resultado sin campo : ["falta el campo requerido 'price_cents'"]
resultado tipo roto : ["'price_cents' debe ser integer, llegó str"]

Notice something important about the third case: "6000.00" is, on any human reading, "the same price" as 6000 — but check_schema doesn't know that, and doesn't care. Its job is to confirm the type is the one the tool's contract promises (int, not str), not that the value "makes sense" to a human reader. That distinction — correct type versus reasonable value — is exactly the line between FORM and QUALITY that lesson 04 develops in depth.

Question 2: did the agent choose the correct tool?

check_tool_choice receives the history run_reservo_agent returns (without touching it) and a list of expected tool names, in order. It extracts the actual sequence of tools called, and compares it against the expected one with a single operator: ==.

def extract_tool_sequence(history):
    """La secuencia LITERAL de tools llamadas por el agente, en el orden en
    que las llamó -- sin juzgar si la elección fue 'razonable', solo
    registrarla."""
    return [
        block["name"]
        for turn in history if not isinstance(turn["content"], str)
        for block in turn["content"] if block["type"] == "tool_use"
    ]


def check_tool_choice(history, expected_tools):
    """Comparación LITERAL: la secuencia de tools obtenida == la secuencia
    esperada, exactamente, elemento por elemento. Nunca un score parcial."""
    actual = extract_tool_sequence(history)
    return actual == expected_tools, actual

With two single-step histories — one that calls get_quote, another that calls book_room instead:

history_a = [
    {"role": "user", "content": "x"},
    {"role": "assistant", "content": [{"type": "tool_use", "id": "t1", "name": "get_quote", "input": {}}]},
]
history_b = [
    {"role": "user", "content": "x"},
    {"role": "assistant", "content": [{"type": "tool_use", "id": "t1", "name": "book_room", "input": {}}]},
]

print("tool esperada get_quote, agente llamó get_quote:", check_tool_choice(history_a, ["get_quote"]))
print("tool esperada get_quote, agente llamó book_room:", check_tool_choice(history_b, ["get_quote"]))

What to expect:

tool esperada get_quote, agente llamó get_quote: (True, ['get_quote'])
tool esperada get_quote, agente llamó book_room: (False, ['book_room'])

check_tool_choice always returns a tuple: the boolean verdict, and the actual sequence — even when the verdict is False. That second part isn't a minor detail: it's what lets you, in lesson 05, build a FAIL message that says exactly which tool was expected and which one was gotten, instead of a plain "something changed" with no clue as to what.

Question 3: did cost and latency stay under the threshold?

These two functions are, on purpose, the simplest of the four — a single numeric comparison each, with no state and no additional logic:

def check_cost_threshold(cost_cents, threshold_cents):
    return cost_cents <= threshold_cents


def check_latency_threshold(latency_ms, threshold_ms):
    return latency_ms <= threshold_ms

print("costo 3 <= umbral 5       :", check_cost_threshold(3, 5))
print("costo 12 <= umbral 5      :", check_cost_threshold(12, 5))
print("latencia 185 <= umbral 250:", check_latency_threshold(185, 250))
print("latencia 185 <= umbral 100:", check_latency_threshold(185, 100))

What to expect:

costo 3 <= umbral 5       : True
costo 12 <= umbral 5      : False
latencia 185 <= umbral 250: True
latencia 185 <= umbral 100: False

There's no mystery in these two functions — and that's, precisely, this lesson's argument: not everything a reliable gate needs has to be complicated. cost_cents comes from cost_for_run (Module 3, untouched); latency_ms comes from the per-tool latency model already established since Module 1 and developed in depth in Module 4. The only new piece here is the <= that decides whether that number, already calculated by another module, is within budget.

Worked example: the three questions, over a real run

With the four functions already confirmed separately, bring them together over a real run — "How much does Focus pro 3h cost?", a single call to get_quote, wrapped in traced_run like any run in this guide:

import json
import reservo_agent as ra
import run_logger as rl
from cost_calculator import cost_for_run

question = "¿Cuánto cuesta Focus pro 3h?"
script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Focus pro 3h cuesta $60.00."}]},
]
with rl.traced_run(question, 1) as trace_id:
    final, history = ra.run_reservo_agent(question, script)

# Pregunta 2 primero: ¿llamó a la tool correcta?
tool_ok, actual_tools = check_tool_choice(history, ["get_quote"])

# Pregunta 1: el resultado de esa tool, ¿tiene la forma correcta?
result = json.loads(_last_result_for_tool(history, "get_quote"))  # helper de la lección 04
schema_errors = check_schema(result, get_quote_schema)

# Pregunta 3: costo y latencia, bajo umbral
report = cost_for_run(trace_id, question, history)
latency_ms = latency_for_run(history)  # helper de la lección 06
cost_ok = check_cost_threshold(report.cost_cents, 5)
latency_ok = check_latency_threshold(latency_ms, 100)

print("pregunta 1 (schema)      :", schema_errors == [], schema_errors)
print("pregunta 2 (tool choice) :", tool_ok, actual_tools)
print("pregunta 3a (costo)      :", cost_ok, f"{report.cost_cents} <= 5 centavos")
print("pregunta 3b (latencia)   :", latency_ok, f"{latency_ms} <= 100 ms")
print("PASS del caso            :", schema_errors == [] and tool_ok and cost_ok and latency_ok)

What to expect:

pregunta 1 (schema)      : True []
pregunta 2 (tool choice) : True ['get_quote']
pregunta 3a (costo)      : True 0 <= 5 centavos
pregunta 3b (latencia)   : True 25 <= 100 ms
PASS del caso            : True

Four deterministic comparisons, no call to a model, and a clear verdict. _last_result_for_tool and latency_for_run are two helper functions — one to find a specific tool's last successful result inside history, the other to sum each step's modeled latency — that lessons 04 and 06 build and explain in detail; here they're used by name so you can see the complete flow before diving into each piece. The rest of this module, lessons 03 through 07, is the disciplined construction of exactly this same pattern, applied to a fixed CASE_SET of five cases instead of a single handwritten one.

Why these three questions, and no others

It's worth asking why this module's gate stops at exactly these three questions, and not, say, "does the final response sound natural?" or "did the agent solve the task the most efficient way possible?" The answer has to do with a property the three questions share, and that no quality question has: each one can be answered with a pure function, with no state, no randomness, and no component that could vary between two identical runs. check_schema({"price_cents": 6000}, schema) is going to return exactly [] today, tomorrow, and a year from now, no matter who runs it. That property — total determinism — is what makes it possible for a regression gate to run thousands of times, in a CI pipeline, with nobody having to manually review any result. A question like "does it sound natural?" doesn't have that property: two human readings of the same response can disagree, and no purely syntactic criterion can resolve that disagreement. That's why that question belongs to a different discipline — evaluation-frameworks-guide — with different tools, specifically designed to deal with that kind of ambiguity.

Common mistakes

1. Thinking check_schema "knows" what each tool's correct schema is. It doesn't — it always receives the schema as an argument. Deciding which schema applies to which result is the calling code's responsibility, not the function's own. This lesson passes it by hand (get_quote_schema); lesson 04 builds the OUTPUT_SCHEMAS dictionary that centralizes that decision for Reservo's four tools.

2. Mixing up check_tool_choice's argument order. The function expects (history, expected_tools) — the real result first, the expectation second. Swapping them doesn't produce a Python error (both are lists/structures compatible with the comparison), but it does produce a confusing diagnostic message if, later on, someone tries to print "expected tool X, got Y" with the values swapped.

3. Using < instead of <= in the threshold checks. A threshold that's touched exactly (cost_cents == threshold_cents) is, by this guide's design, a PASS — the threshold is the maximum acceptable limit, not a forbidden value. check_cost_threshold(5, 5) returns True, not False.

4. Forgetting that check_schema, on a wrong root type, returns immediately, without continuing to check fields. If result isn't the type schema["type"] expects (for example, a list instead of a dict), it makes no sense to keep looking for fields inside a structure that isn't even the right type — the function cuts off right there, with a single error message, instead of trying to iterate over something that might not be iterable the expected way.

5. Thinking these four functions "already are" the complete gate. They're the pieces — the complete gate additionally needs a fixed CASE_SET declaring what's expected of each case (lesson 03), and a function that brings them together and aggregates a verdict per case and per batch (lesson 07). Confusing the loose pieces with the complete system is getting ahead of the lessons that follow.

Exercises

Exercise 1: Validate book_room's schema (Easy)

book_room's output schema is {"type": "object", "properties": {"booking_id": {"type": "integer"}, "confirmed": {"type": "boolean"}}, "required": ["booking_id", "confirmed"]}. Use check_schema to validate three results: one valid ({"booking_id": 1, "confirmed": True}), one with confirmed as a string ("true" instead of True), and one missing booking_id.

book_room_schema = {
    "type": "object",
    "properties": {"booking_id": {"type": "integer"}, "confirmed": {"type": "boolean"}},
    "required": ["booking_id", "confirmed"],
}

print(check_schema({"booking_id": 1, "confirmed": True}, book_room_schema))
print(check_schema({"booking_id": 1, "confirmed": "true"}, book_room_schema))
print(check_schema({"confirmed": True}, book_room_schema))

[]
["'confirmed' debe ser boolean, llegó str"]
["falta el campo requerido 'booking_id'"]

Exercise 2: Build a three-tool sequence and check two different expectations (Medium)

Build a history (by hand, without running the agent) representing the sequence list_rooms → get_quote → book_room. Use check_tool_choice twice: once with the correct sequence as the expectation, once with ["get_quote", "book_room"] (without list_rooms) as the incorrect expectation.

history_three = [
    {"role": "user", "content": "x"},
    {"role": "assistant", "content": [{"type": "tool_use", "id": "t1", "name": "list_rooms", "input": {}}]},
    {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t1", "content": "[]"}]},
    {"role": "assistant", "content": [{"type": "tool_use", "id": "t2", "name": "get_quote", "input": {}}]},
    {"role": "user", "content": [{"type": "tool_result", "tool_use_id": "t2", "content": "{}"}]},
    {"role": "assistant", "content": [{"type": "tool_use", "id": "t3", "name": "book_room", "input": {}}]},
]

print(check_tool_choice(history_three, ["list_rooms", "get_quote", "book_room"]))
print(check_tool_choice(history_three, ["get_quote", "book_room"]))

(True, ['list_rooms', 'get_quote', 'book_room'])
(False, ['list_rooms', 'get_quote', 'book_room'])

Exercise 3: Design cancel_booking's output schema and test the three edge cases (Hard)

cancel_booking returns {"cancelled": bool}. (a) Write its output schema. (b) Test check_schema against: a valid result, a result with an extra key not declared in the schema ({"cancelled": True, "note": "ok"}), and a result with cancelled missing. (c) For the extra-key case: explain, in one sentence, why check_schema — as written in this lesson — reports no error for that key, and whether that's the right design decision for a FORM check.

cancel_booking_schema = {"type": "object", "properties": {"cancelled": {"type": "boolean"}}, "required": ["cancelled"]}

print(check_schema({"cancelled": True}, cancel_booking_schema))
print(check_schema({"cancelled": True, "note": "ok"}, cancel_booking_schema))
print(check_schema({}, cancel_booking_schema))

[]
[]
["falta el campo requerido 'cancelled'"]

Summary and next step

• We built the four functions that are going to be regression/harness.py's heart: check_schema (is the result's shape correct?), check_tool_choice (does the tool sequence literally match?), check_cost_threshold and check_latency_threshold (is the number under the limit?).
• We confirmed, run for real, that all four are pure, deterministic functions: same input, always the same output, with no probabilistic component.
• We brought all four together over a real Reservo run, and produced the whole module's first PASS verdict, with the four comparisons cited.
• We explained why these three questions — and not "does it sound natural?" or "was it efficient?" — are the ones this gate can answer: because, and only because, each one admits an exact comparison against a fixed value.

Next lesson: 03 — The Fixed Case Set. With the four functions already confirmed in isolation, we build regression/golden_cases.json: Reservo's fixed set of five cases this module is going to run, over and over, every time something changes.

Additional resources

1. Anthropic — Tool use (function calling) overview — The input_schema shape this module reuses, with the same vocabulary, to describe an output result's shape.
2. Python — isinstance — check_schema's central function, and why bool is, technically, a subclass of int in Python (something worth checking before blindly trusting a type check).
3. Python — sequence comparison — How Python compares two lists with ==: element by element, in order, with no notion of "subsequence" — the exact foundation of check_tool_choice.
4. Anthropic — Building effective agents — On why a reliable agentic system needs deterministic, repeatable checks, not just manual review.
5. Python 3.14 — What's New — The version every line of code in this lesson ran on.