Form, Not Quality: the Boundary

Description

This is the module's most important lesson, and one of the most important in the entire guide. It builds the missing piece of check_schema — the complete OUTPUT_SCHEMAS dictionary, with the four Reservo tools' output shape, and _last_result_for_tool, the function that finds which result to validate inside a multi-step history — and, with that piece finished, stops to trace, with all the precision this guide's DISEÑO demands, the exact boundary between what this module checks and what it never checks.

That boundary has a name: form versus quality. A form check asks "does this have the right structure?" — a question a program can answer with no ambiguity at all. A quality judgment asks "is this good?" — a question that, almost always, needs some kind of interpretive criterion, human or from a model trained to imitate that criterion. This module, in its entirety, lives on the form side. An LLM agent's semantic quality — does the response actually answer what was asked well? was the reasoning that led to that response sound? — is a real territory, covered in depth in a sister guide: evaluation-frameworks-guide. This lesson doesn't just name it once — it demonstrates it, with a run example that lets you see, with real numbers, exactly where what this module can answer stops.

Connection to the module

This lesson completes check_schema with OUTPUT_SCHEMAS, the dictionary for the four tools, and adds _last_result_for_tool and check_expected_output — the three pieces lesson 07 is going to use, unchanged, inside run_case.

OUTPUT_SCHEMAS: the four tools' output shape

agent-fundamentals already gave you each tool's input_schema — what shape what the model asks for has. This module adds the missing half: what shape what each tool returns has. The vocabulary is the same (type/properties/required), now applied to the result instead of the argument:

OUTPUT_SCHEMAS = {
    "list_rooms": {"type": "array", "items": {
        "type": "object",
        "properties": {"room": {"type": "string"}, "rate_cents": {"type": "integer"}},
        "required": ["room", "rate_cents"],
    }},
    "get_quote": {
        "type": "object", "properties": {"price_cents": {"type": "integer"}}, "required": ["price_cents"],
    },
    "book_room": {
        "type": "object",
        "properties": {"booking_id": {"type": "integer"}, "confirmed": {"type": "boolean"}},
        "required": ["booking_id", "confirmed"],
    },
    "cancel_booking": {
        "type": "object", "properties": {"cancelled": {"type": "boolean"}}, "required": ["cancelled"],
    },
}

list_rooms is the only one of the four with an "array"-type schema — it returns a list of objects, not a single object. check_schema (lesson 02) already knows how to handle this: when the root type is "array", it validates every element in the list against schema["items"], and accumulates errors with an item[i] prefix that says exactly which element failed.

def check_schema(result, schema):
    errors = []
    root_type = _PY_TYPE.get(schema.get("type"))
    if root_type and not isinstance(result, root_type):
        return [f"tipo raíz debe ser {schema['type']}, llegó {type(result).__name__}"]
    if schema.get("type") == "array":
        item_schema = schema.get("items", {})
        for i, item in enumerate(result):
            errors.extend(f"item[{i}].{e}" for e in check_schema(item, item_schema))
        return errors
    props = schema.get("properties", {})
    for name in schema.get("required", []):
        if name not in result:
            errors.append(f"falta el campo requerido '{name}'")
    for name, value in result.items():
        if name in props:
            expected = _PY_TYPE.get(props[name].get("type"))
            if expected and not isinstance(value, expected):
                errors.append(f"'{name}' debe ser {props[name]['type']}, llegó {type(value).__name__}")
    return errors

Confirm the array case with a list_rooms result deliberately broken — a rate stored as a string instead of an integer, in the first element:

broken = [{"room": "Focus", "rate_cents": "2500"}, {"room": "Studio", "rate_cents": 4000}]
print(check_schema(broken, OUTPUT_SCHEMAS["list_rooms"]))

What to expect:

["item[0].'rate_cents' debe ser integer, llegó str"]

The message points out, precisely, which of the three list elements has the problem — information a plain "the schema doesn't validate" wouldn't give.

_last_result_for_tool: which result to validate, inside a multi-step run

A case like book_focus_pro_3h_ana calls three tools in sequence — list_rooms, get_quote, book_room. To validate the last one's result, you need a function that walks history and finds, specifically, the most recent successful tool_result that matches that tool:

def _last_result_for_tool(history, tool_name):
    """El content del último tool_result EXITOSO de `tool_name` en este
    run -- el resultado que se valida contra su OUTPUT_SCHEMAS."""
    tool_use_name = {}
    last_content = None
    for turn in history:
        content = turn["content"]
        if isinstance(content, str):
            continue
        for block in content:
            if block["type"] == "tool_use":
                tool_use_name[block["id"]] = block["name"]
            elif block["type"] == "tool_result" and not block.get("is_error"):
                if tool_use_name.get(block["tool_use_id"]) == tool_name:
                    last_content = block["content"]
    return last_content

The pattern — tool_use_name, a dictionary that pairs each tool_use's id with its name — is the same one you already used in cost_for_run (Module 3) to reconstruct which tool each tool_result belongs to. The only difference is the filter: instead of accumulating everything, this function keeps only the last content that matches tool_name, discarding any earlier attempt rejected via is_error.

Run Ana's complete case and validate every tool result against its OUTPUT_SCHEMAS, not just the last one:

case = CASE_SET[2]  # book_focus_pro_3h_ana
reset_reservo_state()
with rl.traced_run(case["question"], 1) as trace_id:
    final, history = ra.run_reservo_agent(case["question"], case["model_script"])

tool_use_name = {}
for turn in history:
    content = turn["content"]
    if isinstance(content, str):
        continue
    for block in content:
        if block["type"] == "tool_use":
            tool_use_name[block["id"]] = block["name"]
        elif block["type"] == "tool_result" and not block.get("is_error"):
            name = tool_use_name[block["tool_use_id"]]
            result = json.loads(block["content"])
            errors = check_schema(result, OUTPUT_SCHEMAS[name])
            print(f"{name:15} resultado={result}  errores={errors}")

What to expect:

list_rooms      resultado=[{'room': 'Focus', 'rate_cents': 2500}, {'room': 'Studio', 'rate_cents': 4000}, {'room': 'Boardroom', 'rate_cents': 8000}]  errores=[]
get_quote       resultado={'price_cents': 6000}  errores=[]
book_room       resultado={'booking_id': 1, 'confirmed': True}  errores=[]

All three steps of the run pass their form check: list_rooms returns an array of three objects, each with room (string) and rate_cents (integer); get_quote returns price_cents as an integer; book_room returns booking_id (integer) and confirmed (boolean). This is exactly the gate lesson 07 runs over the five complete cases.

🛑 The exact boundary, with numbers: check_schema doesn't catch a wrong price

Here's this lesson's central demonstration, and it's worth reading carefully. Imagine that, for whatever reason — a real business decision, or a transcription error in reservo_tools.py — Focus's base rate changes from 2500 to 2600 cents per hour:

rt.ROOM_RATE_CENTS["Focus"] = 2600  # cambio real, a propósito, para esta demostración

case = CASE_SET[0]  # quote_focus_pro_3h
reset_reservo_state()
with rl.traced_run(case["question"], 1) as trace_id:
    final, history = ra.run_reservo_agent(case["question"], case["model_script"])

result = json.loads(_last_result_for_tool(history, "get_quote"))
schema_errors = check_schema(result, OUTPUT_SCHEMAS["get_quote"])
output_errors = check_expected_output(result, case["expected_output"])

print("resultado real                       :", result)
print("check_schema (forma)                 :", schema_errors)
print("check_expected_output (ancla literal):", output_errors)

What to expect:

resultado real                       : {'price_cents': 6240}
check_schema (forma)                 : []
check_expected_output (ancla literal): ["'price_cents' esperado=6000, obtenido=6240"]

Here's the boundary, with evidence: check_schema returns [] — with no error at all. {"price_cents": 6240} has exactly the right shape: a dict, with the price_cents key, of integer type. To check_schema, this result is just as valid as {"price_cents": 6000} — because, rightly so, it is: the form didn't break. The price changed, not the contract.

What does catch this change is check_expected_output — a second function, deliberately different from check_schema — which compares the exact value against the anchor the case declares: 6000 expected, 6240 obtained, a precise error. Note, carefully, what kind of check this is: it's still a literal, deterministic comparison against a fixed value — never a judgment of "is this price reasonable?" check_expected_output doesn't know, and doesn't care, whether 2600 cents per hour is a fair price for a coworking room — it only knows this specific CASE_SET case anchors to 6000, and that 6240 isn't 6000.

def check_expected_output(result, expected_output):
    """Comparación LITERAL contra un valor fijo -- cada clave declarada en
    el caso debe coincidir EXACTO con el resultado real. Esto es lo que
    ancla, por ejemplo, get_quote(Focus, pro, 3h) a 6000 centavos: si algo
    en la aritmética de Reservo cambiara, este chequeo lo atrapa."""
    errors = []
    for key, expected_value in expected_output.items():
        actual_value = result.get(key)
        if actual_value != expected_value:
            errors.append(f"'{key}' esperado={expected_value!r}, obtenido={actual_value!r}")
    return errors

Three layers, three different questions — and only two of them live in this module

This demonstration lets you see, with complete clarity, that there are at least three possible questions about the same result, and that confusing them is this entire module's most dangerous mistake:

1. "Does it have the right shape?" — check_schema. Deterministic, with no reference value at all — just type and structure. This is the most permissive layer: almost any "reasonable" price passes it.
2. "Does it exactly match the value this specific case anchored to?" — check_expected_output. Also deterministic, but against a fixed value, known in advance: 6000, not "a reasonable price." This layer is strict, but still needs no interpretive criterion at all — it's an == comparison, nothing more.
3. "Is this a reasonable price for a coworking room, in this market, in this city?" — this question has no function in this module, and never will. Answering it requires an external criterion — market research, an acceptable range a human defines, maybe a model that compares against real data — that isn't a form comparison or a literal comparison against a fixed anchor. This question, and any with the same structure (is the response clear? is the tone appropriate? is the explanation correct?), belongs entirely to evaluation-frameworks-guide.

The difference between question 2 and question 3 is subtle but decisive, and it's worth saying once more, plainly: both compare a value against a reference, but question 2's reference is a fixed number, written in the CASE_SET, while question 3's reference doesn't exist as a single value — it depends on context, on judgment, on criteria. check_expected_output never "decides" whether 6000 is a good price — a human already decided that when writing the case; the function only confirms the system still produces that exact number. The day the question turns into "is the price the agent produced reasonable, with nobody having fixed in advance what the correct number is?", that question no longer has a deterministic answer, and at that point, without exception, the job moves to evaluation-frameworks-guide — its evaluating-agents module is designed, specifically, for questions with that structure.

The complete statement, to repeat in every lesson that brushes up against this boundary: this gate checks that the form didn't break — schema, correct tool, threshold — and, where applicable, that a value matches a fixed, known anchor. It never evaluates whether the response is good in any sense that depends on interpretation. That's evaluation-frameworks-guide.

Why this boundary matters in practice

This isn't an academic technicality. A team that confuses these two layers runs one of two real risks, in opposite directions: if it tries to solve quality questions with form tools — for example, trying to anticipate, with a fixed CASE_SET, every possible variation of a "good" response — it ends up with an unmaintainable case file, growing without limit and still not covering the real question. If it tries to solve form questions with quality tools — for example, using a model to "judge" whether check_schema should pass — it introduces a source of non-determinism exactly where the entire guide insists there should be none: a CI gate that sometimes passes and sometimes fails, with nothing having changed, is worse than having no gate at all. This lesson's boundary isn't a curiosity — it's what keeps every tool doing the job it's designed for.

Common mistakes

1. Thinking that because check_expected_output compares an exact value, it "already is" quality evaluation. It isn't — it's still a literal comparison against a fixed number, written in advance in the CASE_SET, with no interpretation at all. Quality evaluation starts when there's no longer a single "correct" value known in advance to compare against.

2. Using check_schema to try to catch a price change. This lesson's worked example demonstrates it: check_schema can't catch this, by design — its job is form, not value. If a case needs to anchor a specific value, that's expected_output's job, never OUTPUT_SCHEMAS's.

3. Extending OUTPUT_SCHEMAS with range constraints ("price_cents must be between 1000 and 100000") to try to catch "reasonable" values. This would be a real step into quality territory, disguised as form — an arbitrary range has no deterministic foundation, and this guide deliberately avoids it. If a value needs to be compared against a real business range, that design decision belongs to an explicit conversation about what kind of check is being built, not a silent extension of check_schema.

4. Confusing a check_expected_output FAIL with "the agent did something wrong." The worked example deliberately uses a legitimate business change (ROOM_RATE_CENTS["Focus"] = 2600) to produce the FAIL — the agent responded exactly what it should have responded with the new data. The gate's FAIL doesn't judge whether the change was good or bad — it only confirms something changed relative to the previous anchor, and leaves it to a human to decide whether that anchor needs updating.

5. Thinking the boundary with evaluation-frameworks-guide only applies to get_quote's value. It applies to any question about the agent's response that depends on interpretation: is the final text response clear? was the order in which the agent explored options the most efficient one possible? does the explanation it gave the user make sense? None of those questions has a function in this module, no matter which tool or field they're asked about.

Exercises

Exercise 1: Confirm cancel_booking also tells form apart from value (Easy)

Run the complete book_and_cancel_studio_basic_1h_diego case. Validate cancel_booking's result with check_schema (it should pass). Then, build a handmade result, {"cancelled": False}, and compare it against the case's expected_output ({"cancelled": True}) with check_expected_output.

case = CASE_SET[4]  # book_and_cancel_studio_basic_1h_diego
reset_reservo_state()
with rl.traced_run(case["question"], 1) as trace_id:
    final, history = ra.run_reservo_agent(case["question"], case["model_script"])

result = json.loads(_last_result_for_tool(history, "cancel_booking"))
print("resultado real   :", result)
print("check_schema     :", check_schema(result, OUTPUT_SCHEMAS["cancel_booking"]))

resultado_falso = {"cancelled": False}
print("check_expected_output sobre un valor hecho a mano:",
      check_expected_output(resultado_falso, case["expected_output"]))

resultado real   : {'cancelled': True}
check_schema     : []
check_expected_output sobre un valor hecho a mano: ["'cancelled' esperado=True, obtenido=False"]

Exercise 2: Design one question from each category about book_room (Medium)

For the book_room tool, write three different questions about its result — one about form, one about literal value, one about semantic quality — following this lesson's three-layer pattern. For each one, say which function (or which guide, for the quality question) would answer it.

Exercise 3: Trigger a form FAIL on book_room and compare it against this lesson's value FAIL (Hard)

Build a handmade result where book_room returns booking_id as a string ("1" instead of 1) while confirmed stays True. Validate it with check_schema and with check_expected_output (against {"booking_id": 1, "confirmed": True}). Compare the two error messages, and explain in one sentence why both checks, in this particular case, end up flagging the same field (booking_id) but for different reasons.

broken_result = {"booking_id": "1", "confirmed": True}
expected = {"booking_id": 1, "confirmed": True}

print("check_schema          :", check_schema(broken_result, OUTPUT_SCHEMAS["book_room"]))
print("check_expected_output :", check_expected_output(broken_result, expected))

check_schema          : ["'booking_id' debe ser integer, llegó str"]
check_expected_output : ["'booking_id' esperado=1, obtenido='1'"]

Summary and next step

• We completed check_schema with OUTPUT_SCHEMAS, the dictionary for Reservo's four tools, including list_rooms's special case (an array of objects, validated element by element).
• We built _last_result_for_tool, the function that finds which result to validate inside a multi-step history, and check_expected_output, the literal comparison against each case's fixed anchor.
• We demonstrated, with a real business change and run numbers, this module's exact boundary: check_schema can't, and shouldn't, catch a value change ({"price_cents": 6240} is perfectly valid in form); check_expected_output does catch it, because it compares against a fixed anchor, not because it judges whether the value is reasonable.
• We stated, precisely, the boundary with evaluation-frameworks-guide: this module checks form (schema, correct tool, threshold) and, where applicable, exact match against a fixed anchor — never semantic quality, never a judgment of "is it reasonable?", never a dataset with fuzzy ground truth.

Next lesson: 05 — Checking Tool Choice. With the form-versus-quality boundary already established over a tool's result, we apply the same criterion to the tool choice itself: check_tool_choice in depth, with the first complete demonstration of a real FAIL — a prompt regression that makes the agent choose the wrong tool.

Additional resources

1. Anthropic — Tool use (function calling) overview — The same schema vocabulary (type/properties/required) this module reuses to describe an output's shape, not just an input's.
2. Python — isinstance — The foundation of every type check inside check_schema, including the recursive array case.
3. Python — equality comparisons (==) — The exact operator behind check_expected_output, and why "1" != 1 in Python even though both "mean the same thing" to a human reader.
4. Anthropic — Building effective agents — On the difference between verifying an agentic system behaves predictably and evaluating whether its decisions are, in substance, the best possible ones.
5. Python 3.14 — What's New — The version every line of code in this lesson ran on, including the price-change demonstration.