Checking Tool Choice

Description

Lesson 04 traced the form-versus-quality boundary over a tool's result. This lesson applies exactly the same criterion to a different question, just as central to an agent: does the agent still choose the correct tool, in the correct order, against a turn script you already know? check_tool_choice — built in its final form in lesson 02 — is the answer, and this lesson puts it to the test in depth: first over the CASE_SET's five real cases, all PASS, and then over two "after a change" scripts — two different ways an agent can regress without its code having changed at all — that produce this entire module's first real FAIL.

This FAIL matters especially, because it's the same kind of evidence this guide's DISEÑO demands citing precisely: not a "something failed," but an exact message — expected tool get_quote, got book_room — that tells whoever reads it, with no ambiguity, what broke.

Connection to the module

This lesson doesn't add any new function to regression/harness.py — check_tool_choice and extract_tool_sequence were already complete in lesson 02. What it adds is the first complete demonstration of run_case, the function lesson 07 is going to finish assembling: run a CASE_SET case against a substituted script, simulating "the agent's version after a prompt change."

Recalling check_tool_choice

def extract_tool_sequence(history):
    """La secuencia LITERAL de tools llamadas por el agente, en el orden en
    que las llamó."""
    return [
        block["name"]
        for turn in history if not isinstance(turn["content"], str)
        for block in turn["content"] if block["type"] == "tool_use"
    ]


def check_tool_choice(history, expected_tools):
    """Comparación LITERAL: la secuencia de tools obtenida == la secuencia
    esperada, exactamente, elemento por elemento."""
    actual = extract_tool_sequence(history)
    return actual == expected_tools, actual

Two properties of this function are worth repeating before putting it to the test in depth. First: the comparison is over an ordered list, not a set — ["get_quote", "book_room"] and ["book_room", "get_quote"] are, to check_tool_choice, two completely different sequences, even though they contain the same two tools. Second: the function always returns the real sequence, even when the verdict is False — that second part of the tuple is what makes it possible to build a precise FAIL message, instead of a plain "doesn't match."

Worked example, part 1: the five real cases, all PASS

Before failing anything on purpose, confirm the complete CASE_SET, run exactly as it stands — with no substitution — passes check_tool_choice on all five cases:

print("--- check_tool_choice sobre los cinco casos reales del CASE_SET ---")
for i, case in enumerate(CASE_SET, start=1):
    reset_reservo_state()
    with rl.traced_run(case["question"], i) as trace_id:
        final, history = ra.run_reservo_agent(case["question"], case["model_script"])
    ok, actual = check_tool_choice(history, case["expected_tools"])
    print(f"{case['name']:38} {'PASS' if ok else 'FAIL'}  esperado={case['expected_tools']}  obtenido={actual}")

What to expect:

--- check_tool_choice sobre los cinco casos reales del CASE_SET ---
quote_focus_pro_3h                     PASS  esperado=['get_quote']  obtenido=['get_quote']
quote_focus_basic_3h                   PASS  esperado=['get_quote']  obtenido=['get_quote']
book_focus_pro_3h_ana                  PASS  esperado=['list_rooms', 'get_quote', 'book_room']  obtenido=['list_rooms', 'get_quote', 'book_room']
book_boardroom_pro_1h_sofia            PASS  esperado=['list_rooms', 'get_quote', 'book_room']  obtenido=['list_rooms', 'get_quote', 'book_room']
book_and_cancel_studio_basic_1h_diego  PASS  esperado=['book_room', 'cancel_booking']  obtenido=['book_room', 'cancel_booking']

Five for five. This isn't surprising yet — every model_script in the CASE_SET was handwritten so the agent (concept) calls exactly those tools. The interesting part starts now, when the script that runs stops being the one the case declares.

run_case, with a substituted script: simulating "the version after"

To compare new behavior against expected behavior, you need to be able to run the same case — same question, same expected_tools, same thresholds — against a script different from the one the CASE_SET ships by default. That's the exact reason run_case (which lesson 07 completes) accepts an optional model_script parameter:

def run_case(case, sequence_number, model_script=None):
    """Corre UN caso del CASE_SET: ejecuta run_reservo_agent (sin tocar su
    lógica) con `model_script` (el del caso, o uno sustituido -- la pieza
    que M7 reusa para comparar una versión vieja contra una nueva)."""
    reset_reservo_state()
    script = model_script if model_script is not None else case["model_script"]

    with rl.traced_run(case["question"], sequence_number) as trace_id:
        final, history = ra.run_reservo_agent(case["question"], script)

    tool_choice_ok, actual_tools = check_tool_choice(history, case["expected_tools"])
    # ... el resto de los chequeos (lecciones 04 y 06) se agregan en la lección 07.
    return tool_choice_ok, actual_tools

When model_script is None, run_case runs exactly what the CASE_SET declares — the previous section's behavior. When a different script is passed in, run_case still compares against the original case's same expected_tools, but runs different behavior. This is, precisely, the technique Module 7 is going to reuse to compare an old agent version against a new one: the CASE_SET doesn't change, what changes is the script representing "how the model responds now."

Worked example, part 2: the main FAIL — a regression that skips quoting

Imagine a change to Reservo's system prompt (concept — a real call never runs) makes the model, faced with a simple quote question, decide to be "more proactive" and book directly, without quoting first. It's exactly the kind of behavior regression a well-intentioned prompt change can introduce without anyone noticing until a customer complains about a booking they never asked for.

case = CASE_SET[0]  # quote_focus_pro_3h -- pregunta: "¿Cuánto cuesta Focus pro 3h?"

# Guion "después del cambio": el modelo salta la cotización y reserva directo.
regressed_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro 3h para Ana."}]},
]

tool_choice_ok, actual_tools = run_case(case, 99, model_script=regressed_script)
print("resultado:", "PASS" if tool_choice_ok else "FAIL")
print(f"FAIL: tool esperada {case['expected_tools'][0]}, obtenida {actual_tools[0]}")

What to expect:

resultado: FAIL
FAIL: tool esperada get_quote, obtenida book_room

This is the message this guide's DISEÑO asks you to cite precisely, and now you have it, produced by real code: expected tool get_quote, got book_room. Notice what this FAIL doesn't say: it doesn't say the agent's final response ("Reservé Focus pro 3h para Ana") is badly written, or confusing — in fact, as text, it's perfectly clear. What it says, with total precision, is that the agent took a different action from the expected one: instead of answering a quote question (a read-only operation, with no effect at all), it created a real booking (a write operation, with an effect the user never asked for). That's exactly the kind of behavior regression a form gate — never a quality one — is designed to catch: no matter how well written the final response is, the action that preceded it was the wrong one. If the question instead were "does the final response sound natural and professional?", check_tool_choice would have nothing to say — that question belongs, with its own entire discipline, to evaluation-frameworks-guide.

Worked example, part 3: a subtler FAIL — order matters

Not every regression changes which tools get called — some change the order. Run Ana's case with the same three tools as always, but with list_rooms and get_quote swapped:

case3 = CASE_SET[2]  # book_focus_pro_3h_ana
out_of_order_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro 3h para Ana."}]},
]
tool_choice_ok, actual = run_case(case3, 98, model_script=out_of_order_script)
print("resultado:", "PASS" if tool_choice_ok else "FAIL")
print("esperado :", case3["expected_tools"])
print("obtenido :", actual)

What to expect:

resultado: FAIL
esperado : ['list_rooms', 'get_quote', 'book_room']
obtenido : ['get_quote', 'list_rooms', 'book_room']

The same three tools, exactly — but in a different order, and check_tool_choice catches it just as well as the previous case, with no change to the function at all. This is a useful example for understanding why the literal comparison of complete sequences, and not of sets, is the right decision for this module: an agent that quotes before knowing which rooms exist could be quoting on a room that's actually no longer available — order, in this domain, isn't a cosmetic detail.

Why the comparison is literal, and not "reasonably similar"

It might seem more flexible for check_tool_choice to accept, say, a "similar" sequence — the same tools, in any order, or even with one extra tool if it doesn't affect the final result. This guide deliberately rejects that flexibility, for the same reason it rejected, in lesson 04, any attempt to have check_schema judge whether a value is "reasonable": the moment the comparison stops being exact, it needs some criterion to decide how much difference is acceptable — and that criterion is no longer a form comparison, it's a judgment. Back to the vehicle-inspection analogy: an inspector who accepted "the brakes respond almost always" instead of "the brakes respond, yes or no" isn't doing an inspection anymore — they're doing a risk assessment, a different job, with different tools. check_tool_choice, with its exact ==, deliberately stays on the inspection side.

Common mistakes

1. Thinking a check_tool_choice FAIL always means "the agent got it wrong." As in lesson 04's common mistake, a FAIL only says behavior changed relative to what was expected — lesson 07 shows how that change can be a real regression (a prompt bug) or an intentional behavior change that simply hasn't been reflected in the CASE_SET yet.

2. Comparing set(actual) against set(expected_tools) "so order doesn't matter." This eliminates exactly the information the worked example, part 3, demonstrates matters. If order genuinely didn't matter for a specific case at some point, the right decision would be to document that explicitly on that case — never change the function's default behavior for every case.

3. Forgetting that run_case with model_script=None uses the case's script, not an empty one. Passing model_script=[] by mistake (instead of omitting the argument) would produce an IndexError inside run_reservo_agent when trying to access model_script[0] — a completely different error from a check_tool_choice FAIL, and much more confusing to diagnose if you don't know what caused the difference.

4. Confusing "one extra tool" with "the correct order of the expected tools." If the substituted script called list_rooms, get_quote, book_room, and also cancel_booking at the end (an extra, unexpected tool), check_tool_choice would also flag it as a FAIL — a four-element list is never == to a three-element one, regardless of the first three matching exactly.

5. Running the substituted script without reset_reservo_state first. As lesson 03 already warned, this can produce an unexpected booking_id, contaminating the diagnosis: a genuine check_tool_choice FAIL (the correct tool) can end up hidden behind a spurious check_expected_output FAIL (the wrong id, for a reason that has nothing to do with the real regression being investigated).

Exercises

Exercise 1: Trigger a FAIL on the cancellation case (Easy)

Using case = CASE_SET[4] (book_and_cancel_studio_basic_1h_diego), build a substituted script that books but doesn't cancel — the agent responds with end_turn immediately after book_room. Run check_tool_choice against case["expected_tools"] and confirm the FAIL.

case = CASE_SET[4]
no_cancel_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Studio", "tier": "basic", "hours": 1, "member": "Diego"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Studio basic 1h para Diego."}]},
]
reset_reservo_state()
with rl.traced_run(case["question"], 1) as trace_id:
    final, history = ra.run_reservo_agent(case["question"], no_cancel_script)
ok, actual = check_tool_choice(history, case["expected_tools"])
print("resultado:", "PASS" if ok else "FAIL")
print("esperado :", case["expected_tools"])
print("obtenido :", actual)

resultado: FAIL
esperado : ['book_room', 'cancel_booking']
obtenido : ['book_room']

Exercise 2: Confirm check_tool_choice on the basic case isn't confused with the pro one (Medium)

quote_focus_pro_3h and quote_focus_basic_3h have the same expected_tools (["get_quote"]). Run quote_focus_basic_3h's script (which quotes with tier="basic") but compare it against quote_focus_pro_3h's expected_tools. Confirm check_tool_choice gives PASS (because the tool sequence does match), and explain in one sentence why this doesn't mean the complete case is correct.

case_basic = CASE_SET[1]
case_pro = CASE_SET[0]

reset_reservo_state()
with rl.traced_run(case_basic["question"], 1) as trace_id:
    final, history = ra.run_reservo_agent(case_basic["question"], case_basic["model_script"])

ok, actual = check_tool_choice(history, case_pro["expected_tools"])
print("check_tool_choice (comparado contra el caso pro):", ok, actual)

check_tool_choice (comparado contra el caso pro): True ['get_quote']

Exercise 3: Design a case where the "reasonable" alternative order should also fail (Hard)

For the book_boardroom_pro_1h_sofia case (list_rooms → get_quote → book_room), build an alternative script, just as "reasonable" at first glance, where the agent calls get_quote twice before booking — once, "changes its mind," and quotes again with the exact same arguments before booking. Run check_tool_choice and confirm the FAIL. Then, explain why, even though the final result (the booking) would be identical, this behavior is still a legitimate regression worth catching.

case = CASE_SET[3]  # book_boardroom_pro_1h_sofia
double_quote_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "get_quote",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "book_room",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1, "member": "Sofía"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Boardroom pro 1h para Sofía."}]},
]
reset_reservo_state()
with rl.traced_run(case["question"], 1) as trace_id:
    final, history = ra.run_reservo_agent(case["question"], double_quote_script)
ok, actual = check_tool_choice(history, case["expected_tools"])
print("resultado:", "PASS" if ok else "FAIL")
print("obtenido :", actual)

resultado: FAIL
obtenido : ['list_rooms', 'get_quote', 'get_quote', 'book_room']

Summary and next step

• We confirmed, run for real, that the CASE_SET's five real cases pass check_tool_choice with no change at all — the baseline any future regression gets compared against.
• We built the run_case technique with a substituted model_script: the same case, a different script, simulating "the agent after a change" — the piece Module 7 is going to reuse for version comparisons.
• We produced the entire module's first real FAIL, with the exact message this guide's DISEÑO demands: expected tool get_quote, got book_room — a prompt regression that skips a read-only step and goes straight to an action with real effects.
• We confirmed a second kind of FAIL — the same set of tools, different order — and explained why the literal comparison of complete sequences, never of sets, is the right decision for this check.

Next lesson: 06 — Checking Cost and Latency Thresholds. With tool choice already covered, we complete the gate's third question: check_cost_threshold and check_latency_threshold, reusing cost_for_run (Module 3) and the latency model (Module 4) without modifying them, with a case that fails by exceeding a threshold.

Additional resources

1. Anthropic — Tool use (function calling) overview — The tool_use protocol extract_tool_sequence walks to build each run's real sequence.
2. Python — list comparison — The exact behavior of == over two lists: element by element, in order, sensitive to length — check_tool_choice's complete foundation.
3. Python — nested list comprehensions — The two-for construction that builds extract_tool_sequence in a single expression.
4. Anthropic — Building effective agents — On why the choice of what action to take, not just the final text's quality, is one of a production agent's most consequential decisions.
5. Python 3.14 — What's New — The version every line of code in this lesson ran on, including the three FAILs demonstrated.