What You Cannot See Without Instrumentation

Description

Lesson 02 named the problem in the abstract: nobody's going to be watching when the agent runs for real users. This lesson makes it concrete, with real code, and puts an exact limit on it. It's not that "it would be nice to know more" — there are specific, nameable questions that cannot be answered with what run_reservo_agent gives you today, no matter how hard you try to look.

You're going to try it three different ways: using only the final response (what a real system consumes), inspecting history by hand with print_trace (the most you can do without instrumenting anything), and — the most revealing case — trying to recover anything from a run that failed. All three ways hit the same hard limit in the end.

Connection to the module

This is the lesson that names the analogy holding up the whole module. A car without a dashboard runs — the engine works, the wheels turn — but the driver has no way to know the speed, how much gas is left, or whether the engine is overheating, until the car has already stalled on the shoulder. run_reservo_agent, as it stands, is exactly that car: it works, and works well — you confirmed that in lessons 01 and 02 — but it has no dashboard. This lesson measures, with precision, how far "looking carefully" gets you before you need, no way around it, to instrument something.

Analogy: the car without a dashboard

You drive a car that starts without trouble, accelerates well, brakes when you ask it to. Nothing tells you something's wrong — until, in the middle of the highway, it just shuts off. Did it run out of gas? Did the engine overheat? Had a tire been losing air for an hour? There's no way to know by looking at the car from outside, because you never had a dashboard: no speedometer, no fuel gauge, no temperature light. The car worked — perfectly, in fact, every single time you drove it before. What it lacked wasn't working better. It lacked the ability to tell you, at any moment, how it was doing.

That's exactly run_reservo_agent at this point in the module. Every run you ran in lessons 01 and 02 worked correctly. The problem was never that the agent was poorly built — it's that, whether it runs well or badly, it has no way to tell you except the final text response, which is about as informative about "how it got there" as a silent car is informative about how much gas it has left.

Worked example: three ways to try to see inside, and where each one stops

Way 1: only the final response (what a real system uses)

Go back to the canonical task, and keep only what a real service would keep — the text response:

import reservo_agent as ra

model_script_demo = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Focus", "tier": "premium", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #1."}]},
]

final, history = ra.run_reservo_agent("Reserva Focus pro 3h para Ana", model_script_demo)
print("RESPUESTA:", final["content"][0]["text"])

What to expect:

RESPUESTA: Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #1.

With this, zero of lesson 01's five questions get answered. It's not that they're hard to extract — it's that the final variable doesn't contain them. final["content"][0]["text"] is a string. There's no place in a string where "how many steps," "which tools," or "how much did it cost" could live.

Way 2: inspecting history by hand, with print_trace

history does still exist, as a local variable — because we didn't discard the second half of what run_reservo_agent returns. Use print_trace, carried over unchanged from agent-fundamentals M8, to look inside:

ra.print_trace(history)

What to expect:

  [0] user      pregunta: 'Reserva Focus pro 3h para Ana'
  [1] assistant tool_use(list_rooms): {}
  [2] user      tool_result: [{"room": "Focus", "rate_cents": 2500}, {"room": "Studio", "rate_cents": 4000}, {"room": "Boardroom", "rate_cents": 8000}]
  [3] assistant tool_use(get_quote): {'room': 'Focus', 'tier': 'premium', 'hours': 3}
  [4] user      tool_result [is_error]: 'tier'='premium' no está en enum ['basic', 'pro']
  [5] assistant tool_use(get_quote): {'room': 'Focus', 'tier': 'pro', 'hours': 3}
  [6] user      tool_result: {"price_cents": 6000}
  [7] assistant tool_use(book_room): {'room': 'Focus', 'tier': 'pro', 'hours': 3, 'member': 'Ana'}
  [8] user      tool_result: {"booking_id": 1, "confirmed": true}
  [9] assistant texto final: 'Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #1.'

Now you can answer three of the five questions, by hand: steps (10, counting len(history)), tools called and in what order (list_rooms, get_quote twice, book_room), and whether anything failed (yes, turn [4] carries [is_error]). Confirm it with code, not just by looking:

print("pasos totales      :", len(history))
print("tool calls totales :", sum(
    1 for m in history if m["role"] == "assistant"
    and isinstance(m["content"], list) and m["content"][0]["type"] == "tool_use"
))
print("turnos con is_error :", sum(
    1 for m in history if m["role"] == "user"
    and isinstance(m["content"], list) and any(b.get("is_error") for b in m["content"])
))

What to expect:

pasos totales      : 10
tool calls totales : 4
turnos con is_error : 1

Three out of five. But notice the cost of getting this far: you had to know in advance that history still existed, call print_trace explicitly, and read ten lines with your own eyes (or write three lines of counting yourself). That works for one run, in a notebook, while you're the one deciding to look. It doesn't work for three thousand runs on Tuesday night, that nobody watched at the moment they happened — because by the time someone asks, that specific history no longer exists anywhere: it was a local variable that died when the function returned and the script ended.

Way 3: the two questions not even the full history can answer

With history still on screen, look for cost and latency. You don't need to write code to confirm they're not there — just walk through the trace above, field by field: there's a role, a type, a name, an input, a content, an is_error when it applies. No block anywhere has a token field, a cents field, or a milliseconds field. history records what happened at each step — never how much it cost nor how long that step took. It's not a limitation of how you're reading it: that information was simply never captured, because nothing in run_reservo_agent, dispatch_robust, or any of the functions you already built in agent-fundamentals measures time or counts tokens for anything. There's no bug to fix here — there's an entire layer that hasn't been built yet, and that this guide builds starting in Module 2.

The most revealing case: when the run fails, you lose even the little you had

The three ways above all assume the run finished, successfully or with some is_error along the way, and returned something. What happens when the run doesn't even finish — when the iteration cap runs out?

stuck_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": f"toolu_0{n}", "name": "list_rooms", "input": {}}]}
    for n in range(1, 4)
]
try:
    ra.run_reservo_agent("Reserva algo", stuck_script, max_iterations=2)
except RuntimeError as exc:
    print("RuntimeError capturado:", exc)
    print("¿'history' quedó definida en este scope?", "history" in dir())

What to expect:

RuntimeError capturado: max_iterations alcanzado (2)
¿'history' quedó definida en este scope? False

This is the harshest version of the problem. When run_reservo_agent exhausts its cap, messages — the list being built step by step, inside the function — never reaches a return. It's lost completely, along with the rest of the function's stack frame, the instant the RuntimeError is raised. Not even Way 2 — inspecting history by hand — is possible here: there's no history to inspect, because the function never handed it to you. The car didn't just shut off without warning — it shut off and also confiscated the dashboard (had it ever had one) at the same moment. This is, precisely, one of the reasons Module 2 doesn't wait for a run to finish before recording what's happening: if you wait until the end, a run that fails leaves you with no record at all.

Common mistakes

1. Thinking "I could inspect history if I needed to" is the same as having observability. Way 2 in this lesson worked because you, deliberately, saved history into a variable and looked at it, in the same process, at the same instant. In production, nobody does that for every run — and if something fails three days later, there's no history saved anywhere to go look at.

2. Believing that "logging the full print(history)" is enough and the problem's solved. It's a step in the right direction, but dumping the entire Python structure to a plain-text log is hard to query, hard to aggregate over a batch of thousands of runs, and doesn't solve the underlying problem: it still has no cost or latency, and it still doesn't survive a RuntimeError unless it's captured during the run, not after.

3. Looking for cost or latency inside tool_result, assuming it "must be somewhere." It isn't. None of agent-fundamentals's functions — dispatch_robust, call_with_timeout, run_reservo_agent — measure real time or count tokens. call_with_timeout does use a timeout, but a maximum cap isn't the same as a measurement: it cuts things off if something takes too long, but it never reports how long something that finished on time actually took.

4. Thinking this limit is an oversight of agent-fundamentals. It isn't — that guide never promised observability; it promised an agent that reliably solves multi-step tasks, and it delivered. The absence of instrumentation isn't a bug in that guide: it is, precisely, the exact point where its scope ends and this one's begins.

5. Underestimating the RuntimeError case. It's tempting to think "well, that run failed, but at least the ones that do finish leave me something." The point of this section is that a run that fails is precisely the one you most need to be able to diagnose — and it's exactly the one that, without instrumentation built during execution (not after), leaves you with less of a trail than any other.

Exercises

Exercise 1: Score the five questions, one by one (Easy)

With the worked example's history (Way 2, with the invalid tier) still available, answer in writing, for each of lesson 01's five questions, whether you can answer it (a) with Way 1, (b) with Way 2, or (c) with neither.

Exercise 2: Reproduce the RuntimeError case with a different tool (Medium)

Repeat the worked example's "the run fails" scenario, but with a script that repeats get_quote with valid arguments indefinitely (no end_turn), and max_iterations=3. Confirm the RuntimeError is raised and that, again, no history remains available outside the function.

stuck_get_quote = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": f"toolu_0{n}", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]}
    for n in range(1, 5)
]
try:
    ra.run_reservo_agent("Cotiza Focus pro 3h", stuck_get_quote, max_iterations=3)
except RuntimeError as exc:
    print("RuntimeError capturado:", exc)
    print("¿'history' quedó definida en este scope?", "history" in dir())

RuntimeError capturado: max_iterations alcanzado (3)
¿'history' quedó definida en este scope? False

Exercise 3: Design, in prose, the minimum requirement that would solve the RuntimeError case (Hard)

Without writing code yet — that starts in lesson 08, and gets developed in depth in Module 2 — describe in a paragraph what would need to change about how a run's information gets recorded so that, even when run_reservo_agent ends in a RuntimeError, a trail is left of the steps it did manage to take before failing.

Summary and next step

• We tried three ways of "seeing inside" a run: only the final response (zero of five questions answered), history inspected by hand (three of five), and a run that fails with RuntimeError (none, because history doesn't even survive to be inspected).
• We confirmed, with real code, that cost and latency are nowhere in history — it's not a problem of how you look at it, it's that that information was never captured.
• The RuntimeError case is the harshest version of the problem: capturing information at the end of a run is useless for the runs you most need to diagnose, the ones that don't reach a clean end.

Next lesson: 04 — The Operational Signals That Matter. With the problem now named with precision, we define the four signals worth capturing — error rate, per-tool failure rate, cost per run, latency per run — and calculate the first two over real runs.

Additional resources

1. Anthropic — Tool use (function calling) overview — The exact shape of tool_use/tool_result that history contains, and on top of which all of this guide's future observability is built.
2. Python — sys.exc_info and exception handling — Why a function's local state is lost when an exception propagates unhandled within that same function.
3. Python — Local variables and a stack frame's lifecycle — The exact technical basis for why history/messages stops existing the moment run_reservo_agent returns or raises an exception.
4. Python 3.14 — What's New — The version every code block in this lesson ran on, including the real RuntimeError.