The Honesty Problem: Modeled vs. Real Clock

Description

This is the lesson that precisely justifies the hard rule lesson 01 already previewed: why this guide — and this module in particular — never uses time.time() or time.perf_counter() in any code block later presented as executed. It isn't a technical limitation — nothing would stop this guide from wrapping every call with a real stopwatch — it's a deliberate decision, with a concrete, verifiable reason: a real stopwatch would break this guide's central promise, the one stated in every "What to expect" across the 64 lessons that make it up: "this is what you're going to get if you run this code on your own machine."

Connection to the module

This module's lessons 02 and 04 use TOOL_LATENCY_MS as if it were obvious that fixed data is the right choice. This lesson is the one that justifies that choice, laying the problem out in detail and with run-tested confirmation that modeled latency really is reproducible — the exact property a real stopwatch could never offer.

Analogy: the bakery scale, not the kitchen clock

A bakery that needs to standardize its recipes doesn't use a stopwatch to decide "how long" bread takes to make — it uses a scale, and states the recipe in grams: 500g of flour, 10g of salt, 7g of yeast. Those numbers are fixed, written into the recipe, and are the same today, tomorrow, and a year from now — they don't depend on how fast the baker on shift kneads, or on whether today's oven runs a bit hotter than yesterday's. If the bakery measured its recipes with a stopwatch instead of a scale — "knead for however long Juan kneaded last week" — every batch would come out different, because how long a human takes to knead depends on factors the recipe can't control.

TOOL_LATENCY_MS is this guide's scale. It doesn't measure "how long it took on your machine, at this instant" — that would be the stopwatch, and it varies — it declares, like a recipe, how much each tool "weighs" in time, fixed. Lesson 04 is going to show the exact four numbers; this lesson stays with the underlying problem: why a scale (fixed data) is the right tool for this guide, and a stopwatch (the real clock) isn't.

The problem, precisely: what would happen if this guide used the real clock

Imagine, for a moment, that the function summing a run's latency didn't consult TOOL_LATENCY_MS, but instead wrapped every tool call with time.perf_counter(), like this (this is an illustrative example, to reason about the problem — no code block in this module actually does this):

In prose, not in executed code: the idea would be to mark an instant right before calling the tool's real function (start = time.perf_counter()), call it, mark another instant right after (end = time.perf_counter()), and subtract the two to get how long that specific call took, in milliseconds.

With that idea (never implemented in this guide), three runs of the same script, on the same machine, a minute apart from each other, could give something like this:

CONCEPTO ILUSTRATIVO -- no es salida real de ningun codigo de esta guia:

corrida 1: list_rooms=1.8ms  get_quote=0.3ms  book_room=2.1ms  total=4.2ms
corrida 2: list_rooms=3.1ms  get_quote=0.4ms  book_room=1.9ms  total=5.4ms
corrida 3: list_rooms=1.2ms  get_quote=0.9ms  book_room=6.7ms  total=8.8ms

Three different "total latency" numbers, for exactly the same script, on the same machine. None of the three is "the correct number" — each one is an honest snapshot of how long that specific run took, at that specific instant, competing for CPU with whatever your operating system happened to be doing at that moment. That is exactly what makes a real stopwatch correct for production — there, you do want to know how long each run really took, with all its variability — and incorrect for a lesson's "What to expect" — here, you need the number you see on this page to be the same number you're going to see in your terminal, with no exceptions.

What's gained by modeling, confirmed with real code

TOOL_LATENCY_MS, on the other hand, never varies — because it doesn't measure anything, it declares something. Confirm it by running exactly the same run twice, in the same process:

import reservo_agent as ra

TOOL_LATENCY_MS = {
    "list_rooms": 40,
    "get_quote": 25,
    "book_room": 120,
    "cancel_booking": 90,
}


def total_run_latency_ms(history):
    latency_ms = 0
    tool_use_name = {}
    for turn in history:
        if turn["role"] != "assistant" or not isinstance(turn["content"], list):
            continue
        for block in turn["content"]:
            if block["type"] == "tool_use":
                tool_use_name[block["id"]] = block["name"]
    for turn in history:
        if turn["role"] != "user" or not isinstance(turn["content"], list):
            continue
        for block in turn["content"]:
            if block["type"] == "tool_result" and not block.get("is_error"):
                name = tool_use_name.get(block["tool_use_id"])
                latency_ms += TOOL_LATENCY_MS.get(name, 0)
    return latency_ms


script_sofia = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "book_room",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1, "member": "Sofía"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Boardroom pro por 1 hora para Sofía. Total $64.00. Confirmación #1."}]},
]

for intento in range(1, 4):
    final, history = ra.run_reservo_agent("Reserva Boardroom pro 1h para Sofía", script_sofia)
    latency_ms = total_run_latency_ms(history)
    print(f"intento {intento}: latencia total = {latency_ms} ms")

What to expect:

intento 1: latencia total = 185 ms
intento 2: latencia total = 185 ms
intento 3: latencia total = 185 ms

185, 185, 185 — exactly the same number, three times, with no variation at all, even though every attempt calls book_room for real again and creates a new booking (with a different booking_id each time, because BOOKINGS's state does advance between calls). The booking_id changes; the modeled latency doesn't — because TOOL_LATENCY_MS doesn't depend on which booking_id resulted, only on which tools got called. Compare it against the previous section's three illustrative numbers (4.2ms, 5.4ms, 8.8ms), which never matched each other. That difference — zero variation versus real variation — is, in a word, the honesty problem this lesson solves: modeled latency is reproducible by design, because it's declared data, not a measurement.

What's lost by modeling, stated with the same honesty

It would be dishonest to present this as a cost-free decision. TOOL_LATENCY_MS models each tool with a single fixed number, and that hides, on purpose, several things that do matter in a real system:

• A single tool's variability, run after run. In production, book_room doesn't always take exactly the same — it can depend on the database's load behind it, on contention with another simultaneous write, on network latency toward that service. TOOL_LATENCY_MS["book_room"] = 120 is a single point; reality is a distribution of values around that point (and sometimes far from it).
• Tail latency spikes. A tool that normally takes 120 ms can, occasionally, take 2,000 ms — because of a garbage collector pausing the process, because of a network connection dropping and retrying. Those spikes are, often, the most important part of a system's real latency, and a model of fixed numbers, by definition, can't represent them.
• Environment dependency. The same tool, running on a more loaded server or in a different geographic region, can have a completely different base latency. TOOL_LATENCY_MS assumes a single environment, always.

None of this invalidates the model — it contextualizes it. This module's purpose isn't teaching you how long book_room takes in a real system (only the real clock, on your own real system, answers that question); it's teaching you what to do with a latency measurement once you have it: how to sum it per run, how to read a percentile, how to identify which tool dominates. Those skills are exactly the same, whether applied to modeled data or to data really measured.

Common mistakes

1. Thinking "modeled" means "the exact number doesn't matter." It does matter — TOOL_LATENCY_MS is a fixed, cited constant, just like Module 3's claude-sonnet-5 pricing. Changing it mid-lesson, or making up a new value for a tool not in the dictionary, breaks the reproducibility this lesson demonstrated.

2. Adding time.perf_counter() "just to compare" inside an exercise. It's the easiest trap to fall into in this module — it looks harmless, "just to see how long it really takes on my machine." The problem is that, the instant you do it, your exercise's "What to expect" stops being reproducible for anyone else running it on a different machine.

3. Believing the "What would happen" section's example is real code from this guide. It isn't — it's explicitly marked CONCEPTO ILUSTRATIVO, in a text block, not in an executable Python block. No code block in this lesson — or in any other in this module — uses time.time() or time.perf_counter().

4. Thinking a model with a single fixed number per tool is "simpler" and therefore "less accurate" than measuring with the real clock. It is less accurate, yes — the previous section says so bluntly — but that's exactly the point: it sacrifices a single run's precision in exchange for the whole lesson's reproducibility. A real system needs both — the real clock to operate, the fixed model to teach and for Module 5's regression gate — and confusing which one belongs to which context is the underlying mistake this lesson tries to prevent.

5. Underestimating tail latency because this guide's model doesn't represent it. It's tempting, after working with TOOL_LATENCY_MS, to forget real variability exists and that occasional spikes are, often, what a real customer experiences as "the system felt slow." This module's lesson 06, on percentiles, exists precisely so that habit of thinking — looking beyond the average — sticks with you, even though this module's data is fixed.

Exercises

Exercise 1: Confirm reproducibility over a different script (Easy)

Run Ana's script (list_rooms, get_quote with tier="premium" rejected, get_quote with tier="pro", book_room) from Module 1 three times, and confirm total_run_latency_ms gives exactly the same number all three times.

script_ana = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Focus", "tier": "premium", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #1."}]},
]
for intento in range(1, 4):
    final, history = ra.run_reservo_agent("Reserva Focus pro 3h para Ana", script_ana)
    print(f"intento {intento}: latencia total = {total_run_latency_ms(history)} ms")

intento 1: latencia total = 185 ms
intento 2: latencia total = 185 ms
intento 3: latencia total = 185 ms

Exercise 2: Write, in one sentence, the difference between "modeled" and "estimated" (Medium)

Module 3 uses the word estimated for cost (len(text) // 4 is a deliberate approximation of a real token count that exists, even though this guide doesn't use it). This module uses the word modeled for latency. Explain, in one sentence, the difference between both terms, using this guide's two concrete cases.

Exercise 3: Describe, in prose, how this guide's "What to expect" would have to change if it used the real clock (Hard)

Without writing code — this exercise's point is to reason, not implement — if this guide decided, from this module onward, to measure latency with a real time.perf_counter(), describe in one paragraph how the following lessons' "What to expect" block format would have to change to remain honest with the reader, given that the exact number would no longer be reproducible.

Summary and next step

• We confirmed, with an illustrative example (never executed), that measuring latency with the real clock gives a different number every run — the variability is real information in production, but breaks a lesson's reproducibility.
• We confirmed, run three times over the same script, that total_run_latency_ms with TOOL_LATENCY_MS gives exactly the same result always — 185 ms, with no variation at all, across all three attempts.
• We honestly named what's lost by modeling: a single tool's run-to-run variability, tail latency spikes, environment dependency — none of the three is represented in a fixed-numbers model, and this guide states it explicitly instead of hiding it.
• With this problem solved, the rest of the module can build on top of TOOL_LATENCY_MS without justifying again why it's the right choice for this guide.

Next lesson: 04 — Tool Latency as Fixed Data. With the honesty problem now solved, we formalize the dictionary you're going to reuse unchanged until Module 8's close: where its four numbers come from, and why book_room is, by a wide margin, the most expensive.

Additional resources

1. Python — time — The official reference for time.perf_counter(), the function this lesson names in detail and never executes.
2. Anthropic — Building effective agents — On why a real agentic system's latency varies with the environment, the load, and the specific tool being called.
3. Python — determinism and reproducible tests — The random documentation, useful by contrast: it's exactly the kind of non-deterministic source this guide bans from any data, for the same reason it bans the real clock.
4. Python 3.14 — What's New — The version every code block in this lesson ran on, three times with the same result.