A First Look at Cost, Latency, and Errors

Description

Lesson 04's first two signals — error rate, per-tool failure rate — are calculated by counting what's already in history. The two remaining ones — cost per run and latency per run — are different: they're not in history in any format, so they have to be estimated (cost) or modeled (latency) with a new engineering layer. This lesson builds that layer, minimal but real, and runs it for the first time over Reservo's canonical run.

It's not the in-depth development of either one — that's this guide's Modules 3 and 4, with a breakdown per tool call, aggregation over large batches, and percentiles. It is, literally, what the title says: a first look, honest about its limits, that completes the module's four signals for the first time.

Connection to the module

With this lesson, lesson 04's four signals — error rate, per-tool failure rate, cost, latency — end up calculated, all of them, at least once, over a real run. That's exactly what lesson 08 (the mini-project) is going to wrap into a single reusable function.

Analogy: the two gauges the dashboard was still missing

Lesson 03 left the dashboard with two empty gauges: how much gas you consumed and how hot the engine ran. This lesson fills them in. The fuel gauge is cost: how much "fuel" — tokens, and their equivalent in cents — this particular trip consumed. The temperature gauge is latency: how hard the engine was pushed — how much time each component took — along the way. Neither gauge tells you on its own whether the trip was good or bad; just like in a car, they matter together with the other two, and above all they matter when you look at them across many trips, not just one.

Worked example: cost and latency, calculated for the first time

Cost: claude-sonnet-5's fixed formula

claude-sonnet-5's list price, verified against the official Claude documentation, is $3.00 per million input tokens and $15.00 per million output tokens. As a fixed constant, in cents:

INPUT_PRICE_CENTS_PER_MILLION_TOKENS = 300    # $3.00 / 1M tokens
OUTPUT_PRICE_CENTS_PER_MILLION_TOKENS = 1500  # $15.00 / 1M tokens


def estimate_cost_cents(input_tokens, output_tokens):
    return (
        input_tokens * INPUT_PRICE_CENTS_PER_MILLION_TOKENS
        + output_tokens * OUTPUT_PRICE_CENTS_PER_MILLION_TOKENS
    ) // 1_000_000

Getting to input_tokens/output_tokens requires a text estimation, with the same honest convention agent-fundamentals established back in its Module 6: len(text) // 4, an order of magnitude, never an exact count from a real tokenizer.

import json


def estimate_run_tokens(history):
    """Primera mirada, orden de magnitud (len//4): suma lo que entra al
    modelo como INPUT (la pregunta + cada tool_result) y lo que el modelo
    genera como OUTPUT (cada tool_use + el texto final)."""
    input_chars = 0
    output_chars = 0
    for turn in history:
        content = turn["content"]
        if isinstance(content, str):
            input_chars += len(content)
            continue
        for block in content:
            if block["type"] == "tool_result":
                input_chars += len(block["content"])
            elif block["type"] == "tool_use":
                output_chars += len(json.dumps(block["input"]))
            elif block["type"] == "text":
                output_chars += len(block["text"])
    return input_chars // 4, output_chars // 4

Notice the classification: the original question and each tool_result are text the model reads — input; every tool_use (what the model decides to request) and the final text are what the model generates — output. This is a deliberate simplification for a first look: it doesn't account for the fact that, in a real API, every turn resends the entire conversation history as part of that call's input — the exact turn-by-turn breakdown is Module 3's work. Here, it just sums the text that flowed in each direction over the whole run, once.

Latency: modeled, with an honest rule about when it counts

TOOL_LATENCY_MS = {
    "list_rooms": 40,
    "get_quote": 25,
    "book_room": 120,
    "cancel_booking": 90,
}


def estimate_run_latency_ms(history):
    """Suma la latencia modelada de cada tool que se EJECUTÓ de verdad.
    Un tool_use rechazado por validación (is_error, sin ejecutar la función
    real) no le agrega latencia al run -- nunca llegó a la tool."""
    total_ms = 0
    tool_use_name = {}
    for turn in history:
        if turn["role"] != "assistant" or not isinstance(turn["content"], list):
            continue
        for block in turn["content"]:
            if block["type"] == "tool_use":
                tool_use_name[block["id"]] = block["name"]
    for turn in history:
        if turn["role"] != "user" or not isinstance(turn["content"], list):
            continue
        for block in turn["content"]:
            if block["type"] == "tool_result" and not block.get("is_error"):
                name = tool_use_name.get(block["tool_use_id"])
                total_ms += TOOL_LATENCY_MS.get(name, 0)
    return total_ms

TOOL_LATENCY_MS is fixed data, not a measurement — the explicit honesty this guide has held since its DISEÑO: in real production, this is measured with a real clock (time.perf_counter() around each execute_tool); here it's modeled so the example is byte-for-byte reproducible on your machine. The design decision worth noting is the not block.get("is_error") condition: a tool_use that check_input_v2 rejects on validation — like the demo's tier="premium" — never runs the real Python function, so it makes no sense to load it with that tool's modeled latency. Only the tool calls that actually did execute — including ones that later result in a business-level is_error, like a cancel_booking that can't find the booking — contribute latency to the total.

All together, over the canonical run

import reservo_agent as ra

model_script_demo = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Focus", "tier": "premium", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #1."}]},
]

final, history = ra.run_reservo_agent("Reserva Focus pro 3h para Ana", model_script_demo)

in_tok, out_tok = estimate_run_tokens(history)
cost_cents = estimate_cost_cents(in_tok, out_tok)
latency_ms = estimate_run_latency_ms(history)
tool_calls = sum(1 for m in history if m["role"] == "assistant" and isinstance(m["content"], list)
                  and m["content"][0]["type"] == "tool_use")
tool_errors = sum(1 for m in history if m["role"] == "user" and isinstance(m["content"], list)
                   and any(b.get("is_error") for b in m["content"]))

print("RESPUESTA        :", final["content"][0]["text"])
print(f"tokens estimados  : input={in_tok} output={out_tok} (len//4, orden de magnitud)")
print(f"costo estimado    : {cost_cents} centavos")
print(f"latencia modelada : {latency_ms} ms")
print(f"tool calls        : {tool_calls}  (errores: {tool_errors})")

What to expect:

RESPUESTA        : Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #1.
tokens estimados  : input=64 output=56 (len//4, orden de magnitud)
costo estimado    : 0 centavos
latencia modelada : 185 ms
tool calls        : 4  (errores: 1)

Lesson 04's four signals, together, for the first time: 0 errors at the run level (completed), 1 of 4 tool calls with an error (25%), 0 cents in cost, 185 milliseconds of latency.

Why the cost came out 0, and why that's correct

0 centavos isn't a calculation error — it's the honest answer to a real question: how much does a text exchange of barely 64 input tokens and 56 output tokens cost, at claude-sonnet-5's list price? Do the math by hand: (64 * 300 + 56 * 1500) // 1_000_000 = (19200 + 84000) // 1_000_000 = 103200 // 1_000_000 = 0. The real cost, without rounding down to a cents integer, would be on the order of $0.001 — a tenth of a cent. With money represented in int cents (the hard convention of this guide and of the whole Reservo family of guides), that value rounds down to 0.

This doesn't mean cost doesn't matter — it means a single simple run, at this pricing, costs a fraction of a cent, and the signal only becomes visible in aggregate. Multiply that same token pattern by volume, without running any additional runs — it's simple arithmetic, not a real run:

for n in [1, 100, 10_000]:
    print(n, "runs similares ~", estimate_cost_cents(in_tok * n, out_tok * n), "centavos")

What to expect:

1 runs similares ~ 0 centavos
100 runs similares ~ 10 centavos
10000 runs similares ~ 1032 centavos

Ten thousand runs like Ana's would cost, in total, about $10.32 — a number that starts to mean something for a business decision. This is exactly why this guide's Module 3 aggregates cost over real batches of runs, not over just one: the signal exists, but only becomes useful at the scale where it actually matters.

Why latency stays the same even when the number of steps changes

Notice something worth confirming with your own eyes before moving on: if you ran lesson 04's Run C — the one with five tool calls and two errors, instead of four tool calls and one — that run's modeled latency also comes out to 185 milliseconds, exactly the same as Run A's. That's not a coincidence of the code: it's the direct consequence of the rule you defined above. Both runs, no matter how many validation-rejected attempts they had along the way, end up running exactly the same sequence of real tools — list_rooms, one valid get_quote, book_room — and that real sequence is what determines the latency. An invalid tier or an hours=0 cost steps in the trace and turns in the history, but they don't cost real time, because check_input_v2 stops them before the tool ever runs. This is one of the reasons per-tool failure rate and latency are independent signals: a high failure rate doesn't necessarily mean a slower run.

Common mistakes

1. Seeing 0 centavos and assuming the calculation is wrong. It isn't — it's the correct answer for a run this size at this pricing. The real mistake would be not verifying it by hand (as the previous section did) before dismissing it as a bug.

2. Charging latency to a tool_use that never ran. If estimate_run_latency_ms summed the latency of every tool_use without checking is_error on its tool_result, the rejected tier="premium" would add an extra 25 ms to the total — time that, in reality, never happened, because check_input_v2's validation is a local Python function, not a call that takes time.

3. Confusing this lesson's token estimation with an exact count. len(text) // 4 is, by design, an order of magnitude — useful for having an approximate cost figure, useless as a source of truth for a real invoice. Any business decision that depends on an exact token count needs a real tokenizer's actual count, not this estimate.

4. Multiplying by volume and thinking that's "already" Module 3's aggregation. This lesson's scaling section is simple arithmetic over a single pattern repeated n times — it's not the same as summing the cost of n distinct runs, each with different tokens, which is what Module 3 actually does over a real batch.

5. Forgetting to label the pricing as "list price." A promotional price different from the list price existed, and could exist again — any cost figure you share outside this guide should clarify which price it was calculated against, so it isn't mistaken for a real billing quote.

Exercises

Exercise 1: Calculate Run B's cost and latency (Easy)

Using estimate_run_tokens, estimate_cost_cents, and estimate_run_latency_ms, calculate the four signals for Sofía's script from lesson 04 — Boardroom, pro, 1 hour, with no error along the way.

script_b = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "book_room",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1, "member": "Sofía"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Boardroom pro por 1 hora para Sofía. Total $64.00. Confirmación #2."}]},
]
final_b, history_b = ra.run_reservo_agent("Reserva Boardroom pro 1h para Sofía", script_b)
in_b, out_b = estimate_run_tokens(history_b)
print(f"tokens: input={in_b} output={out_b}")
print("costo  :", estimate_cost_cents(in_b, out_b), "centavos")
print("latencia:", estimate_run_latency_ms(history_b), "ms")

tokens: input=53 output=49
costo  : 0 centavos
latencia: 185 ms

Exercise 2: How many runs like Sofía's does it take to spend a dollar? (Medium)

Using the Exercise 1 result, calculate how many runs identical to Sofía's it takes for the total cost to exceed 100 cents ($1.00). Don't run any real runs — use the scaling arithmetic from the "Why the cost came out 0" section.

in_b, out_b = 53, 49
for n in [1_000, 5_000, 10_000, 20_000]:
    print(n, "runs ->", estimate_cost_cents(in_b * n, out_b * n), "centavos")

1000 runs -> 89 centavos
5000 runs -> 447 centavos
10000 runs -> 894 centavos
20000 runs -> 1788 centavos

Exercise 3: Design a script that maximizes modeled latency without adding any errors (Hard)

With the four tools and their latencies (list_rooms=40, get_quote=25, book_room=120, cancel_booking=90), design a turn script — valid, with no is_error at all — that results in the highest possible modeled latency using each of the four tools exactly once. Calculate the expected latency before running it, and confirm.

script_all_four = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Studio", "tier": "basic", "hours": 1}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "book_room",
         "input": {"room": "Studio", "tier": "basic", "hours": 1, "member": "Nico"}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "cancel_booking", "input": {"id": 3}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé y cancelé Studio basic 1h para Nico."}]},
]
final_4, history_4 = ra.run_reservo_agent("Reserva y cancela Studio basic 1h para Nico", script_all_four)
print("latencia:", estimate_run_latency_ms(history_4), "ms")

latencia: 275 ms

Summary and next step

• For the first time in this guide, we built a cost estimator (len(text)//4 + claude-sonnet-5's list price, $3.00/$15.00 per million tokens) and a latency model (TOOL_LATENCY_MS, summed only over tool calls that actually ran).
• We ran both over Ana's canonical run: 0 cents, 185 ms — and confirmed, doing the math by hand, why 0 cents is the correct answer for a run of that size, not a bug.
• We discovered, with real execution, that modeled latency depends on the sequence of tools that actually ran, not the number of attempts in the trace — an invalid tier doesn't add time to the run, because it never reaches the tool.
• With this, lesson 04's four signals are now calculated, all of them, over the same run — the exact foundation lesson 08 is going to wrap into a single function.

Next lesson: 06 — Operating vs. Building: The Boundary. With the four signals now in hand, we precisely trace where what agent-fundamentals already built ends and this guide begins — and where this guide (the agent) ends and sre-and-incident-response-guide (the infrastructure) begins.

Additional resources

1. Anthropic — Pricing — The official source for claude-sonnet-5's list price ($3.00/$15.00 per million tokens) that this lesson fixes as a constant.
2. Anthropic — Token counting — The exact token count from a real tokenizer, versus the order-of-magnitude estimate (len//4) this lesson uses.
3. Python — json — json.dumps, used to estimate the text size of each tool_use.
4. Python 3.14 — What's New — The version every real calculation in this lesson ran on.