Scaling Cost to Thousands of Runs

Description

Every CostReport you calculated in lesson 05 ended up with the same thing: cost_cents = 0. That's not a bug — it's the honest answer for a run this size, the same conclusion Module 1 already reached with run_and_observe. A Reservo run typically moves between fifty and a hundred and twenty tokens total; at claude-sonnet-5's list price, that's such a tiny fraction of a cent that integer arithmetic rounds it down, hopelessly, to zero.

This lesson picks back up lesson 01's electricity-meter analogy: an iron left on for half an hour "barely costs anything," looked at alone — but a building with hundreds of irons, every day, does generate a real bill. This lesson makes that same leap with the Reservo agent: from a run that costs 0 cents to a projection of what 1,000, 10,000, and 100,000 runs of the same type would cost — and confirms, with real numbers, why the only correct way to make that leap is summing tokens before rounding, never summing cents figures that were already rounded separately.

Connection to the module

This lesson adds batch aggregation to observability/cost_calculator.py: a function that sums several CostReports' tokens and applies estimate_cost_cents once over the total, plus the scale-projection pattern that reuses lesson 05's same formula with no change at all.

Aggregating a real batch: four runs, one honest total

Before projecting anything into the future, aggregate the cost of a batch of runs that actually ran, for real, in this lesson. Four different Reservo tasks, each with its own trace_id:

import logging
import reservo_agent as ra
import run_logger as rl

rl.logger.setLevel(logging.CRITICAL)  # silenciamos el detalle de traced_run para este resumen

script_a = [  # Ana: Focus pro 3h, con un tier inválido corregido en el camino
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Focus", "tier": "premium", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #1."}]},
]
script_b = [  # Sofía: Boardroom pro 1h, limpio
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "book_room",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1, "member": "Sofía"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Boardroom pro por 1 hora para Sofía. Total $64.00. Confirmación #2."}]},
]
script_d = [  # Diego: reserva y cancela, dos pasos
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Studio", "tier": "basic", "hours": 1, "member": "Diego"}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "cancel_booking", "input": {"id": 4}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé y luego cancelé Studio basic 1h para Diego."}]},
]
# Carla: compara las seis combinaciones de sala/tier antes de decidir -- el run más largo del lote.
rooms, tiers = ["Focus", "Studio", "Boardroom"], ["basic", "pro"]
script_compare = [{"stop_reason": "tool_use", "content": [
    {"type": "tool_use", "id": "toolu_00", "name": "list_rooms", "input": {}}]}]
for i, (room, tier) in enumerate([(r, t) for r in rooms for t in tiers], start=1):
    script_compare.append({"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": f"toolu_{i:02d}", "name": "get_quote",
         "input": {"room": room, "tier": tier, "hours": 2}}]})
script_compare.append({"stop_reason": "tool_use", "content": [
    {"type": "tool_use", "id": "toolu_99", "name": "book_room",
     "input": {"room": "Boardroom", "tier": "pro", "hours": 2, "member": "Carla"}}]})
script_compare.append({"stop_reason": "end_turn", "content": [
    {"type": "text", "text": "Comparé las seis combinaciones de sala y tier. Reservé Boardroom pro por 2 horas para Carla. Total $128.00. Confirmación #1."}]})

tasks = [
    ("Reserva Focus pro 3h para Ana", script_a),
    ("Reserva Boardroom pro 1h para Sofía", script_b),
    ("Reserva y cancela Studio basic 1h para Diego", script_d),
    ("Compara todas las salas antes de reservar la mejor opción para Carla", script_compare),
]

reports = []
for i, (question, script) in enumerate(tasks, start=1):
    with rl.traced_run(question, i) as trace_id:
        final, history = ra.run_reservo_agent(question, script)
    reports.append(cost_for_run(trace_id, question, history))

print(f"{'trace_id':<18} {'in':>4} {'out':>4} {'total':>6} {'cost_cents':>11}  pregunta")
for r in reports:
    total = r.input_tokens + r.output_tokens
    print(f"{r.trace_id:<18} {r.input_tokens:>4} {r.output_tokens:>4} {total:>6} {r.cost_cents:>11}  {r.question!r}")

What to expect:

trace_id             in  out  total  cost_cents  pregunta
run-8487582448eb     64   56    120           0  'Reserva Focus pro 3h para Ana'
run-ae6ff85cf0b0     53   49    102           0  'Reserva Boardroom pro 1h para Sofía'
run-2c27934d8a39     27   31     58           0  'Reserva y cancela Studio basic 1h para Diego'
run-cecde864aa84     88  118    206           0  'Compara todas las salas antes de reservar la mejor opción para Carla'

Four runs, four different trace_ids, and four times cost_cents = 0 — each one, individually, honest. Now aggregate the whole batch:

def aggregate_reports(reports):
    """Suma los tokens de TODOS los reports primero, y aplica
    estimate_cost_cents UNA SOLA VEZ sobre el total -- nunca suma
    cost_cents individuales, ya redondeados."""
    total_input = sum(r.input_tokens for r in reports)
    total_output = sum(r.output_tokens for r in reports)
    return total_input, total_output, estimate_cost_cents(total_input, total_output)


total_in, total_out, total_cost = aggregate_reports(reports)
print("input_tokens totales :", total_in)
print("output_tokens totales:", total_out)
print("costo total del lote :", total_cost, "centavos")

What to expect:

input_tokens totales : 232
output_tokens totales: 254
costo total del lote : 0 centavos

Still 0 — four runs is still very few accumulated tokens (232 + 254 = 486) to cross the threshold where integer arithmetic starts producing anything other than zero (remember lesson 04: it takes numbers in the thousands or millions of tokens for // 1_000_000 to stop flattening everything to 0). This result is still correct and honest — the real leap comes next.

From four runs to a hundred thousand: the projection

The Reservo agent, in production, doesn't handle four tasks a day — it handles thousands. Take the batch's most representative run — Ana's, 64 input tokens and 56 output — and project its cost if that same pattern repeated 1,000, 10,000, and 100,000 times:

per_run_input, per_run_output = 64, 56  # el run de Ana, del ejemplo trabajado

print(f"{'runs':>9} {'input_tokens':>13} {'output_tokens':>14} {'cost_cents':>11} {'dólares':>10}")
for n in (1, 10, 100, 1_000, 10_000, 100_000):
    input_tokens = per_run_input * n
    output_tokens = per_run_output * n
    cost = estimate_cost_cents(input_tokens, output_tokens)
    print(f"{n:>9} {input_tokens:>13} {output_tokens:>14} {cost:>11} {f'${cost/100:.2f}':>10}")

What to expect:

     runs  input_tokens  output_tokens  cost_cents    dólares
        1            64             56           0      $0.00
       10           640            560           1      $0.01
      100          6400           5600          10      $0.10
     1000         64000          56000         103      $1.03
    10000        640000         560000        1032     $10.32
   100000       6400000        5600000       10320    $103.20

There's the analogy's whole leap: one run costs $0.00 by this arithmetic — an iron left on for half an hour — but 100,000 runs of the same type cost $103.20 — the whole building's month-end bill. Notice the 10-run row too: it's the first one where the cost stops being 0 (1 cent) — the exact point where an individual run's tiny fraction starts accumulating into something integer arithmetic can already represent.

The real mistake: summing already-rounded cents, instead of tokens

This is why lesson 03 insisted, with small examples, on the difference between summing rounded fragments and concatenating before rounding. Here that difference stops being a curiosity and becomes a real budgeting mistake. Compare the two ways of projecting the cost of 1,000, 10,000, and 100,000 runs of Ana's type:

per_run_cost = estimate_cost_cents(per_run_input, per_run_output)
print("costo de UN run, redondeado:", per_run_cost, "centavos")
print()
print(f"{'runs':>9} {'naive (cost_cents * n)':>24} {'correcto (tokens * n primero)':>32}")
for n in (1_000, 10_000, 100_000):
    naive_total = per_run_cost * n
    correct_total = estimate_cost_cents(per_run_input * n, per_run_output * n)
    print(f"{n:>9} {naive_total:>24} {correct_total:>32}")

What to expect:

costo de UN run, redondeado: 0 centavos

     runs   naive (cost_cents * n)   correcto (tokens * n primero)
     1000                        0                              103
    10000                        0                             1032
   100000                        0                            10320

The "naive" method — taking a run's already-rounded cost (0 cents) and multiplying it by the number of runs — predicts, in all three cases, a total cost of $0.00, no matter how many runs get projected. It's arithmetically correct (0 * n always gives 0), but it's a false answer about the real budget: the correct method — multiplying the tokens by n first, and applying estimate_cost_cents once over that total — shows that 100,000 runs of this type do cost $103.20, a figure any real budget needs to know. The exact cause is the same one you already saw in lesson 03: every individual run's rounding-down discards a fraction of a cent that, in isolation, looks irrelevant, but that multiplying by thousands makes significant. Rounding before scaling loses that information forever; rounding after scaling preserves it.

Common mistakes

1. Reporting "this run costs $0.00, so it doesn't matter" without projecting to scale. This lesson's example directly disproves it: $0.00 per individual run and $103.20 per every 100,000 runs are the same information, seen at two different scales — neither one is "the truth" without the other.

2. Multiplying an already-calculated CostReport's cost_cents by the expected number of runs. This is, precisely, this lesson's "naive" mistake. The correct way is multiplying input_tokens/output_tokens by n, and applying estimate_cost_cents once over those scaled totals.

3. Assuming every run in a real batch has exactly the same tokens as the run used for projecting. The worked example used Ana's run (120 total tokens) as representative, but the real batch had runs ranging from 58 to 206 tokens. A serious projection uses the batch's average — or, better, aggregates the whole batch's real tokens and scales that total — instead of a single hand-picked run.

4. Confusing "aggregating a batch" (aggregate_reports, over runs that actually happened) with "projecting to scale" (multiplying a representative run's tokens by n). They're two related but distinct operations: the first sums real tokens from runs that already ran; the second multiplies a run's tokens (real or averaged) by a hypothetical number of future repetitions. Mixing them up without clarifying which one you're doing is an easy source of confusion when reading a report.

5. Thinking this scaling already "is" predicting production cost. It's a simple arithmetic projection, useful for having an order-of-magnitude figure — again, the same honesty from lesson 03 — not a real traffic model. The real number of runs per day, the real mix of simple and complex tasks, and each conversation's real variability are business questions this guide doesn't try to answer — Module 8 is going to show how this same calculation applies over real data from a larger batch, without pretending it replaces a complete business projection.

Exercises

Exercise 1: Project Sofía's run's cost to 1,000 runs (Easy)

Using Sofía's run's tokens (53 input, 49 output, from this lesson's worked example), calculate the projected cost of 1,000 runs of that same type.

print(estimate_cost_cents(53 * 1_000, 49 * 1_000))

89

Exercise 2: Confirm the naive method never detects Diego's cost, not even at 50,000 runs (Medium)

Diego's run (27 input tokens, 31 output) is the batch's cheapest. Calculate its rounded individual cost, and compare the naive method against the correct one for 50,000 runs of that type.

per_run_cost_diego = estimate_cost_cents(27, 31)
naive_50k = per_run_cost_diego * 50_000
correct_50k = estimate_cost_cents(27 * 50_000, 31 * 50_000)

print("costo de UN run de Diego (redondeado):", per_run_cost_diego, "centavos")
print("naive a 50.000 runs   :", naive_50k, "centavos")
print("correcto a 50.000 runs:", correct_50k, "centavos =", f"${correct_50k/100:.2f}")

costo de UN run de Diego (redondeado): 0 centavos
naive a 50.000 runs   : 0 centavos
correcto a 50.000 runs: 2730 centavos = $27.30

Exercise 3: Find the number of Diego runs at which the correct cost stops being 0 (Hard)

Using Diego's run's tokens (27/31), write a loop that finds the first value of n (starting at 1) for which estimate_cost_cents(27 * n, 31 * n) stops being 0. Confirm your result by calculating the cost at n - 1 and at n.

n = 1
while estimate_cost_cents(27 * n, 31 * n) == 0:
    n += 1

print("primer n con costo > 0:", n)
print(f"costo en n={n - 1}:", estimate_cost_cents(27 * (n - 1), 31 * (n - 1)), "centavos")
print(f"costo en n={n}    :", estimate_cost_cents(27 * n, 31 * n), "centavos")

primer n con costo > 0: 19
costo en n=18: 0 centavos
costo en n=19: 1 centavos

Summary and next step

• We aggregated a real batch of four Reservo runs with aggregate_reports: sum tokens first, apply estimate_cost_cents once — 232 input tokens, 254 output, 0 cents, a result still honest at this scale.
• We projected Ana's run's cost to 1,000, 10,000, and 100,000 runs: from $0.00 to $103.20 — the electricity-meter analogy, confirmed with real arithmetic.
• We confirmed, with real execution, this scaling's central mistake: multiplying an already-rounded cost_cents by the number of runs always predicts $0.00, no matter the scale; multiplying the tokens first and rounding at the end reveals the real cost.
• Exercise 3 quantified the exact threshold: it takes 19 repetitions of the batch's cheapest run for integer arithmetic to stop showing 0 — the underlying reason this guide talks about "thousands of runs," not a handful.

Next lesson: 07 — Cost as an Operational Signal. With aggregation and scaling now solved, we treat cost as one more signal, alongside Module 1's error rate and per-tool failure rate: how to detect an abnormally expensive run, and the exact boundary with the guide that teaches you how to reduce it.

Additional resources

1. Python — sum() — The function used in aggregate_reports to sum several CostReports' tokens before applying pricing.
2. Python — arithmetic operators (//, *) — The exact foundation of why summing before rounding gives a different — and more correct — result than rounding before summing.
3. Anthropic — Pricing — The source for claude-sonnet-5's list price, reused unchanged from lesson 04 in every calculation in this lesson.
4. Anthropic — Building effective agents — On why projecting an agentic system's cost at production scale, not just measuring it per run, is part of operating it responsibly.
5. Python 3.14 — What's New — The version every line of code in this lesson ran on.