Module 3: Measuring Cost and Tokens per Run

Module 3: Measuring Cost and Tokens per Run

Next →

Description

Module 2 solved a real problem: a run that failed with RuntimeError stopped losing all its information. traced_run records every step of the loop — every tool_use, every tool_result, the run's close — the exact instant it happens, with a deterministic trace_id that correlates everything, even when the whole run crashes. By the end of that module, you had a complete observability/run_logger.py and a real RUN_LOG.jsonl on your disk, able to reconstruct any run's complete story.

But there's a question that file, as it stands, cannot answer. Run the same clean run from Module 2 again — "book Focus pro 3h for Ana," with no error — and look carefully at what fields each line carries:

import logging
import run_logger as rl
import reservo_agent as ra

rl.logger.setLevel(logging.INFO)

script_a = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro 3h para Ana. Confirmación #1."}]},
]
with rl.traced_run("Reserva Focus pro 3h para Ana", 1) as trace_id:
    final, history = ra.run_reservo_agent("Reserva Focus pro 3h para Ana", script_a)

What to expect:

{"seq": 1, "trace_id": "run-8487582448eb", "event": "run_started", "question": "Reserva Focus pro 3h para Ana", "tool_errors": 0, "error": ""}
{"seq": 2, "trace_id": "run-8487582448eb", "event": "tool_use", "step": 1, "tool": "list_rooms", "is_error": false, "content": ""}
{"seq": 4, "trace_id": "run-8487582448eb", "event": "tool_result", "step": 1, "tool": "list_rooms", "is_error": false, "content": "[{\"room\": \"Focus\", \"rate_cents\": 2500}, {\"room\": \"Studio\", \"rate_cents\": 4000}, {\"room\": \"Boardroom\", \"rate_cents\": 8000}]"}
{"seq": 6, "trace_id": "run-8487582448eb", "event": "tool_use", "step": 2, "tool": "get_quote", "is_error": false, "content": ""}
{"seq": 8, "trace_id": "run-8487582448eb", "event": "tool_result", "step": 2, "tool": "get_quote", "is_error": false, "content": "{\"price_cents\": 6000}"}
{"seq": 10, "trace_id": "run-8487582448eb", "event": "tool_use", "step": 3, "tool": "book_room", "is_error": false, "content": ""}
{"seq": 12, "trace_id": "run-8487582448eb", "event": "tool_result", "step": 3, "tool": "book_room", "is_error": false, "content": "{\"booking_id\": 1, \"confirmed\": true}"}
{"seq": 14, "trace_id": "run-8487582448eb", "event": "run_finished", "question": "Reserva Focus pro 3h para Ana", "tool_errors": 0, "error": ""}

Eight lines, complete, correlated by run-8487582448eb. You know with certainty how many steps the run took, which tool each one called, whether anything failed. And yet, not one of the eight lines — not a single key in any JSON object — says how much this run cost. RunEvent has seq, trace_id, event, question, tool_errors, error. ToolCallEvent has seq, trace_id, event, step, tool, is_error, content. Nowhere is there a tokens field, a cost_cents field, or anything resembling one. Module 2 completely solved the question "what happened in this run?" — and left the question "how much did it cost for it to happen?" entirely untouched. That is, precisely, this module's problem.

Connection to the module

This module doesn't replace anything from Module 2 — it uses it. Every lesson that follows reuses traced_run and the deterministic trace_id exactly as they stood, and builds, alongside them, a new artifact: observability/cost_calculator.py. The bridge is literal: a CostReport in this module is identified by the same trace_id you already saw in RUN_LOG.jsonl — the same correlation as always, now applied to cost.

Hard rule for this guide (inherited, with one new precision)

Modules 1 and 2's hard rule stays exactly the same: the model's decision is never executed. This module adds one precision, because the topic now is money, and getting money wrong — even estimated money — is worse than not calculating it:

  • Cost is estimated, and the cost arithmetic really DOES run, for real. The text-to-tokens conversion uses, unchanged, the honest convention you already saw named in the previous two modules: len(text) // 4, always labeled as an order-of-magnitude estimate — never as a real tokenizer's exact count. This module is where that convention, mentioned only in passing so far, becomes a real, tested function, used to calculate money.
  • claude-sonnet-5's pricing is a fixed, cited constant. $3.00 per million input tokens, $15.00 per million output tokens — the list price, verified against the official Claude documentation. This module fixes that constant once, in lesson 04, and the rest of the guide reuses it without citing it again.
  • The LLM call is still concept. The Claude API is never called to measure a real cost — cost is always calculated with the fixed formula over estimated tokens.
  • No randomness and no real clock. No random, no datetime.now(), no uuid4(), no time.time(). The trace_ids remain Module 2's deterministic ones.

Hold onto this sentence, because you'll use it in every lesson that follows: Module 2 tells you WHAT happened in a run; this module tells you HOW MUCH it cost for it to happen — with the same correlation, the same trace_id, without rebuilding anything about the loop.

Where we are in the ecosystem

Agents in production — operating the Reservo agent
├── Module 1: Why Operating Is Different From Building
├── Module 2: Structured Logging and Tracing a Run
├── Module 3: Measuring Cost and Tokens per Run  ← YOU ARE HERE
│   → tokens as the unit of cost, len(text)//4, claude-sonnet-5's
│     fixed pricing, cents per run, scaling to thousands of runs,
│     cost as an operational signal
├── Module 4: Measuring Latency Honestly
├── Module 5: Regression Evals as a Production Gate
├── Module 6: Failures at Scale — Backoff, Circuit Breakers, and Rate Limits
├── Module 7: Versioning and Safe Rollout
└── Module 8: Project — the Reservo Agent in Production

This is the second of the seven modules that build Module 1's four disciplines: observe → measure → gate → harden+version. Module 2 built the observe layer. This module is the first half of the measure layer — the second half, latency, is Module 4. observability/cost_calculator.py, the artifact built here, gets reused unchanged from Module 4 onward, exactly as run_logger.py is reused starting in this module.

This module's central analogy: the electricity meter, before the bill arrives

An appliance — an iron, a refrigerator, a charger — consumes electricity while it's on, and that consumption is measured in a precise unit: kilowatt-hours. A single iron, on for half an hour, consumes such a tiny fraction of a kilowatt-hour that, if you looked only at that half hour, you'd say it "barely costs anything" — and you'd be right, for that isolated use. But an electric company doesn't bill one iron; it bills an entire building, with hundreds of appliances turned on, every day, all month long. The bill that arrives at month's end is the sum of thousands of those tiny fractions — and that sum is a figure a business does need to budget carefully for.

A Reservo agent run is exactly that iron, on for half an hour: it consumes tokens — a language model's unit of cost, as real as an appliance's kilowatt-hour — and that fraction of a cent, looked at one run at a time, seems insignificant. But Reservo doesn't serve one user a day — it serves thousands, each generating its own token consumption. Measuring cost per run, before the full month's "bill" arrives, is exactly what this module teaches you to do: read each individual iron's meter, so you can budget the whole building's bill before it arrives — not after, when it's already too late to act on it.

The case that keeps accompanying the guide: Reservo, without rebuilding anything

The four tools are the same as always — list_rooms(), get_quote(room, tier, hours), book_room(room, tier, hours, member), cancel_booking(id) — and the two price anchors remain intact: Focus basic 3h = 7500 cents, Focus pro 3h = 6000 cents. This module doesn't declare a single new tool, doesn't touch reservo_tools.py, reservo_contracts.py, reservo_robust.py, or reservo_agent.py, and doesn't touch Module 2's observability/run_logger.py either — it imports it, as is, and builds a new artifact alongside it:

  • Lesson 03 builds estimate_tokens(text), the function that applies len(text) // 4 for real, with its limits confirmed over real Reservo texts.
  • Lesson 04 fixes claude-sonnet-5's pricing constant — cited, with its honest note on the promotional price — and the two per-million-token cents constants.
  • Lesson 05 combines both pieces into estimate_cost_cents and cost_for_run, run over a real Reservo run, with a per-tool-call cost breakdown within the run.
  • Lesson 06 adds batch aggregation and scaling to thousands of runs — the part of the analogy where the tiny fraction becomes a real bill.
  • Lesson 07 treats cost as one more operational signal, alongside Module 1's error rate and per-tool failure rate, and traces the boundary with cost-optimization-caching-guide — the guide that teaches you to reduce cost, not to measure it.
  • Lesson 08 closes the module with a complete observability/cost_calculator.py, run over a batch of runs with trace_ids, in a comprehensive cost report.

As in every module of this guide: identifiers and code in English; prose and comments in Spanish; money, always in int cents.

Prerequisites

Required knowledge:

  • ✅ Having completed Module 2 of this guide, especially lessons 04 (make_trace_id, open_run) and 05-06 (complete traced_run, with ToolCallEvent). This module imports run_logger without re-explaining any of its pieces.
  • ✅ Having completed (or knowing well) agent-fundamentals-and-tool-calling-guide M8: run_reservo_agent, and history's format (user/assistant turns, tool_use/tool_result/text blocks).
  • ✅ Python: functions, dataclasses, a basic grasp of integer arithmetic (//, the integer-division operator you already used for Reservo's pro discount, * 80 // 100).

Recommended:

  • ✅ Having seen, at some point, an invoice from a usage-based API service and asked yourself "exactly where does this number come from?" — that question is, precisely, what this module answers for the Reservo agent.

NOT required:

  • ❌ You don't need an API key or an internet connection: the model's decision is still concept, and all of this module's cost calculation runs 100% locally, with pure integer arithmetic.
  • ❌ You don't need to install tiktoken or any real tokenizer. This guide deliberately uses the len(text) // 4 estimate — lesson 03 precisely explains why, and what's lost by doing it this way.
  • ❌ You don't need to know anything about real LLM-provider billing beyond the price table lesson 04 fixes as a constant — there are no contracts, volume-discount tiers, or anything like that within this module's scope.

Environment:

  • ✅ Python 3.14.0 with its standard library (dataclasses, json, itertools, statistics). Nothing to install.
  • ✅ Module 2's observability/ directory, with run_logger.py already built and working.

Module roadmap

Lesson 01 — Module introduction (this one)

The exact limit Module 2 leaves behind — a complete RUN_LOG.jsonl, with no cost field at all — the electricity-meter analogy, and the map of the eight lessons.

Lesson 02 — Tokens Are the Unit of Cost

What a token is, why an LLM provider charges per token instead of per call or per second, and the difference between counting characters, words, and tokens over the same text.

Lesson 03 — Estimating Tokens with len // 4

estimate_tokens(text), run over real Reservo texts, with its limits confirmed: why it's an order-of-magnitude estimate, never an exact count, and in which cases it's most wrong.

Lesson 04 — The claude-sonnet-5 Pricing

The fixed, cited constant: $3.00/$15.00 per million input/output tokens, list price — with the honest note on the promotional price, and why output costs five times more than input.

Lesson 05 — Cost per Run, in Cents

estimate_cost_cents and cost_for_run, run over a real Reservo run correlated by its trace_id, with a cost breakdown for every tool call within the run.

Lesson 06 — Scaling Cost to Thousands of Runs

From a run that costs a fraction of a cent to a projection over 1,000, 10,000, and 100,000 runs — and the real mistake of summing already-rounded cents instead of summing tokens first.

Lesson 07 — Cost as an Operational Signal

Cost alongside Module 1's error rate and per-tool failure rate: how to detect an abnormally expensive run, and the boundary with cost-optimization-caching-guide.

Lesson 08 — Mini-Project: A Cost Report for Reservo Runs

A complete observability/cost_calculator.py, run over a batch of traced Reservo runs, with a comprehensive cost report: per run, per batch, and scaled.

Progression map

Lesson 01 (this)  → Module 2's limit, the meter analogy
Lesson 02         → What a token is, why it's billed per token
Lesson 03         → estimate_tokens(text) = len(text) // 4, run for real
Lesson 04         → claude-sonnet-5's fixed pricing
Lesson 05         → Cost per run, with a per-tool-call breakdown
Lesson 06         → From cents per run to the bill for thousands of runs
Lesson 07         → Cost as a signal, the boundary with "reduce cost"
Lesson 08         → Mini-project: the complete cost report

Difficulty: ⭐⭐ ──────────────────▶ ⭐⭐⭐

What you'll achieve in this module

By completing the 8 lessons, you'll be able to:

  1. Explain why an LLM provider charges per token, and tell a token apart from a character and from a word over the same text.
  2. Estimate tokens with len(text) // 4, and precisely explain why it's an order-of-magnitude estimate — never an exact count — and in which cases it drifts furthest from reality.
  3. Cite claude-sonnet-5's pricing ($3.00/$15.00 per million input/output tokens, list price) and explain why output cost weighs more than input.
  4. Calculate a run's cost in cents, with a breakdown per tool call, correlated by its trace_id.
  5. Scale a run's cost to a batch of thousands, and avoid the mistake of summing already-rounded figures instead of summing tokens before rounding.
  6. Use cost as an operational signal, detect an abnormally expensive run, and trace the boundary with the guide that teaches you to reduce cost, not measure it.

Before and after

BEFORE the module:
→ "a run's cost is a billing detail, not something you
  calculate with code"
→ "if a run costs a fraction of a cent, it's not worth
  measuring"
→ "a model's price is a number I look up when I need it, not
  something I fix in the code"
→ "all tokens cost the same, whether input or output"

AFTER the module:
→ cost per run is a first-class operational signal, calculated
  with the same integer arithmetic as the rest of the system
→ a fraction of a cent, multiplied by thousands of runs, is a
  real budget figure -- measuring it per run means reading it
  BEFORE the bill arrives
→ pricing is a fixed constant, cited once, reused throughout
  the guide
→ an output token costs five times more than an input one --
  an asymmetry with real design consequences

Traps to avoid in this module

1. "This module is going to call the Claude API to measure real cost"

No. This guide's hard rule — inherited from agent-fundamentals, context-engineering, and this same guide's previous two modules — prohibits network calls in all of its engineering. Cost is estimated: tokens with len(text) // 4, fixed price, integer arithmetic. Lesson 03 is explicit about what's lost by not using a real tokenizer, and why that loss is an acceptable cost for this guide.

2. "If cost_cents comes out 0, something's broken"

No — it's, often, the correct and honest answer for a run of this size. Lesson 05 confirms it with real execution: a typical Reservo run, with a few dozen input and output tokens, costs a fraction of a cent so small that integer division rounds it down to 0. That's not a bug in the code — it's the reality of integer arithmetic with int, and lesson 06 shows exactly why that 0 is still real information, not noise.

3. "Input cost and output cost are, roughly, the same"

Lesson 04 confirms it with numbers: claude-sonnet-5's output price is five times its input price ($15.00 versus $3.00 per million tokens). That asymmetry isn't a detail — it changes which part of a run is worth watching closely.

4. "Measuring a run's cost is the first step toward lowering it"

Not in this guide. This module stops, precisely, at "how much did this run cost and why" — never at "how do I lower it." Prompt caching, cost-based model selection, batching: that's cost-optimization-caching-guide, precisely named in lesson 07. Confusing measuring with optimizing is the most common boundary mistake in this entire module.

5. "With cost already solved, Module 4 has nothing new to add"

Cost and latency are two distinct operational signals, calculated over different data, with different arithmetic. This module never measures how long a run took — that is, precisely, Module 4's job, which comes next and which does not use any real stopwatch, for exactly the same honest reason this guide already flagged back in Module 1.

How to work through this module

  1. Run every example yourself, calculator at hand. This module's arithmetic is simple — multiplication and integer division — but confirming by hand that estimate_cost_cents gives what you expect is the best way to make the formula stick.
  2. Don't confuse "estimated" with "made up." When a lesson says cost is estimated, it doesn't mean the number is arbitrary — it means it depends on a declared approximation (len(text) // 4) and a fixed price, never on a real API call.
  3. The mini-project (lesson 08) is the synthesis. There you're going to run a complete observability/cost_calculator.py over a real batch of Reservo runs, correlated by their trace_id, with a comprehensive cost report.

Estimated time:

Lesson 01 (this)  →  20 min reading
Lesson 02         →  20 min reading
Lesson 03         →  25 min + running the example
Lesson 04         →  20 min + running the example
Lesson 05         →  30 min + running the example
Lesson 06         →  30 min + running the example
Lesson 07         →  25 min + running the example
Lesson 08         →  35 min + building the complete mini-project

Total: ~3.5 hours

Evidence of success

Before moving on to Module 4 (Measuring Latency Honestly), you should be able to:

  • ✅ Explain what a token is and why it's a language model's unit of cost, not the character or the word.
  • ✅ Calculate estimate_tokens(text) by hand over a short text, and confirm it with code.
  • ✅ Cite claude-sonnet-5's pricing from memory: $3.00/$15.00 per million input/output tokens, list price.
  • ✅ Calculate a real Reservo run's cost in cents, with its per-tool-call breakdown.
  • ✅ Scale a run's cost to 1,000, 10,000, and 100,000 runs, summing tokens before rounding.
  • ✅ Trace the boundary between measuring cost (this guide) and reducing cost (cost-optimization-caching-guide).

Summary

  • This module measures the Reservo agent's cost per run, over the same trace_id correlation Module 2 already built — without touching the loop or the logger again.
  • We confirmed, with real execution, that Module 2's RUN_LOG.jsonl — complete, correlated, eight lines for a clean run — has no cost field at all. That is, precisely, the limit this module closes.
  • The central analogy is the electricity meter: a run costs a fraction of a cent, insignificant on its own, but multiplied by thousands of users it's a real bill — measuring it per run means reading the meter before that bill arrives.
  • The module's pieces: estimate_tokens (L03), the fixed pricing (L04), estimate_cost_cents + cost_for_run with a breakdown (L05), scaling to thousands of runs (L06), cost as an operational signal (L07), and the mini-project's complete report (L08).

Next lesson: 02 — Tokens Are the Unit of Cost. Before writing the formula, we answer the underlying question: what exactly is a token, and why does an LLM provider charge per that unit and not another?

Additional resources

  1. Anthropic — Pricing — The source for claude-sonnet-5's list price that lesson 04 fixes as a constant.
  2. Anthropic — Token counting — How the Claude API really counts tokens, the reference lesson 03 measures this guide's len // 4 estimate against.
  3. Python — dataclasses — CostReport and StepCost, the structures this module builds starting in lesson 05.
  4. Anthropic — Building effective agents — On why measuring an agentic system's cost is a discipline of its own, not a billing detail.
  5. Python 3.14 — What's New — The version all of this module's arithmetic runs on.
← PreviousMini-Project: A Traced Reservo RunNext →Tokens Are the Unit of Cost