Module 4: Measuring Latency Honestly

Description

Module 3 closed the first half of the measure discipline: every Reservo run now has a cost in cents, calculated with a fixed formula over estimated tokens. But if you look back at a CostReport from that module's lesson 08, you're going to notice something missing: no field says how long the run took. cost_cents answers "how much did it consume?"; no other business question — "did the customer wait too long?", "which tool felt slow?" — has an answer yet.

This module closes that second half. But before writing a single line of code, there's a problem to solve that didn't exist in Module 3 in the same way: measuring time for real would break this entire guide's reproducibility. Cost is estimated with a deterministic formula (len(text) // 4 plus a fixed price) that gives the same number on your machine, on mine, and on anyone's who runs the same code. The real time a function takes to run doesn't work that way — it depends on how loaded your CPU is at this exact instant, on what other processes are competing for it, on factors neither you nor I control. If this guide measured latency with a real clock, every "What to expect" in this module would be a lie: a number you're never going to be able to reproduce exactly in your own terminal.

This guide's solution — previewed since Module 1, and put into real practice starting here — is to model latency: declare, in a fixed dictionary, how "long" each tool "takes," and sum those fixed values instead of timing anything. This module isn't a half-hearted exercise in honesty. Each of its eight lessons explicitly says the same sentence: this is modeled, not measured; in real production it's measured with a real clock; here it's modeled so the example is reproducible and the focus stays on the analysis, not the stopwatch.

This module's hard rule (the strictest in the entire guide)

Modules 1 through 3 already prohibited random, datetime.now(), and uuid4() in any of this guide's data. This module adds the most important prohibition of the eight: no executed code block in this module uses time.time() or time.perf_counter(). None. Not even "just to show what it would look like." Whenever this guide needs to name how latency is really measured in production, it's going to do so in prose, citing the function's name — never inside a code block that later gets presented as executed.

The reason isn't aesthetic. It's the same reason this guide banned random back in Module 1: any data source that isn't deterministic breaks this guide's central promise — that you can run exactly the same code you see here and get exactly the same result. time.perf_counter() is, precisely, a non-deterministic data source: call it twice in a row on the same machine and you're going to get two different numbers, neither reproducible by anyone else.

Instead, this module works with a fixed value you already know from Module 1:

TOOL_LATENCY_MS = {
    "list_rooms": 40,
    "get_quote": 25,
    "book_room": 120,
    "cancel_booking": 90,
}

These four numbers didn't measure anything — they're a design declaration, as deliberate as the price anchors (Focus basic 3h = 7500) you already know. And they're, on purpose, plausible: book_room (a write, with real effects) is the most expensive of the four; get_quote (an in-memory calculation, touching no state) is the cheapest. This guide doesn't invent these numbers in every lesson — it fixes them once, here, and reuses them unchanged until Module 8's close.

Where we are in the ecosystem

Agents in production — operating the Reservo agent
├── Module 1: Why Operating Is Different From Building
├── Module 2: Structured Logging and Tracing a Run
├── Module 3: Measuring Cost and Tokens per Run
├── Module 4: Measuring Latency Honestly  ← YOU ARE HERE
│   → what latency gets measured (per tool and total), the
│     modeled-vs-real-clock honesty problem, TOOL_LATENCY_MS as
│     fixed data, a run's total latency, p50/p95 percentiles
│     over a batch, latency as an operational signal
├── Module 5: Regression Evals as a Production Gate
├── Module 6: Failures at Scale — Backoff, Circuit Breakers, and Rate Limits
├── Module 7: Versioning and Safe Rollout
└── Module 8: Project — the Reservo Agent in Production

This is the second and final module of the measure layer (Modules 3-4), the second of this guide's four disciplines: observe → measure → gate → harden+version. With this module closed, the two "how much" signals — cost and latency — are complete, and Module 5 is going to use both, alongside Module 1's per-tool failure rate, as the thresholds for a deterministic regression gate.

This module's central analogy: the practice stopwatch

A coach preparing a team for a relay race doesn't wait for competition day to start training baton exchanges. They use a practice stopwatch, on the training track, under controlled conditions: the same distance, the same handoff point, no variable wind or wet track. That practice stopwatch doesn't measure "the real race" — race day will have wind, nerves, a different track — it measures something more useful for training: which leg of the relay consumes the most time, consistently, run after run, so the team knows where to focus practice.

That is exactly TOOL_LATENCY_MS's role in this module. It isn't production's real stopwatch — that stopwatch exists, it's called time.perf_counter(), and it gets named in every lesson of this module without ever being used in code. It's the practice stopwatch: fixed, reproducible conditions, designed so you can train on the question that actually matters — which tool dominates the total? which percentile best summarizes the real experience? — without a real measurement's noise (a busy CPU at this exact instant) getting in the way of the learning.

The case that keeps accompanying the guide: Reservo, without rebuilding anything

The four tools are the same as always — list_rooms(), get_quote(room, tier, hours), book_room(room, tier, hours, member), cancel_booking(id) — and Module 3's two price anchors remain intact: Focus basic 3h = 7500 cents, Focus pro 3h = 6000 cents. This module doesn't declare a single new tool, doesn't touch reservo_tools.py, reservo_contracts.py, reservo_robust.py, or reservo_agent.py, and doesn't touch observability/run_logger.py (Module 2) or observability/cost_calculator.py (Module 3) either — it imports them, as they stand, and builds a new artifact alongside them: observability/latency_model.py.

Lesson 02 precisely defines what latency this module measures: an individual tool call's, and a run's total latency, as two related but distinct questions.
Lesson 03 is the module's heart: the modeled-vs-real-clock honesty problem, with reproducibility confirmed, run for real, over the same run run twice.
Lesson 04 fixes TOOL_LATENCY_MS as data — where each number comes from, why book_room is the most expensive, and how it's looked up per individual tool call.
Lesson 05 builds total_run_latency_ms, the sum over a complete run, with the nuance Module 1 already previewed: a tool_use rejected on validation never gets to run the real tool, so it doesn't add a single millisecond to the total.
Lesson 06 scales up to a batch of twelve real runs and calculates p50 and p95 — the first time this guide uses percentiles, not just averages.
Lesson 07 treats latency as one more operational signal, identifies which tool dominates a batch's total, and traces the boundary with sre-and-incident-response-guide.
Lesson 08 closes the module with a complete observability/latency_model.py, run over the same batch of twelve runs, in a comprehensive latency report.

As in every module of this guide: identifiers and code in English; prose and comments, in Spanish; money, always in int cents; latency, always in int milliseconds.

Prerequisites

Required knowledge:

✅ Having completed Module 1 of this guide, especially lesson 05 (TOOL_LATENCY_MS, estimate_run_latency_ms, calculated for the first time over Ana's canonical run). This module picks that function back up and develops it in depth — it doesn't re-explain it from scratch.
✅ Having completed (or knowing well) agent-fundamentals-and-tool-calling-guide M8: run_reservo_agent, and history's format (user/assistant turns, tool_use/tool_result/text blocks, the is_error field).
✅ Python: functions, dictionaries, lists, a basic grasp of sorted() and the statistics module.

NOT required:

❌ You don't need an API key or an internet connection: the model's decision is still concept, and all of this module's engineering runs 100% locally, with pure integer arithmetic.
❌ You don't need to measure real latency at any point in this module. time.perf_counter() gets named, in prose, several times — never executed.
❌ You don't need to know about infrastructure, load balancers, or how an observability provider (Datadog, New Relic) calculates its percentiles internally — the principles you learn here are the same, with any tool.

Environment:

✅ Python 3.14.0 with its standard library (statistics, json, dataclasses, math). Nothing to install.
✅ Modules 2 and 3's observability/ directory, with run_logger.py and cost_calculator.py already built.

Module roadmap

Lesson 01 — Module introduction (this one)

The exact limit Module 3 leaves behind — complete cost, no time field at all — the hard rule on time.time()/time.perf_counter(), the practice-stopwatch analogy, and the map of the eight lessons.

Lesson 02 — What Latency Are We Measuring?

Two related but distinct questions: the latency of one tool call, and a run's total latency — and why neither one lived in history before this module.

Lesson 03 — The Honesty Problem: Modeled vs. Real Clock

The central lesson: why measuring with the real clock would break this guide's reproducibility, what's gained and what's lost by modeling, and the run-tested confirmation that the same run gives the same latency, always.

Lesson 04 — Tool Latency as Fixed Data

The dictionary's structure, where its four numbers come from, and how an individual tool call's latency is looked up.

Lesson 05 — Total Run Latency

total_run_latency_ms, run over Ana's canonical run, with the nuance inherited from Module 1: a tool_use rejected on validation contributes no latency, because it never gets to run the real tool.

Lesson 06 — Percentiles: p50 and p95

The average hides those who wait longer. p50 and p95, calculated by hand and with statistics, over a real batch of twelve runs — with the analogy of customer number 95 out of every 100.

Lesson 07 — Latency as an Operational Signal

Which tool dominates a batch's total, how to read that signal, and the boundary with sre-and-incident-response-guide (infrastructure latency) and with this same guide's Module 6 (what to do when a tool is consistently slow).

Lesson 08 — Mini-Project: A Latency Report

A complete observability/latency_model.py, run over the previous lessons' batch of twelve runs, with a comprehensive report: per tool, per run, and batch percentiles.

Progression map

Lesson 01 (this)  → Module 3's limit, the hard rule, the practice stopwatch
Lesson 02         → Per-tool-call latency vs. a run's total latency
Lesson 03         → Modeled vs. real clock: the honesty problem
Lesson 04         → TOOL_LATENCY_MS, the fixed data
Lesson 05         → total_run_latency_ms + the rejected-tool_use nuance
Lesson 06         → p50 and p95, over a real batch of twelve runs
Lesson 07         → Latency as a signal, the boundary with SRE
Lesson 08         → Mini-project: the complete latency report

Difficulty: ⭐⭐ ──────────────────▶ ⭐⭐⭐

What you'll achieve in this module

By completing the 8 lessons, you'll be able to:

Tell an individual tool call's latency apart from a run's total latency, and calculate both over real data.
Precisely explain why this guide models latency instead of measuring it with the real clock — and why that decision isn't laziness, but the same reproducibility discipline you already saw with random and datetime.now().
Use TOOL_LATENCY_MS as a fixed design value, cited once, reused unchanged throughout the rest of the guide.
Calculate a run's total latency, correctly applying the nuance that a tool_use rejected on validation contributes not a single millisecond.
Calculate p50 and p95 over a batch of runs, by hand and with statistics, and explain why the average alone isn't enough to know whether a system "feels slow."
Identify which tool dominates a batch's total latency, and trace the boundary between this guide (an agent's step latency) and sre-and-incident-response-guide (infrastructure latency).

Before and after

BEFORE the module:
→ "measuring latency is putting a stopwatch on it, I already
  know how"
→ "the latency average already tells me whether the system is
  fast"
→ "a run's latency is a single number, it has no breakdown"
→ "a tool_use that failed shouldn't matter to latency, or should
  it?"

AFTER the module:
→ knowing WHICH signals matter (which tool dominates, which
  percentile to use) is the real work; putting a stopwatch on
  something is trivial
→ the average hides the users having the worst experience -- p95
  is the figure a real business needs to make an honest promise
→ a run's total latency is the sum of its ACTUALLY executed tool
  calls, with a breakdown per tool
→ a tool_use rejected on validation never reaches the tool --
  and that's why it never adds time to the run, no matter how
  many invalid attempts are in the trace

Traps to avoid in this module

1. "This module is going to use `time.perf_counter()` to make the example more realistic"

No. Not a single executed code block in this module ever shows time.time() or time.perf_counter(). When a lesson needs to name how real latency is measured, it does so in a prose sentence — "in production, this gets wrapped with time.perf_counter() around each call" — never inside a block that later gets presented as executed.

2. "If latency is modeled, then it's made up, and it's not worth taking seriously"

Modeled isn't the same as made up. TOOL_LATENCY_MS is a design value, just as real and just as cited as Module 3's fixed claude-sonnet-5 pricing — the difference is that one models money and the other models time, and both do so with the same explicit honesty about what they represent and what they don't.

3. "A run's latency is just one number — breaking it down makes no sense"

It does, and it's exactly what this module's lesson 07 demonstrates with real data: in a batch of twelve runs, a single tool (book_room) is responsible for more than half of the batch's total milliseconds, even though it represents barely a quarter of all calls. Without the per-tool breakdown, that signal is invisible.

4. "With twelve runs I already have enough to calculate a reliable p95"

No, and lesson 06 confirms it with real numbers: with a sample that small, p95 almost always ends up being, literally, the batch's slowest run — not a robust estimate of "95% of cases." This guide uses twelve runs because that's what fits in one lesson, not because it's a statistically solid sample; lesson 06 is explicit about that limitation.

5. "Module 5 (regression evals) doesn't need anything from this module"

It does. Module 5's regression gate is going to compare a new agent version's latency against a fixed threshold — and that threshold gets built, precisely, over the same total_run_latency_ms and the same p50/p95 notions this module develops. Without this module, that gate would have no latency figure to compare against.

How to work through this module

Run every example yourself. This module's central promise is that modeled latency is reproducible — confirm it with your own eyes, not just by reading the "What to expect."
Don't look for time.perf_counter() in any executed code block. If you see it, it isn't from this module — the only time it appears in the whole guide is named in prose.
The mini-project (lesson 08) is the synthesis. There you're going to run a complete observability/latency_model.py over the same batch of twelve runs accompanying lessons 06 and 07 — the same data correlation you already saw get built piece by piece.

Estimated time:

Lesson 01 (this)  →  20 min reading
Lesson 02         →  20 min + running the example
Lesson 03         →  25 min + running the example
Lesson 04         →  20 min + running the example
Lesson 05         →  25 min + running the example
Lesson 06         →  35 min + running the example
Lesson 07         →  25 min + running the example
Lesson 08         →  35 min + building the complete mini-project

Total: ~3.5 hours

Evidence of success

Before moving on to Module 5 (Regression Evals as a Production Gate), you should be able to:

✅ Explain why this guide models latency instead of measuring it with the real clock, without a moment's doubt about whether time.perf_counter() appears in any executed code in this module (it doesn't).
✅ Calculate an individual tool call's latency and a run's total latency, with the correct nuance about rejected tool_uses.
✅ Calculate p50 and p95 over a batch of runs, by hand and with statistics, and explain the difference between the two.
✅ Identify which tool dominates a batch's total latency, with numeric evidence, not intuition.
✅ Trace the boundary between an agent's step latency (this guide) and infrastructure latency (sre-and-incident-response-guide).

Summary

This module measures the Reservo agent's latency, per tool call and per run's total, over Module 1's same foundation — TOOL_LATENCY_MS — developed in depth.
Hard rule, the strictest in the guide: no executed code block uses time.time() or time.perf_counter(); both get named, only in prose, as the real way to measure in production.
The central analogy is the practice stopwatch: fixed, reproducible conditions, designed to train which signals matter, not to replicate the real race.
The module's pieces: what latency gets measured (L02), the honesty problem (L03), TOOL_LATENCY_MS as data (L04), the total sum with its nuance (L05), p50/p95 percentiles (L06), latency as an operational signal (L07), and the mini-project's complete report (L08).

Next lesson: 02 — What Latency Are We Measuring?. Before writing any formula, we precisely distinguish two questions that sound similar but aren't: how long did this tool call take?, and how long did the whole run take?

Additional resources

Anthropic — Building effective agents — On why an agentic system's perceived latency depends on how many steps — and which ones — it needed to take, not just how fast the model responds.
Python — statistics — statistics.median and statistics.quantiles, the core of this module's lesson 06.
Python — time — The reference for time.perf_counter(), named in several of this module's lessons and never executed in its code.
Python 3.14 — What's New — The version all of this module's engineering runs on.

← PreviousMini-Project: A Cost Report for Reservo Runs Next →What Latency Are We Measuring?