The Cost and Latency Report

Description

With Lesson 3's RUN_LOG.jsonl already written, this lesson sets the capstone's second discipline in motion: measuring. cost_for_run (M3) and total_run_latency_ms (M4) read the same history every run in Lesson 3 produced — replacing nothing, wrapping nothing, just reading what run_reservo_agent already returned — and answer, with real numbers, the two questions that opened this guide: how much did this run cost? how long did it take? This lesson goes one step further than repeating M3 and M4 separately: it brings both answers together into a single per-run report, and aggregates the complete batch into a single summary, with total cost and latency percentiles — this capstone's second real artifact.

Connection to the module

This lesson delivers ops/metrics_summary.py: RunMetrics (a run's cost + latency) and BatchMetrics (the batch summary). Neither structure rebuilds the cost formula, the latency model, or the percentile function — each one calls, exactly once, M3's and M4's corresponding piece. Lesson 5 is going to use these same numbers — every case's cost and latency — as part of the regression gate.

Analogy: the accountant and the stopwatch, reviewing the same ticket

Lesson 3's inspector — the scanner at every station — already left its complete trail in RUN_LOG.jsonl. Now two different people come in to review exactly those same tickets: the accountant, who adds up the real ingredient cost every dish used, and the kitchen stopwatch, which already knows — from memory, with no need to measure again with a real clock — how long every station takes. Neither of the two cooks anything again — both read, once service has already passed, exactly the same tickets the scanner recorded. And at the end of the night, someone brings both sheets — the accountant's, the stopwatch's — together into a single report: how much it cost and how long it took, ticket by ticket, plus a summary of the whole night.

Worked example: cost and latency, run by run

Lesson 3's three completed runs, with their history available

Pick back up the three runs that did finish in Lesson 3 — Ana, Sofía, and canceling booking 999 — and calculate, for each one, its CostReport (M3) and its total latency (M4):

import cost_calculator as cc
import latency_model as lm

runs = [
    ("run-8487582448eb", "Reserva Focus pro 3h para Ana", history_ana),
    ("run-c720132bf969", "Reserva Boardroom pro 1h para Sofia", history_sofia),
    ("run-8d26276b0d45", "Cancela la reserva 999", history_cancel),
]

for trace_id, question, history in runs:
    report = cc.cost_for_run(trace_id, question, history)
    latency_ms = lm.total_run_latency_ms(history)
    print(f"{trace_id}  in={report.input_tokens:>3} out={report.output_tokens:>3} "
          f"cost={report.cost_cents}c  latency={latency_ms:>3}ms  -- {question}")

What to expect:

run-8487582448eb  in= 64 out= 56 cost=0c  latency=185ms  -- Reserva Focus pro 3h para Ana
run-c720132bf969  in= 53 out= 48 cost=0c  latency=185ms  -- Reserva Boardroom pro 1h para Sofia
run-8d26276b0d45  in= 10 out=  8 cost=0c  latency= 90ms  -- Cancela la reserva 999

Three runs, three different sizes, the same honest answer as always: 0 cents each, the real scale of an individual run this size. Latency does tell them apart clearly: 185 ms for the two three-step bookings (list_rooms + get_quote + book_room), 90 ms for the single-step cancellation — exactly TOOL_LATENCY_MS["cancel_booking"], no surprises at all.

run 4 has no report — and that absence is information

try:
    with rl.traced_run("Reserva algo ambiguo", 4):
        final, history_stuck = ra.run_reservo_agent("Reserva algo ambiguo", stuck_script, max_iterations=2)
except RuntimeError as exc:
    print("run 4 -- sin CostReport, sin latencia:", exc)
    print("razón: RuntimeError se lanza ANTES de que run_reservo_agent retorne -- no hay history que leer.")

What to expect:

run 4 -- sin CostReport, sin latencia: max_iterations alcanzado (2)
razón: RuntimeError se lanza ANTES de que run_reservo_agent retorne -- no hay history que leer.

This isn't a bug in cost_for_run or total_run_latency_ms — it's the exact same limit run_and_observe (M1) already showed: when run_reservo_agent raises an exception, Python discards its local state before the function can return, and there's no history left for any external function to read. RUN_LOG.jsonl (Lesson 3) still has run 4's complete trail — run_started and run_failed, with the two steps that did get to run — but neither this run's cost nor its latency can be calculated with M3's/M4's tools over a history that never got to exist outside the function. This is, precisely, the kind of failure this module's Lesson 6 (the resilience layer) exists to reduce, and that Lesson 5's gate exists to catch before it ever reaches production.

RunMetrics and BatchMetrics: one report, two disciplines together

ops/metrics_summary.py rebuilds no formula — it calls, once each, cost_for_run (M3) and total_run_latency_ms (M4), and brings the result together into a single per-run structure:

# ops/metrics_summary.py
import statistics
from dataclasses import dataclass, field

import cost_calculator as cc
import latency_model as lm


@dataclass
class RunMetrics:
    """Costo Y latencia de UN run, en un solo lugar -- reusa cost_for_run
    (M3) y total_run_latency_ms (M4), nunca reconstruye ninguna fórmula."""
    trace_id: str
    question: str
    input_tokens: int
    output_tokens: int
    cost_cents: int
    latency_ms: int


def build_run_metrics(trace_id, question, history):
    report = cc.cost_for_run(trace_id, question, history)
    return RunMetrics(
        trace_id=trace_id, question=question,
        input_tokens=report.input_tokens, output_tokens=report.output_tokens,
        cost_cents=report.cost_cents, latency_ms=lm.total_run_latency_ms(history),
    )


@dataclass
class BatchMetrics:
    """El resumen de costo y latencia de un LOTE completo."""
    n_runs: int
    total_cost_cents: int
    mean_latency_ms: float
    p50_latency_ms: int
    p95_latency_ms: int


def aggregate_metrics(runs):
    """Suma TOKENS primero, aplica estimate_cost_cents UNA sola vez sobre
    el total (M3, lección 06) -- nunca suma cost_cents ya redondeados de
    cada run. Percentiles con la MISMA percentile() de M4, lección 06."""
    total_input = sum(r.input_tokens for r in runs)
    total_output = sum(r.output_tokens for r in runs)
    latencies = sorted(r.latency_ms for r in runs)
    return BatchMetrics(
        n_runs=len(runs),
        total_cost_cents=cc.estimate_cost_cents(total_input, total_output),
        mean_latency_ms=round(statistics.mean(latencies), 1),
        p50_latency_ms=lm.percentile(latencies, 50),
        p95_latency_ms=lm.percentile(latencies, 95),
    )

Notice aggregate_metrics's comment, because it guards against the easiest mistake to make here: summing every RunMetrics's r.cost_cents instead of summing their tokens and applying estimate_cost_cents once at the end. M3 (Lesson 6) already demonstrated, with a real difference of $103.20 over a hundred thousand runs, why summing values already rounded down loses precision — aggregate_metrics reuses that same discipline, it doesn't repeat it from scratch.

Run for real: the three-run batch's summary

from metrics_summary import build_run_metrics, aggregate_metrics

metrics = [
    build_run_metrics("run-8487582448eb", "Reserva Focus pro 3h para Ana", history_ana),
    build_run_metrics("run-c720132bf969", "Reserva Boardroom pro 1h para Sofia", history_sofia),
    build_run_metrics("run-8d26276b0d45", "Cancela la reserva 999", history_cancel),
]

for m in metrics:
    print(f"{m.trace_id}  cost={m.cost_cents}c  latency={m.latency_ms}ms")

batch = aggregate_metrics(metrics)
print()
print("=== BatchMetrics -- lote de", batch.n_runs, "runs ===")
print("costo total       :", batch.total_cost_cents, "centavos")
print("latencia promedio  :", batch.mean_latency_ms, "ms")
print("latencia p50       :", batch.p50_latency_ms, "ms")
print("latencia p95       :", batch.p95_latency_ms, "ms")

What to expect:

run-8487582448eb  cost=0c  latency=185ms
run-c720132bf969  cost=0c  latency=185ms
run-8d26276b0d45  cost=0c  latency=90ms

=== BatchMetrics -- lote de 3 runs ===
costo total       : 0 centavos
latencia promedio  : 153.3 ms
latencia p50       : 185 ms
latencia p95       : 185 ms

Stop at p50 and p95: both match, at 185 ms. It isn't a bug in percentile — it's the same honesty M4 (Lesson 6) already warned about with small samples, now taken to the extreme: with n=3, ceil(0.5 * 3) = 2 and ceil(0.95 * 3) = ceil(2.85) = 3 land on different positions of the sorted list ([90, 185, 185]), but position 2 and position 3 happen to share the same value — 185 — because two of the three runs (Ana and Sofía) share exactly the same latency sequence. With a three-run batch, neither p50 nor p95 is yet a reliable signal about "the typical experience" — they are, precisely, what there is: three numbers, sorted, at the positions the nearest-rank method demands.

Common mistakes

1. Summing latency_ms instead of calculating its percentile. BatchMetrics reports p50/p95/mean — never a total sum of latencies, which would have no operational meaning at all (unlike cost, where summing DOES make sense: it's the batch's total spend).

2. Summing cost_cents from every RunMetrics to get the batch's cost. Exactly the mistake aggregate_metrics's comment flags — with batches as small as this (0 cents each run) it makes no difference, but at the scale of thousands of runs the difference is real and already got measured in M3.

3. Trying to calculate RunMetrics for run 4. As the worked example showed, there's no history available for that run — build_run_metrics would fail with a NameError (the history_stuck variable never got assigned outside the try) if attempted. The absence of metrics for a run that didn't complete is valid information, not a bug to fix.

4. Thinking total_cost_cents=0 means the report "was useless." It served exactly what M3 (Lesson 7) already taught: confirming, with evidence, that cost stays in the expected range for this batch size — the same discipline a real system uses to detect when cost stops being 0 for no apparent reason.

5. Calculating BatchMetrics over a batch mixing traced_run runs with runs that never went through it. build_run_metrics needs a real, deterministic trace_id, from traced_run (M2) — passing it a hand-made identifier breaks the correlation with RUN_LOG.jsonl the rest of this capstone depends on.

Exercises

Exercise 1: Confirm Sofía's run's per-step breakdown (Easy)

Using cc.cost_for_run over history_sofia, print the per-step breakdown (report.steps) and confirm which of the three steps — list_rooms, get_quote, book_room — has the highest total tokens (input_tokens + output_tokens).

report_sofia = cc.cost_for_run("run-c720132bf969", "Reserva Boardroom pro 1h para Sofia", history_sofia)
for s in report_sofia.steps:
    print(f"  paso {s.step}: {s.tool:<12} total={s.input_tokens + s.output_tokens} tokens")

busiest = max(report_sofia.steps, key=lambda s: s.input_tokens + s.output_tokens)
print("paso más costoso:", busiest.tool, "con", busiest.input_tokens + busiest.output_tokens, "tokens")

  paso 1: list_rooms   total=30 tokens
  paso 2: get_quote    total=17 tokens
  paso 3: book_room    total=25 tokens
paso más costoso: list_rooms con 30 tokens

Exercise 2: Project the batch's cost over a thousand rounds of similar traffic (Medium)

Using M3's (Lesson 6) scaling technique — multiplying total tokens by n before applying estimate_cost_cents, never multiplying the already-rounded cost — project how much a thousand batches identical to this three-run one would cost.

n = 1000
total_input = sum(m.input_tokens for m in metrics)
total_output = sum(m.output_tokens for m in metrics)

naive = batch.total_cost_cents * n
correct = cc.estimate_cost_cents(total_input * n, total_output * n)
print("método naive  (cost_cents * n):", naive, "centavos")
print("método correcto (tokens * n)  :", correct, "centavos", f"(${correct / 100:.2f})")

método naive  (cost_cents * n): 0 centavos
método correcto (tokens * n)  : 206 centavos ($2.06)

Exercise 3: Design flag_slow_or_expensive_runs, combining both signals (Hard)

Write a flag_slow_or_expensive_runs(runs, latency_threshold_ms, cost_threshold_cents) function receiving a list of RunMetrics and returning the ones exceeding either threshold (latency OR cost). Test it over metrics with latency_threshold_ms=100 and cost_threshold_cents=0 — you should get two runs flagged for latency (Ana and Sofía, both at 185 ms), none for cost.

def flag_slow_or_expensive_runs(runs, latency_threshold_ms, cost_threshold_cents):
    flagged = []
    for r in runs:
        reasons = []
        if r.latency_ms > latency_threshold_ms:
            reasons.append(f"latency={r.latency_ms}ms > {latency_threshold_ms}ms")
        if r.cost_cents > cost_threshold_cents:
            reasons.append(f"cost={r.cost_cents}c > {cost_threshold_cents}c")
        if reasons:
            flagged.append((r.trace_id, reasons))
    return flagged


flagged = flag_slow_or_expensive_runs(metrics, latency_threshold_ms=100, cost_threshold_cents=0)
for trace_id, reasons in flagged:
    print(trace_id, "--", "; ".join(reasons))
print("total marcados:", len(flagged), "de", len(metrics))

run-8487582448eb -- latency=185ms > 100ms
run-c720132bf969 -- latency=185ms > 100ms
total marcados: 2 de 3

Summary and next step

• We calculated CostReport (M3) and total_run_latency_ms (M4) over Lesson 3's three completed runs, confirming both functions read history with no need for any change or additional wrapping at all.
• We confirmed, run for real, that run 4 — the one that fails with RuntimeError — has neither calculable cost nor latency, because of the exact same limit run_and_observe (M1) already showed: with no history, there's nothing to measure.
• We built ops/metrics_summary.py: RunMetrics (a run's cost + latency) and BatchMetrics (the aggregated summary), reusing estimate_cost_cents, cost_for_run, total_run_latency_ms, and percentile without rebuilding any formula — this capstone's second real artifact.

Next lesson: 05 — The Regression Gate in the Capstone. With cost and latency already measured, we run M5's gate against the agent exactly as it stands — PASS — and, then, the same criteria applied to comparing a new prompt version (M7) — NO-GO, with the rollback run.

Additional resources

1. Python — dataclasses — RunMetrics's and BatchMetrics's foundation, the same pattern as CostReport (M3) and LatencyReport (M4).
2. Python — statistics — statistics.mean, reused unchanged in aggregate_metrics.
3. cost-optimization-caching-guide — for when the question stops being "how much did it cost" and becomes "how do I reduce it" — this lesson measures, it never optimizes; this module's Lesson 7 names that boundary precisely.
4. sre-and-incident-response-guide — for when flagged (Exercise 3) needs to turn into a real alert, with a notification channel and a runbook — this lesson stops at identifying the signal, not operating the alert.