Checking Cost and Latency Thresholds

Description

Lessons 04 and 05 covered two of the gate's three questions: the result's form, and tool choice. This lesson closes the third: did this run's cost and latency stay within a budget known in advance? Unlike the two previous questions, this one doesn't check what the agent did — it checks how much it cost to do it, reusing, without changing a single line, the engineering you already built in Modules 3 and 4: cost_for_run for cost, and the TOOL_LATENCY_MS model for latency.

This lesson builds the missing piece — latency_for_run, which sums a run's modeled latency step by step, cost_for_run's exact counterpart but for time instead of money — and puts both thresholds to the test over the CASE_SET's five real cases, and over two scenarios designed on purpose to fail: one on latency, one on cost.

Connection to the module

This lesson delivers latency_for_run, the last new piece of regression/harness.py before lesson 07 assembles everything into run_case and run_regression_gate. check_cost_threshold and check_latency_threshold were already complete in lesson 02 — here they get tested in depth, integrated with the rest of the harness.

latency_for_run: the same idea as cost_for_run, applied to time

cost_for_run (Module 3) walks history and sums estimated text per direction (input, output). latency_for_run walks the same history with almost identical logic, but sums a different number: the modeled latency of every tool that responded successfully.

TOOL_LATENCY_MS = {
    "list_rooms": 40,
    "get_quote": 25,
    "book_room": 120,
    "cancel_booking": 90,
}


def latency_for_run(history):
    """Latencia modelada de un run: suma de TOOL_LATENCY_MS por cada
    tool_result exitoso. NUNCA time.time()/time.perf_counter() -- la misma
    honestidad del Módulo 4: en producción de verdad esto se mide con el
    reloj real; aquí se modela para que el ejemplo sea reproducible."""
    total_ms = 0
    tool_use_name = {}
    for turn in history:
        content = turn["content"]
        if isinstance(content, str):
            continue
        for block in content:
            if block["type"] == "tool_use":
                tool_use_name[block["id"]] = block["name"]
            elif block["type"] == "tool_result" and not block.get("is_error"):
                total_ms += TOOL_LATENCY_MS.get(tool_use_name.get(block["tool_use_id"]), 0)
    return total_ms

TOOL_LATENCY_MS is the same dictionary, with the same four values, you already saw run since Module 1 and that Module 4 develops in depth with percentiles over large batches — this module doesn't invent a new latency model, it reuses the one that already exists. The only thing that adds nothing to a tool_result with is_error: True: an attempt rejected on validation, as you already saw in Module 3 with cost, does consume tokens (and therefore cost), but in this guide's latency model, a validation rejection resolves before the real tool ever runs — dispatch_robust cuts it off right there, running nothing — so there's no tool latency to add for that step.

Worked example, part 1: cost and latency over the five real cases

print("--- costo y latencia sobre los cinco casos reales ---")
for i, case in enumerate(CASE_SET, start=1):
    reset_reservo_state()
    with rl.traced_run(case["question"], i) as trace_id:
        final, history = ra.run_reservo_agent(case["question"], case["model_script"])
    report = cost_for_run(trace_id, case["question"], history)
    latency_ms = latency_for_run(history)
    cost_ok = check_cost_threshold(report.cost_cents, case["cost_threshold_cents"])
    latency_ok = check_latency_threshold(latency_ms, case["latency_threshold_ms"])
    print(f"{case['name']:38} cost={report.cost_cents:>2}c (<= {case['cost_threshold_cents']}) {cost_ok}   "
          f"latency={latency_ms:>3}ms (<= {case['latency_threshold_ms']}) {latency_ok}")

What to expect:

--- costo y latencia sobre los cinco casos reales ---
quote_focus_pro_3h                     cost= 0c (<= 5) True   latency= 25ms (<= 100) True
quote_focus_basic_3h                   cost= 0c (<= 5) True   latency= 25ms (<= 100) True
book_focus_pro_3h_ana                  cost= 0c (<= 5) True   latency=185ms (<= 250) True
book_boardroom_pro_1h_sofia            cost= 0c (<= 5) True   latency=185ms (<= 250) True
book_and_cancel_studio_basic_1h_diego  cost= 0c (<= 5) True   latency=210ms (<= 250) True

Five for five, on both thresholds. The real cost of each of these small runs is 0 cents — the same honest response that's been accompanying this guide since Module 1 — and each threshold (5 cents, deliberately generous) confirms it with margin to spare. Latency varies depending on how many tools each case calls: 25 ms for a single quote (get_quote), 185 ms for a complete three-step booking, 210 ms to book and cancel.

Worked example, part 2: a latency FAIL — a threshold that's too strict

The real CASE_SET's thresholds are generous, and pass with margin. To see the FAIL mechanism in action, swap book_focus_pro_3h_ana's latency threshold — normally 250 ms — for a much stricter one, 100 ms, well below the 185 ms that case actually needs:

case3 = CASE_SET[2]  # book_focus_pro_3h_ana
strict_case = {**case3, "latency_threshold_ms": 100}
result = run_case(strict_case, 50)
print("resultado:", "PASS" if result.passed else "FAIL")
print(f"latency_ms={result.latency_ms}  umbral={strict_case['latency_threshold_ms']}  latency_ok={result.latency_ok}")

What to expect:

resultado: FAIL
latency_ms=185  umbral=100  latency_ok=False

{**case3, "latency_threshold_ms": 100} builds a copy of the case with a single field changed — a useful technique for experimenting with a threshold without touching golden_cases.json. The agent behaved exactly like always: same three tools, same order, same result. The only thing that changed was the budget this specific case demands — and the gate flags it as a FAIL with the same seriousness as if it had chosen the wrong tool. A latency threshold isn't a secondary detail of the gate: for a real system, a tool that starts taking a lot longer than expected — even if it keeps returning the correct result — is a legitimate operational signal that something, in the infrastructure behind that tool, is degrading.

Worked example, part 3: a cost FAIL — a genuinely verbose run

The real CASE_SET's five cases cost 0 cents because they're small — the same honest scale that accompanies this entire guide. To see a cost FAIL with a real CostReport (not just handmade numbers, like in lesson 02), build a script with a deliberately long final response — simulating a verbose agent, the same profile Module 3 identified as generating the most cost, because of the 5x asymmetry between output and input tokens:

verbose_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Focus pro 3h cuesta $60.00. " + (
            "Este es un texto de relleno para inflar el costo de salida de este run de ejemplo. " * 40
        )}]},
]
case_verbose = {**CASE_SET[0], "model_script": verbose_script, "cost_threshold_cents": 0}
result_v = run_case(case_verbose, 51)
print("resultado:", "PASS" if result_v.passed else "FAIL")
print(f"cost_cents={result_v.cost_cents}  umbral={case_verbose['cost_threshold_cents']}  cost_ok={result_v.cost_ok}")

What to expect:

resultado: FAIL
cost_cents=1  umbral=0  cost_ok=False

With a 0-cent threshold — a deliberately tight budget, for this demonstration — the filler text (a response repeated forty times, simulating an agent that became much more verbose than necessary) pushes the estimated cost to 1 cent, and the gate flags it as a FAIL. With the CASE_SET's real threshold (5 cents), this same run would have passed without a problem — every case's threshold is a design decision, not a universal value: a case that's expected to be cheap (a simple quote) can carry a tight threshold; a case involving several steps can carry a more generous one, with the discipline that, whatever it is, it gets declared explicitly in the CASE_SET, never improvised at gate-run time.

Why reuse cost_for_run and TOOL_LATENCY_MS, instead of rebuilding them here

It's worth noting, precisely, what this lesson didn't do: it didn't rebuild estimate_cost_cents's formula, it didn't redeclare claude-sonnet-5's pricing, it didn't invent a new latency model. Each of those pieces already exists, is already tested, and is already cited with its source in the corresponding modules. Rebuilding them here — even with the same code, copied and pasted — would introduce exactly the risk this guide avoids in every module: two copies of the same logic that, over time, someone updates in one place and forgets to update in the other. This module's regression gate measures with the same tools as the rest of the guide — it never reinvents them.

And, as in every lesson in this module, it's worth repeating the boundary once more, now applied to a number: check_cost_threshold/check_latency_threshold answer "is this number under the limit?" — a FORM comparison, as mechanical as <=. They never answer "is this cost reasonable for the value the agent delivered?" or "is this latency acceptable for the user's experience?" — those are quality questions, depending on business context and human judgment, and they belong to evaluation-frameworks-guide's discipline, not to this gate.

Common mistakes

1. Also summing latency for a tool_result with is_error: True. latency_for_run's code explicitly filters not block.get("is_error") — an attempt rejected on validation never gets to run the real tool, so it has no tool time to add. Adding it anyway would artificially inflate the latency of a run with validation errors.

2. Forgetting a small run's cost is, almost always, 0 cents. It isn't a calculation error — it's the honest scale this guide has repeated since Module 1. A cost_threshold_cents: 5 threshold on a case that always costs 0 isn't "a useless threshold that never fails" — it's a correct threshold for that case's real size, ready to catch the day that cost stops being 0 for no apparent reason.

3. Using {**case, "field": value} on a nested dict expecting it to also copy the inner structures. {**case3, "latency_threshold_ms": 100} creates a shallow copy: the new dict has its own latency_threshold_ms, but model_script still points to the same list as the original case — that's fine for this use, because model_script isn't modified, but it would be a real mistake if you tried to mutate that script in place expecting the original case to stay unchanged.

4. Confusing a strict threshold "to test the mechanism" with a real production threshold. The 100 ms threshold in worked example, part 2, is artificially low, chosen on purpose to produce a didactic FAIL — it doesn't reflect any real acceptable latency limit for a three-step booking. The CASE_SET's real thresholds (100/250 ms) are the ones used from lesson 07 onward.

5. Thinking check_cost_threshold/check_latency_threshold "know" where the number they receive comes from. They don't, and they shouldn't — they receive an already-calculated integer (cost_cents, latency_ms) and a threshold integer, and make a single comparison. All the responsibility for calculating those numbers correctly lives in cost_for_run and latency_for_run, not in the threshold functions.

Exercises

Exercise 1: Find the minimum latency threshold that still gives PASS for each case (Easy)

For the CASE_SET's five cases, calculate latency_for_run over each one (you already did this in the worked example) and, for each case, say what the lowest possible latency threshold is that would still give PASS (that is, exactly equal to that case's real latency).

for i, case in enumerate(CASE_SET, start=1):
    reset_reservo_state()
    with rl.traced_run(case["question"], i) as trace_id:
        final, history = ra.run_reservo_agent(case["question"], case["model_script"])
    latency_ms = latency_for_run(history)
    print(f"{case['name']:38} umbral mínimo para PASS: {latency_ms} ms")

quote_focus_pro_3h                     umbral mínimo para PASS: 25 ms
quote_focus_basic_3h                   umbral mínimo para PASS: 25 ms
book_focus_pro_3h_ana                  umbral mínimo para PASS: 185 ms
book_boardroom_pro_1h_sofia            umbral mínimo para PASS: 185 ms
book_and_cancel_studio_basic_1h_diego  umbral mínimo para PASS: 210 ms

Exercise 2: Calculate how many filler-text repeats it takes to cross different thresholds (Medium)

Using worked example part 3's pattern, calculate the resulting cost_cents with 10, 40, and 100 repeats of the filler text. Find, by trying values, how many repeats it takes for the cost to cross 1 cent.

def cost_with_repeats(n):
    script = [
        {"stop_reason": "tool_use", "content": [
            {"type": "tool_use", "id": "toolu_01", "name": "get_quote",
             "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
        {"stop_reason": "end_turn", "content": [
            {"type": "text", "text": "Focus pro 3h cuesta $60.00. " + (
                "Este es un texto de relleno para inflar el costo de salida de este run de ejemplo. " * n
            )}]},
    ]
    case = {**CASE_SET[0], "model_script": script, "cost_threshold_cents": 999}
    result = run_case(case, 60)
    return result.cost_cents

for n in (10, 40, 100):
    print(f"repeticiones={n:>3}  cost_cents={cost_with_repeats(n)}")

repeticiones= 10  cost_cents=0
repeticiones= 40  cost_cents=1
repeticiones=100  cost_cents=2

Exercise 3: Design a case that fails on cost AND latency at once (Hard)

Build a script combining long filler text in the final response and a longer-than-normal tool sequence (for example, list_rooms called three times in a row before get_quote, something a real agent would never do on purpose, but useful for this demonstration). Adjust the case's cost and latency thresholds so both fail. Confirm, with the complete CaseResult, that cost_ok and latency_ok are both False, and that passed is too.

double_fail_script = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Focus pro 3h cuesta $60.00. " + (
            "Relleno para inflar el costo de este run de ejemplo hasta cruzar el umbral. " * 40
        )}]},
]
double_fail_case = {
    **CASE_SET[0], "model_script": double_fail_script,
    "expected_tools": ["list_rooms", "list_rooms", "list_rooms", "get_quote"],
    "cost_threshold_cents": 0, "latency_threshold_ms": 50,
}
result = run_case(double_fail_case, 70)
print("passed    :", result.passed)
print("cost_ok   :", result.cost_ok, "cost_cents:", result.cost_cents)
print("latency_ok:", result.latency_ok, "latency_ms:", result.latency_ms)

passed    : False
cost_ok   : False cost_cents: 1
latency_ok: False latency_ms: 145

Summary and next step

• We built latency_for_run, cost_for_run's (Module 3) exact counterpart for time instead of money, reusing unchanged the TOOL_LATENCY_MS model already established since Module 1 and developed in depth in Module 4.
• We confirmed, run for real, that the CASE_SET's five real cases pass their cost and latency thresholds with margin — 0-cent cost, latency between 25 and 210 ms, depending on how many tools each case involves.
• We produced a latency FAIL (an artificially strict threshold on a real case) and a cost FAIL (a genuinely verbose run, with filler text that crosses a 0-cent budget) — both with real CostReport/latency, not handmade numbers.
• We confirmed why this module reuses cost_for_run and TOOL_LATENCY_MS without rebuilding them: a single source of truth for each calculation, cited where it was defined, never duplicated.

Next lesson: 07 — The Gate: Pass or Fail the Build. With the gate's three questions already complete — form, tool choice, thresholds — we bring them together into run_case and run_regression_gate: the complete CASE_SET's PASS verdict, and the FAIL verdict when a real regression gets simulated.

Additional resources

1. Python — statistics — The library Module 4 uses for percentiles over large latency batches; this module uses only the simple per-run sum, that same model's most basic piece.
2. Anthropic — Token counting — The real token count the len(text)//4 estimate (Module 3) approximates, the foundation of every cost_cents cited in this lesson.
3. Python — dictionary unpacking (**) — The {**case, "field": value} technique used to experiment with thresholds without modifying golden_cases.json.
4. Anthropic — Building effective agents — On why an agent's cost and latency are first-class operational signals, taken just as seriously as its functional correctness.
5. Python 3.14 — What's New — The version every line of code in this lesson ran on, including the two FAILs demonstrated.