Cost as an Operational Signal

Description

Module 1, lesson 04, named four operational signals this module and the next one calculate precisely: error rate (at the run level), per-tool failure rate (at the tool-call level), cost per run, and latency per run. This module's lessons 05 and 06 solved cost — but only as a number: how much a run cost, how much thousands would cost. This lesson takes the last step: treating that number as a signal, with exactly the same seriousness as error rate, able to tell you when something in a run — or in a type of run — deserves attention, before it turns into a real budget problem.

You're going to reuse, unchanged, lesson 06's batch of four runs — Ana, Sofía, Diego, Carla — and you're going to find, with a simple, reproducible criterion, which of the four is a warning sign: a run that consumed substantially more tokens than its peers, without that necessarily meaning something failed.

Connection to the module

This lesson doesn't add any new piece to estimate_tokens or estimate_cost_cents — it uses them, as they stand, to build a decision criterion: flag_expensive_runs, this module's first function that doesn't just measure, but classifies. It's, in a precise sense, the bridge to Module 5: there, a criterion similar to this one becomes a deterministic PASS/FAIL gate.

Cost and correctness are distinct signals

Before building the criterion, one important precision is worth making: an expensive run isn't the same as a run with errors. Go back over lesson 06's batch: Ana's run had a tool_result with is_error: True (the invalid tier) and yet isn't the batch's most expensive. Carla's run — comparing six room-and-tier combinations before deciding — had no errors at all, and is, by a clear margin, the most expensive of the four. A run can be perfectly correct and still costly — because it did more work, not because something failed; and a run can fail without being particularly expensive — like the invalid-tier attempt, rejected before even running anything.

This distinction matters for the design of any alerting system: error rate (Module 1) answers "is the system malfunctioning?"; cost (this module) answers an orthogonal question, "is the system operating in a more expensive way than expected?" Both signals, together, give a more complete picture than either one alone.

Worked example: finding the batch's anomalous run

Go back to lesson 06's four CostReports — same trace_ids, same tokens — and calculate how far each one is from the batch's average:

import statistics

# Los mismos cuatro reports de la lección 06 (reejecutados aquí para esta lección).
reports_summary = [
    ("run-8487582448eb", 64, 56, "Reserva Focus pro 3h para Ana"),
    ("run-ae6ff85cf0b0", 53, 49, "Reserva Boardroom pro 1h para Sofía"),
    ("run-2c27934d8a39", 27, 31, "Reserva y cancela Studio basic 1h para Diego"),
    ("run-cecde864aa84", 88, 118, "Compara todas las salas antes de reservar la mejor opción para Carla"),
]

totals = [input_tokens + output_tokens for _, input_tokens, output_tokens, _ in reports_summary]
avg_tokens = statistics.mean(totals)

print(f"promedio del lote: {avg_tokens} tokens totales")
print()
print(f"{'trace_id':<18} {'total_tokens':>12} {'ratio vs promedio':>18}")
for trace_id, input_tokens, output_tokens, question in reports_summary:
    total = input_tokens + output_tokens
    ratio = total / avg_tokens
    print(f"{trace_id:<18} {total:>12} {ratio:>17.2f}x")

What to expect:

promedio del lote: 121.5 tokens totales

trace_id             total_tokens  ratio vs promedio
run-8487582448eb              120              0.99x
run-ae6ff85cf0b0               102              0.84x
run-2c27934d8a39                58              0.48x
run-cecde864aa84               206              1.70x

Carla's run — run-cecde864aa84 — uses 1.70 times the batch average. It isn't an accident: its script compares six room-and-tier combinations before booking, generating twice the tool calls of the typical run. Diego's, at the other extreme, uses less than half the average (0.48x) — a short, cheap run, only two steps.

flag_expensive_runs: a simple, deterministic criterion

Turn that observation into a reusable function — a fixed threshold, with no model call at all to "judge" whether a run is expensive:

def flag_expensive_runs(reports, threshold_ratio=1.5):
    """Marca los CostReport cuyo total de tokens supera threshold_ratio
    veces el promedio del lote. Un criterio de FORMA, determinista -- sin
    ningún juicio semántico sobre el contenido del run."""
    totals = [r.input_tokens + r.output_tokens for r in reports]
    avg = statistics.mean(totals)
    flagged = []
    for r, total in zip(reports, totals):
        ratio = total / avg
        if ratio >= threshold_ratio:
            flagged.append((r, ratio))
    return flagged


flagged = flag_expensive_runs(reports, threshold_ratio=1.5)
print(f"runs marcados como anómalamente caros (>= 1.5x el promedio del lote): {len(flagged)}")
for r, ratio in flagged:
    print(f"  {r.trace_id}: {ratio:.2f}x -- {r.question!r}")

What to expect:

runs marcados como anómalamente caros (>= 1.5x el promedio del lote): 1
  run-cecde864aa84: 1.70x -- 'Compara todas las salas antes de reservar la mejor opción para Carla'

A 1.5x threshold over the batch average — an arbitrary number, chosen here only as a reasonable example — flags exactly one run: Carla's. This is, precisely, the kind of criterion that becomes part of a regression gate in Module 5: not "is the answer good?" — that's a semantic judgment, outside this guide's scope — but "does the cost stay under a fixed threshold, yes or no?"

Why this signal matters at scale: the anomalous run's cost, projected

A 1.70x over four runs looks like a minor detail. Project it, with lesson 06's same technique, to the scale Reservo actually operates at:

n = 100_000
avg_input = sum(r.input_tokens for r in reports) / len(reports)
avg_output = sum(r.output_tokens for r in reports) / len(reports)
carla_report = next(r for r in reports if r.trace_id == "run-cecde864aa84")

cost_typical = estimate_cost_cents(int(avg_input * n), int(avg_output * n))
cost_carla_type = estimate_cost_cents(carla_report.input_tokens * n, carla_report.output_tokens * n)

print(f"costo proyectado, run TÍPICO x {n}      : {cost_typical} centavos = ${cost_typical / 100:.2f}")
print(f"costo proyectado, run tipo-Carla x {n}  : {cost_carla_type} centavos = ${cost_carla_type / 100:.2f}")
print(f"diferencia atribuible a este patrón     : {cost_carla_type - cost_typical} centavos = ${(cost_carla_type - cost_typical) / 100:.2f}")

What to expect:

costo proyectado, run TÍPICO x 100000      : 11265 centavos = $112.65
costo proyectado, run tipo-Carla x 100000  : 20340 centavos = $203.40
diferencia atribuible a este patrón        : 9075 centavos = $90.75

If the "compare every combination before deciding" pattern became common among Reservo users — not an isolated exception — the cost difference against a typical run would be almost $91 per every 100,000 runs of that type, just from the way that pattern queries the system. This is, precisely, the real usefulness of treating cost as a signal: not to judge whether run-cecde864aa84 "did something wrong" — it didn't, it solved the task correctly — but to know that that usage pattern, if it repeats at scale, has a measurable, quantified budget impact.

The boundary: measuring cost, not reducing it

This lesson — and this entire module — stop exactly here. Identifying that a run (or a pattern of runs) is abnormally expensive is this guide's entire responsibility on cost. What this lesson does not do, on purpose, is propose any way to lower that cost: it doesn't cache list_rooms's tool_result between repeated calls, it doesn't suggest using a cheaper model for simple tasks, it doesn't batch similar questions to save repeated system tokens.

All of that — prompt caching, cost-based model selection, batching, the complete anatomy of what makes up a call's cost and how each component is optimized — is cost-optimization-caching-guide's central content, a sibling guide in the AI Engineering ecosystem. Once the question stops being "how much did it cost, and which run cost more than expected?" and becomes "how do I make this cost less?", that's the guide to go to — precisely named here, at the exact point where this guide stops.

Where this signal goes in the rest of the guide

Cost per run, already calculated and already classified, doesn't end at this lesson — the rest of this guide reuses it:

• Module 4 adds latency per run, the second half of the "measure" layer Module 1 promised — with the same honesty discipline about what's measured and what's modeled.
• Module 5 turns a threshold like flag_expensive_runs's into part of a regression gate: a run whose cost exceeds a fixed limit can fail the build, exactly like an invalid schema or an incorrectly chosen tool — never a judgment about whether the response is "good."
• Module 6, when a circuit breaker stops calls to a tool that keeps failing, also saves the cost of those attempts — a measurable consequence with this module's tools, even though the circuit breaker itself gets built for resilience reasons, not cost ones.
• Module 8 closes the guide with a metrics report that includes cost, alongside the other signals, over the complete Reservo agent.

Common mistakes

1. Treating "expensive run" and "run with an error" as synonyms. The worked example directly disproves it: the batch's most expensive run (Carla) had no errors; the run with an error (Ana) wasn't the most expensive. Both signals are calculated and read separately.

2. Choosing a flag_expensive_runs threshold without justifying it. 1.5x in this lesson is a reasonable example, not a universal rule — a real system would choose that threshold based on the real cost the business can tolerate, not an arbitrary figure copied from a lesson.

3. Confusing "identifying an expensive run" with "identifying a malicious or abusive run." flag_expensive_runs flags runs by their cost, with no judgment at all about the intent of whoever generated them — a legitimate user with a genuinely complex task produces the same signal as any other expensive pattern. Telling legitimate use apart from abuse is a different problem, outside this lesson's scope.

4. Proposing a cost optimization the moment an expensive run gets detected. This lesson deliberately stops at identifying and quantifying — never at proposing a caching, batching, or model-selection solution. That boundary, precisely named above, is one of the easiest to cross without noticing.

5. Calculating the batch average over too few runs and trusting the resulting threshold. With only four runs, a single outlier — like Carla's — also pulls the average upward, which makes the threshold less stable than over a batch of thousands. Module 8 is going to show this same technique over a more representative batch.

Exercises

Exercise 1: Try a stricter threshold (Easy)

Using flag_expensive_runs and the same batch of four CostReports, try a threshold of 1.2 instead of 1.5. How many runs get flagged now?

flagged_strict = flag_expensive_runs(reports, threshold_ratio=1.2)
print(f"runs marcados con umbral 1.2x: {len(flagged_strict)}")
for r, ratio in flagged_strict:
    print(f"  {r.trace_id}: {ratio:.2f}x")

runs marcados con umbral 1.2x: 1
  run-cecde864aa84: 1.70x

Exercise 2: Find the minimum threshold that flags NO run (Medium)

Find, with code, the threshold_ratio value right above which flag_expensive_runs stops flagging any run in the batch — that is, the batch's most expensive run's exact ratio.

totals = [r.input_tokens + r.output_tokens for r in reports]
avg = statistics.mean(totals)
max_ratio = max(total / avg for total in totals)
print(f"ratio del run más caro del lote: {max_ratio:.4f}x")
print(f"con threshold_ratio > {max_ratio:.4f}, ningún run del lote se marca")

# Confirmación:
sin_marcar = flag_expensive_runs(reports, threshold_ratio=max_ratio + 0.01)
print("runs marcados justo por encima de ese umbral:", len(sin_marcar))

ratio del run más caro del lote: 1.6955x
con threshold_ratio > 1.6955, ningún run del lote se marca
runs marcados justo por encima de ese umbral: 0

Exercise 3: Design a composite criterion, cost AND errors (Hard)

Write a flag_concerning_runs(reports, error_counts, cost_threshold=1.5) function that receives this lesson's CostReports along with an error_counts dictionary (trace_id -> number of tool_errors, from Module 2), and returns the runs that are abnormally expensive OR had at least one error — a union of both signals, not an intersection. Test it with error_counts = {"run-8487582448eb": 1, "run-ae6ff85cf0b0": 0, "run-2c27934d8a39": 0, "run-cecde864aa84": 0} (the batch's real tool_errors, per Module 2).

def flag_concerning_runs(reports, error_counts, cost_threshold=1.5):
    """Marca un run si es anómalamente caro (costo) O si tuvo al menos un
    tool_error (correctitud) -- unión de dos señales independientes,
    ninguna sustituye a la otra."""
    expensive = {r.trace_id for r, _ in flag_expensive_runs(reports, cost_threshold)}
    concerning = []
    for r in reports:
        is_expensive = r.trace_id in expensive
        has_errors = error_counts.get(r.trace_id, 0) > 0
        if is_expensive or has_errors:
            concerning.append((r.trace_id, is_expensive, has_errors))
    return concerning


error_counts = {
    "run-8487582448eb": 1, "run-ae6ff85cf0b0": 0,
    "run-2c27934d8a39": 0, "run-cecde864aa84": 0,
}
result = flag_concerning_runs(reports, error_counts)
for trace_id, is_expensive, has_errors in result:
    print(f"{trace_id}: caro={is_expensive}  con_errores={has_errors}")

run-8487582448eb: caro=False  con_errores=True
run-cecde864aa84: caro=True  con_errores=False

Summary and next step

• We confirmed cost and correctness are distinct signals: the batch's most expensive run had no errors; the run with an error wasn't the most expensive.
• We built flag_expensive_runs: a deterministic criterion, based on a fixed threshold over a run's token ratio against the batch average — no semantic judgment at all, the same SHAPE discipline Module 5 is going to demand of every gate in this guide.
• We projected an expensive usage pattern's real impact — Carla's run, 1.70x the average — to 100,000 runs: a $90.75 difference against a typical run, a real budget figure, not a curiosity from four examples.
• We precisely traced the boundary: this guide measures cost and uses it as a signal; cost-optimization-caching-guide teaches how to reduce it with caching, model selection, and batching.

Next lesson: 08 — Mini-Project: A Cost Report for Reservo Runs. We close the module by bringing every piece together — estimate_tokens, the fixed pricing, cost_for_run, aggregation, scaling, and this lesson's criterion — into a complete observability/cost_calculator.py, run over a batch of traced runs.

Additional resources

1. Python — statistics.mean — The function used to calculate the batch average, the foundation of flag_expensive_runs.
2. Anthropic — Building effective agents — On why cost, alongside error rate, is one of the signals that determine whether an agentic system is ready to operate at scale.
3. Python — set and dict comprehensions — The {r.trace_id for r, _ in ...} pattern used in Exercise 3 to build a composite criterion.
4. Sibling guide — cost-optimization-caching-guide (AI Engineering): the complete anatomy of a call's cost and how to reduce it — prompt caching, model selection, batching. The exact point where this guide stops and that guide continues.
5. Python 3.14 — What's New — The version every line of code in this lesson ran on.