The Operational Signals That Matter

Description

Lesson 03 confirmed, with real code, that the problem isn't "it would be nice to know more" — there are concrete questions with no answer. This lesson gives those questions a name and a shape: four signals, each with a precise definition, that together are the equivalent of the analogy's complete dashboard. Not all four are equally easy to capture today — two of them (error rate, per-tool failure rate) can already be calculated with what history gives you; the other two (cost, latency) need a new layer, which starts in lesson 05.

This lesson sticks with the first two, in depth, and leaves the other two set up for the next one. The reason for splitting it this way isn't arbitrary: error rate and per-tool failure rate are signals you count over data that already exists; cost and latency are signals you have to estimate or model, with a new engineering layer on top. It's worth mastering the first half — the simpler one — before building the second.

Connection to the module

This is the lesson where the module stops naming the problem and starts building the vocabulary to solve it. This lesson's four signals are, literally, the four business questions lesson 07 is going to lay out as a formal brief, and the four things lesson 08's mini-project is going to measure together over a batch of runs.

Analogy: four gauges, not a hundred

A car's dashboard doesn't show you everything happening inside the engine — that would be a shop's diagnostic panel, overwhelming and useless while you're driving. It shows you a handful of gauges, chosen carefully because each one answers a specific question you need to be able to answer in seconds: how fast am I going? how much gas do I have left? is the engine overheating? is any warning light on?

An agent's operational signals are that same kind of deliberate choice. It's not about logging absolutely everything history contains and hoping someone makes sense of it later — it's about choosing a handful of numbers that answer, in seconds, the questions that actually matter. This lesson chooses four:

ERROR RATE                -> out of every hundred runs, how many did NOT
                              finish (RuntimeError, cap exhausted)?
PER-TOOL FAILURE RATE     -> within the runs that DID finish, what
                              fraction of their tool calls brought is_error?
COST PER RUN               -> how much, in cents, did solving this
                              task cost? (lesson 05)
LATENCY PER RUN            -> how long, modeled, did solving this
                              task take? (lesson 05)

Worked example: two signals, precisely defined and calculated over real runs

The distinction you need clear before writing a single line of code

Notice something easy to mix up: error rate and per-tool failure rate are not the same signal, even though both speak to "something that went wrong." Error rate is at the whole-run level: did the run, as a whole, reach a stop_reason: "end_turn", or did it blow through the iteration cap with a RuntimeError? Per-tool failure rate is at the level of each individual tool-call attempt, within a run that did finish successfully: out of the four times a tool call was attempted, how many brought is_error? A run can have a high per-tool failure rate — several trip-ups along the way — and still count as a success at the error-rate level, because it finished. That is, precisely, what you already saw in agent-fundamentals M8: "completing the task" is not the same as "never getting anything wrong along the way."

RunSignals: a structure for this lesson's two signals

from dataclasses import dataclass
import reservo_agent as ra


@dataclass
class RunSignals:
    """Las señales que importan de UN run: cuántos pasos dio, cuántas tool
    calls hizo, cuántas de esas tool calls trajeron is_error, y si el run
    terminó (end_turn) o reventó el tope de iteraciones."""
    steps: int
    tool_calls: int
    tool_errors: int
    completed: bool

    @property
    def tool_fail_rate(self):
        if self.tool_calls == 0:
            return 0.0
        return self.tool_errors / self.tool_calls


def compute_run_signals(history, completed=True):
    tool_calls = 0
    tool_errors = 0
    for turn in history:
        content = turn["content"]
        if not isinstance(content, list):
            continue
        for block in content:
            if block["type"] == "tool_use":
                tool_calls += 1
            elif block["type"] == "tool_result" and block.get("is_error"):
                tool_errors += 1
    return RunSignals(steps=len(history), tool_calls=tool_calls, tool_errors=tool_errors, completed=completed)

compute_run_signals walks history exactly once, counting two things: how many tool_use blocks appear (each one is a tool-call attempt) and how many tool_result blocks bring is_error: True. The completed parameter is information that does not come from history — it comes from whether the call to run_reservo_agent finished with a normal return or with a RuntimeError; that's why it's received as an argument instead of being computed, something lesson 03 already made clear history cannot tell you on its own when the run fails outright.

Running this over three different runs, and over the whole batch

# Run A: la demo -- Ana, Focus pro 3h, un tier inválido corregido.
script_a = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Focus", "tier": "premium", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #1."}]},
]

# Run B: Sofía, Boardroom pro 1h -- el modelo acierta el tier al primer intento.
script_b = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "book_room",
         "input": {"room": "Boardroom", "tier": "pro", "hours": 1, "member": "Sofía"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Boardroom pro por 1 hora para Sofía. Total $64.00. Confirmación #2."}]},
]

# Run C: Ana otra vez, pero con DOS errores en el camino (tier inválido, hours inválido).
script_c = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "list_rooms", "input": {}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "get_quote",
         "input": {"room": "Focus", "tier": "premium", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_03", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 0}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_04", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_05", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé Focus pro por 3 horas para Ana. Total $60.00. Confirmación #3."}]},
]

batch = [
    ("Ana - Focus pro 3h", "Reserva Focus pro 3h para Ana", script_a),
    ("Sofía - Boardroom pro 1h", "Reserva Boardroom pro 1h para Sofía", script_b),
    ("Ana - Focus pro 3h (2 tropiezos)", "Reserva Focus pro 3h para Ana", script_c),
]

all_signals = []
for label, question, script in batch:
    final, history = ra.run_reservo_agent(question, script)
    signals = compute_run_signals(history)
    all_signals.append(signals)
    print(f"{label:35} {signals}  tool_fail_rate={signals.tool_fail_rate:.0%}")

print()
total_calls = sum(s.tool_calls for s in all_signals)
total_errors = sum(s.tool_errors for s in all_signals)
total_completed = sum(1 for s in all_signals if s.completed)
print(f"lote de {len(all_signals)} runs")
print(f"tool calls totales      : {total_calls}")
print(f"tool errors totales     : {total_errors}")
print(f"tool-fail rate del lote : {total_errors / total_calls:.0%}")
print(f"runs completados        : {total_completed}/{len(all_signals)} ({total_completed / len(all_signals):.0%})")

What to expect:

Ana - Focus pro 3h                  RunSignals(steps=10, tool_calls=4, tool_errors=1, completed=True)  tool_fail_rate=25%
Sofía - Boardroom pro 1h            RunSignals(steps=8, tool_calls=3, tool_errors=0, completed=True)  tool_fail_rate=0%
Ana - Focus pro 3h (2 tropiezos)    RunSignals(steps=12, tool_calls=5, tool_errors=2, completed=True)  tool_fail_rate=40%

lote de 3 runs
tool calls totales      : 12
tool errors totales     : 3
tool-fail rate del lote : 25%
runs completados        : 3/3 (100%)

Read it carefully, because there are two real findings in this output, not just numbers:

1. All three runs finished. runs completados: 3/3 (100%) — the error rate, at the run level, is 0%. Not a single run blew through the iteration cap, no matter how many trip-ups it had along the way.
2. The per-tool failure rate varies enormously run by run (25%, 0%, 40%), but it stabilizes at 25% when you look at it aggregated over the whole batch. A single run tells you very little about the system's general behavior — you need a batch for the number to start meaning something. This is exactly why neither of this lesson's two signals makes sense measured over an isolated run; both are meant to be aggregated over dozens, hundreds, thousands of runs.

Why error rate (run) and per-tool failure rate (tool call) point to different places

These two signals, although related, tell you different things about where to look when something's wrong:

• A high error rate at the run level (many RuntimeErrors, many runs that never reach end_turn) points to the loop's design: is max_iterations too low for the real complexity of the tasks? Is the model falling into a pattern where it repeats the same tool without converging? This is, precisely, the kind of question that adjusting max_iterations solves, or reviewing why the model (concept) isn't deciding to finish.
• A high per-tool failure rate (many is_errors, even though the runs finish fine) points somewhere else: is a specific tool frequently receiving invalid arguments? Is an input_schema's enum too strict for what real users ask for? Does one particular tool concentrate most of the failures? This signal, broken down by tool name — not just the aggregate number — is what this guide's Module 6 uses to decide when a circuit breaker should open for a specific tool.

Confusing the two leads to wrong diagnoses: if the per-tool failure rate rises because get_quote is frequently receiving tier="premium", the fix isn't touching max_iterations — it's reviewing why the model (concept) keeps proposing a tier that doesn't exist, maybe because the system prompt doesn't make clear what the valid values are. That's exactly the kind of decision Module 7 (versioning) formalizes: change the prompt, and compare the per-tool failure rate BEFORE and AFTER the change, with the same gate.

Common mistakes

1. Calculating the per-tool failure rate over a single run and drawing a conclusion. The worked example showed it with real numbers: 25%, 0%, 40% — three runs, three completely different numbers. None of the three, on its own, tells you whether the system "is fine" or "is broken." Only the aggregate over the batch (25%) starts to be a signal you can reason with.

2. Treating tool_calls == 0 as if it were a division error. RunSignals.tool_fail_rate explicitly checks if self.tool_calls == 0 before dividing — a run that never called any tool at all (for example, a question the model answered directly, without needing tools) has a failure rate of 0.0, not an error, and shouldn't be treated as "no data" when aggregating the batch.

3. Mixing "error rate" (run) with "per-tool failure rate" (tool call) into the same number. They're signals from different layers, with different causes and different remedies, as the previous section explained. A dashboard that mixes them into a single "% of problems" hides exactly the information you need to know where to look first.

4. Thinking completed=True means "with no is_error at all." No — it means the run reached a stop_reason: "end_turn" within the iteration cap, no matter how many is_errors it had along the way. Run C in the worked example is exactly that case: completed=True with tool_errors=2.

5. Expecting this lesson to already calculate cost and latency. It doesn't calculate them — it names them, and leaves them set up. compute_run_signals operates exclusively over what history already contains (steps, tool calls, is_error); cost and latency need an estimation layer that doesn't exist yet at this point in the module. That is, precisely, lesson 05.

Exercises

Exercise 1: Calculate a fourth run's signals (Easy)

Using compute_run_signals, calculate the signals for "Diego's" script — books Studio, basic, 1 hour, then cancels it — with this script:

script_d = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Studio", "tier": "basic", "hours": 1, "member": "Diego"}}]},
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_02", "name": "cancel_booking", "input": {"id": 4}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Reservé y luego cancelé Studio basic 1h para Diego."}]},
]

Before running it: how many tool_calls do you expect? How many tool_errors?

final_d, history_d = ra.run_reservo_agent("Reserva y cancela Studio basic 1h para Diego", script_d)
signals_d = compute_run_signals(history_d)
print(signals_d, "tool_fail_rate:", signals_d.tool_fail_rate)

RunSignals(steps=6, tool_calls=2, tool_errors=0, completed=True) tool_fail_rate: 0.0

Exercise 2: Extend RunSignals with a new signal (Medium)

Add a unique_tools field to RunSignals — how many distinct tools (not counting repeats) were called in the run — and adjust compute_run_signals to calculate it. Confirm it over Run C from the worked example (which calls get_quote three times, but it's the same tool).

from dataclasses import dataclass, field

@dataclass
class RunSignalsV2:
    steps: int
    tool_calls: int
    tool_errors: int
    unique_tools: int
    completed: bool

    @property
    def tool_fail_rate(self):
        return self.tool_errors / self.tool_calls if self.tool_calls else 0.0


def compute_run_signals_v2(history, completed=True):
    tool_calls = 0
    tool_errors = 0
    names = set()
    for turn in history:
        content = turn["content"]
        if not isinstance(content, list):
            continue
        for block in content:
            if block["type"] == "tool_use":
                tool_calls += 1
                names.add(block["name"])
            elif block["type"] == "tool_result" and block.get("is_error"):
                tool_errors += 1
    return RunSignalsV2(steps=len(history), tool_calls=tool_calls, tool_errors=tool_errors,
                         unique_tools=len(names), completed=completed)


final_c, history_c = ra.run_reservo_agent("Reserva Focus pro 3h para Ana", script_c)
print(compute_run_signals_v2(history_c))

RunSignalsV2(steps=12, tool_calls=5, tool_errors=2, unique_tools=3, completed=True)

Exercise 3: Design a signal that combines the two, and explain why you should NOT merge them into one (Hard)

Someone on your team proposes replacing error rate and per-tool failure rate with a single metric: "% of perfect runs" — the percentage of runs that finished (completed=True) and had tool_errors=0. Calculate that metric over the worked example's batch of three runs, and then explain in a paragraph what information is lost by merging the two signals into one.

perfect_runs = sum(1 for s in all_signals if s.completed and s.tool_errors == 0)
print(f"% de runs perfectos: {perfect_runs}/{len(all_signals)} = {perfect_runs/len(all_signals):.0%}")

% de runs perfectos: 1/3 = 33%

Summary and next step

• We precisely defined two of this guide's four operational signals: error rate (at the whole-run level, RuntimeError vs. end_turn) and per-tool failure rate (at the level of each tool-call attempt, is_error).
• We built RunSignals and compute_run_signals, and ran them over three real runs: individual rates of 25%, 0%, and 40% that stabilize at 25% when aggregated over the batch — proof that a single run says little, and a batch starts to say something.
• We explained why mixing the two signals into a single number (like Exercise 3's "% of perfect runs") hides exactly the distinction needed to know whether the problem is in the loop's design or in argument validation.
• Cost per run and latency per run — the other two signals — were named and left pending for the estimation layer the next lesson builds.

Next lesson: 05 — A First Look at Cost, Latency, and Errors. With claude-sonnet-5's fixed pricing formula and a latency model declared in data, we calculate for the first time, for real, how much the canonical Reservo run cost and how long it took.

Additional resources

1. Python — dataclasses — @dataclass and @property, the foundation of RunSignals and its computed tool_fail_rate.
2. Anthropic — Tool use (function calling) overview — The shape of tool_use/tool_result that every signal in this lesson is counted over.
3. Anthropic — Building effective agents — On why a reliable agent is measured with specific signals, not a single fuzzy notion of "it works well."
4. Python 3.14 — What's New — The version every code block in this lesson ran on.