The Resilience Layer

Description

With the agent instrumented, measured, and gated, this lesson sets the fourth discipline in motion: hardening. None of the previous three prepares Reservo for what this lesson genuinely simulates: book_room going down sustainedly, across several consecutive "users," while the rest of the system keeps running. M6's CircuitBreaker — the CLOSED/OPEN/HALF_OPEN three-state machine — is the piece that gives the agent the memory it was missing: not just retrying with backoff within a single attempt, but remembering, across different users' attempts, that a tool has already proven dead.

This lesson reuses, without changing a line, the complete CLOSED → OPEN → HALF_OPEN → CLOSED cycle M6 (Lesson 4) already built and ran, and integrates it with the rest of this capstone's operations layer: every attempt gets recorded with M2's same logging discipline, and the savings the breaker produces get read with M3's/M4's same cost-and-latency vocabulary.

Connection to the module

This lesson builds no new mechanism — it picks back up CircuitBreaker, CircuitOpenError, call_with_breaker, and retry_with_backoff from resilience/tool_circuit_breaker.py (M6), and flaky_book_room (M6, Lesson 2), and confirms the connection point this module's Lesson 2 called "Height 2" — replacing a specific tool's real function — works exactly as described, with the observability layer (Height 1) recording every attempt.

Analogy: the backup generator, with a log book beside it

This module's introduction's restaurant has a backup generator that turns on by itself the moment a kitchen station stops responding. But a generator that turns on silently, with nobody noticing, is only half useful — what really matters, for a business that bills and reports, is the log book sitting beside it: what time it turned on, how many times the station tried to reconnect before giving up, and how much it cost (in time, in wasted ingredients) to discover the outage before the generator took over. This lesson connects the two pieces: the generator (CircuitBreaker, M6) and the log book (log_event, M2) — not because one depends on the other to function, but because together they answer the complete question a real business needs: not just "did the system stay up?", but "what exactly happened while it was down, and how much did it cost?"

Worked example: the complete cycle, reused from M6

book_room, sustainedly going down — nine real calls before recovering

import reservo_tools as rt
import reservo_agent as ra
from tool_circuit_breaker import CircuitBreaker, CircuitOpenError, call_with_breaker

_book_room_real = rt.book_room
_state = {"count": 0}
OUTAGE_CALLS = 9


def flaky_book_room(room, tier, hours, member):
    """La misma tool inestable de M6 (Lección 2): falla las primeras nueve
    llamadas con un error TRANSITORIO, después se recupera sola."""
    _state["count"] += 1
    if _state["count"] <= OUTAGE_CALLS:
        raise ConnectionError(f"timeout de red simulado (llamada real #{_state['count']})")
    return _book_room_real(room, tier, hours, member)


breaker = CircuitBreaker("book_room", failure_threshold=3, cooldown_calls=2)

print("=== 7 runs independientes, book_room caído las primeras 9 llamadas reales ===")
for run_n in range(1, 8):
    print(f"--- run {run_n} (estado del breaker ANTES: {breaker.state}) ---")
    try:
        result = call_with_breaker(
            breaker, flaky_book_room, room="Focus", tier="pro", hours=3, member=f"user{run_n}",
            max_retries=3, base_delay_ms=100,
        )
        print(f"  OK -> {result}")
    except CircuitOpenError as exc:
        print(f"  RECHAZADO SIN LLAMAR A LA TOOL: {exc}")
    except ConnectionError as exc:
        print(f"  FALLO (tope de reintentos agotado): {exc}")
    print(f"  estado del breaker DESPUÉS: {breaker.state} (failure_count={breaker.failure_count})")

print()
print(f"llamadas reales totales a book_room: {_state['count']}")

What to expect:

=== 7 runs independientes, book_room caído las primeras 9 llamadas reales ===
--- run 1 (estado del breaker ANTES: CLOSED) ---
  FALLO (tope de reintentos agotado): timeout de red simulado (llamada real #3)
  estado del breaker DESPUÉS: CLOSED (failure_count=1)
--- run 2 (estado del breaker ANTES: CLOSED) ---
  FALLO (tope de reintentos agotado): timeout de red simulado (llamada real #6)
  estado del breaker DESPUÉS: CLOSED (failure_count=2)
--- run 3 (estado del breaker ANTES: CLOSED) ---
  FALLO (tope de reintentos agotado): timeout de red simulado (llamada real #9)
  estado del breaker DESPUÉS: OPEN (failure_count=3)
--- run 4 (estado del breaker ANTES: OPEN) ---
  RECHAZADO SIN LLAMAR A LA TOOL: book_room: circuito abierto (OPEN), llamada rechazada sin tocar la tool
  estado del breaker DESPUÉS: OPEN (failure_count=3)
--- run 5 (estado del breaker ANTES: OPEN) ---
  RECHAZADO SIN LLAMAR A LA TOOL: book_room: circuito abierto (OPEN), llamada rechazada sin tocar la tool
  estado del breaker DESPUÉS: OPEN (failure_count=3)
--- run 6 (estado del breaker ANTES: OPEN) ---
  OK -> {'booking_id': 1, 'confirmed': True, 'price_cents': 6000}
  estado del breaker DESPUÉS: CLOSED (failure_count=0)
--- run 7 (estado del breaker ANTES: CLOSED) ---
  OK -> {'booking_id': 1, 'confirmed': True, 'price_cents': 6000}
  estado del breaker DESPUÉS: CLOSED (failure_count=0)

llamadas reales totales a book_room: 11

Walk through it with M6's thermal switch in mind: runs 1 and 2 each fail, exhausting their three backoff retries, without the breaker opening yet (failure_count rises to 1, then 2). Run 3 fails again — the third consecutive failure — and the breaker trips to OPEN. Runs 4 and 5 don't even touch book_room — rejected at that exact instant, no backoff, no timeout at all. Run 6: the two-rejection cooldown gets met, the breaker lets a HALF_OPEN probe through, the tool has already recovered, and the breaker goes back to CLOSED. Run 7: normal, start to finish. Total: 11 real calls for seven runs — not 21, what seven runs of up to three attempts each would cost with no breaker at all.

Why the CircuitBreaker never gets installed INSIDE TOOLS

Before connecting this cycle to the rest of the operations layer, it's worth resolving a question this module's Lesson 2 left open: why does call_with_breaker get called directly at the point where the agent would request the tool, instead of replacing TOOLS's "book_room" entry with a "protected" version? The answer is a real integration friction, and it's worth seeing precisely: dispatch_robust (agent-fundamentals M7) already has its own internal retry — up to max_retries=3 attempts, no backoff, catching ConnectionError/FutureTimeoutError — before turning a failure into a tool_result with is_error: True. If TOOLS["book_room"] were a function already internally retrying with retry_with_backoff (M6), a failure exhausting those internal retries would still be a real ConnectionError rising up to dispatch_robust — and dispatch_robust, seeing it, would retry again, triggering a second complete retry_with_backoff cycle for each of its own attempts. The result would be nested retries — up to 3 × 3 = 9 real attempts for a single tool_use — exactly the noise this module exists to avoid, not multiply.

That's why call_with_breaker gets called at the point where the loop, not dispatch_robust, would invoke the tool — the same pattern M6 (Lesson 4) already used, and this lesson reproduces unchanged. CircuitOpenError, on the other hand, does travel safely through dispatch_robust: it isn't ConnectionError or FutureTimeoutError, so it falls into the generic except Exception branch, which never retries — it immediately turns into a tool_result with is_error: True. That's, precisely, the safe integration this module's Lesson 2 (Exercise 3) already traced: the breaker's rejection does reach M2's log clean; M6's backoff retries, on the other hand, live in a layer that never crosses paths with M7's own retry.

Integrating the log book: every attempt, recorded with M2's discipline

With that boundary clear, this lesson adds the log book to the generator — reusing RunEvent/ToolCallEvent and log_event from M2 directly, without going through traced_run (which patches dispatch_robust, a layer this cycle deliberately doesn't use):

import itertools
import logging

import run_logger as rl

_trace_id = rl.make_trace_id("book_room caido -- lote de 7 usuarios", 1)
_seq = itertools.count(1)

breaker2 = CircuitBreaker("book_room", failure_threshold=3, cooldown_calls=2)
_state2 = {"count": 0}
OUTAGE_CALLS_2 = 9

def flaky_book_room_2(room, tier, hours, member):
    _state2["count"] += 1
    if _state2["count"] <= OUTAGE_CALLS_2:
        raise ConnectionError(f"timeout de red simulado (llamada real #{_state2['count']})")
    return _book_room_real(room, tier, hours, member)

rl.log_event(logging.INFO, rl.RunEvent(seq=next(_seq), trace_id=_trace_id, event="run_started",
                                        question="book_room caido -- lote de 7 usuarios"))
for run_n in range(1, 8):
    step_before = breaker2.state
    try:
        result = call_with_breaker(breaker2, flaky_book_room_2, room="Focus", tier="pro", hours=3,
                                    member=f"user{run_n}", max_retries=3, base_delay_ms=100)
        rl.log_event(logging.INFO, rl.ToolCallEvent(
            seq=next(_seq), trace_id=_trace_id, event="tool_result", step=run_n, tool="book_room",
            is_error=False, content=str(result)))
    except (CircuitOpenError, ConnectionError) as exc:
        rl.log_event(logging.ERROR, rl.ToolCallEvent(
            seq=next(_seq), trace_id=_trace_id, event="tool_result", step=run_n, tool="book_room",
            is_error=True, content=f"{type(exc).__name__}: {exc}"))
rl.log_event(logging.INFO, rl.RunEvent(seq=next(_seq), trace_id=_trace_id, event="run_finished",
                                        question="book_room caido -- lote de 7 usuarios"))

What to expect (each line, a real JSON event):

{"seq": 1, "trace_id": "run-1efeabb7e560", "event": "run_started", "question": "book_room caido -- lote de 7 usuarios", "error": ""}
{"seq": 2, ..., "event": "tool_result", "step": 1, "tool": "book_room", "is_error": true, "content": "ConnectionError: timeout de red simulado (llamada real #3)"}
{"seq": 3, ..., "event": "tool_result", "step": 2, "tool": "book_room", "is_error": true, "content": "ConnectionError: timeout de red simulado (llamada real #6)"}
{"seq": 4, ..., "event": "tool_result", "step": 3, "tool": "book_room", "is_error": true, "content": "ConnectionError: timeout de red simulado (llamada real #9)"}
{"seq": 5, ..., "event": "tool_result", "step": 4, "tool": "book_room", "is_error": true, "content": "CircuitOpenError: book_room: circuito abierto (OPEN), llamada rechazada sin tocar la tool"}
{"seq": 6, ..., "event": "tool_result", "step": 5, "tool": "book_room", "is_error": true, "content": "CircuitOpenError: book_room: circuito abierto (OPEN), llamada rechazada sin tocar la tool"}
{"seq": 7, ..., "event": "tool_result", "step": 6, "tool": "book_room", "is_error": false, "content": "{'booking_id': 1, 'confirmed': True, 'price_cents': 6000}"}
{"seq": 8, ..., "event": "tool_result", "step": 7, "tool": "book_room", "is_error": false, "content": "{'booking_id': 1, 'confirmed': True, 'price_cents': 6000}"}
{"seq": 9, "trace_id": "run-1efeabb7e560", "event": "run_finished", "question": "book_room caido -- lote de 7 usuarios", "error": ""}

This log book says, at a glance and with no rerun at all, something M6's cycle on its own never left written anywhere: the breaker's rejections (run 4, run 5) have exactly the same event/is_error as a genuine tool failure (run 1, run 2, run 3) — the difference lives only in content, with the exception's name. To any code reading RUN_LOG.jsonl later — like summarize_by_trace (Lesson 3) — both count the same as "a step that failed"; the distinction between "the tool genuinely failed" and "the breaker didn't even let it try" only shows up if someone reads content carefully, or if the trace_id gets cross-referenced with the CircuitBreaker's state at that exact attempt's moment — information this capstone DOES have, in breaker.state, but that the log file, by design, doesn't duplicate.

What the breaker really saves, with M3's and M4's vocabulary

11 real calls against 21 with no breaker at all isn't just an abstract "efficiency" figure — it is, precisely, the same kind of saving M3 and M4 already taught you to measure. Each of the ten avoided calls (runs 4 and 5, plus the attempts runs 1-3 would have spent extra if the breaker had taken longer to open) is a call that, in a real system connected to a database or an external service, would have consumed waiting time up to the timeout — what M4 measures in milliseconds — and, if book_room were part of a tool_result the model (concept) has to read, real input tokens — what M3 measures in cents. The breaker doesn't change either discipline's formula; it changes how many times that formula has to get applied, stopping the count before it starts, at the exact instant it decides a call isn't even worth trying.

Common mistakes

1. Wrapping TOOLS["book_room"] with call_with_breaker so dispatch_robust uses it automatically. As explained above, this produces nested retries — M7's internal retry on top of M6's backoff retry — multiplying real calls instead of reducing them. call_with_breaker gets called at the point where the loop would request the tool, never inside the registry dispatch_robust consults.

2. Thinking CircuitOpenError needs special handling inside dispatch_robust. It doesn't — it naturally falls into the generic except Exception branch, which never retries and always produces a tool_result with is_error: True. M7's uniform error protocol already solves this with no change at all.

3. Using traced_run (which patches dispatch_robust) around a cycle that never calls run_reservo_agent. traced_run's patch stays installed, but inert, because nothing in this cycle goes through dispatch_robust — the right instrumentation for this pattern is direct log_event, with RunEvent/ToolCallEvent, as this lesson shows.

4. Confusing a tool_result's content when it's rejected by the breaker with a genuine tool failure. Both share is_error: True in the log — the difference (breaker versus real failure) lives only in the message text. A real system needing to tell them apart to alert differently would have to parse that text, or record an additional field — a reasonable improvement, outside this guide's scope.

5. Sharing a single CircuitBreaker between this lesson and Lesson 5 (the gate). Every demonstration in this guide creates its own instance (breaker, breaker2) — a CircuitBreaker that already accumulated failures from an earlier experiment would contaminate the next one's result, exactly the same kind of state leak reset_reservo_state() (M5) exists to prevent in the gate.

Exercises

Exercise 1: Count how many log events have is_error: true (Easy)

Without running the cycle again: of this lesson's seven tool_result log lines, how many have is_error: true? Name them by run_n.

Exercise 2: Calculate the "detection toll" with a different failure_threshold (Medium)

Repeat this lesson's complete cycle (the seven runs) with failure_threshold=5 instead of 3, keeping OUTAGE_CALLS=9. How many real calls get spent before the breaker opens, and how many runs get resolved with a genuine success (without going through an open breaker) before that opening?

_state3 = {"count": 0}
def flaky_book_room_3(room, tier, hours, member):
    _state3["count"] += 1
    if _state3["count"] <= 9:
        raise ConnectionError(f"timeout de red simulado (llamada real #{_state3['count']})")
    return _book_room_real(room, tier, hours, member)

breaker3 = CircuitBreaker("book_room", failure_threshold=5, cooldown_calls=2)
for run_n in range(1, 8):
    try:
        result = call_with_breaker(breaker3, flaky_book_room_3, room="Focus", tier="pro", hours=3,
                                    member=f"user{run_n}", max_retries=3, base_delay_ms=100)
        print(f"run {run_n}: OK -> {result}")
    except CircuitOpenError as exc:
        print(f"run {run_n}: RECHAZADO -- {exc}")
    except ConnectionError as exc:
        print(f"run {run_n}: FALLO -- {exc}")
print("llamadas reales:", _state3["count"], "estado final:", breaker3.state)

run 1: FALLO -- ... (llamada real #3)
run 2: FALLO -- ... (llamada real #6)
run 3: FALLO -- ... (llamada real #9, agota OUTAGE_CALLS -- la 10a llamada real ya recupera)
run 4: OK -> {'booking_id': ..., 'confirmed': True, 'price_cents': 6000}
...
llamadas reales: 12
estado final: CLOSED

Exercise 3: Design the log event that tells apart a breaker rejection from a genuine failure (Hard)

Without changing ToolCallEvent (M2) — which has no dedicated field for this — propose how a real system could tell apart, in a monitoring dashboard, how many of a period's tool_results with is_error: true were breaker rejections versus genuine tool failures, using only what's already in RUN_LOG.jsonl. Write the function doing that count.

def count_breaker_rejections_vs_real_failures(events):
    rejected = 0
    real_failures = 0
    for e in events:
        if e["event"] != "tool_result" or not e["is_error"]:
            continue
        if e["content"].startswith("CircuitOpenError"):
            rejected += 1
        else:
            real_failures += 1
    return {"rechazados_por_breaker": rejected, "fallos_reales": real_failures}


# Simulando los eventos de esta lección como una lista de dicts:
events_demo = [
    {"event": "tool_result", "is_error": True, "content": "ConnectionError: timeout de red simulado (llamada real #3)"},
    {"event": "tool_result", "is_error": True, "content": "ConnectionError: timeout de red simulado (llamada real #6)"},
    {"event": "tool_result", "is_error": True, "content": "ConnectionError: timeout de red simulado (llamada real #9)"},
    {"event": "tool_result", "is_error": True, "content": "CircuitOpenError: book_room: circuito abierto (OPEN), llamada rechazada sin tocar la tool"},
    {"event": "tool_result", "is_error": True, "content": "CircuitOpenError: book_room: circuito abierto (OPEN), llamada rechazada sin tocar la tool"},
    {"event": "tool_result", "is_error": False, "content": "{'booking_id': 1, 'confirmed': True, 'price_cents': 6000}"},
]
print(count_breaker_rejections_vs_real_failures(events_demo))

{'rechazados_por_breaker': 2, 'fallos_reales': 3}

Summary and next step

• We reused, without changing a single line, M6's complete CLOSED → OPEN → HALF_OPEN → CLOSED cycle: seven runs, three real failures, two instant rejections, one successful probe — 11 real calls against the 21 it would have cost with no breaker at all.
• We explained, with technical precision, why call_with_breaker never gets installed inside TOOLS: dispatch_robust (M7) already retries ConnectionError internally, and composing both retries would nest up to 3 × 3 real attempts per tool call — the real reason behind Height 2, which this module's Lesson 2 traced.
• We confirmed CircuitOpenError does travel safely through dispatch_robust (it falls into the generic branch, with no retry), and built the seven attempts' complete log book with M2's RunEvent/ToolCallEvent — the breaker's rejection and a genuine failure share is_error: true, distinguishable only by their content.
• We translated the breaker's savings into M3's/M4's vocabulary: every avoided call is time and, in a genuinely connected system, tokens that never got spent.

Next lesson: 07 — What Your Agent Still Needs. With the four disciplines already operating together, we close the ecosystem map: where to go when this operated agent has to face infrastructure incidents, semantic judgment, real cost reduction, or hardening against attacks.

Additional resources

1. resilience-and-reliability-patterns-guide (Module 5, "Circuit Breakers") — the canonical source for the CLOSED/OPEN/HALF_OPEN vocabulary and its measurement in depth with thread-pool exhaustion, retry storms, and bulkheads, for any generic HTTP dependency beyond an agent tool.
2. Anthropic — Tool use error handling — The uniform is_error protocol that makes it possible for CircuitOpenError to travel clean through dispatch_robust, with no special case at all.
3. Python — Exceptions and the Exception hierarchy — Why except (ConnectionError, FutureTimeoutError) doesn't catch CircuitOpenError, this lesson's boundary's technical foundation.
4. sre-and-incident-response-guide — for when this same "detect, isolate, recover" pattern needs to operate at the level of complete infrastructure (a downed service, not an agent tool), with incident roles and postmortems.