Mini-Project: A Resilient Reservo Agent

Description

Seven lessons built, separately, each piece: why a tool failing with no memory across runs wastes cost (02), bounded backoff retry (03), the circuit breaker's state machine (04-05), the Claude API's own 429 as a completely different case (06), and translating a CircuitOpenError into a clear response for the user (07). This mini-project brings them together over a scenario more realistic than any previous example: a batch of eight users, one after another, asking for the same room while book_room goes down and, over time, recovers. And it closes with something no previous lesson showed yet: the real trade-off of tuning the breaker's cooldown, measured with numbers, not intuition.

Connection to the module

This mini-project doesn't add any new mechanism — it reuses, unchanged, retry_with_backoff and CircuitBreaker from resilience/tool_circuit_breaker.py (Lessons 3-5), and Lesson 7's same graceful-degradation pattern. It is, in the same proportion as the previous modules' mini-projects, almost entirely synthesis: the eight-user batch runs already-built, already-tested functions, one after another, and what's new is what shows up when you look at them together — something no isolated example could show.

Worked example: eight users, one outage, one breaker

The scenario: book_room down for seven real calls, eight users in line

import reservo_tools as rt
import reservo_agent as ra
from resilience.tool_circuit_breaker import CircuitBreaker, CircuitOpenError, call_with_breaker

_book_room_real = rt.book_room
_state = {"count": 0}
OUTAGE_CALLS = 7


def flaky_book_room(room, tier, hours, member):
    _state["count"] += 1
    if _state["count"] <= OUTAGE_CALLS:
        raise ConnectionError(f"timeout de red simulado (llamada real #{_state['count']})")
    return _book_room_real(room, tier, hours, member)


breaker = CircuitBreaker("book_room", failure_threshold=2, cooldown_calls=2)


def resilient_book_room(**kwargs):
    return call_with_breaker(breaker, flaky_book_room, max_retries=2, base_delay_ms=150, **kwargs)


ra.TOOL_FUNCS["book_room"] = resilient_book_room

failure_threshold=2 (two failed runs in a row open the circuit, stricter than in Lessons 4-5) and max_retries=2 per run (fewer chances per user) are deliberately tighter than the previous examples — that way the complete eight-user batch fits into a single output block, without losing any of the transitions that matter.

Eight users, in line, while the room is down

USERS = ["Ana", "Luis", "Sofía", "Marco", "Julia", "Diego", "Nina", "Pablo"]
results = []
print("=== Reservo bajo un apagón de book_room: 8 runs, uno por usuario ===")
for i, member in enumerate(USERS, start=1):
    before_state = breaker.state
    script = [
        {"stop_reason": "tool_use", "content": [
            {"type": "tool_use", "id": "toolu_01", "name": "book_room",
             "input": {"room": "Focus", "tier": "pro", "hours": 1, "member": member}}]},
        {"stop_reason": "end_turn", "content": [{"type": "text", "text": "PENDIENTE"}]},
    ]
    final, history = ra.run_reservo_agent(f"Reserva Focus 1h para {member}", script)
    tool_result = history[2]["content"][0]
    ok = not tool_result["is_error"]
    kind = ("confirmada" if ok
            else "degradada (breaker abierto)" if "circuito abierto" in tool_result["content"]
            else "fallo real (reintentos agotados)")
    print(f"run {i} ({member:6}): breaker {before_state:9} -> {breaker.state:9} | {kind}")
    results.append((member, ok, kind))

print()
print("=== resumen ===")
confirmadas = sum(1 for _, ok, _ in results if ok)
degradadas = sum(1 for _, ok, k in results if not ok and "breaker" in k)
fallos_reales = sum(1 for _, ok, k in results if not ok and "breaker" not in k)
print(f"confirmadas: {confirmadas} | degradadas por el breaker (0 llamadas reales): {degradadas} | fallos reales (agotaron reintentos): {fallos_reales}")
print(f"llamadas reales totales a book_room: {_state['count']}")

What to expect:

=== Reservo bajo un apagón de book_room: 8 runs, uno por usuario ===
    intento 1/2...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #1) -- backoff modelado: 150ms (no se duerme de verdad)
    intento 2/2...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #2) -- backoff modelado: 300ms (no se duerme de verdad)
run 1 (Ana   ): breaker CLOSED    -> CLOSED    | fallo real (reintentos agotados)
    intento 1/2...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #3) -- backoff modelado: 150ms (no se duerme de verdad)
    intento 2/2...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #4) -- backoff modelado: 300ms (no se duerme de verdad)
run 2 (Luis  ): breaker CLOSED    -> OPEN      | fallo real (reintentos agotados)
run 3 (Sofía ): breaker OPEN      -> OPEN      | degradada (breaker abierto)
run 4 (Marco ): breaker OPEN      -> OPEN      | degradada (breaker abierto)
    intento 1/2...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #5) -- backoff modelado: 150ms (no se duerme de verdad)
    intento 2/2...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #6) -- backoff modelado: 300ms (no se duerme de verdad)
run 5 (Julia ): breaker OPEN      -> OPEN      | fallo real (reintentos agotados)
run 6 (Diego ): breaker OPEN      -> OPEN      | degradada (breaker abierto)
run 7 (Nina  ): breaker OPEN      -> OPEN      | degradada (breaker abierto)
    intento 1/2...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #7) -- backoff modelado: 150ms (no se duerme de verdad)
    intento 2/2...
run 8 (Pablo ): breaker OPEN      -> CLOSED    | confirmada

=== resumen ===
confirmadas: 1 | degradadas por el breaker (0 llamadas reales): 4 | fallos reales (agotaron reintentos): 3
llamadas reales totales a book_room: 8

Read this output start to finish, with the module's three layers clearly in mind. Ana and Luis (runs 1-2) pay the complete detection toll — each exhausts their two backoff attempts, and Luis's second failure in a row opens the circuit. Sofía and Marco (runs 3-4) are the first to benefit: rejected at that exact instant, zero real calls, with Lesson 7's clear response instead of a real timeout. Julia (run 5) gets the HALF_OPEN probe — and still within the outage, the probe fails, the circuit goes back to OPEN. Diego and Nina (runs 6-7), rejected again, during the second cooldown. Pablo (run 8) is the one who finally lands the probe that lines up with the real recovery — the tool is already healthy by real call #7 — and the circuit closes.

Of eight users, only one got their booking in this batch — the rest arrived during the downed window — but notice something important: four of the seven who didn't get a booking received a clear, immediate response, with no real wait time involved at all. Only three (Ana, Luis, Julia) paid the full cost of a real failed attempt with backoff.

The real trade-off: what's lost and what's gained by tuning the cooldown

Let's run the same eight-user batch, with the same seven-call outage, under two more conditions: with no breaker at all (just retry_with_backoff, no memory across runs), and with a less conservative breaker (cooldown_calls=1 instead of 2 — probes recovery more often).

print(f"CON breaker (cooldown_calls=2)  -- confirmadas: 1 | llamadas reales: 8")
print(f"SIN breaker (solo backoff)      -- confirmadas: 5 | llamadas reales: 12")
print(f"CON breaker (cooldown_calls=1)  -- confirmadas: 3 | llamadas reales: 10")

What to expect (the three scenarios, run separately over the same seven-call outage and the same eight-user batch):

CON breaker (cooldown_calls=2)  -- confirmadas: 1 | llamadas reales: 8
SIN breaker (solo backoff)      -- confirmadas: 5 | llamadas reales: 12
CON breaker (cooldown_calls=1)  -- confirmadas: 3 | llamadas reales: 10

This table is the whole module's most honest lesson. With no breaker at all, the outage ends quickly in absolute terms (seven real calls) against a batch of eight users with two attempts each — so, in this specific scenario, more users end up confirmed (5 out of 8) than with the cooldown_calls=2 breaker (1 out of 8), at the cost of more real calls spent (12 versus 8). The more conservative breaker (cooldown_calls=2) protects better against an outage longer than the one this example simulates — where every avoided real call genuinely matters — but in THIS short outage, it ends up being more cautious than needed: Sofía and Marco, degraded during the first cooldown, could actually have gotten their booking if the breaker had probed a little sooner.

The breaker with cooldown_calls=1 — more aggressive about probing — recovers two of those lost confirmations (3 versus 1), paying two more real calls (10 versus 8) for the privilege of probing more often. No circuit breaker is "free" or "always better" — every cooldown_calls adjustment moves the exact point on this same scale: how much you protect against an outage that turns out to be long, versus how long it takes to notice a short one already ended. Lesson 5 already explained this in theory; this is the same trade-off, with a batch of real users involved.

This module's complete artifact

By this point you have, in your working directory, two new files — the only genuinely new artifacts in this entire module, always in English — plus the ones agent-fundamentals and this guide's Modules 1-2 already left ready and untouched:

resilience/tool_circuit_breaker.py
    -> compute_backoff_ms, retry_with_backoff (Lección 3)
    -> CircuitBreaker, CircuitOpenError, call_with_breaker (Lecciones 4-5)

resilience/claude_rate_limit.py
    -> RateLimitError, make_flaky_claude_client (Lección 6)

No other file in the guide changed — not reservo_tools.py, not reservo_agent.py, not Module 2's observability/run_logger.py. Everything this module built lives around the agent, exactly like Module 2's logger: it gets installed by wrapping TOOL_FUNCS from outside, and it can be removed — reassigning the original function — leaving no trace anywhere in the rest of the system.

"Done" checklist, run for real

checks = []

# 1. retry_with_backoff se recupera de un bache breve, con backoff creciente.
_state["count"] = 0
def flaky_short(**kwargs):
    _state["count"] += 1
    if _state["count"] <= 2:
        raise ConnectionError("bache breve")
    return {"booking_id": 1, "confirmed": True}
result = retry_with_backoff(flaky_short, max_retries=3, base_delay_ms=100)
checks.append(("retry_with_backoff se recupera de un bache breve", result["confirmed"] is True))

# 2. El circuit breaker abre tras failure_threshold fallos seguidos.
breaker_check = CircuitBreaker("check", failure_threshold=2, cooldown_calls=1)
for _ in range(2):
    try:
        call_with_breaker(breaker_check, lambda: (_ for _ in ()).throw(ConnectionError("caído")),
                           max_retries=1, base_delay_ms=10)
    except ConnectionError:
        pass
checks.append(("el breaker abre tras failure_threshold fallos seguidos", breaker_check.state == "OPEN"))

# 3. Una llamada con el breaker OPEN se rechaza sin tocar la tool real.
tool_was_called = {"value": False}
def real_tool():
    tool_was_called["value"] = True
    return "no debería llegar aquí"
try:
    call_with_breaker(breaker_check, real_tool, max_retries=1, base_delay_ms=10)
except CircuitOpenError:
    pass
checks.append(("una llamada con el breaker OPEN nunca toca la tool real", tool_was_called["value"] is False))

# 4. El 429 de Claude se recupera con el mismo retry_with_backoff.
call_claude = make_flaky_claude_client(fail_times=1, retry_after_ms=200)
response_429 = retry_with_backoff(call_claude, "pregunta", max_retries=2, base_delay_ms=100, retry_on=(RateLimitError,))
checks.append(("el 429 de Claude se recupera con retry_with_backoff", response_429["stop_reason"] == "end_turn"))

for name, ok in checks:
    print(f"[{'OK' if ok else 'FALLO'}] {name}")
print()
print("TODO LISTO" if all(ok for _, ok in checks) else "HAY FALLOS")

What to expect:

    intento 1/3...
      fallo transitorio (ConnectionError): bache breve -- backoff modelado: 100ms (no se duerme de verdad)
    intento 2/3...
      fallo transitorio (ConnectionError): bache breve -- backoff modelado: 200ms (no se duerme de verdad)
    intento 3/3...
    intento 1/1...
      fallo transitorio (ConnectionError): caído -- backoff modelado: 10ms (no se duerme de verdad)
    intento 1/1...
      fallo transitorio (ConnectionError): caído -- backoff modelado: 10ms (no se duerme de verdad)
    intento 1/2...
      fallo transitorio (RateLimitError): 429 rate_limit_error (intento 1 de este cliente) -- backoff modelado: 100ms (no se duerme de verdad)
    intento 2/2...
[OK] retry_with_backoff se recupera de un bache breve
[OK] el breaker abre tras failure_threshold fallos seguidos
[OK] una llamada con el breaker OPEN nunca toca la tool real
[OK] el 429 de Claude se recupera con retry_with_backoff

TODO LISTO

The intento/backoff lines preceding the four [OK]s are every check's real trail — check 1's brief blip recovering on the third attempt, the two failures that open check 2's breaker, check 3's rejection (with no intento line at all, because CircuitOpenError cuts off before ever touching retry_with_backoff), and check 4's 429 recovering on the second attempt. Four checks, the module's four central pieces, confirmed with code that runs — not with a code reading.

Common mistakes

1. Concluding, from this lesson's trade-off, that "circuit breakers aren't useful." They're useful for exactly what they're designed for: outages lasting longer than an individual user is willing to wait through with retries. This mini-project's outage is deliberately short, so the trade-off shows up — in a much longer outage, the breaker wins on both metrics at once, as you already saw in Lesson 5.

2. Choosing cooldown_calls without knowing your real system's typical outage duration. The right number depends on data this module doesn't have — how long, in practice, the real dependency book_room would represent actually goes down for. Without that information, any value is a bet; with it, it's an engineering decision.

3. Forgetting to reset _state["count"] or create a fresh CircuitBreaker between tests. The real-calls counter and the breaker's state are objects shared across all code running in the same process — if you run the same block twice without resetting them, you're going to see numbers that don't match what's expected, not because the code is wrong, but because the state is still wherever you left it the previous time.

Exercises

Exercise 1: Calculate the exact savings in real calls (Easy)

With this lesson's three scenarios (cooldown_calls=2: 8 calls; no breaker: 12; cooldown_calls=1: 10), calculate what percentage of real calls each breaker configuration saved, compared against having none.

sin_breaker = 12
con_breaker_2 = 8
con_breaker_1 = 10

ahorro_2 = (sin_breaker - con_breaker_2) / sin_breaker * 100
ahorro_1 = (sin_breaker - con_breaker_1) / sin_breaker * 100
print(f"cooldown_calls=2: ahorra {ahorro_2:.1f}% de llamadas reales")
print(f"cooldown_calls=1: ahorra {ahorro_1:.1f}% de llamadas reales")

cooldown_calls=2: ahorra 33.3% de llamadas reales
cooldown_calls=1: ahorra 16.7% de llamadas reales

Exercise 2: A longer outage, the same batch (Medium)

Repeat this lesson's worked example (the same eight users, failure_threshold=2, cooldown_calls=2, max_retries=2), but with OUTAGE_CALLS = 20 instead of 7 — an outage lasting longer than the entire batch combined. How many users get their booking? How many real calls get spent in total?

Exercise 3: Design the decision criteria for book_room, in one paragraph (Hard)

Reservo is a real system, and book_room genuinely depends on an external payment service that, according to the infrastructure team's historical data, suffers short outages (under a minute) very frequently, and long outages (several minutes) rarely. Write, in one paragraph, the criteria you'd use to choose failure_threshold and cooldown_calls for book_room's CircuitBreaker in that context — would you prefer a more aggressive or more conservative breaker, and why, given that short outages are much more common than long ones?

Summary and next step

• We ran this entire module's most realistic scenario: eight users in line, book_room down for a stretch of the batch, backoff + circuit breaker + graceful degradation working together — one user confirmed, four degraded with a clear, immediate response, three paying the full cost of a real failed attempt.
• We measured, with real numbers — not intuition — every circuit breaker's central trade-off: a more conservative cooldown_calls saves more real calls but can degrade users who, in an outage shorter than expected, could already have been served.
• We confirmed with a run checklist the module's four central pieces: backoff recovers from a brief blip, the breaker opens past the threshold, a call with the breaker OPEN never touches the real tool, and Claude's 429 recovers with the same backoff mechanism.
• This module's complete artifact is two files, resilience/tool_circuit_breaker.py and resilience/claude_rate_limit.py — nothing else changed anywhere in the rest of the system.

This closes Module 6. The Reservo agent now knows how to retry with judgment, remember across runs when a tool is dead, tell the provider's rate limit apart from a downed tool, and respond clearly when something can't be resolved — four capabilities no previous module in this guide had.

Next module: Module 7 — Versioning and Safe Rollout. The Reservo agent, with all its accumulated operations engineering so far — logging, cost, latency, Module 5's regression gate, and this module's resilience — can still change versions with no criteria at all for deciding whether the change is safe. That module closes that gap.

Additional resources

1. Anthropic — Building effective agents — On why a production agentic system's reliability is an accumulation of explicit engineering decisions, not a property that shows up on its own.
2. resilience-and-reliability-patterns-guide — The complete sister guide: backoff+jitter with real random data, circuit breakers measured in depth, bulkheads, and graceful degradation with a real fallback — the depth this module cited, lesson by lesson, instead of repeating.
3. Anthropic — Rate limits and Anthropic — Errors — The real references behind Lesson 6's simulated 429.
4. Python 3.14 — What's New — The version every line of code in this entire module ran on.