Module 6: Failures at Scale — Backoff, Circuit Breakers, and Rate Limits
Graceful Degradation
Description
Lessons 4 and 5 built a CircuitBreaker that knows, with certainty, when book_room has been failing sustainedly — and the moment it knows, it rejects any new call in microseconds, without touching the dead tool. That certainty is valuable, but only if the agent does something useful with it. A breaker that just throws CircuitOpenError outward, with nobody catching it with any judgment, leaves the agent exactly where it was before: a raw, untranslated technical error, at the worst possible moment. This lesson closes that loop — not with a new mechanism, but by reusing the is_error protocol agent-fundamentals already built, so the agent tells the user, clearly and at that exact instant, why their booking couldn't be confirmed.
Connection to the module
This lesson doesn't add any new component to resilience/tool_circuit_breaker.py — everything it needs already exists since Lesson 4: CircuitOpenError, call_with_breaker, and agent-fundamentals M7's dispatch_robust mechanism, which catches any real exception from a tool and turns it into a tool_result with is_error: true. The only thing that changes in this lesson is where that exception comes from — before it was a real ConnectionError from the tool; now it's also, potentially, a CircuitOpenError from the breaker — and what the model (concept) does with the information that is_error gives it.
🛑 Boundary — what this lesson does and doesn't do
resilience-and-reliability-patterns-guide (Module 7, "Graceful Degradation and Load Shedding") develops this topic in depth, with a complete system: fallback (an alternative response when the main dependency isn't there), degrade (lowering service quality instead of denying it entirely — for example, a deferred shipment instead of an immediate one), and load shedding (deliberately rejecting traffic when the entire system is overloaded, not just a single dependency), all measured with a real success_rate against the Mercado case. That's the complete reference, and this lesson cites it instead of repeating it.
What this lesson builds is much narrower: what the agent tells the user, in a single response, when a specific tool's circuit breaker is OPEN. There's no fallback to an alternative service, no deferred-retry queue, no load-shedding mechanism over the system's general traffic — that stays entirely in the sister guide. What there is, is a translation: from CircuitOpenError (a technical exception) to a clear response (a sentence the user understands), using the same is_error protocol you already know.
Analogy: getting on with life once the switch has tripped
Go back to the thermal breaker. When it trips and cuts power to a circuit, you have two ways to react. The bad one: keep plugging the same broken appliance back in, over and over, every time you forget it already tripped — gaining nothing, learning nothing. The good one: realizing, right then, that circuit is cut, and acting accordingly — using a flashlight if needed, telling whoever needs to fix it, and getting on with your night without pretending the light is still on. Degrading gracefully is exactly that second reaction, applied to the agent: the moment it knows book_room is OPEN, it doesn't try again, and it explains to the user, clearly, what happened and what to expect — instead of hanging, blindly retrying, or responding with a technical message that helps nobody.
Worked example: the same is_error, now with the breaker inside
The breaker is already open, from previous runs' failures
We simulate the moment when several users already tripped the breaker — like in Lessons 4 and 5 — and now a new user, Ana, arrives with the circuit already OPEN.
import reservo_tools as rt
import reservo_agent as ra
def always_down_book_room(room, tier, hours, member):
raise ConnectionError("timeout de red simulado -- book_room sigue caído")
def make_resilient_book_room(breaker, real_fn, max_retries=3, base_delay_ms=100):
def resilient_book_room(**kwargs):
return call_with_breaker(breaker, real_fn, max_retries=max_retries, base_delay_ms=base_delay_ms, **kwargs)
return resilient_book_room
# El breaker YA está OPEN, por fallos acumulados en runs anteriores
breaker = CircuitBreaker("book_room", failure_threshold=3, cooldown_calls=5)
breaker.state = "OPEN"
ra.TOOL_FUNCS["book_room"] = make_resilient_book_room(breaker, always_down_book_room)
The technique is, again, the same as always: we replace TOOL_FUNCS["book_room"] from outside, without touching dispatch_robust or run_reservo_agent. The only new thing is that the function we register now wraps the real tool with call_with_breaker — so any exception coming out of it, whether a real ConnectionError or a CircuitOpenError from the breaker, is going to reach dispatch_robust through exactly the same path, and dispatch_robust is going to catch it exactly the same way it catches any other real exception: without changing a single line of its own code.
The script: the model sees the is_error, and responds clearly
script = [
{"stop_reason": "tool_use", "content": [
{"type": "tool_use", "id": "toolu_01", "name": "book_room",
"input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
{"stop_reason": "end_turn", "content": [
{"type": "text", "text": "Las reservas de Focus están temporalmente pausadas por mantenimiento del sistema. Guardé tu pedido y te aviso apenas se restablezca -- no hace falta que lo repitas."}]},
]
final, history = ra.run_reservo_agent("Reserva Focus pro 3h para Ana", script)
ra.print_trace(history)
print()
print("RESPUESTA:", final["content"][0]["text"])
print()
tool_result = history[2]["content"][0]
print(f"tool_result real: is_error={tool_result['is_error']} content={tool_result['content']!r}")
What to expect:
[0] user pregunta: 'Reserva Focus pro 3h para Ana'
[1] assistant tool_use(book_room): {'room': 'Focus', 'tier': 'pro', 'hours': 3, 'member': 'Ana'}
[2] user tool_result [is_error]: CircuitOpenError: book_room: circuito abierto (OPEN), llamada rechazada sin tocar la tool
[3] assistant texto final: 'Las reservas de Focus están temporalmente pausadas por mantenimiento del sistema. Guardé tu pedido y te aviso apenas se restablezca -- no hace falta que lo repitas.'
RESPUESTA: Las reservas de Focus están temporalmente pausadas por mantenimiento del sistema. Guardé tu pedido y te aviso apenas se restablezca -- no hace falta que lo repitas.
tool_result real: is_error=True content='CircuitOpenError: book_room: circuito abierto (OPEN), llamada rechazada sin tocar la tool'
Look at step [2]: the tool_result with is_error: true carries, inside it, the exact message CircuitOpenError produced — "circuit open (OPEN), call rejected without touching the tool." That's real, specific information, not a generic timeout — the agent (concept, claude-sonnet-5) knows, with that information, it isn't worth retrying within this same run, and that the cause isn't invalid user data but an outage in the system itself. Step [3] is this example's scripted response — in a real system, the model would produce text like this from seeing exactly that is_error, the same way agent-fundamentals M7 already showed the model self-correcting or responding with judgment when facing an error tool_result. No new mechanism at all — the same protocol, with a new error source.
Comparison: without the breaker, the same outage costs three real attempts per user
def unprotected_book_room(**kwargs):
return retry_with_backoff(always_down_book_room, max_retries=3, base_delay_ms=100, **kwargs)
ra.TOOL_FUNCS["book_room"] = unprotected_book_room
script_sin_breaker = [
{"stop_reason": "tool_use", "content": [
{"type": "tool_use", "id": "toolu_01", "name": "book_room",
"input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
{"stop_reason": "end_turn", "content": [
{"type": "text", "text": "No pude confirmar la reserva de Focus -- el sistema no respondió tras varios intentos."}]},
]
print("=== SIN circuit breaker: cada run gasta max_retries llamadas reales antes de rendirse ===")
final, history = ra.run_reservo_agent("Reserva Focus pro 3h para Ana", script_sin_breaker)
print("RESPUESTA:", final["content"][0]["text"])
What to expect:
=== SIN circuit breaker: cada run gasta max_retries llamadas reales antes de rendirse ===
intento 1/3...
fallo transitorio (ConnectionError): timeout de red simulado -- book_room sigue caído -- backoff modelado: 100ms (no se duerme de verdad)
intento 2/3...
fallo transitorio (ConnectionError): timeout de red simulado -- book_room sigue caído -- backoff modelado: 200ms (no se duerme de verdad)
intento 3/3...
fallo transitorio (ConnectionError): timeout de red simulado -- book_room sigue caído -- backoff modelado: 400ms (no se duerme de verdad)
RESPUESTA: No pude confirmar la reserva de Focus -- el sistema no respondió tras varios intentos.
Both final responses say, deep down, the same thing — "couldn't book right now" — but they got there in completely different ways. Without the breaker: three real calls, each waiting its own backoff (100ms, 200ms, 400ms of modeled wait — in a real system, real time, with the user waiting for a response), before giving up. With the breaker: zero real calls, an immediate response, and — the detail that matters most — a response that also tells the user there's no need to retry ("I saved your request, I'll let you know"), instead of leaving them wondering whether it's worth asking again. That's, with numbers, Lessons 4 and 5's complete payoff, put to work for the end user in this lesson.
Common mistakes
-
Letting
CircuitOpenErrorpropagate without the model ever seeing it. If the code callingrun_reservo_agentcaughtCircuitOpenErroroutside the agent's loop — instead of lettingdispatch_robustturn it into atool_resultwithis_error— the model would never find out why it failed, and couldn't produce an informed response. The breaker's message is valuable information; losing it in generic exception handling wastes it. -
Confusing "degrading gracefully" with "hiding the failure." A response that just said "booking confirmed" when nothing was actually confirmed isn't graceful degradation — it's a lie. Degrading gracefully means being honest about what happened, fast, and with a clear way out — never faking a success that didn't happen.
-
Building a different degradation message for every tool, by hand, in every corner of the code. The real message the user sees (step
[3]in the example) is the model's responsibility, not this module's — this module only guarantees the right information (CircuitOpenErrorwith a clear message) reaches it. Trying to hardcode, in Python, every possible variant of "what to tell the user" for every tool and every breaker state is rebuilding, by hand, something theis_errorprotocol plus the model already solve better.
Exercises
Exercise 1: Confirm the savings in history steps (Easy)
Without running Python: compare len(history) between the run with the breaker OPEN (this lesson's first example) and the run with no breaker (the second example). Both have the same script structure (one tool call, one final text) — why is the number of steps in history the same in both, even though the internal path was so different?
See solution
len(history) is 4 in both cases: [0] the question, [1] the tool_use, [2] the tool_result, [3] the final text. The number of steps in history reflects the protocol's structure — how many turns there were between the model and the tools — not how much internal work dispatch_robust did to produce step [2]'s tool_result. Whether that single tool_result involved zero real calls (breaker OPEN) or three real calls with backoff (no breaker), the history doesn't distinguish it — to see that, you need to look at the run's real output (the intento lines), not history's shape. It's the same distinction Lesson 5 underlined between the breaker's "before/after" state and what genuinely happened inside a run.
Exercise 2: A degradation message for cancel_booking (Medium)
cancel_booking doesn't have a circuit breaker in this guide — the module focused on book_room — but the same pattern would apply equally if it did. Write the complete script (model_script) for a run where cancel_booking's breaker is OPEN, and the model responds with a suitable degradation message — think about what information differs between canceling and booking: what matters more for the user to be told when what failed is a cancellation, not a new booking?
See solution
breaker_cancel = CircuitBreaker("cancel_booking", failure_threshold=3, cooldown_calls=5)
breaker_cancel.state = "OPEN"
script_cancel = [
{"stop_reason": "tool_use", "content": [
{"type": "tool_use", "id": "toolu_01", "name": "cancel_booking", "input": {"id": 1}}]},
{"stop_reason": "end_turn", "content": [
{"type": "text", "text": "No pude procesar la cancelación de tu reserva #1 -- el sistema está temporalmente fuera de servicio. Tu reserva SIGUE ACTIVA por ahora; guardé tu pedido de cancelación y lo proceso apenas se restablezca el servicio."}]},
]
Explanation: the key difference from book_room's message is that, in a new booking, the default state if something fails is "nothing happened" — there's no ambiguity at all; in a cancellation, the default state if something fails is that the original booking is still active, and that's exactly what the user needs to know as clearly as possible, so they don't mistakenly assume it's already canceled. A good degradation message isn't a generic template ("the system isn't responding, try later") — it has to reflect what's genuinely important to the user in THAT specific type of operation.
Exercise 3: Why does this lesson never build a deferred-retry queue? (Hard)
The worked example's message says "I saved your request, I'll let you know as soon as it's restored" — but this lesson's code does not implement any real queue that saves that request and retries it later. Explain, in one paragraph, why this lesson stops there — at the message — without building the mechanism that would make it true, and which guide would handle that piece if it needed to be implemented for real.
See solution
A deferred-retry queue — saving the request in some persistent storage, with a separate process that retries it later and notifies the user once it's finally confirmed — is real infrastructure: it needs a database or a message queue, a worker running outside a single agent run's lifecycle, and a notification mechanism back to the user. None of that is "resilience at an agent's tool-call layer" — this entire module's declared scope since Lesson 1 — it's application infrastructure, of the same order as a queueing system or a scheduler. The message's sentence is honest about the intent — telling the user there's no need to keep insisting — but the module, on purpose, doesn't build the mechanism behind that promise, for exactly the same reason it never builds bulkheads or load shedding: that infrastructure layer, if it ever needed to be implemented for real, is of the size and nature of what resilience-and-reliability-patterns-guide works on — graceful degradation with a real, measured fallback — not this guide, which operates an agent's layer with pure Python and zero cost.
Summary and next step
- Degrading gracefully, in this guide, means one concrete, narrow thing: translating a
CircuitOpenErrorinto a clear, honest response for the user, at that exact instant, without blindly retrying and without faking a success that didn't happen. - There's no new mechanism at all — we reuse
agent-fundamentals's (M4/M7) completeis_errorprotocol: the breaker wraps the real tool, and any exception it produces —ConnectionErrororCircuitOpenError— reachesdispatch_robustthrough exactly the same path as always. - We compared, with real numbers, the cost of responding with no breaker (three real calls with backoff, real wait time for the user) against responding with the breaker already
OPEN(zero real calls, immediate response) — the same final sentence, with a radically different cost behind it. - Boundary: fallback to an alternative service, deferred-retry queues, bulkheads, and full load shedding all stay entirely in
resilience-and-reliability-patterns-guide(Module 7) — this lesson never builds them.
Next lesson: 08 — Mini-Project: A Resilient Reservo Agent. We bring backoff, circuit breaker, and graceful degradation together into a single batch of real users, with book_room going down and recovering — and we measure, with numbers, the real trade-off of tuning the breaker's cooldown.
Additional resources
resilience-and-reliability-patterns-guide(Module 7, "Graceful Degradation and Load Shedding") — The complete development of fallback, quality degradation, and load shedding, measured with a realsuccess_rateagainst the Mercado case — the depth this lesson cites instead of repeating.- Anthropic — Tool use (function calling) overview — The
tool_use/tool_resultprotocol withis_error, which this lesson reuses with no modification at all. - Anthropic — Building effective agents — On why a production agent's responses have to be honest about their own limitations, not just technically correct.
- Python 3.14 — What's New — The version every line of code in this lesson ran on.