Module 6: Failures at Scale — Backoff, Circuit Breakers, and Rate Limits

Module 6: Failures at Scale — Backoff, Circuit Breakers, and Rate Limits

Next →

Description

Module 5 gave you a gate: a regression harness that runs a fixed set of cases against the Reservo agent and fails the build when something broke. That gate answers a specific question — "does this agent, exactly as it stands today, still behave as expected?" — but it takes something for granted that this module puts into question: that the agent's tools respond. In production, they don't always respond. A tool that works perfectly today can, tomorrow, start failing — an external dependency down, a saturated network, a service that returns error after error for fifteen minutes straight. And on top of that, there's a type of failure no agent tool can cause or fix: the model provider itself, the Claude API, can respond with a 429 when your organization exceeds its requests-per-minute quota.

This module builds the discipline the Reservo agent lacks to survive both types of failure without wasting money, without hanging, and without lying to the user. Three pieces, in growing order of scope: retries with bounded backoff (when a tool fails, retry it with a wait that grows exponentially, up to a hard cap); a per-tool circuit breaker (when a tool has been failing consistently — not a hiccup, a real outage — stop calling it for a while, and try again later); and the Claude API's own 429 (a case no generic resilience guide covers, because it isn't just any HTTP dependency — it's the LLM provider that sustains the entire system). You're going to build all three, run them against simulated failures deterministically, and close with a Reservo agent that keeps working — with judgment, not blindly — when book_room goes down in the middle of a real traffic day.

Connection to the module

This module rests directly on two pieces you already built, without touching either: agent-fundamentals Module 8's run_reservo_agent runner, and that same guide's basic tool-error handling, Module 7. That Module 7 already solved one question — "what does the agent do when THIS call to THIS tool fails, right now, within this run?" — with a simple retry, no backoff, captured in call_with_retries. This module asks a different, later question: "what does the SYSTEM do when that tool has been failing, run after run, for a good while?" The answer doesn't live inside a single call — it lives in state that persists across runs, something that lesson's call_with_retries, bounded to a single invocation, can't have by design.

Where we are in the ecosystem

Agents in production — operating the Reservo agent
├── Module 1: Why Operating Is Different From Building
├── Module 2: Structured Logging and Tracing a Run
├── Module 3: Measuring Cost and Tokens per Run
├── Module 4: Measuring Latency Honestly
├── Module 5: Regression Evals as a Production Gate
├── Module 6: Failures at Scale — Backoff, Circuit Breakers, and Rate Limits  ← YOU ARE HERE
│   → Retries with bounded backoff (modeled, no real clock);
│     a per-tool circuit breaker with state ACROSS runs;
│     the Claude API's own 429; graceful degradation
├── Module 7: Versioning and Safe Rollout
└── Module 8: Project — the Reservo Agent in Production

The five previous modules built, in order, four capabilities: observing a complete run with structured logging and a trace_id (Module 2); measuring how much it cost and how long it took (Modules 3 and 4); and gating — deciding, with a deterministic criterion, whether the agent still behaves as expected (Module 5). All four capabilities share a silent premise: that when the agent calls a tool, the tool responds. This module is the first to deliberately break that premise, and it builds this guide's third complete discipline — harden — before Module 7 closes the cycle with version.

The problem, in one sentence

An agent that blindly retries every time a tool fails isn't resilient — it's noisy. It keeps sending the same request into a service we already know is down, spends tokens and time on every failed attempt, and — worst of all — learns nothing from one run to the next. Every user who arrives while the tool is down pays, out of their own pocket, the full cost of discovering that outage from scratch. This module solves that with three layers, each built on top of the previous one:

1. BOUNDED BACKOFF    -> within one attempt, wait longer and longer
   (Lesson 3)            before retrying, with a hard cap on attempts.

2. CIRCUIT BREAKER    -> across runs, remember a tool is dead and
   (Lessons 4-5)         stop calling it -- until a periodic check
                         confirms it came back.

3. GRACEFUL           -> while the breaker is open, the agent does NOT
   DEGRADATION            crash or lie -- it responds clearly, fast,
   (Lesson 7)             without touching the dead tool.

And a fourth piece that runs in parallel to the three, because it's a completely different type of failure: the Claude API's own 429 (Lesson 6) — the model's provider, not an agent tool, asking you to wait.

The analogy that holds the whole module together: your house's circuit breaker

Picture your house's electrical panel. Every circuit has a thermal breaker — the one that "trips" when something goes wrong. If you plug in a hair dryer with a short circuit and the breaker didn't exist, every time you plug it in current keeps flowing into a broken appliance, and at some point something really burns — the wire, the outlet, the whole installation. The breaker exists exactly to prevent that: as soon as it detects something's wrong on THAT circuit — not the whole house, THAT specific circuit — it trips (opens) and cuts power right there. The rest of the house keeps working normally. After a while, someone checks it, and if the problem got fixed, they flip the switch back — the circuit conducts again.

Retrying with no breaker is plugging the broken hair dryer back in, over and over, hoping this time no spark flies. This module's circuit breaker is, literally, that switch applied to an agent tool: if book_room short-circuits — fails over and over — the breaker trips (opens, OPEN) and cuts off calls to THAT tool, without affecting get_quote or list_rooms. After a sensible while, it lets one test call through (HALF_OPEN) — like someone going to check the panel. If that test goes well, the circuit goes back to normal (CLOSED); if it keeps failing, it trips again and waits another while. Hold onto this analogy — you're going to use it, with the exact CLOSED/OPEN/HALF_OPEN vocabulary, in lessons 4 and 5.

🛑 A double boundary — read it before writing a single line of code

This module has the densest overlap in the entire guide, and it needs to be traced precisely before starting, because it repeats in every lesson.

Boundary 1 — toward agent-fundamentals Module 7 (within-the-run vs. across-runs)

agent-fundamentals Module 7 already built call_with_retries(fn, *args, max_retries=3, **kwargs): a simple retry, no wait between attempts, bounded by a hard cap, that tells a transient failure (ConnectionError, worth retrying) apart from a validation error (never changes, not worth it). That mechanism still exists, is still correct, and this module doesn't rebuild it. The exact boundary:

  • agent-fundamentals M7 solves: "this call to this tool, right now, within this run — do I retry?" Its scope is a single run; its memory lasts as long as that call does.
  • This module (M6) solves: "this tool has been failing run after run — do I keep calling it?" Its scope spans runs; its memory — the circuit breaker's state — lives in an object that persists while the process is alive, beyond any individual run.

This module's Lesson 3 scales up M7's retry — adding the piece that same lesson deliberately left pending ("this lesson doesn't implement it in detail"): exponential backoff between attempts. Lesson 4 onward builds something M7, by design, couldn't build: memory across runs.

Boundary 2 — toward resilience-and-reliability-patterns-guide (reused vocabulary, narrow scope)

The complete circuit-breaker vocabulary — the CLOSED/OPEN/HALF_OPEN states, backoff with real random jitter, bulkheads, graceful degradation and load shedding measured in depth with thread exhaustion and retry storms — is resilience-and-reliability-patterns-guide's complete content (software-architecture ecosystem, Mercado case — generic HTTP microservices, unrelated to LLMs). That guide is the canonical source for that vocabulary, and this module reuses it, citing it explicitly, without repeating its development. The scope difference is narrow and precise:

  • That guide builds a circuit breaker for any HTTP dependency in a generic distributed system, with real random backoff+jitter (with a fixed seed for its own experiments' reproducibility) and measures it with thread-pool exhaustion, retry storms, bulkheads.
  • This module builds a circuit breaker per agent tool, applied only to the tool-call layer (Lessons 4-5) and to the LLM provider's 429 (Lesson 6, a case that guide never covers because it isn't generic HTTP — it's the Claude API's rate limit). This module's backoff is modeled and deterministic — never random, never a real time.sleep() — because the rest of this entire guide demands byte-for-byte reproducibility in every "What to expect" block.

This module never builds bulkheads, full graceful degradation, or load shedding — those topics stay entirely in the sister guide, named explicitly where relevant (Lesson 7). When you need the complete, measured version of these patterns — or resilience for a system that doesn't involve an LLM — that's the guide to go to.

🛑 This lesson's hard rule (and the whole module's)

All of this module's resilience engineering — the backoff retry, the circuit breaker's state machine, the 429 simulation — runs for real, with Python 3.14.0, and every "What to expect" block shows the real output of having run that code. Three hard constraints, inherited from the rest of this guide and applied here with special care:

  1. Backoff is modeled, never real. You're going to see a function that calculates delay = base * 2**intento and prints it — you're never going to see time.sleep(delay). If this guide actually slept on every retry, every example would take real seconds to run and the result would stop being byte-for-byte reproducible from one machine to another. The lesson that matters — how much the wait grows, and why that protects a downed tool — comes across just as clearly seeing the calculated number as actually waiting for it.
  2. No random. If jitter ever gets mentioned — the technique of adding a random variation to backoff so many clients don't all retry at the exact same instant — it gets named and explained, but never implemented with random in this guide. The reason is the usual one: a random number in a "What to expect" block breaks reproducibility. The real version, with genuine random jitter, is in resilience-and-reliability-patterns-guide — it's cited, not repeated.
  3. The 429 is simulated, never a real API call. There's no network call anywhere in this module. Claude's 429 rate_limit_error gets simulated with a test client that returns the error a set number of times and then responds normally — deterministic, cited, with no dependency on whether your real quota is or isn't exceeded right now.

The case that keeps accompanying the guide: book_room goes down, in the middle of a real day

The system is the same as always — Reservo, coworking room bookings — with agent-fundamentals's same four canonical tools. This module doesn't declare any new business tool; what it does is put a realistic failure on a tool you already know — book_room, the write one, the one the user really needs to work — and build, around it, the resilience layer it's missing.

list_rooms()                          -- solo lectura, no se toca en este módulo
get_quote(room, tier, hours)          -- solo lectura, no se toca en este módulo
book_room(room, tier, hours, member)  -- ESCRITURA: la tool que este módulo endurece
cancel_booking(id)                    -- destructiva, no se toca en este módulo

Across the eight lessons, book_room is going to fail three different, always deterministic ways: a brief blip (two or three calls in a row, then it recovers on its own — Lesson 3); a sustained outage, across several different users' runs — Ana, Luis, Sofía, and several more — arriving one after another while the service stays down (Lessons 4, 5, 7, and 8); and, in parallel, a 429 from Claude itself that has nothing to do with book_room — it's the model layer, not the tools layer (Lesson 6).

What's genuinely new in this module is one more operations artifact, always in English:

  • resilience/tool_circuit_breaker.py — retry_with_backoff, compute_backoff_ms, the CircuitBreaker class (CLOSED/OPEN/HALF_OPEN), CircuitOpenError, and call_with_breaker, which composes both pieces.
  • resilience/claude_rate_limit.py — RateLimitError and a deterministic test client for simulating Claude's 429.

Identifiers and code, always in English (CircuitBreaker, retry_with_backoff, RateLimitError); prose and comments, in Spanish; money, in int cents, as in the rest of the guide. Current models (claude-sonnet-5) wherever the module needs to mention the model — the LLM call remains concept throughout this guide.

Prerequisites

Required knowledge

  • ✅ Having completed (or knowing well) Modules 1 and 2 of this guide: the run_reservo_agent runner, the structured logger, and the "wrap without touching" technique (monkeypatching a module attribute) Module 2 developed in depth. This module reuses it once more, now to inject resilience instead of observability.
  • ✅ Having completed agent-fundamentals-and-tool-calling-guide Module 7, especially call_with_retries (Lesson 6) — this module scales it up, it doesn't repeat it.
  • ✅ Python: exceptions (try/except, custom exception hierarchies), closures, functions that take another function as an argument (fn(*args, **kwargs)).

Recommended

  • ✅ Having felt, at some point, the frustration of a system that keeps insisting on a service that — anyone with two minutes of logs could see — has been down for a good while. That instinct is exactly what this module turns into code.

NOT required

  • ❌ You don't need to know resilience-and-reliability-patterns-guide beforehand — this module cites what's needed, precisely, at the right moment.
  • ❌ You don't need an API key or an internet connection: the 429 is always simulated, and all of this module's engineering runs 100% locally.
  • ❌ You don't need to know about bulkheads, load shedding, or thread-pool exhaustion — that's, entirely, the sister guide's territory.

Environment

  • ✅ Python 3.14.0 with its standard library (enum isn't mandatory — this module uses plain strings for states, just like the sister guide does). Nothing to install.

Module roadmap

Lesson 01 — Module introduction (this one)

The thermal-breaker analogy, the double boundary (agent-fundamentals M7 and resilience-and-reliability-patterns-guide), the determinism hard rule, and the case: book_room goes down in the middle of a real day.

Lesson 02 — A Tool That Fails Repeatedly

The problem laid out with real code: three different users, three independent runs, the same downed tool — and no memory between them. The cost of not remembering.

Lesson 03 — Retries with Bounded Backoff

retry_with_backoff: the direct scale-up of agent-fundamentals M7's call_with_retries, with the piece that lesson left pending — exponential, modeled backoff, with a hard cap. And its limits: what happens when the outage lasts longer than the cap.

Lesson 04 — The Circuit Breaker Pattern

The three-state machine, built from scratch: CircuitBreaker, CircuitOpenError. The first complete run: CLOSED → three failures → OPEN → two fast rejections → HALF_OPEN → the probe succeeds → CLOSED.

Lesson 05 — Closed, Open, Half-Open

Zooming into the transitions: what happens when the HALF_OPEN probe also fails (back to OPEN, new cooldown), the real cost in calls saved, and why this breaker's "probing" operates at the complete-run level, not per individual attempt.

Lesson 06 — The Claude 429 Rate Limit

Claude's own API rate limit: RateLimitError, a simulated, deterministic client, and Lesson 3's same retry_with_backoff applied to a completely different kind of failure. Why this layer never carries its own circuit breaker.

Lesson 07 — Graceful Degradation

What the agent tells the user when book_room's breaker is OPEN — without building bulkheads or load shedding, reusing the is_error protocol agent-fundamentals already built. Fast, honest, without touching the downed tool.

Lesson 08 — Mini-Project: A Resilient Reservo Agent

A batch of eight users, a book_room that goes down and recovers, backoff + circuit breaker + graceful degradation working together — and the real trade-off of tuning the cooldown, measured with real numbers.

Progression map

Lesson 01 (this)  → The analogy, the double boundary, the case
Lesson 02         → The problem: failures with no memory across runs
Lesson 03         → retry_with_backoff (scales up M7)
Lesson 04         → CircuitBreaker: the three-state machine
Lesson 05         → The transitions, in depth
Lesson 06         → Claude's 429
Lesson 07         → Graceful degradation
Lesson 08         → Mini-project: everything together

Difficulty: ⭐⭐ ──────────────────▶ ⭐⭐⭐

What you'll achieve in this module

By completing the 8 lessons, you'll be able to:

  1. Precisely tell apart the within-a-run retry (agent-fundamentals M7) from the across-runs circuit breaker (this module) — and explain, with an example, why one doesn't replace the other.
  2. Build and run retry_with_backoff: a bounded retry with modeled exponential backoff, applied both to a failing tool and to the Claude API's own 429.
  3. Build and run a complete CircuitBreaker state machine (CLOSED/OPEN/HALF_OPEN) per tool, with state that persists across runs, citing resilience-and-reliability-patterns-guide's vocabulary precisely.
  4. Explain and demonstrate why Claude's 429 is a different case from a downed tool, and why it never carries its own circuit breaker.
  5. Integrate backoff, circuit breaker, and graceful degradation into a single Reservo agent, and measure — with real numbers, not from memory — the trade-off of tuning the breaker's cooldown.

Before and after

BEFORE the module:
→ "If a tool fails, retrying it again is enough"
→ "A circuit breaker is the same as a retry, with a different name"
→ "Claude's 429 gets handled the same way as any downed tool"
→ "When something fails, tell the user exactly what came out,
  stacktrace included if needed"

AFTER the module:
→ Retrying with no backoff and no memory across runs is noise, not resilience
→ A circuit breaker solves a problem a retry, by design, can't
  solve: remembering across runs
→ Claude's 429 is the model provider's layer, not an agent tool
  -- it gets retried, never cut off with a breaker
→ Degrading gracefully means responding fast and clearly when
  the breaker is open -- not hiding the failure, not retrying blindly

Traps to avoid in this module

1. "I already used call_with_retries in agent-fundamentals — this module repeats the same thing"

No. call_with_retries solves the retry within a run, with no memory from one to the next. This module builds backoff (an improvement on that same mechanism) and, above all, a circuit breaker — something call_with_retries, bounded to a single invocation, could never do by design: remember a tool has been failing across several runs.

2. "A circuit breaker is just a retry with a different name"

A retry decides "how many more chances do I give THIS call?" A circuit breaker decides something different, and earlier: "is it even worth trying this call, or do I already know — from what happened in previous runs — that the tool is dead?" A retry never stops trying on its own; a breaker does, and that's exactly the protection it provides.

3. "I'm going to implement backoff with random so the jitter feels realistic"

Not in this guide. Real random jitter is a legitimate, useful technique — it's developed in depth in resilience-and-reliability-patterns-guide — but it breaks the byte-for-byte reproducibility every "What to expect" block in this entire guide demands. Here backoff gets calculated and shown, never run with time.sleep(), and if jitter gets mentioned, it's done with explicit honesty about that simplification.

4. "Claude's 429 gets solved the same way as a downed tool, with the same circuit breaker"

No. A downed tool is a local problem — THAT dependency, THAT service, stopped responding — and the circuit breaker exists to stop insisting on it. The 429 is Claude's own API telling you "you're asking for more than your quota allows right now" — it isn't down, it keeps working perfectly for everyone else. Cutting off calls to the model with a breaker makes no sense: the model IS the product. The right response to a 429 is to retry with backoff, never to stop calling it.

5. "When the breaker is open, it's better for the agent to fail with a technical error — that way the user knows something's wrong"

The opposite. Failing with a technical error (CircuitOpenError: ...) is exactly what Lesson 7 fixes: the agent knows, with certainty, why book_room isn't available — that information shouldn't get lost in a stacktrace. Degrading gracefully means translating that certainty into a clear, useful response, without pretending everything went fine.

How to work through this module

  1. Run every example yourself. Every lesson brings runnable code with real output. Seeing the breaker jump from CLOSED to OPEN on your own terminal — with the exact number of failures it took — is worth more than reading it in a table.
  2. Follow book_room's thread. It's the same tool, the same simulated outage, reused and expanded lesson after lesson — don't restart the example every time you open a new file.
  3. The mini-project (Lesson 08) is the synthesis with real trade-offs. It isn't just "everything together" — it measures, with numbers, what's gained and what's lost by tuning the breaker's cooldown.

Estimated time

Lesson 01 (this)  →  20 min reading
Lesson 02         →  20 min + running the example
Lesson 03         →  30 min + running the example
Lesson 04         →  30 min + running the example
Lesson 05         →  30 min + running the example
Lesson 06         →  25 min + running the example
Lesson 07         →  25 min + running the example
Lesson 08         →  40 min + building the mini-project

Total: ~3.5 hours

Evidence of success

Before moving on to Module 7 (Versioning and Safe Rollout), you should be able to:

  • ✅ Explain, with the book_room example, why retrying with no memory across runs wastes cost every time a new user arrives while the tool is still down.
  • ✅ Run retry_with_backoff over a flaky tool and read, in the real output, how backoff grows with each attempt.
  • ✅ Run a complete CircuitBreaker and narrate, with the exact vocabulary, every transition: CLOSED → OPEN by threshold, OPEN → HALF_OPEN by cooldown, HALF_OPEN → CLOSED or HALF_OPEN → OPEN depending on the probe's result.
  • ✅ Tell apart, without hesitation, a Claude 429 from a downed tool, and explain why the first one never carries a breaker.
  • ✅ Trace this module's two boundaries: toward agent-fundamentals M7 (within-the-run) and toward resilience-and-reliability-patterns-guide (reused vocabulary, narrow scope).

Summary

  • This module hardens the Reservo agent against two types of failure: a tool that fails sustainedly (book_room) and the Claude API's own 429 — with three pieces: bounded backoff, a per-tool circuit breaker, and graceful degradation.
  • The boundary toward agent-fundamentals M7 is one of temporal scope: that guide solves the retry within a run; this module builds memory that persists across runs.
  • The boundary toward resilience-and-reliability-patterns-guide is one of depth and scope: that guide is the canonical source for the CLOSED/OPEN/HALF_OPEN vocabulary and for backoff+jitter measured in depth with real random data; this module reuses it, citing it, applied only to the agent's tool-call layer and to the LLM provider's 429 — it never builds bulkheads or load shedding.
  • Hard rule: backoff is calculated and shown, never actually slept (time.sleep()); there's never random; the 429 is simulated with a deterministic client, never a real API call.
  • The analogy holding the whole module together: the circuit breaker is your house's thermal switch — it trips to protect, not to punish, and returns to normal as soon as it can confirm the problem passed.

Next lesson: 02 — A Tool That Fails Repeatedly. We see, with run code, the exact problem the rest of the module solves: three different users, three independent runs, the same downed tool — and the cost of remembering nothing from one to the next.

Additional resources

  1. Anthropic — Errors — The complete reference of Claude API error codes, including the 429 rate_limit_error Lesson 6 simulates.
  2. Anthropic — Rate limits — How the Claude API's limits work per account tier, the real foundation behind this module's simulated 429.
  3. Anthropic — Building effective agents — On why a production agent's reliability depends on anticipating failures, not just handling them once they've already happened.
  4. resilience-and-reliability-patterns-guide — The sister guide, canonical source for the circuit-breaker, backoff+jitter, and graceful-degradation vocabulary this module reuses precisely and cites in every lesson where it applies.
  5. Python 3.14 — What's New — The exact version all of this module's engineering runs on.
← PreviousMini-Project: A Regression Gate for ReservoNext →A Tool That Fails Repeatedly