Module 6: Failures at Scale — Backoff, Circuit Breakers, and Rate Limits

Closed, Open, Half-Open

Next →

Description

Lesson 4 showed the circuit breaker's complete cycle with a happy ending: the HALF_OPEN probe succeeded on the first try, and the circuit went back to CLOSED with no surprises. Reality isn't always that tidy — sometimes the tool is still down right when the breaker decides to test again, and the probe also fails. This lesson completes the transition map the previous one left half-drawn: what happens when HALF_OPEN doesn't work, how much every complete open-and-probe cycle costs — in real calls — and why this specific breaker's "probing" operates at a different level than a generic HTTP dependency's breaker does.

Connection to the module

This lesson doesn't add any new mechanism — CircuitBreaker and call_with_breaker are exactly Lesson 4's, without changing a single line. What changes is the scenario: a longer outage, one that survives two probes in a row before the third one finally lines up with the real recovery. It's the same state machine, put to the test against the case Lesson 4 didn't get to show.

The complete transition map

                 failure_count == failure_threshold
        ┌──────────────────────────────────────────────┐
        │                                                ▼
    ┌────────┐                                      ┌────────┐
    │ CLOSED │◀─────────────────────────────────────│  OPEN  │
    └────────┘   the HALF_OPEN probe succeeds        └────────┘
        │                                                  ▲
        │           _calls_while_open >= cooldown_calls    │
        │                        │                          │
        │                        ▼                          │
        │                  ┌───────────┐                    │
        └─────────────────▶│ HALF_OPEN │────────────────────┘
         (never happens     └───────────┘  the HALF_OPEN probe fails
          directly --
          CLOSED only reaches
          HALF_OPEN via OPEN)

Three transitions, and only three: CLOSED → OPEN when failure_count reaches failure_threshold (Lesson 4); OPEN → HALF_OPEN when the rejection cooldown is met; and from HALF_OPEN, two possible exits depending on the probe's result — to CLOSED if it succeeded, back to OPEN (with a fresh cooldown, _calls_while_open back to zero) if it failed. There's no other arrow anywhere in the diagram — any behavior not on this map is a bug.

Run for real: an outage that survives two probes in a row

We use the same flaky_book_room from the previous lessons, this time with OUTAGE_CALLS = 15 — an outage long enough that two complete HALF_OPEN probes (each with its own max_retries=3 real attempts) still fall within the downed window, before the third finally lines up with the recovery.

breaker = CircuitBreaker("book_room", failure_threshold=3, cooldown_calls=2)

print("=== apagón de 15 llamadas reales: DOS sondas HALF_OPEN fallan antes de la que recupera ===")
for run_n in range(1, 14):
    before_state = breaker.state
    before_real = _state["count"]
    try:
        result = call_with_breaker(
            breaker, flaky_book_room, room="Focus", tier="pro", hours=3, member=f"user{run_n}",
            max_retries=3, base_delay_ms=100,
        )
        outcome = "OK -- reserva confirmada"
    except CircuitOpenError:
        outcome = "RECHAZADO SIN LLAMAR A LA TOOL"
    except ConnectionError:
        outcome = "FALLO (tope de reintentos agotado)"
    real_calls_used = _state["count"] - before_real
    print(f"run {run_n:2}: breaker {before_state:9} -> {breaker.state:9} | llamadas reales: {real_calls_used} | {outcome}")

print()
print(f"llamadas reales totales a book_room: {_state['count']}")

What to expect:

=== apagón de 15 llamadas reales: DOS sondas HALF_OPEN fallan antes de la que recupera ===
run  1: breaker CLOSED    -> CLOSED    | llamadas reales: 3 | FALLO (tope de reintentos agotado)
run  2: breaker CLOSED    -> CLOSED    | llamadas reales: 3 | FALLO (tope de reintentos agotado)
run  3: breaker CLOSED    -> OPEN      | llamadas reales: 3 | FALLO (tope de reintentos agotado)
run  4: breaker OPEN      -> OPEN      | llamadas reales: 0 | RECHAZADO SIN LLAMAR A LA TOOL
run  5: breaker OPEN      -> OPEN      | llamadas reales: 0 | RECHAZADO SIN LLAMAR A LA TOOL
run  6: breaker OPEN      -> OPEN      | llamadas reales: 3 | FALLO (tope de reintentos agotado)
run  7: breaker OPEN      -> OPEN      | llamadas reales: 0 | RECHAZADO SIN LLAMAR A LA TOOL
run  8: breaker OPEN      -> OPEN      | llamadas reales: 0 | RECHAZADO SIN LLAMAR A LA TOOL
run  9: breaker OPEN      -> OPEN      | llamadas reales: 3 | FALLO (tope de reintentos agotado)
run 10: breaker OPEN      -> OPEN      | llamadas reales: 0 | RECHAZADO SIN LLAMAR A LA TOOL
run 11: breaker OPEN      -> OPEN      | llamadas reales: 0 | RECHAZADO SIN LLAMAR A LA TOOL
run 12: breaker OPEN      -> CLOSED    | llamadas reales: 1 | OK -- reserva confirmada
run 13: breaker CLOSED    -> CLOSED    | llamadas reales: 1 | OK -- reserva confirmada

llamadas reales totales a book_room: 17

Follow the thread carefully, because there's a detail not even this table's before/after shows directly. Runs 1-3: fail, CLOSED → OPEN on run 3, exactly like in Lesson 4. Runs 4-5: flatly rejected, 0 real calls — the two-rejection cooldown is counting. Run 6: something happens here that the "breaker BEFORE/AFTER" column doesn't show directly — within this single call, the breaker goes from OPEN to HALF_OPEN (the cooldown's already been met), lets the probe through, the probe uses its own three backoff attempts (real calls 10, 11, 12 — still within the fifteen-call outage), the entire probe fails, and the breaker goes back to OPEN at that exact instant — that's why the column shows OPEN → OPEN, even though internally it went through HALF_OPEN and back. Runs 7-8: rejected again, a fresh cooldown. Run 9: second probe, real calls 13, 14, 15 — again within the outage, right at the edge — fails again, back to OPEN. Runs 10-11: rejected, third cooldown. Run 12: third probe — real call 16, already outside the fifteen-call outage — succeeds on the first attempt, and the circuit finally closes. Run 13: normal, CLOSED start to finish.

Seventeen real calls total for thirteen runs — compare that to what it would have cost with no breaker at all: each of the thirteen runs exhausting up to three real attempts, up to 39 calls in the worst case. And of those seventeen, six rejections (runs 4, 5, 7, 8, 10, 11) never touched the tool at all.

Why this breaker's "probe" isn't a single HTTP call

It's worth pausing on something already hinted at in Lesson 4: when resilience-and-reliability-patterns-guide talks about the HALF_OPEN probe, it literally means one HTTP call — a single GET or POST to the dependency, that flips the circuit up or down based on its immediate result. In this module, run 6's "probe" (for example) is actually three real calls to book_room — retry_with_backoff's three backoff attempts, running within that same run — before the breaker finds out whether the probe, as a whole, succeeded or not.

This isn't a mistake — it's a direct consequence of the granularity this breaker operates at: at the run level, not at the individual low-level call level. Every Reservo agent run already brings its own backoff-retry budget (Lesson 3), and the circuit breaker decides something earlier and different: whether that entire run — with all its own backoff included — is worth trying, or whether we already know, from previous runs, that it isn't. It's an honest design decision, not an accident: if you wanted a breaker that cuts off within the same run, after the first failed call, without giving retry_with_backoff its three chances, you'd have to wrap the breaker around every individual attempt instead of around the complete run — a valid design, but different from the one this guide built, and one that would change how many real calls every probe costs.

Choosing failure_threshold and cooldown_calls

The same criterion agent-fundamentals M4 used for max_iterations, and M7 for max_retries, applies here, with the corresponding adjustment: neither number has a universal "correct" value — each is an explicit trade-off.

  • A low failure_threshold (say, 1) opens the circuit almost immediately — protects better against short outages, but runs a bigger risk of opening over an isolated failure that wasn't, in reality, the start of a sustained outage.
  • A high failure_threshold (say, 10) is harder to confuse with noise — it takes a much clearer failure pattern — but pays a more expensive detection toll: more real calls wasted before the breaker reacts.
  • A low cooldown_calls probes recovery more often — detects the tool came back faster — but every failed probe (like runs 6 and 9 in the example) costs the system up to max_retries complete real calls, not just one.
  • A high cooldown_calls wastes less on failed probes, but lets more time pass — more users flatly rejected — before noticing the tool is healthy again.

resilience-and-reliability-patterns-guide develops this exact trade-off in depth, with formulas and measurements over outages of different durations — that's the reference if you need to calibrate these numbers with mathematical, not just intuitive, criteria for a real system.

Common mistakes

  1. Assuming OPEN → OPEN in the table means "nothing happened." As you saw in runs 6 and 9 of the example, OPEN → OPEN can hide a complete OPEN → HALF_OPEN → OPEN cycle — a probe that did get tried, did spend real calls, and did fail. To see it, you need to look at the real-calls-spent column, not just the before/after state.

  2. Thinking that lowering cooldown_calls to 0 is "more aggressive, and therefore better." With cooldown_calls=0, the breaker would probe on the very first call after opening — basically giving the downed tool no breathing room at all, very close to having no breaker at all during that instant. The cooldown exists, precisely, to give a minimal margin before the first test.

  3. Forgetting every failed probe resets the cooldown from zero. on_failure(), in the HALF_OPEN branch, does self._calls_while_open = 0 — the rejection count for the next probe starts over, it doesn't continue where it left off. A long outage can, because of this, generate several complete cooldown cycles before recovering, as in this lesson's example.

Exercises

Exercise 1: Count the complete cycles (Easy)

Without running Python: in this lesson's worked example (OUTAGE_CALLS=15, failure_threshold=3, cooldown_calls=2, max_retries=3), how many complete OPEN → HALF_OPEN → OPEN cycles (failed probe) happen before the final cycle that does close the circuit? Confirm by counting the FALLO lines that show up with the breaker already in the OPEN state.

See solution

Two complete failed-probe cycles: run 6 (real calls 10-12, still within the 15-call outage) and run 9 (real calls 13-15, right at the edge). The third cycle, on run 12 (real call 16), finally falls outside the outage and closes the circuit.

Exercise 2: An outage that ends at exactly the worst moment (Medium)

With OUTAGE_CALLS = 12 (instead of 15) and the same failure_threshold=3, cooldown_calls=2, max_retries=3, run the same thirteen-run loop. On which run does the circuit finally close? (Hint: recalculate which real calls each probe uses, and compare them against the new OUTAGE_CALLS.)

See solution 
_state["count"] = 0
OUTAGE_CALLS = 12
breaker_12 = CircuitBreaker("book_room", failure_threshold=3, cooldown_calls=2)

for run_n in range(1, 14):
    before_real = _state["count"]
    try:
        call_with_breaker(breaker_12, flaky_book_room, room="Focus", tier="pro",
                           hours=3, member=f"user{run_n}", max_retries=3, base_delay_ms=100)
        outcome = "OK"
    except CircuitOpenError:
        outcome = "RECHAZADO"
    except ConnectionError:
        outcome = "FALLO"
    print(f"run {run_n:2}: {outcome} (llamadas reales: {_state['count'] - before_real})")

With the same failure rhythm as the worked example — runs 1-3 fail (calls 1-9, open the circuit), runs 4-5 rejected, run 6 probes with calls 10-12 — the new OUTAGE_CALLS=12 means real calls 10 and 11 still fall within the outage, but 12 also does (12 <= 12), so run 6's probe still fails, exactly like with OUTAGE_CALLS=15. The circuit only closes on run 9, whose probe uses real call 13 — the first one already outside the outage (13 > 12) — and succeeds on the first attempt. The circuit closes three runs earlier than in the original example, simply because the outage ended three real calls earlier.

Exercise 3: Design a breaker that probes without spending the complete backoff (Hard)

This lesson's design makes every HALF_OPEN probe use up to max_retries real calls, because call_with_breaker always invokes retry_with_backoff with the same max_retries, regardless of the breaker's state. Describe, in prose (no need to write the complete code), how you'd change call_with_breaker so that, specifically when the breaker is HALF_OPEN, the probe uses max_retries=1 — a single real chance, no extended backoff — instead of the run's normal max_retries. What new trade-off does that change introduce, compared to the current design?

See solution

The change would consist of call_with_breaker checking breaker.state before calling retry_with_backoff, and if it's HALF_OPEN, passing max_retries=1 instead of the max_retries it received as an argument — something like effective_max_retries = 1 if breaker.state == HALF_OPEN else max_retries. The trade-off is direct: with a single real chance per probe, every HALF_OPEN cycle costs at most one real call instead of up to max_retries — cheaper when the probe fails, like runs 6 and 9 in this lesson — but also more prone to failing on pure bad luck: if the tool already recovered but that single specific call runs into a genuine transient hiccup (unrelated to the outage), the entire probe gets written off with no second chance the normal backoff would have given it. It's exactly the same kind of decision as failure_threshold and cooldown_calls — there's no single right answer, there's a trade-off between probing cost and tolerance for one-off bad luck, and the right choice depends on how much every wasted real call costs in your real system.

Summary and next step

  • We completed the CircuitBreaker's transition map: CLOSED → OPEN by threshold, OPEN → HALF_OPEN by cooldown, and from HALF_OPEN, two exits — CLOSED if the probe succeeds, back to OPEN (with a fresh cooldown) if it fails.
  • We ran a fifteen-real-call outage that survives two complete failed-probe cycles before the third finally lines up with the real recovery — thirteen runs, seventeen real calls, six rejections that never touched the tool.
  • We explained why this specific breaker's "probe" is actually up to max_retries real calls within the same run — a conscious design decision, different from resilience-and-reliability-patterns-guide's generic single-HTTP-call breaker — and what new trade-off changing that granularity would introduce.
  • failure_threshold and cooldown_calls are, both, explicit trade-offs with no universally correct value — the same criterion you already know from max_iterations and max_retries in agent-fundamentals.

Next lesson: 06 — The Claude 429 Rate Limit. We change layers entirely: from the per-tool circuit breaker to the Claude API's own rate limit — a failure that never carries a breaker, and why.

Additional resources

  1. resilience-and-reliability-patterns-guide (Module 5, Lessons 4-7) — The complete development of every transition, with a real cooldown_s, thread-second-savings measurements, and the mathematical criteria for choosing failure_threshold and cooldown_s against outages of different durations.
  2. Anthropic — Building effective agents — On the general criteria for designing safeguards at the system's right granularity, not the first available one.
  3. Python 3.14 — What's New — The version every line of code in this lesson ran on.
← PreviousThe Circuit Breaker PatternNext →The Claude 429 Rate Limit