The Circuit Breaker Pattern

Description

Lesson 3 closed with an uncomfortable fact: not even the best exponential backoff saves an individual run from an outage that lasts longer than its own retry cap. And worse — every new user arriving while the outage is still active repeats exactly the same sequence of failed attempts, because nothing in the system remembers what was already learned. This lesson builds that memory: a circuit breaker — an object that lives beyond any individual run, counts a specific tool's consecutive failures, and when that count crosses a threshold, stops calling it altogether — with no waiting for the timeout, without spending a single more backoff — until a periodic check confirms the tool came back.

Connection to the module

The vocabulary you build in this lesson — CLOSED, OPEN, HALF_OPEN — is exactly the same one resilience-and-reliability-patterns-guide (Module 5, "Circuit Breakers") uses for any generic HTTP dependency in a distributed system. This lesson reuses it, citing it, with a much narrower scope: a breaker per agent tool, applied to book_room. And unlike retry_with_backoff (Lesson 3), which solves "how many times do I retry within this run?", this lesson's CircuitBreaker solves a question that only makes sense across runs: "has this tool already proven, in previous runs, that it's dead?"

Analogy: the thermal switch that learns from the outage

You already know it from the module's Lesson 1, now in more detail. Your house's thermal switch doesn't trip at the first flicker of light — an isolated flicker means nothing, any healthy installation has one every once in a while. It trips when it detects a real pattern: a sustained, out-of-the-ordinary current, a sign that something — a short circuit, a broken appliance — is genuinely wrong, not just bad luck this one time. The moment it trips, it cuts power to that specific circuit, without shutting down the rest of the house. After a sensible while, someone flips the switch back on, as a test — if the problem got fixed, the light comes back normally; if the short circuit is still there, it trips again immediately.

Translated into this lesson's code: "an isolated flicker means nothing" is failure_count getting reset with every success — it never opens over a single stray failure. "A sustained, out-of-the-ordinary current" is failure_count reaching failure_threshold — several failures in a row, with no success in between. "It trips and cuts power to that circuit" is the breaker moving to OPEN and rejecting every call to book_room, without touching get_quote or list_rooms. "Someone flips the switch back, as a test" is HALF_OPEN — after a cooldown, one call gets let through, and that single call's result decides whether the circuit goes back to normal or trips again.

A note on the names, before the code

CLOSED sounds, in everyday language, like "nothing's happening" — but in an electrical circuit it means exactly the opposite: a closed circuit is a complete loop, and current flows. OPEN sounds like "things can happen" — but in a circuit it means the loop is broken, and nothing flows. It's the reverse of a door's intuition, and it makes sense in the electrical world the pattern comes from: closing the circuit completes the loop; opening it cuts it. If you ever doubt it, go back to your house's switch: when it trips to protect you, it stays open (OPEN) and cuts the power; when everything's normal, it's closed (CLOSED) and power flows. This convention — CLOSED = conducts = calls go through; OPEN = cuts = calls get rejected — is the same one resilience-and-reliability-patterns-guide uses, and the one we use here, unchanged.

Worked example: CircuitBreaker, built in full

The three states and the error it raises on rejection

# resilience/tool_circuit_breaker.py

CLOSED, OPEN, HALF_OPEN = "CLOSED", "OPEN", "HALF_OPEN"


class CircuitOpenError(Exception):
    """El breaker rechaza la llamada sin tocar la tool real (falla rápido)."""

CircuitOpenError is the key to "failing fast": when the breaker is OPEN, the call gets rejected at that exact instant, without touching book_room, without waiting for any timeout — nothing resembling the cost of a real attempt. Whoever receives this exception decides what to do with it; Lesson 7 builds that part.

The constructor and the state that persists

class CircuitBreaker:
    """Máquina de tres estados, con estado que persiste ENTRE runs -- a
    diferencia del reintento dentro-del-run de agent-fundamentals M7. El
    cooldown se mide en LLAMADAS rechazadas, no en segundos: no hay reloj
    real en esta guía (ver honestidad más abajo)."""

    def __init__(self, name, failure_threshold=3, cooldown_calls=2):
        self.name = name
        self.failure_threshold = failure_threshold   # fallos seguidos para abrir
        self.cooldown_calls = cooldown_calls          # rechazos antes de una sonda
        self.state = CLOSED                            # arranca sano
        self.failure_count = 0                          # fallos consecutivos en CLOSED
        self._calls_while_open = 0                       # rechazos ya contados en OPEN

Honesty, before moving on: resilience-and-reliability-patterns-guide measures its circuit breaker's cooldown in real seconds, with time.monotonic() — that makes sense there, because its case is HTTP microservices receiving continuous traffic, many calls per second. This guide can't use a real clock in any "What to expect" block (the same rule that bans time.sleep() in backoff), so this CircuitBreaker measures cooldown in rejected calls, not seconds — cooldown_calls=2 means "reject the next two calls, and on the third, let a probe through." It's a deliberate simplification to keep the example deterministic and reproducible; the underlying criterion — give it time before testing again — is the same.

Deciding whether a call goes through: before_call

    def before_call(self):
        if self.state == OPEN:
            if self._calls_while_open >= self.cooldown_calls:
                self.state = HALF_OPEN
                return
            self._calls_while_open += 1
            raise CircuitOpenError(
                f"{self.name}: circuito abierto (OPEN), llamada rechazada sin tocar la tool"
            )

If the breaker is CLOSED or already moved to HALF_OPEN, before_call does nothing — the call proceeds normally. If it's OPEN, it counts how many calls it's already rejected (_calls_while_open); the moment that count reaches cooldown_calls, instead of rejecting again, it moves to HALF_OPEN and lets this call through as a probe.

Recording the result: on_success and on_failure

    def on_success(self):
        if self.state == HALF_OPEN:
            self.state = CLOSED
        self.failure_count = 0

    def on_failure(self):
        if self.state == HALF_OPEN:
            self.state = OPEN
            self._calls_while_open = 0
            return
        self.failure_count += 1
        if self.failure_count >= self.failure_threshold:
            self.state = OPEN
            self._calls_while_open = 0

Notice the self.failure_count = 0 line inside on_success: any success wipes out the consecutive-failure count, regardless of whether the breaker was CLOSED or just coming out of HALF_OPEN. That line is what makes the breaker count consecutive failures, not total failures accumulated over the process's entire lifetime — without it, isolated failures with no relation to each other (a hiccup today, another next week) would end up adding up to the threshold and opening the circuit over a perfectly healthy tool. on_failure, on the HALF_OPEN side, is just as important: if the probe fails, the breaker doesn't start counting from zero again — it goes straight back to OPEN, with a fresh cooldown, without giving that tool an immediate second probe.

Composing the breaker with Lesson 3's backoff

def call_with_breaker(breaker, fn, *args, max_retries=3, base_delay_ms=100,
                       retry_on=(ConnectionError,), **kwargs):
    """Compone el circuit breaker (decide si la llamada pasa) con el
    reintento con backoff (decide cuántas veces intentarla)."""
    breaker.before_call()
    try:
        result = retry_with_backoff(
            fn, *args, max_retries=max_retries, base_delay_ms=base_delay_ms,
            retry_on=retry_on, **kwargs,
        )
    except Exception:
        breaker.on_failure()
        raise
    else:
        breaker.on_success()
        return result

Notice this composition's exact granularity: breaker.before_call() decides, once, whether this call even gets to try at all. If it does, retry_with_backoff runs with its own retry cap — Lesson 3's same max_retries with backoff — and only once that entire run gives up (exhausts its own cap), the breaker finds out and adds one failure. This differs from resilience-and-reliability-patterns-guide's generic breaker, which cuts off before every individual HTTP call: this breaker operates at the complete-run level — within a run that does get past before_call, Lesson 3's backoff keeps working exactly like always. Lesson 5 revisits this detail in more depth.

Run for real: the complete cycle, CLOSED → OPEN → HALF_OPEN → CLOSED

breaker = CircuitBreaker("book_room", failure_threshold=3, cooldown_calls=2)

print("=== 7 runs independientes, book_room caído las primeras 9 llamadas reales ===")
for run_n in range(1, 8):
    print(f"--- run {run_n} (estado del breaker ANTES: {breaker.state}) ---")
    try:
        result = call_with_breaker(
            breaker, flaky_book_room, room="Focus", tier="pro", hours=3, member=f"user{run_n}",
            max_retries=3, base_delay_ms=100,
        )
        print(f"  OK -> {result}")
    except CircuitOpenError as exc:
        print(f"  RECHAZADO SIN LLAMAR A LA TOOL: {exc}")
    except ConnectionError as exc:
        print(f"  FALLO (tope de reintentos agotado): {exc}")
    print(f"  estado del breaker DESPUÉS: {breaker.state} (failure_count={breaker.failure_count})")

print()
print(f"llamadas reales totales a book_room: {_state['count']}")

flaky_book_room is Lesson 2's same tool, this time with OUTAGE_CALLS = 9 — nine real calls down before recovering.

What to expect:

=== 7 runs independientes, book_room caído las primeras 9 llamadas reales ===
--- run 1 (estado del breaker ANTES: CLOSED) ---
    intento 1/3...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #1) -- backoff modelado: 100ms (no se duerme de verdad)
    intento 2/3...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #2) -- backoff modelado: 200ms (no se duerme de verdad)
    intento 3/3...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #3) -- backoff modelado: 400ms (no se duerme de verdad)
  FALLO (tope de reintentos agotado): timeout de red simulado (llamada real #3)
  estado del breaker DESPUÉS: CLOSED (failure_count=1)
--- run 2 (estado del breaker ANTES: CLOSED) ---
    intento 1/3...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #4) -- backoff modelado: 100ms (no se duerme de verdad)
    intento 2/3...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #5) -- backoff modelado: 200ms (no se duerme de verdad)
    intento 3/3...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #6) -- backoff modelado: 400ms (no se duerme de verdad)
  FALLO (tope de reintentos agotado): timeout de red simulado (llamada real #6)
  estado del breaker DESPUÉS: CLOSED (failure_count=2)
--- run 3 (estado del breaker ANTES: CLOSED) ---
    intento 1/3...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #7) -- backoff modelado: 100ms (no se duerme de verdad)
    intento 2/3...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #8) -- backoff modelado: 200ms (no se duerme de verdad)
    intento 3/3...
      fallo transitorio (ConnectionError): timeout de red simulado (llamada real #9) -- backoff modelado: 400ms (no se duerme de verdad)
  FALLO (tope de reintentos agotado): timeout de red simulado (llamada real #9)
  estado del breaker DESPUÉS: OPEN (failure_count=3)
--- run 4 (estado del breaker ANTES: OPEN) ---
  RECHAZADO SIN LLAMAR A LA TOOL: book_room: circuito abierto (OPEN), llamada rechazada sin tocar la tool
  estado del breaker DESPUÉS: OPEN (failure_count=3)
--- run 5 (estado del breaker ANTES: OPEN) ---
  RECHAZADO SIN LLAMAR A LA TOOL: book_room: circuito abierto (OPEN), llamada rechazada sin tocar la tool
  estado del breaker DESPUÉS: OPEN (failure_count=3)
--- run 6 (estado del breaker ANTES: OPEN) ---
    intento 1/3...
  OK -> {'booking_id': 1, 'confirmed': True, 'price_cents': 6000}
  estado del breaker DESPUÉS: CLOSED (failure_count=0)
--- run 7 (estado del breaker ANTES: CLOSED) ---
    intento 1/3...
  OK -> {'booking_id': 1, 'confirmed': True, 'price_cents': 6000}
  estado del breaker DESPUÉS: CLOSED (failure_count=0)

llamadas reales totales a book_room: 11

Walk through it with the thermal switch in mind. Runs 1 and 2: fail, each exhausting its three backoff retries — the breaker stays CLOSED, but failure_count rises to 1, then 2. Run 3: fails again — the third failure in a row — and failure_count reaches failure_threshold (3): the breaker trips to OPEN. Runs 4 and 5: don't even touch book_room — RECHAZADO SIN LLAMAR A LA TOOL, at that exact instant, with no backoff, no timeout at all — the complete savings a circuit breaker exists to provide. Run 6: the two-rejection cooldown has already been met (_calls_while_open reached 2), so the breaker lets this call through as a HALF_OPEN probe — and by this point, the tool has already recovered (real call #10, within this run's attempt cap, is already outside the nine-call downed window): the probe succeeds, and the breaker returns, at that exact instant, to CLOSED. Run 7: normal, CLOSED start to finish, no failures at all.

Notice the total: 11 real calls for seven runs — not 21 (which is what seven runs of up to three attempts each would cost with no breaker at all). Runs 4 and 5, the ones the breaker flatly rejected, didn't spend a single real call.

Common mistakes

1. Counting total failures instead of consecutive ones — forgetting the reset in on_success. If self.failure_count = 0 weren't in on_success, the breaker would open over a healthy tool that simply had two isolated hiccups, weeks apart, with no relation to each other. Counting consecutive ones — and resetting on every success — is what tells "just bad luck this once" apart from "genuinely dead."

2. Setting failure_threshold too high "to be safe." A threshold of 50, for example, lets fifty real calls leak out to their complete retry cap before opening — almost as much cost as having no breaker at all. The right threshold is low enough that the detection toll stays small; a handful of consecutive failures (three, five) is already a clear signal of a sustained outage.

3. Thinking the detection toll (runs 1, 2, and 3 in the example, which did genuinely fail) is a flaw in the breaker. It's inevitable and correct: the breaker can't know book_room is down without letting some runs try first — those runs are how the breaker discovers the outage. What the breaker guarantees is that, after that discovery, it stops paying that cost — not that the cost disappears entirely from the first failure.

4. Sharing a single CircuitBreaker across different tools. This lesson's breaker protects one tool — book_room. If get_quote also needed one, it needs a separate instance (CircuitBreaker("get_quote", ...)) with its own failure_count and its own state — a get_quote failure should never open book_room's circuit, and vice versa.

Exercises

Exercise 1: Calculate the detection toll (Easy)

With failure_threshold=3 and max_retries=3 per run (like in this lesson's example), how many real calls get spent, at minimum, before the breaker opens for the first time? (Hint: every run that fails exhausts its own retry cap before the breaker counts that run as a single failure.)

Exercise 2: A stricter breaker (Medium)

Repeat this lesson's worked example, but with CircuitBreaker("book_room", failure_threshold=1, cooldown_calls=1) — opens on the first failure, tries again after a single cooldown rejection. Run four runs against the same flaky_book_room (OUTAGE_CALLS=9, resetting _state["count"] = 0 first) and confirm on which run the breaker opens, and on which it rejects without touching the tool.

_state["count"] = 0
breaker_strict = CircuitBreaker("book_room", failure_threshold=1, cooldown_calls=1)

for run_n in range(1, 5):
    print(f"--- run {run_n} (ANTES: {breaker_strict.state}) ---")
    try:
        call_with_breaker(breaker_strict, flaky_book_room, room="Focus", tier="pro",
                           hours=3, member=f"user{run_n}", max_retries=3, base_delay_ms=100)
        print("  OK")
    except CircuitOpenError:
        print("  RECHAZADO SIN LLAMAR A LA TOOL")
    except ConnectionError:
        print("  FALLO (tope de reintentos agotado)")
    print(f"  DESPUÉS: {breaker_strict.state}")

--- run 1 (ANTES: CLOSED) ---
  FALLO (tope de reintentos agotado)
  DESPUÉS: OPEN
--- run 2 (ANTES: OPEN) ---
  RECHAZADO SIN LLAMAR A LA TOOL
  DESPUÉS: OPEN
--- run 3 (ANTES: OPEN) ---
  FALLO (tope de reintentos agotado)
  DESPUÉS: OPEN
--- run 4 (ANTES: OPEN) ---
  RECHAZADO SIN LLAMAR A LA TOOL
  DESPUÉS: OPEN

Exercise 3: Why shouldn't the breaker reset failure_count when it receives a CircuitOpenError? (Hard)

Someone proposes modifying call_with_breaker so that, if breaker.before_call() raises CircuitOpenError, the except Exception further below catches it just like any other failure and calls breaker.on_failure(). Explain, in one paragraph, why that would break the state machine — think about what would happen to failure_count and to _calls_while_open while the breaker is OPEN rejecting calls one after another.

Summary and next step

• We built CircuitBreaker: three states (CLOSED/OPEN/HALF_OPEN), the same vocabulary as resilience-and-reliability-patterns-guide, with before_call (decides whether the call goes through), on_success and on_failure (record the result), and CircuitOpenError (fails fast, without touching the real tool).
• call_with_breaker composes the breaker with Lesson 3's retry_with_backoff: the breaker decides, once, whether an entire run gets to try at all; if it does, backoff keeps operating within that run exactly like before.
• We ran the complete cycle against a downed book_room: three failed runs open the circuit (CLOSED → OPEN), two runs get rejected without touching the tool, and the sixth — with the tool already recovered — closes the loop (OPEN → HALF_OPEN → CLOSED). Total: 11 real calls, not 21.
• Honesty: this breaker's cooldown gets measured in rejected calls, not real seconds — a deliberate simplification to keep this guide's reproducibility; the version with a real time.monotonic() is in resilience-and-reliability-patterns-guide.

Next lesson: 05 — Closed, Open, Half-Open. We look more closely at the transitions — what happens when the HALF_OPEN probe also fails, how much gets saved in real calls, and why this breaker's "probing" differs from a generic HTTP dependency's.

Additional resources

1. resilience-and-reliability-patterns-guide (Module 5, "Circuit Breakers") — This pattern's complete canonical source: the same three-state machine, measured in depth against a real microservices outage, with a real cooldown_s and time.monotonic().
2. Anthropic — Building effective agents — On why a production agent needs mechanisms that protect the complete system, not just an individual call.
3. Python — Classes — The mutable-state-object mechanism that makes it possible for CircuitBreaker to remember across calls, within the same process.
4. Python 3.14 — What's New — The version every line of code in this lesson ran on.