Module 5: Failure Modes and Resilience for AI
The circuit breaker over the model
Overview
The fallback (lesson 5) resolves what you answer when the model fails. This lesson resolves a question left open: during a model outage that lasts minutes, does it make sense to try the model on every request —wait for the timeout, fail, fall to the fallback— over and over? No. If the model has failed twenty times in a row, the twenty-first request will almost certainly fail too; trying it only costs you the timeout (latency), the money for the call, and —this is AI-specific— more load on a model that may be rate-limited precisely because of too many calls. The piece that resolves this is the circuit breaker: a component that detects that the model has been failing and stops calling it for a while, going straight to the fallback, until a probe attempt confirms the model has recovered. This lesson applies it to the AI component and —what's specific here— highlights why a breaker over a model has an extra motivation that a breaker over a classic service does not have: the per-call cost and the provider's quotas.
In lesson 1 you saw the breaker open in the incident trace; here you develop it fully and measure its effect. You'll see, executed, two strategies in the face of a model outage: one that blindly retries on every request, and another with a circuit breaker. The second cuts the model calls, the timeouts paid, and the cost, while both answer 100% of requests (thanks to the fallback).
Connection with the module. This lesson crowns the availability family: the timeout (L4) bounds each individual wait, the fallback (L5) provides the alternate route, and the breaker (this one) avoids the waste of trying over and over a route we know is broken. The three work together —you saw it in the lesson 1 example—. The boundary with the resilience guide is hard and here it's explicit: the mechanics of the circuit breaker —the states with precision, the sliding count windows, the thresholds, how it's calibrated— are taught in resilience-and-reliability-patterns-guide M5 (circuit breakers); here we apply it to the model and defer there for the serious implementation. Our breaker is a minimal version, enough to see the pattern.
An analogy: the switch that cuts the power to protect the house
A circuit breaker —the name comes from here— is, literally, the thermomagnetic switch in your home's electrical panel. Its job: when it detects too much current flowing through a circuit —a short, a broken appliance—, it cuts the electricity to that circuit. It doesn't do it to annoy you; it does it to protect: without the breaker, that overcurrent would overheat the wires and could start a fire. The breaker would rather leave you without light in the kitchen for a while than let the house burn down. And notice the clever detail: when you fix the problem, you don't buy a new breaker —you reset it: you flip it back up and, if the circuit is fine now, the light comes back; if the short persists, it trips again—. That "reset attempt" is exactly the HALF_OPEN state.
Now transfer the image to your system calling a downed model. Without a breaker, every request "pushes current" through a broken circuit: it tries the model, waits for the timeout, fails, falls to the fallback. With twenty requests per second during a five-minute outage, that's thousands of useless calls to a model you already know is down —thousands of timeouts paid, thousands of calls charged, and thousands of hits to a provider that may be down precisely because it's saturated with calls—. The breaker is the switch that says: "I detected that this circuit has been failing; I cut the calls to the model, send everything to the fallback, and every so often I probe whether it has recovered". It stops pushing current through the broken circuit. It protects your latency, your budget, and —the AI-specific part— the provider's quota.
Here's the point: the circuit breaker over the model stops hitting a route it knows is broken, so as not to waste resources or make the problem worse, and it resets itself when the route recovers. It's the difference between a stubborn employee who keeps dialing a broken phone every two seconds all afternoon, and a sensible one who, after several failed attempts, stops dialing for ten minutes and uses another channel, checking again from time to time.
Worked example: blind retry vs circuit breaker
Let's measure the waste the breaker avoids. We simulate 16 requests with a model outage from request 3 to 11 (nine requests failing in a row —an outage, not scattered failures—, like a rate-limit window or a provider interruption). Each failing model call costs a timeout (800 ms lost) and counts as a charged call ($0.002). We compare:
- Without breaker (blind retry): every request calls the model, no matter what. During the outage, it pays the full timeout every time.
- With breaker: after 3 failures in a row, the breaker opens and stops calling the model; requests go straight to the fallback. Every 4 requests, it lets one probe attempt through (
HALF_OPEN) to see whether the model recovered.
Both strategies answer 100% thanks to the fallback; what changes is the waste —calls, timeouts, cost—.
# Lesson 6: CIRCUIT BREAKER over the model. If the model has been failing,
# stop calling it and use the fallback. AI-specific: every failed call
# costs latency AND money, and retrying a rate-limited model makes it WORSE
# (more 429s, more quota burned). The breaker mechanics in depth:
# resilience-and-reliability-patterns-guide M5.
MODEL_COST = 0.002 # $ per model call attempt
TIMEOUT_MS = 800 # ms lost each time a failure exhausts the timeout
class ModelError(Exception):
pass
OUTAGE = set(range(3, 12)) # the model is down from request 3 to 11 (rate-limited)
N = 16
def call_model(i):
if i in OUTAGE:
raise ModelError("down / rate-limited")
return "ok"
def fallback(i):
return "fallback"
class CircuitBreaker:
def __init__(self, fail_threshold=3, cooldown=4):
self.fail_threshold = fail_threshold
self.cooldown = cooldown
self.fails = 0
self.state = "CLOSED"
self.opened_at = None
def allow(self, now):
if self.state == "OPEN":
if now - self.opened_at >= self.cooldown:
self.state = "HALF_OPEN"
return True
return False
return True
def on_success(self):
self.fails = 0
self.state = "CLOSED"
def on_failure(self, now):
self.fails += 1
if self.fails >= self.fail_threshold:
self.state = "OPEN"
self.opened_at = now
# --- Strategy 1: NO breaker (blind retry, hits the model every time) ---
calls_naive = timeouts_naive = 0
for i in range(N):
try:
call_model(i)
except ModelError:
timeouts_naive += 1 # pay the full timeout
fallback(i)
calls_naive += 1 # called the model on EVERY request
# --- Strategy 2: WITH breaker ---
cb = CircuitBreaker(fail_threshold=3, cooldown=4)
calls_cb = timeouts_cb = skipped_cb = 0
print(f"{'req':<5}{'breaker':<11}{'called model?':<18}result")
print("-" * 52)
for i in range(N):
if cb.allow(i):
state = cb.state
try:
call_model(i)
cb.on_success()
calls_cb += 1
print(f"{i:<5}{state:<11}{'yes':<18}model ok")
except ModelError:
calls_cb += 1
timeouts_cb += 1
cb.on_failure(i)
fallback(i)
print(f"{i:<5}{state:<11}{'yes':<18}failed -> fallback")
else:
skipped_cb += 1
fallback(i)
print(f"{i:<5}{'OPEN':<11}{'no (avoided)':<18}direct fallback")
print("-" * 52)
print("Comparison (all 16 requests were answered in both strategies):")
print(f" {'':<26}{'no breaker':>14}{'with breaker':>14}")
print(f" {'model calls':<26}{calls_naive:>14}{calls_cb:>14}")
print(f" {'timeouts paid':<26}{timeouts_naive:>14}{timeouts_cb:>14}")
print(f" {'ms lost in timeouts':<26}{timeouts_naive * TIMEOUT_MS:>14}"
f"{timeouts_cb * TIMEOUT_MS:>14}")
print(f" {'call cost ($)':<26}{calls_naive * MODEL_COST:>14.3f}"
f"{calls_cb * MODEL_COST:>14.3f}")
print(f" calls avoided by the breaker: {skipped_cb} "
f"(didn't hit an already rate-limited model)")What to expect. When you run the file, the output is exactly this:
req breaker called model? result
----------------------------------------------------
0 CLOSED yes model ok
1 CLOSED yes model ok
2 CLOSED yes model ok
3 CLOSED yes failed -> fallback
4 CLOSED yes failed -> fallback
5 CLOSED yes failed -> fallback
6 OPEN no (avoided) direct fallback
7 OPEN no (avoided) direct fallback
8 OPEN no (avoided) direct fallback
9 HALF_OPEN yes failed -> fallback
10 OPEN no (avoided) direct fallback
11 OPEN no (avoided) direct fallback
12 OPEN no (avoided) direct fallback
13 HALF_OPEN yes model ok
14 CLOSED yes model ok
15 CLOSED yes model ok
----------------------------------------------------
Comparison (all 16 requests were answered in both strategies):
no breaker with breaker
model calls 16 10
timeouts paid 9 4
ms lost in timeouts 7200 3200
call cost ($) 0.032 0.020
calls avoided by the breaker: 6 (didn't hit an already rate-limited model)
Read the trace and then the comparison, because that's where the whole breaker is.
The trace shows the breaker's lifecycle. Follow the states:
- Requests 0-2 (
CLOSED): the model is healthy, the breaker lets everything through, the calls come back ok. - Requests 3-5 (
CLOSED, failing): the outage begins. The breaker is still closed —it didn't know—, so it calls the model, fails, and falls to the fallback each time, counting failures: 1, 2, 3. On the third failure, it opens. - Requests 6-8 (
OPEN): the breaker is open. It doesn't call the model —"no (avoided)"—, it goes straight to the fallback. Here's the saving: three requests that paid neither timeout nor cost because the breaker already knew the model was down. - Request 9 (
HALF_OPEN): the cooldown passed (4 requests), the breaker lets one probe attempt through. The model is still down (the outage runs to 11), so the attempt fails and the breaker opens again. It's the "reset" that trips again because the short persists. - Requests 10-12 (
OPEN): open again, avoiding calls. - Request 13 (
HALF_OPEN): another probe attempt. Now the model has recovered (the outage ended at 11), so the attempt works, and the breaker closes (CLOSED). - Requests 14-15 (
CLOSED): the system recovered, working normally. It reset itself.
The comparison measures the waste avoided. Both strategies answered the 16 requests (the fallback is always there). But look at the cost of the blind retry versus the breaker:
- Model calls: 16 vs 10. The breaker avoided 6 calls to a downed model.
- Timeouts paid: 9 vs 4. Without breaker, you paid the timeout on all 9 requests of the outage; with breaker, only on the 4 real attempts (3 before opening + 1 failed probe). 5 fewer timeouts.
- Milliseconds lost: 7200 vs 3200. The breaker saved 4 seconds of accumulated wait.
- Cost: $0.032 vs $0.020. The breaker saved 37% of the cost during the outage, by not charging for the 6 avoided calls.
The architectural implication, and here's the AI-specific part: every call avoided by the breaker isn't just latency saved; it's money saved and load you didn't put on a model that was already unwell. In a classic service, the breaker "only" saves you latency and protects your threads. In a model, it saves you money (each call costs tokens) and —more subtly— it prevents you from making a rate limit worse: if the model is down because it's saturated with calls, continuing to call it prolongs its saturation. The breaker cuts that vicious circle. That's why a breaker over a model has a motivation a classic breaker does not have.
Going deeper: the breaker applied to an AI component
The three states, in brief (the mechanics in depth are in the resilience guide). The breaker is a small state machine:
CLOSED(closed, working): calls pass through to the model normally. Failures are counted.OPEN(open, protecting): after crossing the failure threshold, the breaker opens and cuts the calls to the model; everything goes straight to the fallback. It stays this way during a cooldown.HALF_OPEN(half open, probing): once the cooldown passes, it lets one probe attempt through. If it works, it closes (recovered); if it fails, it opens again (still down).
Our implementation is minimal: it counts consecutive failures and uses a fixed cooldown measured in number of requests. A production implementation uses time windows, thresholds by proportion of failures (not just consecutive), and concurrency considerations. All of that is the resilience guide M5. Here the minimal version is enough to see the pattern and its effect.
The breaker needs the timeout and the fallback; it doesn't work alone. Retain the interdependence of the three availability pieces:
- Without timeout (L4), a hang doesn't throw an exception, so the breaker doesn't count the failure —it stays waiting, just like without a breaker—. The timeout turns the hang into the failure the breaker counts.
- Without fallback (L5), when the breaker is open and doesn't call the model, it has nothing to answer with —the breaker just gives you a faster error—. The fallback is what the breaker serves when it cuts.
- The breaker is what avoids paying the timeout and the cost of the fallback over and over during a long outage.
The three together: the timeout bounds each attempt, the breaker stops attempting when it's useless, the fallback answers. Remove one and the others limp.
The AI angle that isn't in a classic breaker: the rate limit. It's worth insisting because it's what's specific to this lesson. A downed classic service normally doesn't get worse because you keep calling it (it's already down, your calls bounce). A rate-limited model does get worse: every call during the rate-limit window counts against your quota and can extend the block, and if you share quota across features, a feature that blindly retries can rate-limit the others. The breaker is the right defense against a 429 precisely because it stops consuming quota while it waits. Retrying a 429 (even with backoff) still consumes; opening the breaker doesn't. That's why, for the rate limit, the breaker isn't just a latency optimization: it's the difference between recovering in a minute or staying blocked for ten.
The honest tradeoff: the breaker delays detecting the recovery. Nothing is free. Look at request 12 in the trace: the model had already recovered at 11 (end of the outage), but the breaker was OPEN in cooldown, so it served fallback at 12 even though the model already worked —an unnecessary degraded response—. The breaker didn't probe until 13. That's the cost: during the cooldown, you serve some extra fallback because you don't know the model came back. It's a deliberate tradeoff: a short cooldown detects recovery fast but probes more often (more probe calls, more risk of reopening falsely); a long cooldown protects more but is slow to notice the recovery. The resilience guide covers how to calibrate it. The lesson: the breaker trades "many useless calls" for "some extra degraded response", a trade that's almost always favorable.
Common mistakes
Blindly retrying during an outage (without a breaker). What happens: the model has a five-minute outage, and the system tries the model on each of the thousands of requests in those five minutes —each one pays the timeout, each one costs, and all together they hammer a provider that may be down from overload—. Why it happens: you have a fallback but no breaker; each request "discovers" the outage on its own, paying the timeout. How to detect it: during an outage, your rate of calls to the model doesn't drop —you keep calling just like when it was healthy—. How to fix it: put a circuit breaker that, after N failures, stops calling and goes straight to the fallback. The example measures it: the breaker cut the calls from 16 to 10 and the cost by 37%.
Retrying a rate limit and making it worse. What happens: the API returns 429, the system retries —maybe with backoff, maybe not—, and each retry consumes more quota, extending the block; a rate limit that would have lasted a minute lasts ten because you didn't stop calling. Why it happens: the 429 is treated as a transient error that "a retry fixes", without seeing that this error feeds on the retries. How to detect it: your handling of the 429 keeps calling the model (retry or not) instead of cutting. How to fix it: on a persistent 429, open the breaker —stop calling entirely while you wait for the quota window to free up— and serve the fallback. The breaker is the only defense that stops consuming quota.
A breaker without a reasonable cooldown (that never probes or probes too much). What happens: two variants. An eternal cooldown leaves the breaker open forever —the model recovered hours ago and you're still serving fallback because you never probed—. A zero cooldown probes on every request —you basically have no breaker, you keep hammering—. Why it happens: the cooldown was set at random without thinking about the detection-vs-protection tradeoff. How to detect it: either your breaker stays open long after the recovery, or you keep calling the downed model almost as if there were no breaker. How to fix it: calibrate the cooldown according to how long an outage typically lasts and how expensive an attempt is —the resilience guide M5 formalizes it—; the HALF_OPEN should probe often enough to notice the recovery, but not so often that it hammers.
Exercises
Exercise 1 — Follow the breaker's state. Using the example's state machine (threshold 3 failures, cooldown 4), trace what state the breaker would have and whether it would call the model on each of these requests, given that the model is down from request 2 to 6 (and healthy the rest): requests 0, 1, 2, 3, 4, 5, 6, 7.
See solution
Model down at 2, 3, 4, 5, 6. Threshold 3, cooldown 4.
- req 0 (
CLOSED): calls, model healthy, ok. fails=0. - req 1 (
CLOSED): calls, healthy, ok. fails=0. - req 2 (
CLOSED): calls, fails (outage). fails=1. Stays CLOSED. - req 3 (
CLOSED): calls, fails. fails=2. Stays CLOSED. - req 4 (
CLOSED): calls, fails. fails=3 → opens (opened_at=4). Serves fallback. - req 5 (
OPEN): 5-4=1 < 4 → doesn't call, direct fallback. - req 6 (
OPEN): 6-4=2 < 4 → doesn't call, direct fallback. (The model is still down, so avoiding the call was correct.) - req 7 (
OPEN): 7-4=3 < 4 → doesn't call, direct fallback. (The model already recovered at 7, but the breaker doesn't know because it's not time to probe yet; it serves extra fallback —the cooldown tradeoff—.)
Summary: it called the model at 0,1,2,3,4 (5 times), avoided calling it at 5,6,7 (3 times). It paid the timeout at 2,3,4 (3 times). The breaker would probe again at request 8 (8-4=4 ≥ 4 → HALF_OPEN), where the now-healthy model would close the breaker. Notice how at req 7 the system served fallback even though the model already worked: that's the cost of the cooldown, the honest tradeoff from the deep dive.
Exercise 2 — Why the breaker over a model is different. A colleague says: "a circuit breaker is a circuit breaker; it doesn't matter whether it's over a payments service or over a model". Give two reasons, specific to an AI component, why the motivation to put a breaker over a model is stronger than over a classic service.
See solution
Two AI-specific reasons:
-
Every call costs money. A model is charged per token/per call. When the blind retry hammers a downed model with thousands of failed calls, many of those calls are still charged (if the provider processed tokens before failing, or simply for the attempt). A classic payments service normally doesn't charge you for a call that bounced. So the breaker over a model saves a resource —money— that the classic breaker doesn't save. The example measured it: 37% of cost saved during the outage.
-
The rate limit gets worse when you call. A model is often "down" because it's rate-limited (429) —a quota, not a breakdown—. Every call during the rate-limit window consumes quota and can extend the block, and if several features share quota, the one that retries can rate-limit the others. A downed classic service normally doesn't get worse because you keep calling it. So over a model, continuing to call isn't just waste: it's counterproductive, it feeds the problem. The breaker, by stopping quota consumption, is the defense that lets the rate-limit window free up.
In both cases, the classic breaker saves latency and protects threads; the breaker over a model, in addition, saves money and avoids making the failure itself worse. The motivation is strictly greater.
Exercise 3 — The three pieces together. Explain, for a system with timeout + fallback + circuit breaker, which piece fails (and what consequence it has) if you remove each one, leaving the other two. Complete: (a) timeout + fallback, no breaker; (b) fallback + breaker, no timeout; (c) timeout + breaker, no fallback.
See solution
- (a) timeout + fallback, no breaker: the system works but wastes. Every request during a long outage discovers the failure on its own: it tries the model, pays the timeout (bounded by the timeout, granted), fails, falls to the fallback. It answers 100%, but pays thousands of unnecessary timeouts and charged calls, and hammers the provider. Consequence: latency and cost wasted throughout the outage, and possible worsening of a rate limit. It's the "no breaker strategy" from the example.
- (b) fallback + breaker, no timeout: it breaks in the face of a hang. A model that hangs (doesn't throw an exception, just waits) makes the request stay blocked indefinitely. The breaker doesn't count the failure because there was no exception —it stays waiting all the same—, so it never opens; the fallback doesn't fire because there was no exception to catch. Consequence: the system freezes in the face of a hang, even though you have a breaker and a fallback. The timeout is what turns the hang into the failure the other two need.
- (c) timeout + breaker, no fallback: the system fails faster, but fails. The timeout bounds the wait, the breaker stops attempting the downed model... but when the breaker cuts or the timeout expires, there's nothing to degrade to: the request falls with an error (faster and cleaner than without a timeout, but it falls). Consequence: high availability of failing fast, not of answering. Without a fallback, the breaker just gives you a fast error. The fallback is what turns "fail fast" into "answer worse".
The lesson: the three are interdependent. The timeout creates the failure, the fallback answers it, the breaker avoids repeating the useless attempt. None alone is enough.
Summary and next step
In this lesson you applied the circuit breaker to the AI component: the piece that detects that the model has been failing and stops calling it for a while, going straight to the fallback, until a probe attempt confirms it has recovered. You saw it with the electrical switch that cuts the power to protect the house and resets itself, and you measured it: in the face of a model outage, the breaker cut the calls from 16 to 10, the timeouts from 9 to 4, the wait from 7200 to 3200 ms, and the cost from $0.032 to $0.020 —by 37%—, while both strategies answered 100% thanks to the fallback. You followed the CLOSED → OPEN → HALF_OPEN → CLOSED cycle in the trace, with its probe attempt that fails while the model is still down and works when it recovers. And you retained what's AI-specific: a breaker over a model saves money and avoids making a rate limit worse, motivations a classic breaker does not have.
Before moving on you should be able to: explain the breaker's three states and what it does in each; argue why the three availability pieces (timeout, fallback, breaker) need each other; give two AI-specific reasons why a breaker over a model has an extra motivation; and recognize the cooldown tradeoff. The breaker's mechanics in depth live in resilience-and-reliability-patterns-guide M5.
With lesson 6 you close the family of availability failures (noisy ones) and their three defenses. Lesson 7 returns to the silent failures, but to the one you hadn't seen: drift —the silent degradation over time—. It's neither an outage nor an exception: the model or the data change little by little and quality drops without anything breaking. You'll see, executed, the search eval drop from 0.94 to 0.75 over seven weeks as queries from new categories arrive, and a monitor that fires the alert in week 4 —two weeks before users complain—. The way slow degradation doesn't take you by surprise.
Resources
resilience-and-reliability-patterns-guide(this ecosystem), M5 "Circuit breakers" — the central reference for the mechanics of the breaker that here we only apply: the states with precision, the sliding count windows, the proportion thresholds, the cooldown calibration, the concurrency. When you implement a real breaker, that's the destination. In Spanish.- Anthropic, Claude documentation — docs.anthropic.com. The rate limits pages describe the per-token and per-request quotas and the behavior of the 429 —the failure mode that makes the breaker over a model more than a latency optimization—. Conceptual, without pinning a version. In English.
- Martin Fowler, "CircuitBreaker" — martinfowler.com/bliki/CircuitBreaker.html. The classic article that gives the pattern its name and shape; useful for the mental model of the states. In English.
- Chip Huyen, AI Engineering (O'Reilly, 2024). The chapters on reliability and inference cost cover provider quotas and the handling of a model's availability failures as design properties. In English.