Module 5: Failure Modes and Resilience for AI
The model down, slow, or rate-limited
Overview
When you placed the LLM behind a boundary (module 1), that boundary hid something we now have to look at head-on: on the other side lives an external API you don't control, served by a third party, with its own availability, its own variable latency, and its own quotas. A component that runs inside your process fails rarely and fails fast; a component that lives behind a network call to a model provider fails in three new ways you have to design explicitly: down (the API doesn't respond, returns a 5xx), slow (it takes longer than your budget, and if you do nothing, your system keeps waiting), and rate-limited (you exhausted your token or request quota, and the API returns a 429). This lesson installs this family's first defense, the most basic and the most forgotten: the timeout —don't wait forever—, and explains why depending on a slow, quota-bound model API changes the design.
In lesson 2 you classified these three as noisy failures —they throw an exception or a timeout turns them into one—. In lesson 3 you saw the silent failure (hallucination). Here we go back to the noisy front with the most elementary of problems: the wait. You're going to see, executed, a flow of requests where some model calls hang (30 seconds, 15 seconds) or fail immediately (down, rate-limited), and you're going to measure how an 800 ms timeout transforms a system that blocks for 48 seconds into one that waits 3.5 seconds and always responds via fallback.
Connection with the module. Lesson 1 showed the complete shell; this one isolates its most basic piece —the timeout— and the problem it solves —the dependency on a slow, quota-bound API—. It's the foundation of lessons 5 and 6: the timeout turns a hang (which throws nothing) into a catchable noisy failure, and only then can the fallback (L5) react and the circuit breaker (L6) count failures. Without a timeout, the other defenses have nothing to grab onto. The boundary with the resilience guide is hard: the mechanics of the timeout —how it's really implemented with threads, with async, with signals, how the value is chosen, how it interacts with retries— is taught in resilience-and-reliability-patterns-guide M2 (timeouts); here we apply it to the AI component and refer there for the detail.
An analogy: the phone that rings and rings
Imagine you call a supplier to confirm an urgent order. You dial and the phone rings. It rings once, twice, five, ten times. Nobody answers. How long do you keep listening to the tone before hanging up and looking for another way to solve it? A sensible person hangs up after a reasonable while —thirty seconds, a minute— and tries something else: calls another supplier, sends an email, resolves it on their own. What nobody does is stay with the phone glued to their ear forever, listening to the tone, blocked, doing nothing else, waiting for maybe someone to answer someday. That would be absurd: while you wait on hold, you serve nobody else, you make no progress on anything, your whole day stops for a call that didn't go through.
A system without a timeout does exactly that: it keeps the phone glued to its ear. When it calls the model and the model doesn't respond —it hung, the network dropped, the provider is saturated—, the thread that made the call keeps waiting indefinitely. And while it waits, that thread serves no other requests; if you have many requests waiting on a hung model, you run out of threads, the queue fills up, and a working service falls in cascade —not because it was broken, but because all its resources are waiting on hold for a call that's never going to go through—.
The timeout is the "hang up after X" rule. You tell the system: "wait for the model at most 800 milliseconds; if it didn't answer, hang up and solve it another way." It turns an infinite wait into a bounded one, and —this is key— it turns a silent hang (the ringing phone, which is neither success nor error, just waiting) into a noisy failure you can catch and handle (a TimeoutError that triggers the fallback). The timeout doesn't make the model answer faster; it makes you stop waiting for it when it no longer makes sense, so you can serve the customer with the alternate route. In Mercado, it's the difference between a search that freezes for twenty seconds because the model hung, and one that waits 800 ms, hangs up, and gives you keyword results.
Worked example: the timeout that cuts the wait
We're going to measure the timeout's effect. We have a flow of 10 model requests. The stub reports, for each one, how long it would have taken (its duration_ms) and whether it fails immediately (down, rate_limited). Two of the calls are hangs: one of 30 seconds (an API that doesn't respond) and another of 15 seconds. Another is simply slow (2.5 s, over the budget). The rest respond fast.
We compare two designs:
- Without a timeout: we wait the full duration of each call. A 30 s hang blocks us for 30 s.
- With an 800 ms timeout: if the call was going to take longer than the budget, we cut it at 800 ms, throw
ModelTimeout, and fall to the fallback. We never wait more than the budget.
Important for reproducibility: the stub doesn't actually sleep —there's no sleep, no network—; it reports a simulated duration, and the timeout is modeled as "if the simulated duration exceeds the budget, it's cut to budget." That way the experiment is deterministic and measures exactly what we want.
# Lesson 4: the model DOWN / SLOW / RATE-LIMITED. Depending on a slow
# external API with quotas. The TIMEOUT avoids waiting forever.
# The LLM is SIMULATED: each request carries a simulated duration and/or a failure;
# there's NO network or real sleeps. The mechanics of the timeout in depth:
# resilience-and-reliability-patterns-guide M2.
LATENCY_BUDGET_MS = 800 # budget: we don't wait more than this
class ModelDown(Exception): pass
class RateLimited(Exception): pass
class ModelTimeout(Exception): pass
# Each request: (kind, simulated_duration_ms). The stub does NOT sleep; it reports
# how long it "would have taken". That way we measure the timeout deterministically.
REQUESTS = [
("ok", 120),
("ok", 340),
("slow", 30000), # a hang: 30 s (an API that doesn't respond)
("rate_limited", 0),
("ok", 210),
("slow", 2500), # slow: exceeds the budget
("down", 0),
("ok", 180),
("ok", 260),
("slow", 15000), # another hang
]
def call_model(kind, duration_ms):
if kind == "down": raise ModelDown()
if kind == "rate_limited": raise RateLimited()
return f"response ({duration_ms} ms)"
def fallback():
return "deterministic response (fallback)"
# --- Mode A: WITHOUT timeout. We wait the full duration of each call. ---
wait_A = 0
worst_A = 0
for kind, dur in REQUESTS:
try:
# Without a timeout, a hang makes us wait its FULL duration.
if kind in ("down", "rate_limited"):
call_model(kind, dur) # fails immediately (0 ms)
else:
wait_A += dur
worst_A = max(worst_A, dur)
except (ModelDown, RateLimited):
fallback()
# --- Mode B: WITH timeout = LATENCY_BUDGET_MS. We cut what exceeds it. ---
wait_B = 0
worst_B = 0
served_B = 0
print(f"{'req':<5}{'kind':<14}{'expected(ms)':<14}{'served by':<13}result")
print("-" * 74)
for i, (kind, dur) in enumerate(REQUESTS):
try:
if kind in ("down", "rate_limited"):
call_model(kind, dur) # throws immediately
waited = 0
result = "ok"
served = "model"
elif dur > LATENCY_BUDGET_MS:
waited = LATENCY_BUDGET_MS # we wait only the budget
raise ModelTimeout(f"cut at {LATENCY_BUDGET_MS} (would have taken {dur})")
else:
waited = dur
call_model(kind, dur)
result = "ok"
served = "model"
except ModelDown:
waited = 0; result = "down"; served = "fallback"
except RateLimited:
waited = 0; result = "rate_limited"; served = "fallback"
except ModelTimeout as e:
result = str(e); served = "fallback"
wait_B += waited
worst_B = max(worst_B, waited)
served_B += 1
print(f"{i:<5}{kind:<14}{dur:<14}{served:<13}{result}")
print("-" * 74)
print(f"All responded: {served_B}/{len(REQUESTS)} (the failed ones, via fallback).")
print()
print("Total simulated wait (the sum of what the system was blocked):")
print(f" WITHOUT timeout : {wait_A:>6} ms (worst case: {worst_A} ms blocked)")
print(f" WITH timeout : {wait_B:>6} ms (worst case: {worst_B} ms blocked)")
print(f" the timeout cut {wait_A - wait_B} ms of waiting and bounded the worst "
f"case from {worst_A} to {worst_B} ms.")What to expect. When you run the file, the output is exactly this:
req kind expected(ms) served by result
--------------------------------------------------------------------------
0 ok 120 model ok
1 ok 340 model ok
2 slow 30000 fallback cut at 800 (would have taken 30000)
3 rate_limited 0 fallback rate_limited
4 ok 210 model ok
5 slow 2500 fallback cut at 800 (would have taken 2500)
6 down 0 fallback down
7 ok 180 model ok
8 ok 260 model ok
9 slow 15000 fallback cut at 800 (would have taken 15000)
--------------------------------------------------------------------------
All responded: 10/10 (the failed ones, via fallback).
Total simulated wait (the sum of what the system was blocked):
WITHOUT timeout : 48610 ms (worst case: 30000 ms blocked)
WITH timeout : 3510 ms (worst case: 800 ms blocked)
the timeout cut 45100 ms of waiting and bounded the worst case from 30000 to 800 ms.
Read the numbers, because the timeout's effect is dramatic and concrete.
The total wait: from 48.6 seconds to 3.5 seconds. Without a timeout, the system was blocked for 48610 ms —almost 49 seconds— summing the waits of the ten calls, dominated by the two hangs (30000 + 15000 = 45 seconds on just those two). With a timeout, the total wait dropped to 3510 ms. The timeout cut 45100 ms of useless waiting: the seconds the system would have spent with the phone glued to its ear listening to the tone. That time isn't abstract: it's time server threads are blocked serving nobody else.
The worst case: from 30 seconds to 800 ms. Just as important as the total is the individual worst case. Without a timeout, a single bad request (the 30 s hang) blocked the system for half a minute —imagine a Mercado customer waiting half a minute for a search to load—. With a timeout, no request blocked more than 800 ms, the budget. The timeout doesn't just lower the average; it bounds the maximum, which is what keeps a one-off hang from dragging down everyone's experience.
And everyone responded. Look at the last column of the table: the four requests that failed (the two hangs, the down, the rate_limited) were served via fallback. The system didn't fall on any of them; it cut the wait and responded with the deterministic route. The table shows you the complete pattern per request: the ok ones were served by the model (waiting their real duration, all under budget), and the failed ones were served by the fallback after cutting. 10 of 10 responded. The timeout made the fallback possible: without cutting the wait, the system would still be hung on request 2 and would never have reached the fallback.
The architectural implication: the timeout is the piece that turns a hang into something manageable. A hang, on its own, is neither success nor error: it's a wait that doesn't end, and against that no other defense works —the fallback doesn't fire because there's no exception, the breaker doesn't count a failure because there's no failure, just waiting—. The timeout puts a limit on the wait and, when it expires, throws a ModelTimeout that is a noisy failure, and then the fallback reacts. That's why the timeout goes first.
Going deeper: why a model API is an especially fragile dependency
The three availability failures, with their AI nuance. All three resemble those of any network dependency, but each has a different weight when the dependency is a model:
- Down (5xx / no response). The provider has an outage, or the network between you and it fails. Just like any down service. The defense: timeout + fallback + (if it repeats) circuit breaker.
- Slow. Here's the big nuance: for an LLM, slowness is the normal case, not the exception. As you saw in module 2, a model call takes from hundreds of milliseconds to several seconds when everything is going well. That means your latency budget and your timeout aren't insurance against oddities; they're a first-class constraint that coexists with normal operation. Choosing the timeout value is a real trade-off: too short and you cut legitimate calls that were going to answer; too long and you tolerate hangs that degrade the experience. (The resilience guide, M2, treats how to choose that value; it's often anchored to a high percentile of the observed latency plus a margin.)
- Rate-limited (429). This is more central than in a classic service. Model providers impose strict quotas —per tokens per minute, per requests per minute—, and when you cross them, they deny you service with a 429. The treacherous part: the rate limit appears exactly when you have the most traffic (a sales spike, a campaign), that is, at the worst moment. And —lesson 6 develops this— blindly retrying a 429 makes it worse: each retry counts against the quota and prolongs the block. That's why the rate limit isn't solved by retrying; it's solved by stopping the calls (breaker) and serving the fallback.
The cost per call gives the timeout extra weight. In a classic service, waiting too long "only" costs latency. In a model, every call that does complete costs money (tokens). This has a subtle consequence: a timeout that cuts a call that was going to complete saves you the wait, but if the provider already processed the tokens, you maybe already paid. The fine design of the timeout with models considers this; for this guide, it's enough to keep in mind that the timeout protects your latency and your availability, and that cost handling is the theme of module 2 (budgets) and of lesson 6 (breaker).
The infinite timeout is an antipattern, not an innocent default. Here's a library trap. Many API clients ship with a very high default timeout (30, 60 seconds) or directly none. A developer who doesn't configure the timeout explicitly is choosing the infinite timeout without knowing it, and inherits all the provider's hangs as their own hangs. The hard rule: every call to an external dependency carries an explicit timeout, chosen from your latency budget, not the library's default. In an AI component this is even more important because the normal latency is already high and variable, so the boundary between "normal slow" and "hung" has to be defined on purpose.
The timeout enables the other defenses. Keep the dependency chain, because it orders the module:
call to the model
│
▼
┌──────────┐ limit expires ┌──────────────┐
│ TIMEOUT │──────────────────► │ ModelTimeout │ (now it's NOISY)
│ (L4) │ │ (exception) │
└──────────┘ └──────────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
FALLBACK (L5) counts toward (if it repeats)
responds with the breaker opens the breaker (L6)
alternate route (L6)
The timeout is the base; the fallback and the breaker are built on top. Without a timeout, a hang never becomes an exception, and neither the fallback nor the breaker learns that there was a problem. That's why this lesson goes before the other two.
Common mistakes
Infinite timeout (or the library's default). What happens: the team calls the model without configuring the timeout, inherits the HTTP client's 60-second default, and the day the provider hangs, every Mercado search request freezes for a minute —the threads run out and the whole service degrades—. Why it happens: the timeout isn't seen in development (the model always answered fast), so nobody configured it, and the default is dangerously high or nonexistent. How to spot it: search your code for the model call; if there's no explicit timeout= anchored to your budget, you have the antipattern. How to fix it: put an explicit timeout, derived from your latency budget (module 2), not the default. The example measures it: without a timeout, 48 s of waiting; with an 800 ms timeout, 3.5 s.
Confusing "slow" with "broken" and killing legitimate calls. What happens: the team, chastened by the hangs, puts an aggressive 200 ms timeout —but the model's normal latency is 400-600 ms—, so it cuts most of the good calls and serves fallback almost always, degrading quality needlessly. Why it happens: the timeout is chosen without looking at the model's real latency distribution, forgetting that for an LLM slow is normal. How to spot it: your timeout rate is high even when the provider is healthy. How to fix it: anchor the timeout to a high percentile of the observed latency (p95/p99) plus a margin, not a random number; the timeout must distinguish "hung" from "normal slow." The resilience guide (M2) treats how to choose that value.
Retrying a rate limit immediately. What happens: the API returns 429, and the code retries immediately, and again, and again —each retry counts against the exhausted quota and prolongs the block, turning a short rate limit into a self-perpetuating storm of retries—. Why it happens: the 429 is treated like any other transient error ("retrying usually works"), without seeing that this error gets worse by retrying. How to spot it: your 429 handling is an immediate retry with no backoff or breaker. How to fix it: on a 429, don't retry immediately —use backoff (resilience guide M3) and, if it persists, open the circuit breaker and serve the fallback (lesson 6)—. The rate limit is solved by stopping the calls, not calling more.
Exercises
Exercise 1 — Choose the timeout. Mercado's search latency budget is 1 second total (module 2), of which the rest of the pipeline (parsing, ranking, render) consumes ~300 ms, leaving ~700 ms for the model call. The model's latency, measured in production, is: p50 = 250 ms, p95 = 600 ms, p99 = 1200 ms. Propose a timeout value for the model call and justify it. What happens with the p99 calls?
See solution
A reasonable timeout would be ~700 ms —the budget available for the call—, which is also above the p95 (600 ms). With that, you cut below your total budget (you respect the 1 s SLA), and you let 95% of the legitimate calls through (the ones that take up to 600 ms).
The p99 (1200 ms) calls would be cut: they exceed the 700 ms budget, so they become ModelTimeout and are served via fallback. And that's fine: they're calls that, if completed, would break your total latency budget (1200 ms of model + 300 ms of pipeline = 1500 ms, over the promised second). It's preferable to serve a fast fallback than to make the customer wait 1.5 s. The ~1% of the p99 is degraded in exchange for protecting the remaining 99%'s budget.
The trade-off: if you raised the timeout to 1200 ms to "reach" the p99, you'd stop cutting those calls but you'd break the total budget and make everyone wait longer. If you lowered it to 300 ms, you'd cut a large part of the legitimate p95 and serve fallback more than needed. ~700 ms balances: it respects the budget and saves the vast majority of the good calls. (The resilience guide, M2, formalizes this choice.)
Exercise 2 — Why the timeout goes before the fallback. A colleague proposes: "let's just put the fallback; if the model fails, we fall to keywords, we don't need a timeout." Explain with this lesson's example why the fallback without a timeout doesn't protect against a hang, and what would happen exactly in request 2 (the 30 s hang) with a fallback but no timeout.
See solution
The fallback fires when the model call throws an exception (or returns an error). But a hang throws nothing: the call simply doesn't return, it keeps waiting. The fallback is written as an except, and that except never fires because there's no exception —there's only waiting—.
In request 2 (the 30 s hang) with a fallback but no timeout: the call_model call would keep waiting for the model's response indefinitely (in the real world, until the socket timeout, which can be minutes, or forever). The thread serving that request stays blocked. The except block that would lead to the fallback never runs, because no exception was thrown: there wasn't an error, there was a wait. The customer sees the search frozen; the thread serves nobody else.
The timeout is what fixes this: when the 800 ms expire, it throws a ModelTimeout —it turns the silent wait into a noisy failure—, and then the except ModelTimeout fires and the fallback responds. That's why the order is timeout → fallback: the timeout creates the exception the fallback needs to react. The fallback without a timeout is an extinguisher with nobody to pull the alarm: it's there, but nothing triggers it.
Exercise 3 — The three failures and their response. For each of the three availability failures, say (a) how the call manifests (what the API returns or does), (b) why retrying immediately is a good or bad idea, and (c) this module's defense: (1) the model is down (503); (2) the model is rate-limited (429); (3) the model is slow (takes 8 s).
See solution
- (1) Down (503): (a) the API returns an explicit 503 error immediately. (b) Retrying can help if the outage is a transient blip (a retry with backoff sometimes lands at a moment where the service already recovered), but if the outage is prolonged, retrying only adds latency. (c) Defense: fallback immediately to respond, and circuit breaker if it repeats to stop trying (lesson 6). A retry with backoff is acceptable (resilience guide M3), but don't block the customer waiting for it: serve them the fallback.
- (2) Rate-limited (429): (a) the API returns 429 immediately. (b) Retrying immediately is a bad idea: each retry counts against the exhausted quota and prolongs the block. (c) Defense: don't retry immediately; open the circuit breaker and serve the fallback until the quota window frees up (lesson 6). The rate limit is solved by stopping the calls.
- (3) Slow (8 s): (a) the call returns nothing; it keeps waiting (a hang). (b) "Retrying" doesn't even apply until you decide to stop waiting; on its own, you wait 8 s. (c) Defense: timeout —you cut at your budget (e.g. 800 ms), turn it into
ModelTimeout, and fall to the fallback—. This lesson. The timeout is the specific defense for slowness, because it's the only failure that doesn't announce itself.
Summary and next step
In this lesson you installed the availability family's first defense: the timeout —don't wait forever for an API that hung—, and you understood why depending on a slow, quota-bound model API is an especially fragile dependency. You saw it with the phone that rings and rings while your day stops, and you measured it: with an 800 ms timeout, the system's total wait dropped from 48610 ms to 3510 ms, the worst case from 30000 ms to 800 ms, and the 10 calls responded —the failed ones, via fallback—. And you saw the key design piece: the timeout turns a silent hang into a noisy failure the fallback and the breaker can handle, that's why it goes first. You kept the three availability failures —down, slow, rate-limited— with their AI nuance: for a model, slow is normal, the rate limit is central and treacherous, and retrying a 429 makes it worse.
Before moving on you should be able to: explain why the infinite timeout (or the library's default) is an antipattern; choose a timeout value from the latency budget; argue why the fallback without a timeout doesn't protect against a hang; and distinguish the right response to a down, a rate-limited, and a slow. The mechanics of the timeout in depth, remember, live in resilience-and-reliability-patterns-guide M2.
Lesson 5 takes the route the timeout enables and develops it: the fallback and graceful degradation. You're going to see, executed, Mercado's semantic search falling to a cascade —semantic → cache → keywords— when the model fails, with availability rising from 70% to 100% and the degraded responses marked for what they are: worse, but valid. The way to make "the model fell" stop meaning "the system fell" and start meaning "the system responded a bit worse."
Resources
resilience-and-reliability-patterns-guide(this ecosystem), M2 "Timeouts" — the central reference for the mechanics of the timeout we only apply here: how to implement it with threads or async, how to choose the value from latency percentiles, how it interacts with retries and with the total budget. In Spanish.- Anthropic, Claude documentation — docs.anthropic.com. See the rate limits pages (quotas per tokens and requests, the 429) and errors pages to understand the availability failure modes of an API-served model, at a conceptual level and without fixing a version. In English.
architecture-for-ai-native-systems-guide, Module 2 (latency and cost as architecture) — the latency budget from which the timeout value is derived, and why the model's slowness is a first-class constraint. In Spanish.- Chip Huyen, AI Engineering (O'Reilly, 2024). The chapters on latency, cost, and inference reliability treat the dependency on a model API with its quotas and variable latency as a design property. In English.