Module 5: Failure Modes and Resilience for AI
Project: make a Mercado AI feature resilient
Overview
This is the module's capstone. In the seven previous lessons you built, piece by piece, the defense against each failure mode of an AI component: you saw the taxonomy (L2), contained hallucination with verification (L3), put the timeout against the down/slow/rate-limited model (L4), degraded with a fallback cascade (L5), avoided waste with the circuit breaker (L6), and monitored drift (L7). Now you integrate them all into a single real Mercado feature and execute it end to end. You take the semantic search and build it the complete resilience shell —timeout + circuit breaker + cascading fallback + degradation—, measure its availability with and without the shell, and document the decision in an ADR. When you finish, you'll have the artifact that demonstrates the module's capability: an AI feature whose model failure doesn't take the system down, with the number that proves it.
Connection with the module. This lesson introduces no new ideas; it synthesizes the previous six into an executed system. It's the counterpart of the module 4 project (where you built the trust boundary) and anticipates the module 8 capstone (where you'll architect a complete AI feature, with this module's resilience as one of its faces). The boundary with the resilience guide holds: the shell you build applies timeout, breaker, and degradation to the model; its mechanics in depth are in resilience-and-reliability-patterns-guide. Here you demonstrate that you know how to place and compose those pieces around an AI component, measure their effect, and justify the decision.
The assignment
You're the architect of Mercado's search. The semantic search —the LLM that understands the intent of a query like "headphones for running in the rain"— launches to production next week. The infrastructure team warns you: the model API has a historical availability of ~92%, with occasional outages of several minutes and rate-limit spikes during peak-traffic hours (which coincide with peak-sales hours). Your boss gives you the assignment in one sentence: "search can't go down when the model goes down".
Your deliverable has four pieces:
- The diagram of the feature's resilience shell.
- The executed code of the shell (timeout + circuit breaker + cascading fallback + degradation), with the literal output.
- An ADR (Architecture Decision Record) that documents the decision.
- The justification of why, with this shell, the model failure doesn't take the system down.
The shell diagram
Before the code, the map. The resilience shell surrounds the AI component (the LLM behind its boundary) with the three availability pieces, and behind it has the fallback cascade:
flowchart TD
Q["user query"] --> CB{"circuit breaker<br/>open?"}
CB -- "OPEN: don't call" --> FB
CB -- "CLOSED/HALF_OPEN" --> TO["call the model<br/>with TIMEOUT"]
TO -- "responds in time" --> OK["semantic result<br/>(optimal)"]
TO -- "fails / timeout / 429" --> REG["record failure<br/>in the breaker"]
REG --> FB["FALLBACK cascade"]
FB --> C1{"in cache?"}
C1 -- "yes" --> CACHE["cached result<br/>(degraded)"]
C1 -- "no" --> KW["keyword search<br/>(degraded, final net)"]
OK --> RESP["response to the user"]
CACHE --> RESP
KW --> RESP
MON["drift monitor<br/>(continuous eval)"] -.observes.-> RESP
Read it top to bottom: each query first passes through the circuit breaker (is it open because the model has been failing?). If it's open, it goes straight to the fallback without touching the model. If it's closed (or probing), it calls the model with a timeout. If the model responds in time, optimal result. If it fails, slips past the timeout, or returns 429, the failure is recorded in the breaker and it drops to the fallback cascade: first cache, and if there's none, keywords (the final net that never fails). The drift monitor observes the quality of the responses over time, outside the request path. Each piece is a lesson from the module; the diagram is the whole module composed.
The executed code
We build the shell over the simulated model. The stream has 30 requests with a model outage from request 12 to 20 (nine requests) and two hangs (requests 5 and 25, which exceed the timeout). We measure availability with the shell (timeout + breaker + fallback) against without it (semantic only). The LLM is simulated by the stub, with no network or real API.
# Lesson 8 (project): Mercado's semantic search made RESILIENT.
# Joins timeout + circuit breaker + cascading fallback + degradation, all
# over the SIMULATED AI component (no network, no API). We measure the
# system's availability WITH vs WITHOUT the resilience shell.
LATENCY_BUDGET_MS = 800
class ModelError(Exception): pass
class ModelTimeout(Exception): pass
CATALOG = [
(1, "wireless noise-cancelling headphones"),
(2, "programmable drip coffee maker"),
(3, "waterproof laptop backpack"),
(4, "backlit mechanical keyboard"),
(5, "water-resistant sports headphones"),
]
CACHE = {"headphones": [1, 5], "coffee maker": [2]}
# Stream of 30 requests. Each one: (query, kind, duration_ms). The model goes
# down in a window (12..20) and there are two slow calls that exceed the timeout.
import itertools
_QUERIES = ["headphones", "coffee maker", "backpack", "keyboard", "headphones water"]
_q = itertools.cycle(_QUERIES)
STREAM = []
for i in range(30):
q = next(_q)
if 12 <= i <= 20:
STREAM.append((q, "down", 0)) # model outage
elif i in (5, 25):
STREAM.append((q, "slow", 9000)) # hang: exceeds the budget
else:
STREAM.append((q, "ok", 150))
def semantic_search(query, kind, dur):
if kind == "down":
raise ModelError("model down")
if dur > LATENCY_BUDGET_MS:
raise ModelTimeout("exceeds the latency budget")
words = query.split()
return [pid for pid, name in CATALOG if any(w in name for w in words)]
def keyword_search(query):
words = query.split()
return [pid for pid, name in CATALOG if any(w in name for w in words)]
class CircuitBreaker:
def __init__(self, fail_threshold=3, cooldown=4):
self.fail_threshold = fail_threshold
self.cooldown = cooldown
self.fails = 0
self.state = "CLOSED"
self.opened_at = None
def allow(self, now):
if self.state == "OPEN":
if now - self.opened_at >= self.cooldown:
self.state = "HALF_OPEN"
return True
return False
return True
def on_success(self):
self.fails = 0
self.state = "CLOSED"
def on_failure(self, now):
self.fails += 1
if self.fails >= self.fail_threshold:
self.state = "OPEN"
self.opened_at = now
def resilient(i, query, kind, dur, cb):
# Shell: breaker -> (timeout+model) -> fallback cascade (cache/keyword).
if cb.allow(i):
try:
hits = semantic_search(query, kind, dur)
cb.on_success()
return ("semantic", hits, False, "model")
except (ModelError, ModelTimeout):
cb.on_failure(i) # falls to the fallback cascade
else:
pass # breaker OPEN: we don't even touch the model
if query in CACHE:
return ("cache", CACHE[query], True, "fallback")
return ("keyword", keyword_search(query), True, "fallback")
# --- WITHOUT shell: semantic only; if the model fails, the request drops ---
responded_A = 0
for i, (q, kind, dur) in enumerate(STREAM):
try:
semantic_search(q, kind, dur)
responded_A += 1
except (ModelError, ModelTimeout):
pass
avail_A = responded_A / len(STREAM) * 100
# --- WITH shell ---
cb = CircuitBreaker(fail_threshold=3, cooldown=4)
tiers = {"semantic": 0, "cache": 0, "keyword": 0}
opened_at_req = None
for i, (q, kind, dur) in enumerate(STREAM):
tier, hits, degraded, _by = resilient(i, q, kind, dur, cb)
tiers[tier] += 1
if opened_at_req is None and cb.state == "OPEN":
opened_at_req = i
responded_B = len(STREAM)
avail_B = responded_B / len(STREAM) * 100
degraded_total = tiers["cache"] + tiers["keyword"]
print("Feature: Mercado's semantic search with a resilience shell")
print(f" stream of {len(STREAM)} requests; model down in 12..20, "
f"slow in 5 and 25")
print()
print("System availability:")
print(f" WITHOUT shell (semantic only) : {responded_A}/{len(STREAM)} = {avail_A:.1f}%")
print(f" WITH shell (breaker+timeout+fallback) : "
f"{responded_B}/{len(STREAM)} = {avail_B:.1f}%")
print()
print("How the 30 requests were served WITH the shell:")
print(f" optimal (semantic) : {tiers['semantic']}")
print(f" degraded (cache) : {tiers['cache']}")
print(f" degraded (keyword) : {tiers['keyword']}")
print(f" total degraded (worse, but valid): {degraded_total}")
print(f" the breaker opened at request {opened_at_req} and stopped hitting "
f"the downed model.")What to expect. When you run the file, the output is exactly this:
Feature: Mercado's semantic search with a resilience shell
stream of 30 requests; model down in 12..20, slow in 5 and 25
System availability:
WITHOUT shell (semantic only) : 19/30 = 63.3%
WITH shell (breaker+timeout+fallback) : 30/30 = 100.0%
How the 30 requests were served WITH the shell:
optimal (semantic) : 18
degraded (cache) : 6
degraded (keyword) : 6
total degraded (worse, but valid): 12
the breaker opened at request 14 and stopped hitting the downed model.
Read the numbers, because they're the demonstration of the assignment fulfilled.
Availability: from 63.3% to 100%. With the model outage (9 requests) and the two hangs, the search without a shell answered only 19 of 30 = 63.3% —more than a third of users would have seen search broken during the incident—. With the shell, it answered 30 of 30 = 100%: no one was left without results. Your boss's assignment —"search can't go down when the model goes down"— is fulfilled and measured. And as throughout the module, the key is that the model has the same (bad) availability in both cases; the shell made the system stop depending on it to answer.
The breakdown by tier shows the honest degradation. Of the 30 requests, 18 were served optimally (semantic, when the model was healthy) and 12 degraded —6 by cache, 6 by keyword—. Those 12 are the requests during the outage and the hangs: they were served worse (without semantic understanding) but valid (with relevant products), instead of failing. Without the shell, those 12 would have been 11 errors (19 answered, not 18, because without a breaker each request during the outage tries and fails individually). The feature degrades gracefully: worse, not broken.
The breaker opened at request 14. The outage begins at 12; the breaker counts 3 failures (12, 13, 14) and opens at 14, stopping the hits to the downed model for the rest of the window —going straight to the fallback and saving timeouts and calls—, with periodic probe attempts until the model recovers past request 20. It's the full breaker lifecycle from lesson 6, now inside the complete feature.
The demonstration: every piece of the module did its part, and together they turned a 63%-available feature into a 100%-available one. The timeout caught the hangs (5 and 25); the breaker stopped hammering the downed model (from 14); the fallback cascade served cache and keywords when the model wasn't there; the degradation marked those 12 responses as what they are. The AI component's non-determinism and fragility were contained by a deterministic shell.
The decision ADR
An ADR (Architecture Decision Record) documents an architectural decision: its context, the decision made, the alternatives, and the consequences. This is the ADR of your deliverable:
# ADR-005: Resilience shell for the semantic search
## Status
Accepted
## Context
Mercado's semantic search depends on a model (LLM) API with ~92% historical
availability, with outages of several minutes and rate-limit spikes during
high-traffic hours (which coincide with high-sales hours). Without
protection, the search's availability would be, at best, the model's
(~92%), and a model outage would leave the search completely non-functional,
with direct loss of traffic and sales. Search is a critical business path.
## Decision
Wrap the model call in a RESILIENCE SHELL with four pieces, applied to the
AI component:
- TIMEOUT (800 ms, derived from the module 2 latency budget): no query
waits for the model longer than the budget.
- CIRCUIT BREAKER over the model (threshold 3 failures, cooldown): stops
calling the model when it has been failing, so as not to waste
latency/cost or make a rate limit worse.
- CASCADING FALLBACK: semantic -> cache -> keywords. The final tier
(keywords) is deterministic and local: it doesn't share the fragile
dependency (the model API), so it never fails.
- MARKED DEGRADATION: the fallback responses are labeled as degraded,
for the monitoring and (optionally) for the user.
Also, a DRIFT MONITOR runs the eval in production continuously to detect
silent quality degradation (module 3 + 7).
The MECHANICS of timeout/breaker/degradation are taken from the resilience
guide; here they are applied to the AI component.
## Alternatives considered
- Direct model call without protection: REJECTED. Search availability =
model availability (~63% measured in the simulated incident); search
goes down with the model. Fails the requirement.
- Fallback only, without timeout or breaker: REJECTED. The fallback
without a timeout doesn't protect against hangs (it freezes); without a
breaker, it wastes timeout/cost/quota on every request during a long
outage.
- Fallback to a smaller model from the same provider as the final tier:
REJECTED as the FINAL TIER. It shares the fragile dependency (the
provider's API); if the entire provider goes down, it goes down with it.
Acceptable as an INTERMEDIATE tier, but the final tier must be
deterministic and local.
## Consequences
+ The search's availability DECOUPLES from the model's:
100% measured vs 63.3% without a shell, in the face of an outage + two hangs.
+ During an outage, the search degrades (keywords/cache) instead of going down.
+ The breaker saves latency, cost, and quota during the outages.
- During degradation, the search quality drops (without semantic
understanding); mitigated by marking and monitoring the degraded proportion.
- Extra complexity: you have to maintain the keyword fallback, the cache,
and calibrate timeout/threshold/cooldown (see resilience guide).
- The drift monitor requires running the eval in production continuously
(operational cost, justified by catching drift early).
The justification
The question that closes the module: why, with this shell, does the model failure no longer take the system down? The answer has three layers, and it's worth stating them explicitly because they're the module's thesis made argument.
First: the system's availability decoupled from the model's. The system's most fragile component —the model, ~92% available, with outages— stopped being the one that determines whether search works. This is achieved by the fallback with a deterministic and local final tier (in-memory keywords), which doesn't share the fragile dependency and therefore responds even if the entire provider goes down. The number proves it: 100% of system over a model that was at 63% during the incident. Resilience didn't eliminate the model failure (the model went down all the same); it decoupled the system from that failure.
Second: failures are contained in the moment and over time. At the moment of the request, the timeout prevents a hang from freezing the system, the breaker prevents a long outage from wasting resources, and the cascade provides the alternate route. Over time, the drift monitor catches the silent degradation that no per-request defense can see. The two dimensions of failure —instantaneous (down/slow/rate-limited/hallucination) and temporal (drift)— are covered.
Third: the degradation is honest and chosen. The system doesn't pretend everything is fine during an outage: it serves degraded responses marked as such, monitors the degraded proportion, and treats degradation as a deliberate design decision —"I'd rather serve worse than not serve"— not as an accident. That's what separates a resilient system from one that simply hides its failures: the resilient one knows when it's degraded and says so.
The synthesis, which links to module 6: the AI component's non-determinism and fragility were contained by a deterministic shell. The LLM proposes the best response when it can; when it can't —because it went down, hung, or is being rate-limited—, a deterministic layer arranges the alternate route. The probabilistic and fragile core was kept small, wrapped, and contained. That's the architecture of an AI-native system that survives its own AI pieces.
Common mistakes
Composing the pieces in the wrong order. What happens: the team puts the fallback but calls the model without a timeout, or puts the breaker after an infinite timeout; the system has all the pieces but poorly ordered, and a hang freezes it all the same because the timeout —which should go first, turning the hang into an exception— isn't there or is poorly placed. Why it happens: the pieces are joined without respecting their interdependence (lesson 6, exercise 3). How to detect it: check that the timeout wraps every call, that the breaker counts the failures the timeout generates, and that the fallback responds when either of the two cuts. How to fix it: the order is breaker (do I call?) → timeout (I call, bounded) → recording the failure → fallback cascade. The example composes it this way.
A fallback that shares the fragile dependency. What happens: the final tier of the cascade calls another external API (or the same provider), and on the day of a full-provider outage, everything goes down together. Why it happens: the fallback was thought of as "another way to solve it" without verifying that it's independent from the failure it covers. How to detect it: your final fallback tier has a network or provider dependency. How to fix it: the final tier must be deterministic and local (in-memory keywords, template, local cache) —the safety net that survives the outage it covers—. The ADR marks it as a rejected alternative.
Delivering the shell without measuring its effect. What happens: the team builds timeout + breaker + fallback, it looks good, and deploys it without ever measuring the availability with and without the shell —so it doesn't know whether it really works nor does it have the number to justify the extra complexity—. Why it happens: "it looks resilient" is confused with "it is resilient". How to detect it: you can't cite your availability with vs without the shell against a simulated incident. How to fix it: simulate the failure and measure —like the example: 63.3% without a shell, 100% with— because the resilience you didn't measure is a hypothesis, not a property. The number is what turns the shell into a defensible decision in the ADR.
Exercises
Exercise 1 — Adapt the shell to the support agent. The project built the shell for the semantic search. Adapt it to Mercado's support agent (which answers customer questions and sometimes touches order data). Say what changes in each piece: (a) the final tier of the fallback; (b) what additional defense the agent needs that search doesn't; (c) why the timeout and the breaker apply the same.
See solution
- (a) The final tier of the fallback: in search it was keywords (deterministic, local). In the support agent, the final tier is escalate to a human —the human agent is the universal safety net that resolves any case—. The cascade would be: LLM agent → template responses for frequent questions → human. The last resort is human, not deterministic, because support touches cases (money, complaints) where a human is the only guarantee (lesson 5, exercise 1).
- (b) The additional defense the agent needs: the verification against the source of truth (lesson 3), because the agent asserts factual data (order status, tracking number) that it may hallucinate. Search returns products (verifiable by "does it exist/is it active?", a light verification), but the agent asserts facts about orders that must be confronted against the real record. Without that verification, the agent would serve hallucinations even if the model were 100% available —it's a content failure, not an availability one—. The availability shell (timeout/breaker/fallback) doesn't cover hallucination; the agent needs both defenses.
- (c) Timeout and breaker apply the same: because the agent also depends on the same slow, quota-limited model API. A down/slow/rate-limited model affects the agent the same as search, so the timeout (don't wait forever) and the breaker (stop hitting a downed model) are identical. What changes is the destination of the fallback (human instead of keywords) and the extra defense (content verification). The availability pieces are the same; the fallback content and the content defenses change according to the feature.
Exercise 2 — The number that justifies the complexity. Your boss asks: "this shell adds complexity —you have to maintain the fallback, the cache, calibrate the breaker—; is it worth it?". Use the project's numbers and a business estimate to justify (or qualify) the decision. Suppose search generates 100,000 requests a day and that the model has incidents totaling ~8 hours a month.
See solution
With the project's number: without a shell, the search's availability drops to ~63% during the incidents (and in normal operation, to the model's ~92%). With a shell, 100% in both cases.
Business estimate: 100,000 requests/day. The model's incidents total ~8 h/month ≈ ~1.1% of the time, but concentrated and, according to the project's statement, during high-traffic and high-sales hours. During those 8 hours, without a shell, more than a third of the searches fail (or, in a total outage, all of them). If during those hours ~30,000 searches drop per month and a fraction of searches convert to a sale, each failed search is a potentially lost sale, plus the damage to trust (a user who sees search broken may not come back). Against that, the cost of the shell is: maintaining a keyword fallback (simple code, you already had it for basic search), a cache (which also serves for latency/cost, module 2), and calibrating three parameters (once, with a guide).
The justification: the shell's complexity is bounded and reusable (the fallback and the cache have other uses), while the cost of not having it is variable and hits at the worst moment (the incidents coincide with the sales hours). The decision is clearly favorable for a critical business path like search. Honest qualification: for a non-critical and low-traffic feature (for example, a description generator that a seller uses once), the complete shell might be over-engineering —a simple fallback without a breaker or cache would be enough—. The rule: the investment in resilience is sized according to the criticality and traffic of the feature. For search, the complete shell is worth it; the number (63% → 100% at the worst moment) backs it.
Exercise 3 — The ADR that missed a consequence. Every honest ADR lists its negative consequences, not just the positive ones. The project's ADR lists several. Identify an additional negative consequence of this shell that the ADR doesn't mention explicitly, explain why it's a real cost, and propose how to mitigate it.
See solution
An additional negative consequence not listed explicitly: the risk that the silent degradation hides a model problem. Since the shell makes the system respond 100% even when the model is down, a team that only looks at "does search respond?" might not notice that the model has been degraded or down for days, because the availability metric looks perfect (100%) while almost everything is served by low-quality fallback. The shell, by doing its job well, hides the underlying failure. It's the flip side of its virtue.
Why it's a real cost: you could be serving keyword search (bad quality) for a week without finding out, losing conversion, because the availability dashboard says "100%, all good". The fallback resolved the symptom (availability) and covered up the disease (the downed model).
How to mitigate it: monitor the degraded proportion as a first-class metric, with its own alert. It's not enough to measure "did it respond?"; you have to measure "did it respond optimally or degraded?", and alert if the degraded proportion exceeds a threshold for a sustained time. In the example, if suddenly 40% of the responses are by fallback, there's a model outage in progress even though the total availability is 100%. This connects with lesson 5 (marking the degraded) and lesson 7 (monitoring). The ADR should add this consequence and its mitigation: "the shell can hide a prolonged model failure; mitigated by alerting on the degraded proportion, not just on the total availability". A good ADR names even the costs of its own virtues.
Summary and next step
In this capstone you integrated the module's six lessons into a real, executed Mercado feature: you built the semantic search the complete resilience shell —timeout + circuit breaker + cascading fallback + degradation, with a drift monitor observing— and demonstrated the assignment with a number: availability went from 63.3% without a shell to 100% with a shell in the face of a model outage and two hangs, with 18 optimal responses and 12 degraded but valid, and the breaker opening at request 14. You delivered the four pieces: the shell diagram, the executed code, the ADR with its alternatives and consequences, and the justification of why the model failure no longer takes the system down —because availability decoupled from the model, failures are contained in the moment and over time, and the degradation is honest and chosen—.
With this you close module 5. You should be able, now, to take any AI feature and design its resilience: name its failure modes (availability, content, temporal), put the corresponding defenses (timeout, fallback, breaker, verification, monitoring), compose them in the correct order, and measure their effect with a simulated incident. And you know where the boundary is: here you applied the resilience patterns to the AI component; their mechanics in depth live in resilience-and-reliability-patterns-guide.
Where the guide goes next: module 6, the deterministic shell, generalizes what you did here —the model proposes, the system disposes— to its strongest form: a deterministic layer that validates what the model suggests before touching money or state (the LLM never executes a refund directly; it proposes, and deterministic rules approve or reject). This module's resilience is one face of that shell. And module 7, the data loop, takes the drift monitor from lesson 7 and turns it into the complete flywheel of observability and feedback. Beyond this guide: to build the AI pieces (RAG, agents, evals, fine-tune) the destination is the AI Engineering ecosystem; for the resilience mechanics in depth, resilience-and-reliability-patterns-guide; for the architectural decisions and tradeoffs, architecture-decisions-and-tradeoffs-guide.
Resources
resilience-and-reliability-patterns-guide(this ecosystem) — the reference for the mechanics of everything you composed: M2 timeouts, M5 circuit breakers, M7 degradation. Its own capstone (M8, "make Mercado's checkout resilient") is the sibling of this project, applied to a classic service instead of an AI component. In Spanish.architecture-for-ai-native-systems-guide, Module 6 (the deterministic shell) and Module 7 (the data loop) — where the guide goes next; they generalize the containment and monitoring this project used. And Module 3 (the eval) and Module 2 (latency/cost), which this project reuses. In Spanish.- Martin Fowler, "Architecture Decision Records" and (Bharani Subramaniam and Martin Fowler) "Emerging Patterns in Building GenAI Apps" — martinfowler.com. The ADR format you used and the map of resilience patterns around an AI component. In English.
- Chip Huyen, AI Engineering (O'Reilly, 2024). The chapters on reliability, cost, and monitoring integrate this module's pieces into the operational design of an application with foundation models. In English.
- To build the AI pieces (RAG, agents, evals, fine-tuning) that here we only treat as components with failure properties: the AI Engineering ecosystem, out of scope for this guide. In Spanish/English.