Module 5: Failure Modes and Resilience for AI
Fallback and graceful degradation
Overview
You already have the timeout that turns a hang into a catchable exception (lesson 4). Now the question is: when the model fails —down, slow, rate-limited—, what do you respond? There are two possible answers and one is much better than the other. The bad one is nothing: the request fails, the customer sees an error, the feature doesn't work. The good one is a worse but valid response: instead of the semantic search that understands the intent, a keyword search that at least finds relevant products; instead of the model's fresh datum, the last cached response. This lesson installs the module's central mitigation: the fallback —degrade to a simpler route when the model fails— and graceful degradation —give a partial or worse response, marked as such, instead of crashing—.
In lesson 1 you measured the effect of a binary fallback (model or keywords). Here you develop it in its complete form: a fallback cascade with several levels —semantic → cache → keywords—, where each level is worse than the previous but better than crashing, and where the degraded response is marked so that the system (and sometimes the user) knows it's not the optimal one. You're going to see, executed, Mercado's search going through that cascade during a model outage, with availability rising from 70% to 100% and the responses labeled by their quality level.
Connection with the module. Lesson 4 gave you the timeout, which enables the fallback (without cutting the wait, there's nothing to degrade to). This lesson builds the fallback on top. And it prepares lesson 6: the circuit breaker decides when to skip the model and go straight to the fallback, but the fallback itself —where you go when you skip the model— is what's here. The boundary with the resilience guide is hard: graceful degradation and load shedding as general patterns are taught in resilience-and-reliability-patterns-guide M7; here we apply them to the AI component —what "degrade" means when the preferred route is an LLM— and refer there for the complete pattern.
An analogy: the elevator, the stairs, and the emergency ladder
Go back to the building of lesson 1, but notice it doesn't have one backup route, but several, ordered from best to worst. The preferred route is the elevator: fast, comfortable, reaches all floors. If the elevator fails, there's the main staircase: slower and tiring, but safe, wide, with a railing, it takes you to any floor. And if for some reason the main staircase is blocked (a repair, a fire in that section), there's the emergency ladder: uncomfortable, narrow, you use it only if there's no other way, but it exists and gets you out of the building. A well-designed building has this cascade: the optimal route, the good one, and the last-resort one. You're never left with no way to go up or down.
Notice two things. First, each level is worse than the previous but infinitely better than "nothing": the emergency ladder is uncomfortable, but compared to being trapped, it's excellent. Second, and this is key, when you use the stairs, you know you're using the stairs: you don't fool yourself thinking you're in an elevator. The degradation is honest —you climb the stairs knowingly, accepting you'll take longer—. A building that made you believe you were in an elevator while you climbed on foot would be worse than one that told you the truth, because you'd plan your time badly.
Here's the point: an AI system's fallback is a cascade of routes, ordered from best to worst, where the degraded response is served honestly marked as degraded. The semantic search is the elevator (it understands the intent); the cached response is the main staircase (good, but it may be a little old); the keyword search is the emergency ladder (it doesn't understand the intent, but it finds products that contain those words). When the model falls, the system goes down the cascade to the first level that works, serves that response, and marks it as degraded —for monitoring, and sometimes for the user ("approximate results; smart search isn't available right now")—. The user is never left trapped; they get the best available route at that moment, knowing it's not the optimal one.
Worked example: the fallback cascade
We're going to build the cascade. Mercado's search has three levels:
semantic(optimal): the LLM understands the query's intent. It's the preferred route. It can fail (model down).cache(degraded): recent stored responses for popular queries. It doesn't depend on the model. It covers the queries already seen.keyword(degraded): literal word search over the catalog. It doesn't depend on the model, doesn't understand intent, but always returns something relevant. It's the final safety net.
resilient_search implements the cascade: it tries semantic; if it fails, it looks in cache; if it's not in cache, it falls to keyword. It's never left with no response. Each result is labeled with its tier and a degraded flag. We simulate a model outage at indices 4, 5, and 6 (3 of 10 requests → model 70% available) and measure availability with and without the cascade.
# Lesson 5: FALLBACK and DEGRADATION. The semantic search falls to a simpler
# route when the model fails: first a cached response, then
# a keyword search. The user ALWAYS gets something (worse, but not an
# error). We measure availability WITH vs WITHOUT fallback and which TIER served each
# request. Graceful degradation in depth: resilience-and-reliability M7.
CATALOG = [
(1, "wireless headphones with noise cancellation"),
(2, "programmable drip coffee maker"),
(3, "rainproof backpack for laptop"),
(4, "backlit mechanical keyboard"),
(5, "waterproof sports headphones"),
]
class ModelError(Exception):
pass
# The semantic model is down in a window (indices 4..6).
OUTAGE = set(range(4, 7))
QUERIES = ["headphones", "coffee", "backpack laptop", "keyboard",
"headphones", "coffee", "gym headphones", "backpack",
"keyboard light", "coffee"]
# Recent cached responses (a fallback route with no model).
CACHE = {"headphones": [1, 5], "coffee": [2]}
def semantic_search(i, query):
if i in OUTAGE:
raise ModelError("semantic model down")
words = query.split()
return [pid for pid, name in CATALOG if any(w in name for w in words)]
def keyword_search(query):
words = query.split()
return [pid for pid, name in CATALOG if any(w in name for w in words)]
def resilient_search(i, query):
# Fallback cascade: semantic -> cache -> keyword. Never falls.
try:
return ("semantic", semantic_search(i, query), False) # tier, hits, degraded
except ModelError:
pass
if query in CACHE:
return ("cache", CACHE[query], True)
return ("keyword", keyword_search(query), True)
# --- Mode A: WITHOUT fallback (semantic only) ---
responded_A = 0
for i, q in enumerate(QUERIES):
try:
semantic_search(i, q)
responded_A += 1
except ModelError:
pass # without fallback: the user sees an error
avail_A = responded_A / len(QUERIES) * 100
# --- Mode B: WITH fallback cascade ---
tiers = {"semantic": 0, "cache": 0, "keyword": 0}
print(f"{'req':<5}{'query':<16}{'tier':<10}{'results':<14}quality")
print("-" * 58)
for i, q in enumerate(QUERIES):
tier, hits, degraded = resilient_search(i, q)
tiers[tier] += 1
quality = "degraded" if degraded else "optimal"
print(f"{i:<5}{q:<16}{tier:<10}{str(hits):<14}{quality}")
responded_B = len(QUERIES)
avail_B = responded_B / len(QUERIES) * 100
print("-" * 58)
print(f"Availability: WITHOUT fallback {responded_A}/{len(QUERIES)} = {avail_A:.0f}% "
f"| WITH fallback {responded_B}/{len(QUERIES)} = {avail_B:.0f}%")
print(f"Tiers served: semantic={tiers['semantic']} "
f"cache={tiers['cache']} keyword={tiers['keyword']}")
print(f"Degraded responses (worse, but valid): "
f"{tiers['cache'] + tiers['keyword']}")What to expect. When you run the file, the output is exactly this:
req query tier results quality
----------------------------------------------------------
0 headphones semantic [1, 5] optimal
1 coffee semantic [2] optimal
2 backpack laptop semantic [3] optimal
3 keyboard semantic [4] optimal
4 headphones cache [1, 5] degraded
5 coffee cache [2] degraded
6 gym headphones keyword [1, 5] degraded
7 backpack semantic [3] optimal
8 keyboard light semantic [4] optimal
9 coffee semantic [2] optimal
----------------------------------------------------------
Availability: WITHOUT fallback 7/10 = 70% | WITH fallback 10/10 = 100%
Tiers served: semantic=7 cache=2 keyword=1
Degraded responses (worse, but valid): 3
Read the table, because it shows the cascade going down level by level.
Outside the outage, everything goes through the optimal level. Requests 0-3 and 7-9 were served by semantic: the model was healthy, understood the intent, gave the best response. The cascade charges no cost when the preferred route works; it only kicks in when needed.
During the outage (requests 4, 5, 6), the cascade drops down. Look at what happened when the model was down:
- Request 4 ("headphones"):
semanticfailed, but "headphones" is in theCACHE, so it was served from the cache —products [1, 5], the last good stored response—. Degraded, but relevant and fast. - Request 5 ("coffee"): same,
semanticfailed, "coffee" is in cache, it was served from the cache [2]. - Request 6 ("gym headphones"):
semanticfailed, and "gym headphones" is not in the cache (it's a query that hadn't been seen), so the cascade dropped one more level, to keyword —literal search—, which found [1, 5] (the two products with "headphones" in the name). The last resort, and still it returned relevant results.
All three were served degraded —marked as such in the "quality" column—, but all three returned products. The user who searched during the outage didn't see an error or an empty page; they saw slightly worse results (without the semantic understanding of the intent), labelable as approximate.
Availability: from 70% to 100%. Without a fallback, the system responded to 7 of 10 (it failed on the 3 outage requests) = 70%. With the cascade, it responded to 10 of 10 = 100%. As in lesson 1, the model has the same availability (70% in this experiment); what changed is that the system stopped depending on the model to respond. And the breakdown by tier tells the complete story: 7 optimal, 2 via cache, 1 via keyword —3 degraded but valid responses that, without the cascade, would have been 3 errors—.
The architectural implication: degrading is qualitatively different from crashing. A system that crashes tells the user "I can't help you"; a system that degrades tells them "I can help you a bit worse." The difference, at Mercado's scale, is enormous: during a twenty-minute model outage, the system without a fallback loses all the search traffic (and the sales that came from it), while the system with the cascade keeps selling, with a less smart but functional search. Resilience isn't avoiding degradation; it's choosing to degrade instead of falling.
Going deeper: how a good fallback cascade is designed
The golden rule: the fallback must not share the fragile dependency. This is what makes availability multiply in your favor (you computed it in exercise 2 of lesson 1). If your fallback also called the model —for example, "if the large model fails, try the small model"— and the outage were of the whole provider (not of a specific model), the fallback would fall together with the preferred route and you'd gain nothing. That's why the cascade's last level must be deterministic and local: the keyword search runs in your own process, no network, no external API. It's the emergency ladder that works even if the power to the whole building is cut. A fallback that depends on the same thing as the preferred route is a fake fallback.
The cascade is ordered from best quality to greatest independence. Notice the order: semantic (best quality, but depends on the model) → cache (good quality, independent of the model but limited to what's already seen) → keyword (worst quality, totally independent and always available). The principle: at the top you put the best even if it's fragile, at the bottom you put the most robust even if it's worse, and the system goes down to the first level that works. The bottommost level must be the most robust of all —your final safety net— because it's the one that catches everything the upper ones drop. If the lowest level could fail, you'd have a hole.
The types of fallback, applied to AI. The example's cascade uses three types, and it's worth naming them because you'll combine them according to the feature:
- Alternate deterministic route (keywords instead of semantics): a simpler, AI-free implementation that solves the same problem worse. It's the most robust fallback because it shares nothing with the AI route.
- Cached response (the last good response): you serve what you already had. Good for repeated queries; useless for new queries. It's the GPS giving you the last known route.
- Simpler / cheaper model (a small model instead of the large one): useful when the failure is of capacity (the large model is saturated) but not when the failure is of the whole provider (both models fall together). Use it with care: it shares the network dependency with the preferred route.
For the support agent, the cascade would be different: model → template response for common questions → escalate to a human (the final degradation, of the employee who escalates instead of inventing, lesson 3). The human is support's "keyword": slower and more expensive, but always resolves.
Honest degradation: mark the degraded. Back to the analogy's lesson: when you use the stairs, you know you're using the stairs. In the example, each response carries a degraded flag. That flag has two uses. One, for monitoring: if suddenly 80% of the responses are degraded, you have a model outage in progress and you want to know it (it connects with lesson 7's monitoring). Two, for the user, when applicable: a discreet label ("approximate results; smart search will be back soon") manages the expectation. You don't always expose the flag to the user —for a search, maybe not—, but you always record it. What you never do is serve a degraded response pretending it's optimal, because that breaks trust when the user notices the difference and doesn't understand why.
Anatomy of the cascade:
user query
│
▼
┌───────────┐ fails ┌───────────┐ not there ┌───────────┐
│ semantic │─────────►│ cache │────────────►│ keyword │
│ (optimal) │ │(degraded) │ │(degraded) │
└───────────┘ └───────────┘ └───────────┘
│ │ │
serves serves serves
(optimal) (if there's a hit) (ALWAYS: final net)
│ │ │
└──────────────────────┴─────────────────────────┘
▼
the user ALWAYS gets results
Common mistakes
Having no fallback and dropping everything when the model falls. What happens: the search is a direct call to the model with no alternate route; the day of a provider outage, Mercado's search stops working entirely and takes a part of the traffic and sales with it. Why it happens: the happy path was designed, the failure was never seen in development. How to spot it: trace what happens when the model call throws an exception; if the answer is "the request fails," you have no fallback. How to fix it: put a fallback cascade with a deterministic, local final level. The example measures it: without a fallback 70%, with the cascade 100%.
A fallback that depends on the same thing as the preferred route. What happens: the team puts as a fallback "if the large model fails, use the small model from the same provider"; the day the whole provider falls, both models fall together and the fallback is useless. Why it happens: the fallback was thought of as "another way to do AI" without seeing that it shares the fragile dependency (the network to the provider). How to spot it: your last fallback level calls an external API or the same provider as the preferred route. How to fix it: the cascade's final level must be deterministic and local, without the fragile dependency —keywords in memory, a template, a locally cached response—. The fallback must survive the outage it's supposed to cover.
Degrading without marking it (or without monitoring it). What happens: the system falls to keywords silently and serves worse results without recording that it's degraded; a model outage of hours goes unnoticed because "the system was responding" —worse, but responding—, and nobody finds out until the conversion metrics drop. Why it happens: the fallback worked too well —it covered the problem instead of degrading visibly—. How to spot it: you don't have a "% degraded responses" metric; you wouldn't know if you're serving 5% or 95% via fallback. How to fix it: mark each response with its tier and monitor the degraded proportion; a degradation spike is an alarm that the preferred route is failing. The example's degraded flag exists precisely for this. Degrading is good; degrading blindly is hiding a fire.
Exercises
Exercise 1 — Design the support agent's cascade. The search has the cascade semantic → cache → keywords. Design the equivalent fallback cascade for Mercado's support agent (which answers customer questions), with at least three levels ordered from best to worst, and say which is the final level that never fails and why.
See solution
A reasonable cascade for the support agent, from best to worst:
- Full LLM agent (optimal): it understands the customer's free question and responds with data verified against the source of truth (lesson 3). Preferred route; depends on the model.
- Template responses for frequent questions (degraded): for the most common questions ("where's my order?", "how do I return a product?"), a deterministic template that fills in data from the database. It doesn't understand rare questions, but it covers the bulk of the volume without the model. Independent of the model.
- Escalate to a human (degraded, last resort): a human agent receives the conversation. Slower and more expensive, but it resolves anything.
The final level that never fails is escalate to a human. It never fails because it doesn't depend on any fragile technical piece (neither the model, nor a template that covers the case): a human can handle any question, even the ones no template foresaw. It's the equivalent of search's keyword —the final safety net, more expensive but universal—. The difference from search is that here the last resort is human instead of deterministic, because support touches cases (money, complaints) where a human is the only guarantee. It's exactly the employee who escalates to the supervisor of lesson 1.
Exercise 2 — Why the cascade order matters. In the example, the cascade is semantic → cache → keyword. Explain what would happen to the quality of the responses if you inverted the order to keyword → cache → semantic, and why the correct order puts the best at the top and the most robust at the bottom.
See solution
If you inverted the order to keyword → cache → semantic, the cascade would always serve via keyword —the first level—, because keyword never fails (it's deterministic and local, always returns something). The cascade would never drop to the other levels, so you would never use the semantic search or the cache, even if the model were perfectly healthy. You'd serve the worst quality 100% of the time, even when the best was available. It would be like always using the emergency ladder while the elevator works.
The correct order puts the best at the top (semantic) because you want to serve the maximum quality whenever possible, and you try that level first. It puts the most robust at the bottom (keyword) because that level is the safety net: you only reach it when all the ones above failed, and you need that final level to not be able to fail. The cascade goes down "looking for the first level that works," so the first must be the best quality and the last the most robust. Inverting the order turns the safety net into the main route, throwing all the quality away.
Exercise 3 — Partial degradation within a response. The example's cascade degrades the whole response to a lower level. But sometimes you can degrade only part of a response. Imagine Mercado's product page, which shows: (a) the price and stock (from the database), (b) the AI-generated description, and (c) "similar products" AI-generated recommendations. If the model falls, which part of the page do you degrade and how, and which part isn't touched? Why is this better than dropping the whole page?
See solution
With the model down, you degrade only the parts that depend on AI, and leave intact the ones that don't:
- (a) Price and stock → NOT touched. They come from the deterministic database, not from the model. The model outage doesn't affect them at all; they're shown normally.
- (b) AI-generated description → degrade. If the description is generated on the fly with the model, it falls to a cached description (the last generated one, stored) or to the raw product attributes from the spec sheet ("Headphones. Wireless. 30 h battery."). Worse worded, but informative.
- (c) AI recommendations → degrade or hide. They fall to non-personalized recommendations (the category's best sellers, computed without AI) or, if there's no good alternative, the section is simply hidden —a page with no "similar products" is still perfectly usable—.
This is better than dropping the whole page because most of the page doesn't depend on the model: the customer can still see the price, the stock, and buy the product. Dropping the complete page over a failure that only affects two secondary sections would be throwing away the sale over a cosmetic problem. Partial degradation isolates the failure to the affected parts and keeps the business core (being able to buy) functional. It's graceful degradation in its finest form: not all or nothing, but "each part degrades to its own fallback, and what doesn't depend on the model doesn't even notice." The resilience guide (M7) covers this pattern in depth.
Summary and next step
In this lesson you installed the module's central mitigation: the fallback —degrade to a simpler route when the model fails— and graceful degradation —a worse but valid response, marked as such, instead of crashing—. You saw it with the building and its cascade of routes (elevator → main staircase → emergency ladder), and you measured it: Mercado's search went down the cascade semantic → cache → keyword during a model outage, availability rose from 70% to 100%, and 3 of 10 responses were served degraded but valid —2 via cache, 1 via keyword— instead of failing. You kept the rules of a good cascade: the final level must be deterministic and local (not share the fragile dependency), it's ordered from best quality at the top to greatest robustness at the bottom, and the degraded is marked —for monitoring and sometimes for the user— never served pretending it's optimal.
Before moving on you should be able to: design a fallback cascade with ordered levels and a robust last resort; explain why the fallback must not share the preferred route's fragile dependency; argue why degrading is qualitatively better than falling; and recognize when a partial degradation (per section) is preferable to a total one. Degradation as a general pattern lives in resilience-and-reliability-patterns-guide M7; here we apply it to AI.
Lesson 6 answers a question the fallback leaves open: during a prolonged outage, does it make sense to keep trying the model on every request —paying the timeout each time— before falling to the fallback? No. The circuit breaker detects that the model has been failing and stops calling it for a while, going straight to the fallback and saving the timeout, the cost per call, and —key in AI— the load on a model that maybe is rate-limited precisely from too many calls. You're going to measure how the breaker cuts the model calls from 16 to 10 and the cost from $0.032 to $0.020 during an outage.
Resources
resilience-and-reliability-patterns-guide(this ecosystem), M7 "Graceful degradation and load shedding" — the central reference for the mechanics of degradation we apply here: how to degrade functionality by priority, how to do partial degradation, how to shed load under pressure. In Spanish.- Martin Fowler and Bharani Subramaniam, "Emerging Patterns in Building GenAI Apps" — martinfowler.com/articles/gen-ai-patterns. It treats fallbacks and degraded routes around an AI component as first-class patterns. In English.
- Chip Huyen, AI Engineering (O'Reilly, 2024). The chapters on reliability treat backup routes (simpler model, cache, default response) as part of the design of a model application. In English.
architecture-for-ai-native-systems-guide, Module 2 (response cache) — the cache we use here as a fallback level was designed there as a latency/cost optimization; here it serves double duty as a degraded route. In Spanish.