Module 5: Failure Modes and Resilience for AI
Module introduction: failure modes and resilience for AI
Why this module exists here
Ask yourself an uncomfortable question about everything you built in the previous four modules: what happens when the model doesn't respond? In module 1 you placed the LLM behind a boundary; in module 2 you gave it a latency and cost budget; in module 3 you gave it an eval gate; in module 4 you guarded its trust boundary. All that work, without saying so, assumed that the AI component is there and answers. But an LLM lives behind an external API that can be down, saturated, slow, or denying you service because you exhausted your quota. And even when it responds, it can answer with a datum it invented —with the same confident grammar and the same professional tone it uses when it's right—. A classic component does none of this: a function sums a cart and gives you the total, or throws an exception you catch. The LLM opens a new family of failures, and some of those failures are invisible: they throw no error, trigger no alarm, they simply deliver garbage that looks like gold.
This module installs the thesis that governs the design from here on: an AI-native system must be designed so that the AI component's failure does NOT become the crash of the whole system, and so that a silent failure doesn't pass for a correct answer. The AI component is the most fragile piece of the system —the slowest, the one that depends on a third party, the one that sometimes lies—, and the architecture has to contain that fragility, not trust that it won't appear. The way to contain it is a set of pieces you may already know from distributed systems —fallback, circuit breaker, timeout, degradation— applied specifically to the AI component and its own failures.
Here it's worth marking this module's hard boundary before continuing, because it defines what you'll learn and what you won't. The mechanics of the resilience patterns —how a real circuit breaker is implemented with its counters and windows, the math of backoff with jitter, how a timeout is done with threads or async— is taught in the resilience-and-reliability-patterns-guide (M2 timeouts, M5 circuit breakers, M7 degradation). This module does not re-teach that mechanics: it applies it to the AI component and refers to that guide for the detail. What is native here, and isn't in the resilience guide, are the AI-specific failures —hallucination as a failure mode, the dependency on a slow API with quotas, silent drift— and how those known pieces arrange themselves around a piece that, besides failing, sometimes lies without warning.
The case, as throughout the guide, is Mercado, with two features at the center:
- Semantic search: the user types "headphones for running in the rain" and the LLM understands the intent. When the model is down, we don't want Mercado's search box to stop working; we want it to degrade to a keyword search —worse, without understanding the intent, but it returns relevant products—. The user gets results, not an error.
- The support agent: it answers customer questions. When it can't verify a datum (does this order exist?, what's its real status?) or when the model fails repeatedly, we don't want it to invent an answer nor to crash; we want it to escalate to a human. The agent that doesn't know escalates; it doesn't guess.
Connection with the module. This is the map-lesson. We don't go deep into any technique yet; we install the thesis (the model's failure shouldn't take down the system; the silent failure shouldn't pass for correct), the vocabulary (hallucination, down/slow/rate-limited, drift, fallback, circuit breaker, timeout, degradation) and the map of how each lesson builds a part. Lesson 2 presents the taxonomy of the new failures and why the silent one is the dangerous one. Lesson 3 treats hallucination and its containment by verification. Lesson 4 treats the model down/slow/rate-limited and the timeout. Lesson 5 is the central mitigation: fallback and degradation. Lesson 6 puts the circuit breaker over the model. Lesson 7 treats drift and its monitoring. And lesson 8 puts you to building the complete resilience shell of a real Mercado feature, executed. The boundary with the resilience guide is HARD: here we apply the patterns to the AI component; their mechanics in depth are in resilience-and-reliability-patterns-guide.
And the promise that's kept throughout the module: nothing is asserted "from memory," everything is executed. Every simulation runs in Python, with the LLM simulated by a deterministic stub that can fail —down, slow, rate-limited, hallucinating—, never a real API, no keys and no network, and fixed data, so the output you see in each "What to expect" block is the literal output of running the code. You can copy it and reproduce it identically.
Three analogies: the elevator, the GPS, and the employee
The elevator that, if it fails, lets you use the stairs. A well-designed building doesn't depend on the elevator working always. When the elevator fails —and all elevators fail sometime—, you don't get stuck between floors nor does the building stop working: you use the stairs. They're slower, more uncomfortable, they don't reach the 40th floor with the same ease; but they take you where you're going. The building was designed to degrade: when the optimal route (the elevator) isn't there, there's a worse but available route (the stairs) that keeps the building standing. A building with no stairs —where the elevator is the only way up— is a building that falls entirely every time the elevator falls. That's an AI system with no fallback: when the model falls, everything falls.
The GPS that, if it loses signal, gives you the last known route. You're driving with the GPS and you enter a tunnel: you lose signal. A well-made GPS doesn't turn off nor leave you blind in the middle of the tunnel; it keeps showing you the last known route and an estimate of where you are, and when you recover signal, it recalibrates. It gives you a degraded answer —less precise, based on old data— instead of no answer. A GPS that turned off entirely when it lost signal would be useless exactly when you need it most. That's a cached response as a fallback: when the model isn't there, you serve the last good answer you had, marking it for what it is.
The employee who, when they don't know, escalates to the supervisor instead of inventing. A good customer-service employee, when asked something they don't know for certain —"when exactly does my order arrive?", "can you refund this charge?"—, doesn't invent an answer to look good. They say "let me check" or escalate to the supervisor. A bad employee invents: they give you a date that sounded good, promise you a refund they can't authorize, and create a bigger problem than admitting they didn't know. The difference between the two isn't intelligence; it's that the good one knows when not to trust their own answer and has somewhere to escalate. That's exactly what an AI component has to be taught: when it can't verify a datum, it escalates or degrades, it doesn't hallucinate.
Here's the point that unites the three: a resilient system always has a way out when the preferred route fails, and that way out is worse but works. The elevator has the stairs; the GPS has the last route; the employee has the supervisor. Your AI system has the fallback: when the model falls, is slow, or can't verify what it says, there's a deterministic route —keywords instead of semantics, cache instead of model, human instead of guessing— that keeps the system standing. Resilience isn't that the model never fails (it will fail); it's that when it fails, the system doesn't.
Worked example: the model falls, the system doesn't
We're not going to say that a fallback keeps the system standing: we're going to execute it and measure it. We model Mercado's semantic search with the complete resilience shell —timeout + circuit breaker + fallback— and simulate a model outage in the middle of a flow of 50 requests. We compare two designs:
- Without fallback: the system depends directly on the model. If the model fails, the request fails.
- With fallback + timeout + circuit breaker: when the model fails, the system degrades to a deterministic route (an in-memory keyword search) that doesn't depend on the model and therefore never fails.
The model is simulated by a stub that raises an error during the outage (indices 20 to 24 of 50 requests, i.e. the model is up 90% of the time). The fallback is a local deterministic function. The circuit breaker stops calling the model when it sees it fail three times in a row, so as not to keep paying the timeout of an API we already know is down.
# Lesson 1 (intro M5): a system calls the LLM with TIMEOUT + CIRCUIT BREAKER
# + FALLBACK. The model down is SIMULATED (no network, no API, no keys).
# We measure the system's availability WITH vs WITHOUT fallback, and see the
# breaker open during a model outage.
class ModelError(Exception):
pass
# The model is DOWN during a window (indices 20..24): a real outage,
# concentrated, not stray failures. 5 of 50 requests -> model 90% up.
OUTAGE = set(range(20, 25))
TIMEOUT_COST_MS = 800 # what it "costs" to wait for a timeout before giving up
def call_model(i):
# Deterministic LLM stub. During the outage it hangs and exhausts the timeout
# (raises ModelError). Outside the outage it responds healthily.
if i in OUTAGE:
raise ModelError("model timeout")
return f"semantic-response-{i}"
# Deterministic fallback: does NOT depend on the model (keyword search in
# memory). That's why it never fails: the system's availability decouples from
# the model's.
def fallback(i):
return f"keyword-response-{i}"
class CircuitBreaker:
# Mechanics in depth: resilience-and-reliability-patterns-guide M5.
def __init__(self, fail_threshold=3, cooldown=3):
self.fail_threshold = fail_threshold
self.cooldown = cooldown
self.fails = 0
self.state = "CLOSED"
self.opened_at = None
def allow(self, now):
if self.state == "OPEN":
if now - self.opened_at >= self.cooldown:
self.state = "HALF_OPEN" # lets ONE test attempt through
return True
return False # OPEN: we don't call the model
return True
def on_success(self):
self.fails = 0
self.state = "CLOSED"
def on_failure(self, now):
self.fails += 1
if self.fails >= self.fail_threshold:
self.state = "OPEN"
self.opened_at = now
N = 50
# --- Mode A: WITHOUT fallback (the system depends directly on the model) ---
responded_A = 0
for i in range(N):
try:
call_model(i)
responded_A += 1
except ModelError:
pass # without fallback: the request fails
avail_A = responded_A / N * 100
# --- Mode B: WITH timeout + breaker + fallback ---
cb = CircuitBreaker(fail_threshold=3, cooldown=3)
responded_B = model_calls = timeouts_paid = skipped_by_breaker = 0
trace = []
for i in range(N):
if cb.allow(i):
decision = cb.state
try:
call_model(i)
cb.on_success()
served = "model"
model_calls += 1
except ModelError:
timeouts_paid += 1 # we pay the timeout before giving up
cb.on_failure(i)
fallback(i)
served = "fallback (model failed)"
else:
decision = "OPEN"
skipped_by_breaker += 1 # breaker OPEN: we don't even call the model
fallback(i)
served = "fallback (breaker OPEN)"
responded_B += 1
if 19 <= i <= 26:
trace.append((i, decision, served))
avail_B = responded_B / N * 100
print("Incident trace (model down at indices 20..24):")
print(f" {'req':<5}{'breaker':<11}served by")
print(" " + "-" * 45)
for i, st, served in trace:
print(f" {i:<5}{st:<11}{served}")
print()
print("System availability (responded / total):")
print(f" WITHOUT fallback : {responded_A}/{N} = {avail_A:.1f}%")
print(f" WITH fallback : {responded_B}/{N} = {avail_B:.1f}%")
print()
print("Breaker cost during the outage:")
print(f" model calls made : {model_calls}")
print(f" timeouts paid : {timeouts_paid} "
f"({timeouts_paid * TIMEOUT_COST_MS} ms waiting)")
print(f" calls avoided (breaker) : {skipped_by_breaker} "
f"({skipped_by_breaker * TIMEOUT_COST_MS} ms of waiting SAVED)")What to expect. When you run the file, the output is exactly this:
Incident trace (model down at indices 20..24):
req breaker served by
---------------------------------------------
19 CLOSED model
20 CLOSED fallback (model failed)
21 CLOSED fallback (model failed)
22 CLOSED fallback (model failed)
23 OPEN fallback (breaker OPEN)
24 OPEN fallback (breaker OPEN)
25 HALF_OPEN model
26 CLOSED model
System availability (responded / total):
WITHOUT fallback : 45/50 = 90.0%
WITH fallback : 50/50 = 100.0%
Breaker cost during the outage:
model calls made : 45
timeouts paid : 3 (2400 ms waiting)
calls avoided (breaker) : 2 (1600 ms of waiting SAVED)
Read the output calmly, because that's the whole module in miniature.
Availability is the number that matters. Without a fallback, the system responded to 45 of 50 requests = 90.0% —exactly the model's availability, because the system is the model: when the model falls, the request falls with it—. With a fallback, the system responded to 50 of 50 = 100.0%. Burn that comparison in: the model has the same availability in both cases (90%); what changed is that in the second, the system stopped depending on the model to respond. The deterministic route —the keyword fallback— is always there, and always responds. That's why the system is more available than its most fragile dependency. This is the central result of the whole module, made a number: a 90%-available model, inside a system with a fallback, produces a system that responds 100% of the time.
The trace shows the circuit breaker doing its job. Look at requests 20 to 24, which is where the model is down. At 20, 21, and 22, the breaker is still CLOSED —it didn't know the model was down—, so it called the model, waited, and paid the timeout (that's why "fallback (model failed)": it tried the model, failed, fell to the fallback). Three failures in a row and the breaker opens. At 23 and 24 the breaker is already OPEN: it doesn't even try the model, it goes straight to the fallback ("fallback (breaker OPEN)"). Notice the saving: it paid the timeout only 3 times (2400 ms), and the breaker avoided 2 more calls to a model it already knew was down (1600 ms of waiting saved). At request 25, after the cooldown, the breaker goes to HALF_OPEN and lets one test attempt through —the model has recovered, so it responds—, and the breaker returns to CLOSED. The system recovered on its own.
Put the two readings together and you have the complete architecture: the fallback kept availability at 100% (nobody was left with no response), the timeout kept each failure from blocking the system forever (it paid a bounded cost, not an infinite one), and the circuit breaker stopped hammering the down model as soon as it detected it (it saved latency and protected the model from more useless load). None of the three pieces alone is enough: without a fallback, the breaker only gives you a faster error; without a timeout, a hang blocks the system even if you have a fallback; without a breaker, you keep paying the full timeout on every request throughout the outage. Together, the model falls and the system doesn't.
The ideas this module installs, and where each one lives
That example touched on, without fully developing them, the module's ideas. It's worth making them explicit, because they're the backbone of the lessons that follow.
1. The new failure modes (lesson 2). An AI component fails in ways a classic component doesn't have: down, slow, rate-limited (noisy failures, which throw an exception) and —the most dangerous— hallucination (a silent failure, which throws nothing). Lesson 2 executes the contrast: a deterministic component fails hard or doesn't fail; the AI one has a family of silent failures a try/except doesn't catch.
2. Hallucination as a failure mode (lesson 3). The model invents a datum with confidence. The containment: verify the claim against a deterministic source of truth, and degrade (escalate, say "I don't know") when it can't be verified. Lesson 3 executes an agent that cites order data —the verified ones are served, the invented ones are blocked—.
3. The model down/slow/rate-limited and the timeout (lesson 4). The dependency on an external API with quotas. The timeout so as not to wait forever. Lesson 4 measures how an 800 ms timeout cuts the wait of a 30 s hang.
4. Fallback and degradation (lesson 5). The central mitigation: degrade to a simpler route instead of crashing. The cascade (semantic → cache → keywords) and the degraded but valid response. Lesson 5 measures availability rising from 70% to 100%.
5. The circuit breaker over the model (lesson 6). Stop calling a model that's been failing. The AI angle: each failed call costs latency and money, and retrying a rate-limited model makes it worse. Lesson 6 measures the calls and the cost saved.
6. Drift (lesson 7). The silent degradation over time. The defense: measure with the eval continuously. Lesson 7 fires a drift alert two weeks before the users complain.
Keep this map; it's the module's route:
Idea Lesson Key concept
─────────────────────────────────────── ──────── ──────────────────────────────
AI's new failure modes L2 noisy (exception) vs
silent (hallucination)
Hallucination as a failure mode L3 verify vs source of truth;
degrade if not possible
Down / slow / rate-limited + timeout L4 don't wait forever;
refers to resilience M2
Fallback and degradation L5 worse but valid route;
refers to resilience M7
The circuit breaker over the model L6 stop hammering the model;
refers to resilience M5
Drift: the silent degradation L7 measure over time; alert
─────────────────────────────────────── ──────── ──────────────────────────────
Make a Mercado feature resilient L8 the mini-project, executed
The map: where this module sits in the guide and in the ecosystem
This module is the fifth piece of the deterministic shell that surrounds the AI component. Here's how it connects with the rest of the guide:
flowchart TD
M1["M1 · Place the component<br/>(contract, boundary, core/shell)"]
M2["M2 · Latency and cost as architecture"]
M3["M3 · The eval as a fitness function"]
M4["M4 · Guardrails and the trust boundary"]
M5["M5 · Failure modes and resilience for AI"]
M6["M6 · The deterministic shell"]
M7["M7 · The data and feedback loop"]
M8["M8 · Project: architect an AI feature"]
M1 --> M2 --> M3 --> M4 --> M5 --> M6 --> M7 --> M8
Read it like this: in M1 you placed the component; in M2 you gave it a budget; in M3 you gave it a quality gate; in M4 you guarded its trust boundary; here (M5) you make it resilient to its own failures: you accept that it's going to hallucinate, go down, and drift, and you design so that none of those failures takes down the system. In M6 you'll see the deterministic shell in depth —the model proposes, the system disposes— of which this module's resilience is a central part; and in M7 you'll close the data loop, of which lesson 7's drift monitoring is the first stitch.
And the boundary with the resilience guide, which must be respected and is HARD: the mechanics of timeouts, retries with backoff, circuit breakers, bulkheads, and degradation are not taught here in depth. That lives in resilience-and-reliability-patterns-guide (M2 timeouts, M5 circuit breakers, M7 degradation), and this guide links to it in every lesson where it applies. What we do treat is how those patterns are applied to the AI component and —this is what's native here— the AI-specific failures that don't exist in a classic component: hallucination, the dependency on a model API with quotas, and drift. When lesson 6 talks about the circuit breaker, it's not going to give you a circuit-breaker course: it's going to show you why a breaker over a model has an extra motivation —the cost per call and the provider's quotas— that a breaker over a classic service doesn't have. The distinction is the same as throughout the ecosystem: here we treat the architectural property of resilience applied to AI, not the complete resilience discipline.
Common mistakes
Assuming the LLM always responds. What happens: the team writes semantic search as a direct call to the model —"I pass it the query, it returns results"— with no alternate route. The day the model's API has a twenty-minute outage, Mercado's search box stops working entirely, and with it a huge part of the traffic and the sales. Why it happens: in development the model always responded, so the failure was never seen; it was designed for the happy path. How to spot it: trace what happens in your code when the model call throws an exception or hangs; if the answer is "the request fails" or "it keeps waiting," you have no resilience. How to fix it: design assuming the model will fail —down, slow, rate-limited— and put a deterministic fallback route, as the building puts stairs next to the elevator. Lesson 5 executes it: with a fallback, availability rises from 70% to 100%.
Having no fallback and taking down the whole system when the model falls. What happens: a variant of the previous one, but worse: the AI feature isn't isolated, and its fall drags down other parts of the system —the thread waiting for the model stays blocked, requests pile up, the connection pool is exhausted, and a service that didn't even use AI falls in cascade—. Why it happens: the AI component was treated like any other reliable dependency, without isolating or bounding it. How to spot it: a single down model can degrade services that don't depend on it. How to fix it: isolate the AI component (the resilience guide calls this a bulkhead), give it a timeout and a breaker, and give it a fallback so its fall is local and degraded, not global and total. This module applies it; the mechanics of isolation in depth are in the resilience guide.
Trusting a hallucinated output as if it were true. What happens: the support agent tells a customer "your order arrives Thursday and ships with tracking TRK-4521," the customer believes it, and it turns out that order doesn't have that tracking —the model invented it—. The failure threw no exception, triggered no alarm; the system delivered a false datum with total naturalness. Why it happens: hallucination is a silent failure; unlike an outage, it doesn't announce itself, and since the output sounds good, nobody reviews it. How to spot it: your system serves data the model asserts without verifying it against a source of truth. How to fix it: treat hallucination as a failure mode —verify every factual claim against a deterministic source before serving it, and degrade (escalate to a human, say "I can't confirm it") when you can't verify—. Lesson 3 executes it: the verified data are served, the invented ones are blocked.
Exercises
Exercise 1 — The elevator, the GPS, and the employee. For each of the module's three analogies, identify (a) which is the preferred route that fails, (b) which is the degraded fallback route, and (c) which architectural piece of this module it represents. Then say, for Mercado's semantic search, which is its preferred route and which its fallback.
See solution
- The elevator: (a) preferred route = the elevator (fast, comfortable); (b) fallback = the stairs (slower, but they take you); (c) it represents the fallback / degradation —when the optimal route falls, there's a worse but available route—. The key lesson: without stairs, the building falls with the elevator.
- The GPS: (a) preferred route = navigation with live signal; (b) fallback = the last known route (old data, less precise); (c) it represents the cached response as a fallback —you serve the last good thing you had instead of nothing—.
- The employee: (a) preferred route = answer with certainty; (b) fallback = escalate to the supervisor / say "let me check"; (c) it represents the degradation against hallucination —when you don't know for certain, you escalate instead of inventing—.
- Mercado's semantic search: preferred route = semantic search with the LLM (it understands the intent of "headphones for running in the rain"); fallback = keyword search (it doesn't understand the intent, but it finds products containing those words). Worse, but the user gets results. It's exactly the elevator/stairs pattern applied to AI.
Exercise 2 — The availability number. In the worked example, the model has 90% availability and the system with a fallback reached 100%. Explain why the system can be more available than its most fragile dependency (the model), and what property the fallback must meet for that to work. Then answer: if the keyword fallback failed 1% of the time (for example, a problem with the local database), what would the system's availability be approximately?
See solution
The system can be more available than the model because it doesn't depend only on the model to respond: when the model fails, there's a second route (the fallback) that handles the request. The system's availability isn't the model's; it's the probability that at least one of the routes works. For the system to reach 100%, the fallback must meet a key property: not depend on the model (nor on anything that falls together with the model). In the example, the fallback is an in-memory keyword search —no network, no external API—, so it's independent of the model's outage and always responds.
If the fallback failed 1% of the time, the system would only fall when both routes fail at once. The model falls 10% and, in that 10%, the fallback falls 1%: the joint fall is approximately 0.10 × 0.01 = 0.001 = 0.1%. The system's availability would be ~99.9%. The general lesson: availability multiplies in your favor when the routes are independent. That's why it matters so much that the fallback doesn't share the fragile dependency (the model) with the preferred route; if the fallback also called the model, there'd be no gain.
Exercise 3 — Noisy vs silent. Classify each of these AI-component failures as noisy (it throws an exception you can catch) or silent (it delivers an output that looks valid) and say, for each, what the appropriate architectural defense is: (a) the model's API returns HTTP 429 (rate limit); (b) the model takes 40 seconds to respond; (c) the model states that order A-1002 was delivered, when it's actually in progress; (d) the model's API returns a 503 error (service unavailable).
See solution
- (a) HTTP 429 (rate limit) → NOISY. The API returns an explicit error; you can catch it with an
except. Defense: circuit breaker (stop calling the model while it's rate-limited, so as not to make it worse) + fallback (serve the deterministic route meanwhile). Lesson 6 treats it. - (b) Takes 40 seconds → NOISY (with a timeout). On its own, a hang throws nothing —it leaves you waiting—, but the timeout turns it into a noisy failure: you cut at 800 ms and throw a
TimeoutErroryou catch. Defense: timeout + fallback. Lesson 4 treats it. - (c) States a false order status → SILENT. It's a hallucination: it throws no exception, it delivers a response that sounds perfect and is wrong.
try/exceptdoesn't catch it. Defense: verify the claim against the source of truth (the order's real status) and degrade if it doesn't match. Lesson 3 treats it. It's the most dangerous failure precisely because it's silent. - (d) HTTP 503 → NOISY. Explicit down-service error. Defense: fallback (degrade to the deterministic route) + circuit breaker (if it repeats, stop trying). Lesson 5 treats it.
The pattern: (a), (b), and (d) are availability failures —noisy, catchable— and are defended with timeout/breaker/fallback; (c) is a content failure —silent— and is defended with verification. The module covers both fronts.
Summary and next step
In this lesson you installed the thesis that holds up the module: an AI-native system must be designed so that the AI component's failure doesn't take down the whole system, and so that a silent failure doesn't pass for a correct answer. You saw it with three analogies —the elevator with its stairs, the GPS with its last route, the employee who escalates instead of inventing— and you measured it: a system with timeout + circuit breaker + fallback facing a 90%-available model responded to 100% of the requests, while the same system without a fallback stayed at the model's 90%. You saw the circuit breaker open after three failures, stop hammering the down model, and recover on its own. And you saw the distinction that organizes the whole module: noisy failures (down, slow, rate-limited, which an exception catches) and silent failures (hallucination, which only a verification sees).
Before moving on you should be able to: explain why a system can be more available than its most fragile dependency; name the three pieces of the resilience shell (timeout, circuit breaker, fallback) and what each does; distinguish a noisy failure from a silent one; and locate the hard boundary with the resilience guide (here it's applied, there the mechanics are taught).
Lesson 2 takes the first idea and develops it in depth: the new failure modes of an AI component. You're going to see, executed, the contrast between a classic component —which fails hard (a catchable exception) or doesn't fail— and an AI component, which opens a new family of failures, including the most dangerous: hallucination, a silent failure that throws no exception and that only a validation detects. With code, so that "the LLM fails differently" stops being a warning and becomes a taxonomy you can see and count.
Resources
resilience-and-reliability-patterns-guide(this same ecosystem) — the central reference for the mechanics of everything this module applies: M2 timeouts, M3 retries with backoff, M5 circuit breakers, M7 graceful degradation and load shedding. When you want to implement a real circuit breaker (counters, sliding windows, states) or a robust timeout, that guide is the destination. Here we apply it to the AI component; there it's taught in depth.- Anthropic, Claude documentation — docs.anthropic.com. See the rate limits and API errors pages to understand, at a conceptual level, the availability failure modes of an API-served model (429 for quota, 5xx errors, retries) —without fixating on a specific model version—. In English.
- Chip Huyen, AI Engineering (O'Reilly, 2024). The chapters on reliability and monitoring of foundation-model applications treat AI's own failures —including hallucination and drift— as design properties that must be measured and contained. In English.
- Martin Fowler and Bharani Subramaniam, "Emerging Patterns in Building GenAI Apps" — martinfowler.com/articles/gen-ai-patterns. It places resilience and fallbacks around an AI component on the complete architectural map. In English.