Module 8: Project — Architect an AI Feature in Mercado
Resilience and fallback
Overview
Steps 2, 3, and 4 protected the cost, the quality, and the security of the support agent —all with the model working—. Step 5 protects the availability, which is what happens when the model doesn't work: it goes down, gets slow, or gets rate-limited. This lesson fills the sheet's fallback field with module 5's mitigations: a fallback cascade (model → FAQ template → escalate to human) and a circuit breaker over the model. You'll see, executed, how a model outage doesn't take the feature down —the availability holds at 100% with the cascade versus going down without it—, and how the breaker stops hitting a rate-limited model.
And there's a containment decision this lesson makes explicit and that is specific to a tolerance-3 feature: the fallback's last resort never auto-approves a refund. When the model goes down and a ticket asking for money arrives, the system doesn't invent a response nor approve blindly —it escalates to a human—. Degrading for the support agent doesn't mean "answer worse whatever it takes"; it means "answer worse without ceasing to be safe". A search that goes down can degrade to keywords with no risk; an agent that touches money must degrade to a human when the model isn't there, because the alternative —auto-approving without the model or the human— is exactly what the shell exists to prevent.
Connection with the module. This lesson protects everything the previous ones built: a cheap agent (M2), of proven quality (M3), and well-armored (M4) is of no use if it goes down when the model provider has an outage. The fallback cascade uses the cache from lesson 3 (which now serves a double function as a degraded route) and prepares lesson 7, where the deterministic shell validates the proposals that do arrive. The boundary with the resilience guide is HARD: the mechanics of the patterns —the circuit breaker's states with precision, the count windows, the calibration— are taught in resilience-and-reliability-patterns-guide; here we apply them to the AI component and defer there for the serious implementation.
An analogy: the hospital when the power goes out
A hospital can't afford to stop functioning when the power goes out. That's why it has a backup cascade, ordered from best to worst. The preferred source is the power grid: cheap, unlimited, always available… until it isn't. When the grid goes down, the diesel generators start: more expensive and noisier, but they keep the operating rooms running for hours. And for the equipment that can't blink for even a second —a ventilator, a life-support machine—, there are batteries that cover the instant between the grid going out and the generators starting. Each level is worse than the previous one (more expensive, more limited) but infinitely better than "nothing": an operating room with a generator is worse than with the grid, but much better than an operating room in the dark in the middle of an operation.
Now notice the detail that makes the hospital special: there are procedures that, if there's no reliable power, simply aren't done. A non-urgent elective surgery isn't started during a blackout with only generators; it's postponed or escalated to a center with full power. The hospital degrades its capacity, but doesn't lower its safety standards: it prefers to postpone a procedure than to do it without the guarantees. Degrading isn't "operate whatever it takes"; it's "operate only what can be done safely, and for the rest, escalate".
The support agent is that hospital. The fallback cascade is grid → generator → battery: the model is the grid (the best response, but it can go down), the FAQ template is the generator (answers the common without the model), and the human is the battery for the critical (always available, resolves anything). And the hospital's rule translates directly: when the model goes down and a ticket asking for a refund arrives —the "critical procedure"—, the system doesn't auto-approve (it doesn't "operate without reliable power"); it escalates to a human. It degrades its response capacity, but doesn't lower the safety standard. This lesson sets up that cascade for the support agent and measures that the feature doesn't go down.
Worked example: the cascade holds a model outage
We're going to simulate a model outage and measure two things: that the availability holds at 100% with the fallback cascade, and that the circuit breaker stops hitting a downed model. The model goes down from request 3 to 11 (nine requests failing in a row, like a rate-limit window). Each ticket tries the model (governed by the breaker); if the model fails, it drops to the FAQ template for common questions, or escalates to a human for the rest. The human never auto-approves.
# M8 Lesson 6 — RESILIENCE (M5) of the support agent: when the model goes down,
# the system does NOT go down. Fallback cascade (model -> FAQ template -> escalate
# to human, which never auto-approves) + circuit breaker over the model. The human
# NEVER auto-approves a refund: escalating is safe degradation. Model STUB;
# no network or APIs. Mechanics in depth: resilience-and-reliability-patterns-guide.
MODEL_COST = 0.002 # $ per model call attempt
TIMEOUT_MS = 800 # ms lost when a failure exhausts the timeout
class ModelError(Exception):
pass
N = 16
OUTAGE = set(range(3, 12)) # the model is down from request 3 to 11 (rate-limited)
# Incoming tickets. Some are common questions (the FAQ template covers them);
# others are cases only a human can resolve.
COMMON = {"tracking", "return", "shipping"}
QUERIES = ["tracking", "return", "shipping", "tracking", "shipping",
"complex_refund", "return", "tracking", "shipping",
"complex_refund", "tracking", "return", "shipping",
"tracking", "complex_refund", "return"]
def call_model(i):
if i in OUTAGE:
raise ModelError("down / rate-limited")
return "model response"
def fallback(query):
# Fallback cascade: FAQ template for the common; escalate to human the rest.
# The human is the final net: never fails and NEVER auto-approves a refund.
if query in COMMON:
return "template"
return "human"
class CircuitBreaker:
def __init__(self, fail_threshold=3, cooldown=4):
self.fail_threshold = fail_threshold
self.cooldown = cooldown
self.fails = 0
self.state = "CLOSED"
self.opened_at = None
def allow(self, now):
if self.state == "OPEN":
if now - self.opened_at >= self.cooldown:
self.state = "HALF_OPEN"
return True
return False
return True
def on_success(self):
self.fails = 0
self.state = "CLOSED"
def on_failure(self, now):
self.fails += 1
if self.fails >= self.fail_threshold:
self.state = "OPEN"
self.opened_at = now
# --- Mode A: NO fallback (model only). During the outage, the ticket drops. ---
responded_naive = 0
for i, q in enumerate(QUERIES):
try:
call_model(i)
responded_naive += 1
except ModelError:
pass # no fallback: the customer sees an error
# --- Mode B: fallback cascade + circuit breaker. ---
cb = CircuitBreaker(fail_threshold=3, cooldown=4)
tiers = {"model": 0, "template": 0, "human": 0}
model_calls = 0
print(f"{'req':<5}{'query':<20}{'breaker':<11}{'tier served':<14}quality")
print("-" * 66)
for i, q in enumerate(QUERIES):
if cb.allow(i):
state = cb.state
try:
call_model(i)
cb.on_success()
model_calls += 1
tiers["model"] += 1
print(f"{i:<5}{q:<20}{state:<11}{'model':<14}optimal")
continue
except ModelError:
model_calls += 1
cb.on_failure(i)
else:
state = "OPEN"
tier = fallback(q)
tiers[tier] += 1
print(f"{i:<5}{q:<20}{state:<11}{tier:<14}degraded")
print("-" * 66)
avail_naive = responded_naive / N * 100
print(f"Availability: WITHOUT fallback {responded_naive}/{N} = {avail_naive:.0f}%"
f" | WITH cascade {N}/{N} = 100%")
print(f"Tiers served: model={tiers['model']} template={tiers['template']}"
f" human={tiers['human']}")
print(f"Model calls (with breaker): {model_calls} "
f"(without the breaker they'd have been {N}: the breaker avoided {N - model_calls})")
print("The system answered the 16 tickets during the outage; the breaker stopped")
print("hitting a rate-limited model; the human covered what the template didn't.")What to expect. When you run the file, the output is exactly this:
req query breaker tier served quality
------------------------------------------------------------------
0 tracking CLOSED model optimal
1 return CLOSED model optimal
2 shipping CLOSED model optimal
3 tracking CLOSED template degraded
4 shipping CLOSED template degraded
5 complex_refund CLOSED human degraded
6 return OPEN template degraded
7 tracking OPEN template degraded
8 shipping OPEN template degraded
9 complex_refund HALF_OPEN human degraded
10 tracking OPEN template degraded
11 return OPEN template degraded
12 shipping OPEN template degraded
13 tracking HALF_OPEN model optimal
14 complex_refund CLOSED model optimal
15 return CLOSED model optimal
------------------------------------------------------------------
Availability: WITHOUT fallback 7/16 = 44% | WITH cascade 16/16 = 100%
Tiers served: model=6 template=8 human=2
Model calls (with breaker): 10 (without the breaker they'd have been 16: the breaker avoided 6)
The system answered the 16 tickets during the outage; the breaker stopped
hitting a rate-limited model; the human covered what the template didn't.
Read the trace and then the metrics, because together they show the cascade and the breaker working.
Outside the outage, everything goes through the model. Requests 0-2 and 13-15 were served by model, in quality optimal: the model was healthy, the cascade charged no cost. The resilience doesn't take quality from the happy path; it only kicks in when the model fails.
During the outage, the cascade drops a level according to the ticket. Look at requests 3-12. The ones asking common things —tracking, return, shipping— drop to the FAQ template (template): a deterministic response, without the model, that covers the bulk of the volume. The ones asking for something only a human can resolve —complex_refund— escalate to a human (human): slower and more expensive, but safe. Notice the containment decision in requests 5 and 9: they're complex refunds that arrived during the model outage, and the system didn't auto-approve them —it escalated them to a human—. That's the hospital's rule: when the reliable power (the model) is missing, the critical procedure (the refund) isn't done blindly; it's escalated. The human is the only guarantee for an action that touches money when the model isn't there.
The availability: from 44% to 100%. Without fallback, the system answered 7 of 16 —it failed on the 9 requests of the outage— = 44%. With the cascade, it answered 16 of 16 = 100%. The model had the same availability in both cases; what changed is that the system stopped depending on the model to answer. And the breakdown by tier tells the story: 6 optimal, 8 by template, 2 by human —10 degraded but valid responses that, without the cascade, would have been errors—. Degrading is qualitatively different from going down: a system that goes down tells the customer "I can't help you"; one that degrades tells them "I can help you a little worse" —or, for the refund, "a human will attend to you"—.
The circuit breaker: 10 calls instead of 16. Follow the breaker column. In requests 3-5 the breaker is CLOSED —it didn't know about the outage yet—, so it calls the model, fails, and counts failures (1, 2, 3). On the third failure, it opens (OPEN): in requests 6-8 it doesn't call the model, it goes straight to the fallback —there's the saving, three avoided calls to a model we already knew was down—. On 9 (HALF_OPEN) it lets a probe attempt through, which fails (the model is still down), and it reopens. On 13 (HALF_OPEN) the attempt works (the model recovered) and it closes. Total: 10 model calls instead of 16 —the breaker avoided 6—. And here's what's AI-specific: each avoided call isn't just latency saved; it's money saved (each call costs tokens) and load you didn't put on a model that may be down precisely because it's saturated with calls. The breaker cuts that vicious circle.
Going deeper: the rules of a good cascade applied to support
The last level must be the most robust, and for support that level is human. Module 5's golden rule: the lowest level of the cascade must be the most robust of all, because it's the one that catches what the ones above let fall. In the semantic search, that level is keyword —deterministic, local, always returns something—. In the support agent it's the human, and the difference matters. The search can have a deterministic last resort because a slightly worse order of results doesn't harm anyone; support touches money and complaints, where a human is the only universal guarantee —they can attend to any ticket, even the ones no template foresaw, and they're the only "authority" who can approve a refund when the model isn't there—. The last resort is sized by what the feature needs: deterministic for the tolerant, human for the intolerant.
The fallback shouldn't share the fragile dependency. Another module 5 rule: the final level of the cascade can't depend on the same thing as the preferred route. If your fallback were "if the big model fails, use the small model from the same provider", the day the entire provider goes down, both models go down together and the fallback is useless. That's why the agent's cascade ends in two routes that don't depend on the model: the FAQ template (deterministic, runs in your process) and the human (a person, not an API). Both survive the outage they're supposed to cover. A fallback that depends on the same thing as the preferred route is a fake fallback.
Degrade honestly: mark the degraded. In the trace, each response carries its quality (optimal or degraded). That marking has two uses. One, for the monitoring (lesson 7's observability): if suddenly 80% of the responses are degraded, you have a model outage in progress and you want to know it. Two, for the user, when it applies: a customer whose ticket was escalated to a human deserves to know "an agent will attend to you shortly", not an invented response pretending to be from the normal system. What you never do is serve a degraded response pretending it's optimal. And for support there's a third use, specific to a feature that touches money: the human marking on a complex refund is the audit trace that that refund was approved by a person, not the downed model.
The breaker over a model has an extra motivation. A classic circuit breaker saves latency and protects threads. A breaker over a model also saves money (each call costs tokens) and avoids making a rate limit worse (if the model is down from being saturated, continuing to call it prolongs its saturation). Retrying a 429 keeps consuming quota; opening the breaker doesn't. That's why, for the rate limit, the breaker isn't just a latency optimization: it's the difference between recovering in a minute or staying blocked for ten. The mechanics of the states and their calibration live in resilience-and-reliability-patterns-guide; here it's enough to see that the breaker, applied to a model, protects lesson 3's budget during an outage.
Common mistakes
Not having a fallback and dropping everything when the model goes down. What happens: the agent is a direct call to the model with no alternate route; the day of a provider outage, Mercado's support stops functioning completely, exactly when the most customers are writing (a model outage usually coincides with incidents that generate tickets). Why it happens: the happy path was designed, the failure was never seen in development. How to detect it: trace what happens when the model call throws an exception; if the answer is "the ticket fails", you have no fallback. How to fix it: put a cascade with a robust final level —for support, the human—. The example measures it: without fallback 44%, with cascade 100%.
Auto-approving during degradation "so as not to bother the human". What happens: the team, seeking the feature to "keep working" during an outage, makes the fallback automatically approve small refunds without the model or the human —"they're small amounts anyway"—. An attacker (or a bug) who knows that during outages there's auto-approval exploits exactly that moment, and refunds go out without any real validation. Why it happens: "degrading the response" is confused with "degrading the security", and auto-approving seems more convenient than escalating. How to detect it: your fallback route has a path that executes an action that touches money without the model or a human. How to fix it: degrade the response quality, yes; degrade the security, never. The last resort for an action that touches money is to escalate to a human, not auto-approve. It's the hospital's rule: the critical procedure is postponed, not done without guarantees.
A fallback that shares the fragile dependency. What happens: the team puts as a fallback "if the model fails, use a simpler model from the same provider"; the day the entire provider goes down, both go down together and the fallback is useless. Why it happens: the fallback was thought of as "another way to do AI" without seeing that it shares the fragile dependency (the network to the provider). How to detect it: your last fallback level calls an external API or the same provider as the preferred route. How to fix it: the final level must be local or human, without the fragile dependency —the FAQ template runs in your process, the human is a person—. The fallback must survive the outage it's supposed to cover.
Exercises
Exercise 1 — Why the complex refund escalates to a human and not to a template. In the cascade, the common questions drop to the FAQ template, but the complex_refund escalates to a human. Explain why it couldn't drop to a template, and what principle of the intolerant feature (tolerance 3) justifies it.
See solution
A complex refund can't drop to a template because a template can't make the decision the refund requires. FAQ templates work for questions whose answer is fixed and the same for everyone ("how long does shipping take?" → "3 to 5 days"). But a complex refund —"it arrived broken, I paid with two cards, I want a partial refund to one and the rest to store credit"— requires understanding several conditions and deciding an action that touches money, with logic specific to that case. No template foresaw that combination, and a template that "guessed" a response to a refund would be worse than not answering —it could promise something incorrect or, worse, approve a wrong amount—.
The principle of the intolerant feature (tolerance 3) that justifies it: when the model isn't there, an action that touches money can only be authorized by a human, never by a degraded automatic rule. Tolerance 3 means every refund proposal must be validated with full rigor —and during a model outage, the rigor that's missing (the understanding of the case plus the shell's validation) is only provided by a human—. Escalating isn't a luxury; it's the only way to keep the feature available without lowering its safety standard. For a tolerant feature (the search), the last resort can be deterministic because there's no money decision to make; for the refunds agent, the last resort is human because there is one. The cascade is sized by the tolerance, just like the shell.
Exercise 2 — The breaker's cooldown tradeoff. Look at request 12 of the trace: the breaker is OPEN and serves template, even though the model already recovered at request 11. Explain why the breaker served an "extra" fallback, what tradeoff it represents, and why it's almost always worth it.
See solution
At request 12 the breaker served a fallback even though the model already worked because the breaker didn't know yet. The outage ended at request 11, but the breaker was in OPEN in its cooldown (it reopened at request 9 after the failed probe attempt, and its cooldown of 4 hadn't elapsed: 12 − 9 = 3 < 4), so it didn't probe the model at 12 —it went straight to the fallback—. The breaker doesn't probe until request 13, where it detects the recovery and closes.
The tradeoff it represents: during the cooldown, the breaker serves some extra fallback —degraded responses when the model was already available— in exchange for not hitting the downed model over and over during the real outage. A short cooldown detects the recovery fast (fewer extra fallbacks) but probes more often (more probe calls, more risk of reopening falsely); a long one protects more but is slow to notice the recovery. It's a deliberate trade: "some extra degraded response" for "many fewer useless calls".
Why it's almost always worth it: during an outage, the number of useless calls avoided (and the money, and the load you don't put on the rate-limited model) far exceeds the cost of a few extra degraded responses at the end of the cooldown. In the example, the breaker avoided 6 calls in exchange for serving one extra degraded response (request 12). That trade is favorable in almost any real scenario, where outages last much longer than the cooldown. The exact calibration —how much cooldown— lives in resilience-and-reliability-patterns-guide.
Exercise 3 — Partial degradation of an agent response. The support agent, besides answering the ticket, shows in its response: (a) the order's status (from the database), (b) a response drafted by the model, and (c) a "related products" suggestion generated by AI. If the model goes down, which part do you degrade and how, and which part isn't touched? Why is it better than dropping the whole response?
See solution
With the model down, you degrade only what depends on the AI and leave intact what doesn't:
- (a) Order status → NOT touched. It comes from the deterministic database, not the model. The model outage doesn't affect it; it's shown normally. The customer still sees where their order is.
- (b) Response drafted by the model → degrade. It drops to a FAQ template if the question is common ("how do I return this?" → the returns template), or to an escalation message ("an agent will attend to you shortly") if it isn't. Worse drafted, but informative and honest.
- (c) AI related-products suggestion → degrade or hide. It drops to non-personalized recommendations (the best-sellers, computed without AI) or the section is simply hidden —a support response without "related products" is still perfectly useful—.
Why it's better than dropping the whole response: most of what the customer needs doesn't depend on the model. Their order's status —the most important information in a support ticket— comes from the database and is available even if the model goes down. Dropping the whole response over a failure that only affects the drafted part and a secondary suggestion would deny the customer the critical information (where their order is) over a problem in the accessory parts. Partial degradation isolates the failure to the affected parts and keeps the core functional. It's graceful degradation in its finest form: each part degrades to its own fallback, and what doesn't depend on the model doesn't even notice. The mechanics of this pattern live in resilience-and-reliability-patterns-guide.
Summary and next step
In this lesson you filled the sheet's fallback field: the resilience of the support agent. With the hospital and its power cascade (grid → generator → battery, and the critical procedures that are postponed instead of done without guarantees), you saw that degrading isn't lowering the safety standard. And you executed it: a 9-request model outage, and the fallback cascade (model → FAQ template → escalate to human) held the availability at 100% versus 44% without fallback —6 optimal, 8 by template, 2 escalated to human—. The circuit breaker trimmed the calls from 16 to 10, stopping hitting a rate-limited model. And you saw the containment decision specific to a tolerance-3 feature: when the model goes down and a refund arrives, the system doesn't auto-approve —it escalates to a human—, because the last resort for an action that touches money is human, not an automatic rule. Degrade honestly instead of going down, without lowering the safety standard.
Before moving on you should be able to: design a fallback cascade with a robust last resort (human for support); explain why the fallback shouldn't share the fragile dependency; argue why degrading is better than going down and why the security isn't degraded; and recognize the extra motivations of a circuit breaker over a model (money, rate limit).
Lesson 7 sets up the two fields that close the containment: the deterministic shell and the data loop. You'll see the shell measured in money —the model proposes, the system disposes, and a deterministic layer protects $5,190 of dangerous proposals— and you'll close the data loop: the quality observability a server log doesn't see, and the feedback loop that turns the live thumbs_down into new eval-set cases —closing the circle back toward lesson 4—. The same shell that protects the money feeds the quality.
Resources
resilience-and-reliability-patterns-guide(this ecosystem) — the central reference for the mechanics of degradation, circuit breakers, and load shedding that here we only apply: the states with precision, the count windows, the cooldown calibration, the partial degradation by priority. In Spanish.- Martin Fowler and Bharani Subramaniam, "Emerging Patterns in Building GenAI Apps" — martinfowler.com/articles/gen-ai-patterns. It treats fallbacks and degraded routes around an AI component as first-class patterns. In English.
- Anthropic, Claude documentation — docs.anthropic.com. The rate limits pages describe the quotas and the behavior of the 429 —the failure mode that makes the breaker over a model more than a latency optimization—. Conceptual, without pinning a version. In English.
architecture-for-ai-native-systems-guide, Module 5 (this ecosystem) — the in-depth treatment of the fallback, the circuit breaker, the timeout, the degradation, and the drift. This lesson applies to the support agent what M5 developed. In Spanish.