Module 8: Project — Architect an AI Feature in Mercado

The budget, the cascade, and the cache

Next →

Overview

With the component placed and its sheet issued, the method moves to step 2: make the feature affordable. The support agent is slow (hundreds of milliseconds per turn) and costs money (each model call is charged per token), so before armoring it you have to fit it into its latency and cost budget —because a feature that doesn't fit in its budget doesn't reach production, however good it is—. This lesson fills the latency_budget and cost_budget fields of the sheet with module 2's two levers: the model cascade (cheap first, escalate only if needed) and the cache (don't repay the repeated), measured over 1000 agent tickets.

And it brings the honest lesson module 2 anticipated and that almost no one explains: no single lever is a silver bullet. You'll see three strategies —send everything to the expensive model, the cascade alone, and cascade + cache— compared against the feature's budget, and you'll discover that the cascade alone trims the cost but stays above the margin, and that you need to compose the cascade with the cache to fit the feature inside. It's the difference between "I applied an optimization technique" and "I designed the feature's budget".

Connection with the module. This is the first layer of the shell built on lesson 2's sheet, and the most "operational" one —it deals with cost and latency, not security or quality—. It's also where two pieces the following lessons reuse appear: the cascade's classifier (which lives before the LLM and decides which to call) and the cache (which in lesson 6 will serve a double function as a fallback route). The boundary with AI Engineering holds: here we treat the component's cost and latency as architecture constraints; the optimization of the inference on the inside —quantization, batching, GPU— is infra/AI Engineering, not this module.

An analogy: the call center with a script of frequent responses

Go back to module 2's call center, the one that serves with cheap juniors and escalates only the hard stuff to the expensive senior. That's the cascade, and you already know it. But a well-run call center has a second economy, even more powerful, that the cascade doesn't capture: a script of frequent responses. Most calls ask the same thing —"what are your hours?", "how do I track my order?", "how do I return this?"—, and for those, the agent doesn't even think: they read the response already written in the script. It's not that a cheap junior answers it; it's that no one answers it from scratch —the response already existed, computed beforehand—.

That's the cache. The cascade cheapens each call (junior instead of senior); the cache eliminates entire calls (the response was already written, no need to generate it). And notice that the two economies compose: first the script catches the repeated questions (free), and only what's not in the script passes to the cascade, which routes it to the junior or the senior according to its difficulty. A call center that had only the cascade —no script— would have a junior draft the hours response from scratch a thousand times a day; one that had only the script —no cascade— would send all new questions to the senior. The two together are what makes the call center economically viable. In Mercado's support agent, the cascade picks the model by difficulty and the cache serves the repeated questions in one shot; this lesson measures how much each saves and why both are needed.

Worked example: three strategies against the budget

We're going to measure the support agent over 1000 tickets and compare three strategies against its budget (latency 4000 ms, cost $3000/mo). The workload is realistic: 30% of the tickets are hard (they need the expensive model) and 40% are repeated questions (cache candidates). We measure: all_strong (everything to the expensive model, the baseline), cascade (classify and route), and cascade_cache (the two levers together). All with module 2's cost/latency model, fixed seed.

# M8 Lesson 3 — budget + model cascade + cache (M2), over the support
# agent. We measure three strategies over 1000 tickets and compare against
# the feature's latency/cost budget. LLM STUB: all SIMULATED, fixed seed.
import random

# Cost/latency model identical to module 2's (consistency).
MODELS = {
    "cheap":  dict(usd_in=0.0008, usd_out=0.004, base_ms=90,  ms_per_tok=0.4),
    "strong": dict(usd_in=0.008,  usd_out=0.040, base_ms=300, ms_per_tok=3.0),
}

# The feature's budget (M2). The component must fit here.
LATENCY_BUDGET_MS = 4000          # the customer doesn't wait more than ~4 s
COST_BUDGET_USD_MONTH = 3000      # monthly margin for the support agent


def call_llm(model, in_tokens, out_tokens):
    m = MODELS[model]
    latency_ms = m["base_ms"] + out_tokens * m["ms_per_tok"]
    cost_usd = (in_tokens / 1000) * m["usd_in"] + (out_tokens / 1000) * m["usd_out"]
    return latency_ms, cost_usd


random.seed(7)
N = 1000
IN_TOK = 300
OUT_EASY, OUT_HARD = 120, 300
CLASSIFIER_MS = 3

# The workload: 1000 tickets. 30% hard (reasoning). Also, 40% are REPEATED
# questions (tracking, returns, hours) -> cache candidates.
tickets = []
for _ in range(N):
    true_hard = random.random() < 0.30
    repeated = (not true_hard) and (random.random() < 0.55)   # the easy ones repeat
    tickets.append((true_hard, repeated))


def classify(true_hard):
    # The cheap classifier: right 92% of the time (imperfect, as in reality).
    return true_hard if random.random() < 0.92 else (not true_hard)


def out_tokens_for(true_hard):
    return OUT_HARD if true_hard else OUT_EASY


def pct(sorted_vals, p):
    return sorted_vals[int(len(sorted_vals) * p)]


def measure(strategy):
    total_cost, lats, cache_hits, model_calls = 0.0, [], 0, 0
    for true_hard, repeated in tickets:
        if strategy == "cascade_cache" and repeated:
            # Cache HIT: response already computed. No model call.
            lats.append(5.0)          # serving from cache: ~5 ms
            cache_hits += 1
            continue
        if strategy == "all_strong":
            model = "strong"
            extra_ms = 0.0
        else:  # cascade or cascade_cache
            model = "strong" if classify(true_hard) else "cheap"
            extra_ms = CLASSIFIER_MS
        lat, cost = call_llm(model, IN_TOK, out_tokens_for(true_hard))
        total_cost += cost
        lats.append(lat + extra_ms)
        model_calls += 1
    lats.sort()
    return total_cost, sum(lats) / len(lats), pct(lats, 0.95), cache_hits, model_calls


rows = []
for name in ("all_strong", "cascade", "cascade_cache"):
    rows.append((name,) + measure(name))

print(f"{'strategy':<16}{'cost_1k':>10}{'avg_ms':>9}{'p95_ms':>9}{'cache':>7}{'calls':>7}")
for name, cost, avg, p95, hits, calls in rows:
    print(f"{name:<16}{cost:>10.4f}{avg:>9.1f}{p95:>9.1f}{hits:>7}{calls:>7}")

base = rows[0][1]
best = rows[2][1]
print(f"\nCost savings cascade+cache vs all_strong: "
      f"{(1 - best/base)*100:.1f}%  (${base:.4f} -> ${best:.4f} for {N} tickets)")

# Monthly scale: 20k tickets/day (realistic support volume).
DAILY_TICKETS = 20_000
scale = DAILY_TICKETS * 30 / N
print("\n=== Against the budget (M2) ===")
print(f"  latency_budget = {LATENCY_BUDGET_MS} ms   cost_budget = ${COST_BUDGET_USD_MONTH}/mo")
for name, cost, avg, p95, hits, calls in rows:
    monthly = cost * scale
    lat_ok = "OK" if p95 <= LATENCY_BUDGET_MS else "EXCEEDS"
    cost_ok = "OK" if monthly <= COST_BUDGET_USD_MONTH else "EXCEEDS"
    print(f"  {name:<16} ${monthly:>8,.0f}/mo [{cost_ok:<7}]   p95={p95:>6.0f}ms [{lat_ok}]")
print("\nOnly cascade+cache fits the feature into its cost budget. The cascade")
print("alone trims, but isn't enough: the cache —not repaying the repeated— closes the gap.")

What to expect. When you run the file, the output is exactly this:

strategy           cost_1k   avg_ms   p95_ms  cache  calls
all_strong          9.5544    836.6   1200.0      0   1000
cascade             5.0638    479.5   1203.0      0   1000
cascade_cache       4.6454    415.8   1203.0    383    617

Cost savings cascade+cache vs all_strong: 51.4%  ($9.5544 -> $4.6454 for 1000 tickets)

=== Against the budget (M2) ===
  latency_budget = 4000 ms   cost_budget = $3000/mo
  all_strong       $   5,733/mo [EXCEEDS]   p95=  1200ms [OK]
  cascade          $   3,038/mo [EXCEEDS]   p95=  1203ms [OK]
  cascade_cache    $   2,787/mo [OK     ]   p95=  1203ms [OK]

Only cascade+cache fits the feature into its cost budget. The cascade
alone trims, but isn't enough: the cache —not repaying the repeated— closes the gap.

Read the top table and then the comparison against the budget, because together they tell the complete story of step 2.

Each lever trims, and the trimming composes. The all_strong baseline (everything to the expensive model) costs $9.55 per 1000 tickets. The cascade lowers it to $5.06 —almost half—, routing the 70% easy ones to the cheap model. And cascade_cache lowers it to $4.65, because it also serves 383 of the 1000 tickets from the cache (calls drops from 1000 to 617: 383 tickets never called the model). The total savings is 51.4% versus sending everything to the expensive one. Notice that the two levers attack different things: the cascade cheapens the calls that are made (from $5000 to $500 on the easy ones), the cache eliminates entire calls. That's why they compose —the cache trims what the cascade left—.

But the number that matters is the budget one, and there's the honest lesson. Look at the "Against the budget" section. At a scale of 20,000 tickets/day:

  • all_strong costs $5,733/mo → EXCEEDS the $3,000 margin. Predictable: sending everything to the expensive one is expensive.
  • cascade costs $3,038/mo → EXCEEDS. And this is the point: the cascade trimmed almost half and still stays above the budget, barely —by 38 dollars, but above—. A feature that costs $3,038 when the margin is $3,000 doesn't reach production; "almost fits" is not fitting.
  • cascade_cache costs $2,787/mo → OK. Only by composing the two levers does the feature enter its budget.

That's the whole argument of step 2: no single lever is a silver bullet. The cascade is the most famous and biggest lever, and yet alone it isn't enough for this feature —it trims 47% and stays a hair above the margin—. You need the cache to close the gap. An architect who applies only the cascade and declares "done, I optimized the cost" puts into production a feature that exceeds its budget by a little; one who composes cascade + cache fits it inside. The difference between the two is exactly those 38 dollars that separate "EXCEEDS" from "OK".

And a note on latency, honest as in module 2. The p95 of the three strategies is practically identical (~1200 ms) and they all fit in the 4000 ms latency_budget. The cascade and the cache lowered the average (from 837 to 416 ms) but not the tail latency —the hard queries still go to the slow model in all strategies—. Here that's not a problem (1200 ms fits comfortably in support's 4-second budget), but it's the same nuance as module 2: the cascade fixes the cost and the average latency, not the tail. If the latency budget were strict (as in the search), streaming or other techniques would be needed —but this feature has slack, so the cascade + cache resolve its step 2 in full—.

Going deeper: composing levers and the classifier as a new piece

The budget is a business constraint, not a technical number. Where do the $3,000/mo and the 4,000 ms come from? Not from engineering. The cost_budget comes from the margin the business can spend on AI support; the latency_budget, from how long a customer is willing to wait before getting frustrated. Just as the eval's threshold (lesson 4) comes from the risk, the budget comes from the business. Engineering doesn't invent the budget; it receives it and designs to fit inside. That's why the "EXCEEDS/OK" verdict isn't a technical opinion —it's the feature against a line the business drew—. A budget engineering invents without talking to whoever pays the bill is a sign with no authority, as module 2 warned.

The classifier is a new, deterministic, cheap piece that governs the expensive ones. The cascade introduced something the following lessons reuse: a piece —the classifier— that lives before the LLM and decides which to call. It's deterministic (a heuristic or a tiny model), cheap (it doesn't cost a big-model call), and its job is only to route, not to answer. It's the first example of a pattern that runs through the whole capstone: deterministic and cheap components that govern the probabilistic and expensive ones. The classifier governs which model each ticket goes to; the guardrails (lesson 5) govern which proposals pass; the shell (lesson 7) governs which actions are executed. All are deterministic code surrounding and controlling the probabilistic core.

The cascade has a quality risk the eval (lesson 4) covers. The classifier is imperfect (92% in the example), and sometimes sends a hard query to the cheap model, which may answer it worse. The cascade, then, saves but introduces a quality risk —the same one lesson 4 will catch when the "regression" that fails the eval gate turns out to be exactly a poorly calibrated cost savings (dropping too much to the cheap model)—. Here's the connection between the two lessons: the cascade gives you the savings, the eval confirms the savings didn't come at the cost of quality. That's why the classifier is calibrated so that when in doubt it escalates (prefers to overpay for an easy one than to answer a hard one badly), and that's why step 3 (budget) and step 4 (eval) go together: optimizing the cost without an eval watching the quality is optimizing blindly.

Common mistakes

Applying a single lever and declaring the budget solved. What happens: the team puts in the cascade, sees the cost drop almost in half, and concludes "we already optimized, it fits in budget" without comparing against the real margin. In this example, the cascade alone leaves the feature at $3,038/mo when the margin is $3,000 —it exceeds by a little, but exceeds—, and at month's end the bill comes in over budget. Why it happens: a big trim (47%) feels like enough, and it's easy not to verify against the business's exact number. How to detect it: you applied an optimization technique but never compared the resulting cost against the sheet's cost_budget. How to fix it: measure against the budget, not against the baseline. "It dropped 47%" isn't the verdict; "it fits in $3,000/mo" or "it doesn't fit" is. And if one lever isn't enough, compose another —cascade + cache is what closes the gap—.

Trusting a perfect classifier. What happens: the design assumes the classifier always gets it right and doesn't plan for the routing errors. In production, the classifier sends some hard queries to the cheap model, which answers them worse, and the quality degrades without anyone noticing —until the eval gate (lesson 4) catches it, or worse, until a customer complains—. Why it happens: in the diagram the classifier is drawn as a box that "decides the difficulty", and it's easy to forget that it makes mistakes. How to detect it: your cascade doesn't have an eval that verifies the quality of the queries routed to the cheap one. How to fix it: calibrate the classifier to escalate when in doubt (a cheap false positive, not an expensive false negative) and trust the eval gate to confirm that cheapening didn't break the quality. The cascade without an eval is optimizing the cost without a safety net.

Caching without thinking about freshness or personalization. What happens: the team caches aggressively to maximize the hit-rate, and serves from cache a response that no longer applies —the order's status changed, the returns policy was updated, or the response was specific to another customer—. The savings is real but the response is wrong. Why it happens: the hit-rate is optimized as if every response were cacheable, without distinguishing what's stable (an hours FAQ) from what's volatile or personal (the status of your order). How to detect it: your cache doesn't distinguish stable responses from state- or user-dependent responses. How to fix it: cache only what's stable and impersonal (FAQs, policy responses), and for the volatile, either don't cache or use a short TTL. In the example, the cacheable repeated questions are the general-policy ones (hours, how to return), not "where is my order A-1234?" —that one is computed fresh each time—.

Exercises

Exercise 1 — How much cache would it take to fit with cache alone? Suppose you remove the cascade (everything to the expensive model) but raise the cache's hit-rate. With all_strong at $9.5544/1000 tickets and a $2,787/mo margin as the target (the one cascade+cache achieved) at 20,000 tickets/day, what fraction of tickets would the cache have to serve to fit? Reason through the calculation and comment on whether it's realistic.

See solution

At 20,000 tickets/day, the month has 600,000 tickets (a scale factor of 600 over the experiment's 1000). With all_strong at $9.5544/1000, the monthly cost without cache is 9.5544 × 600 = $5,732.6/mo. To drop to $2,787/mo, you need to eliminate the corresponding fraction of cost:

  • fraction of cost to remove = (5732.6 − 2787) / 5732.6 = 2945.6 / 5732.6 ≈ 0.514, that is 51.4%.

Since each cached ticket eliminates its entire cost (it doesn't call the model), serving 51.4% of the tickets from cache would achieve the same cost as cascade+cache. That is: you'd need a hit-rate of ~51%.

Is it realistic? It depends on the workload, but it's high —higher than the 38.3% the cache achieved in the example (383/1000)—. A 51% hit-rate would require more than half the tickets to be repeated and cacheable questions, which is optimistic for support (many tickets are specific to an order, not cacheable). The moral: you could fit with cache alone if the hit-rate were very high, but it's more robust to compose —the cascade cheapens what isn't cached, and the cache eliminates the repeated—, because that way no single lever has to do all the work alone. Composing two moderate levers (cascade 47% + cache 38%) is more achievable than squeezing a single one to 51%.

Exercise 2 — The budget that doesn't fit even with both levers. Imagine the business lowers the cost_budget to $2,000/mo (instead of $3,000). With cascade+cache at $2,787/mo, the feature no longer fits. Name three architecture decisions (not inference-optimization ones, which are AI Eng) you could make to fit it, and the trade-off of each.

See solution

Three architecture decisions to fit the feature into $2,000/mo:

  1. Raise the cache's hit-rate with a semantic cache. Instead of caching only identical queries, cache by intent (two ways of asking "where is my order?" share a template response). It raises the hit-rate and lowers the cost. Trade-off: a semantic cache can serve the wrong response if two queries seem similar but aren't —you have to calibrate the similarity threshold, and that answers to the eval—.
  2. Move more traffic to the deterministic fallback for the FAQs. The most common questions (tracking, returns policy) could be answered always with deterministic templates (no AI), reserving the model only for what really needs reasoning. Trade-off: templates are more rigid and less natural; if they cover a case poorly, quality drops (the eval gate watches it).
  3. Reserve the expensive model for a higher difficulty threshold. Recalibrate the classifier so fewer queries are considered "hard", sending more to the cheap one. It lowers the cost. Trade-off: it's the dangerous one —more false negatives (hard ones to the cheap one) degrade the quality—; it's only acceptable if the eval gate confirms the quality holds. It's exactly the cost/quality tension lesson 4 governs.

What does not count as an architecture decision here: quantizing the model, changing the inference hardware, or fine-tuning a smaller and cheaper model —all of that is inference optimization, which lives in AI Engineering/infra, on the other side of the boundary—. The architect composes cache, deterministic fallback, and cascade calibration; the AI engineer optimizes the model on the inside.

Exercise 3 — Why the p95 didn't change and why it doesn't matter here. In the table, the p95_ms of the three strategies is ~1200 ms —the cascade and the cache didn't lower it—. Explain why the cascade doesn't lower the tail latency, and why in this feature (unlike the search) it's not a problem.

See solution

The cascade doesn't lower the tail latency because the p95 —the latency of the worst 5%— is dominated by the hard queries, and those keep going to the expensive and slow model in all strategies. The cascade cheapens and speeds up the easy ones (which go to the cheap model), but the hard ones take as long as they took (300 ms base + 300 tokens × 3 ms/token = 1200 ms). Since the p95 is set precisely by the hard ones, it doesn't move. The cache doesn't lower it either: it serves the repeated ones in one shot (lowering the average), but the non-repeated hard ones still pay their full latency. It's module 2's honest nuance: the cascade fixes the cost and the average latency, not the tail.

Why it doesn't matter in this feature: the support agent's latency_budget is 4000 ms, and the p95 is 1200 ms —it fits with a great deal of slack—. A customer who opens a support ticket expects a response in seconds, not milliseconds; 1.2 seconds in the worst case is perfectly acceptable. In the semantic search, on the other hand, the latency budget is strict (the user expects results almost instantly, hundreds of milliseconds), and there a p95 of 1200 ms would be a problem —you'd need streaming or another technique for the tail—. The lesson: if the technique fixes the wrong problem (the cascade doesn't fix the tail), don't blame it; verify whether the problem it doesn't fix matters in this feature. Here, with latency slack, the cascade + cache resolve step 2 in full. The same technique, evaluated against each feature's budget.

Summary and next step

In this lesson you filled the latency_budget and cost_budget fields of the sheet with module 2's two levers: the model cascade (cheap first) and the cache (don't repay the repeated), measured over 1000 support agent tickets. You saw, with the call center and its script of frequent responses, how the cascade cheapens each call and the cache eliminates entire calls, and how they compose. And you learned step 2's honest lesson: no single lever is enough —the cascade trimmed 47% and stayed a hair above the margin ($3,038 against $3,000), and only cascade + cache fit the feature into its budget ($2,787/mo, OK)—. You measured against the budget, not against the baseline, because "almost fits" is not fitting. And you saw that the cascade's classifier is the first deterministic piece that governs the probabilistic core, a pattern that runs through the whole capstone.

Before moving on you should be able to: measure a feature's cost against its budget (not against the baseline); explain why the cascade and the cache attack different things and compose; argue why no single lever is a silver bullet; and recognize that the cascade's savings answer to the eval (cheapening without verifying the quality is optimizing blindly).

Lesson 4 takes the eval_gate field of the sheet and sets up the quality gate that governs the deploy: the eval gate. You'll execute the gate over two versions of the agent —the production candidate, which passes, and a regression that fails— and you'll see that the regression that's blocked is exactly the poorly calibrated cost savings from this lesson (dropping too much to the cheap model). The budget and the eval, tied: the cascade gave you the savings, and now the eval confirms the savings didn't break the quality.

Resources

  • martinfowler.com — "Emerging Patterns in Building GenAI Applications", Bharani Subramaniam and Martin Fowler — the frame of treating the building blocks of a GenAI app as architecture decisions; the conceptual backing of this lesson's step 2. (The routing between models and the cache are measured by this step's code; Anthropic's prompt-caching docs cover the cache in detail.) In English.
  • Chip Huyen — AI Engineering (O'Reilly) — the systematic treatment of routing by difficulty, the cache, and the cost/quality trade-off that this lesson's two levers exploit. In English.
  • Anthropic — Models overview (Claude docs) — the panorama of model families by capability and price, the real ladder (small → medium → large) on which a cascade is built, without pinning a version. In English.
  • architecture-for-ai-native-systems-guide, Module 2 (this ecosystem) — the in-depth treatment of the budget, the cascade, the cache, and streaming, including the nuance of why the p95 doesn't drop with the cascade. This lesson applies to the support agent what M2 developed. In Spanish.
← PreviousPlace the component and its contractNext →The eval gate