Module 2: Latency and Cost as Architecture

4. The model cascade: cheap first, escalate only if needed

Next →

Overview

By the end of this lesson you'll know how to design and measure the highest-leverage pattern on an AI feature's cost: the model cascade. The idea is simple and powerful: don't send all queries to the most expensive model; send the easy ones to the cheap one and only the hard ones to the expensive one. For that you need one more piece —a classifier that looks at each query and decides the difficulty, to route it to the right model—. Since most queries in a real system are easy, and since the cheap model costs close to ten times less than the expensive one, routing well saves a huge fraction of the cost without losing quality where it matters. You're going to build a cascade executed over 1000 Mercado searches and measure the saving versus "everything to the expensive": you're going to see the cost and the average latency drop notably, and you're going to see an honest detail almost nobody explains —why the p95 barely drops—.

This matters because "everything to the most expensive model just in case" is the most common and most expensive cost mistake when putting AI into a system. It's made for a reasonable reason —the expensive model gives better quality— but with a wrong conclusion —"better for the hard ones" doesn't imply "better for everything"—. Most queries don't need the big model's power: "iphone" or "cheap laptop" are resolved by the small model just as well, at a tenth of the cost. Paying the premium price for every trivial query is exactly what makes Mercado's semantic search cost $25,200/month instead of a fraction. The cascade is the technique that corrects that: it reserves the expensive model for where its extra quality really pays, and charges the cheap one for the rest. It's the difference between paying specialist rates for every query and paying it only for the cases that warrant it.

Connection with the module: this is the first technique for respecting the budget of lesson 3. There you saw that "everything to the strong model" violates the cost budget ($25,200 against $3,000); here you attack it head-on, routing the easy traffic to the cheap one. It's also the first time a new architecture piece appears —the classifier— that lives before the LLM and decides which one to call: a deterministic, cheap component that governs the probabilistic, expensive ones. Lesson 5 (cache) is the other big cost lever, and it operates complementarily: the cascade makes each call cheaper, the cache eliminates repeated calls. Lesson 7 composes them and shows that together —not either alone— they bring the feature within its budget. And the p95 detail you'll see here connects directly with lesson 6: the cascade doesn't fix the tail latency, for that you need streaming.

The call center that handles with the junior and escalates to the senior

Think of it this way. A well-run call center receives thousands of calls a day, and has two types of agent: juniors, many, cheap, fast with the routine; and seniors, few, expensive, experts in the complicated. The novice manager's temptation is to put the seniors to handle everything —"that way I guarantee the best service on every call"—. It's an economic disaster: most calls are easy ("what are your hours?", "how do I reset my password?"), and a senior resolving an hours question is a very expensive specialist doing a beginner's work. The serious manager does something else: all calls come in through the juniors, who resolve most in seconds; and only when the junior detects a hard case —a complex complaint, something they can't resolve— it escalates to the senior. The rule is "handle with the cheap one, escalate to the expensive one only when needed."

The model cascade is exactly that call center. The cheap model is the army of juniors: fast, cheap, perfectly capable with most of the traffic. The strong model is the handful of seniors: slow, expensive, reserved for the hard. And there's a piece that acts as receptionist: the classifier, which looks at each incoming query and decides whether it's a junior case (easy → cheap) or a senior case (hard → strong). The classifier is cheap —a heuristic or a tiny model, not the big model— because its job is only to route, not to answer.

There are two ways to set up the routing, both valid and both present in the call center. The first is classify-and-route: the receptionist listens to the first sentence and decides junior or senior before passing the call. The second is try-cheap-and-escalate: the call always goes first to the junior, and the junior itself, if it sees it can't handle it, passes it to the senior. The first is cleaner to measure and it's the one we execute in the example; the second has its own cost nuance (the hard ones pay junior and senior) that we'll see in the deep-dive. In both cases, the principle is the same, and it's the heart of the saving: don't put the senior to answer what time you open.

Worked example: the cascade that measures its saving

We're going to set up the cascade over a realistic workload and measure how much it saves versus sending everything to the strong model. The workload is 1000 searches, of which 30% are "hard" (they need the big model's reasoning) and 70% easy. A cheap classifier —which costs ~3 ms and $0 because it's a heuristic, not an LLM call— predicts the difficulty and routes. The classifier is imperfect: it's right 92% of the time (sometimes it sends an easy one to the expensive one, or a hard one to the cheap one), because no real classifier is perfect and it's honest to measure with that imperfection included. The hard queries ask for longer responses (300 output tokens) than the easy ones (120), which makes the hard ones intrinsically slower and more expensive regardless of the model.

Everything with a fixed seed so the numbers are reproducible. We measure two strategies: the baseline (everything to the strong model) and the cascade (classify and route).

# Lesson 04 — the model cascade (cheap first, escalate only if needed)
# LLM STUB: everything SIMULATED. Zero network, zero API, zero keys. Fixed seed.
import random

MODELS = {
    "cheap":  dict(usd_in=0.0008, usd_out=0.004, base_ms=90,  ms_per_tok=0.4),
    "strong": dict(usd_in=0.008,  usd_out=0.040, base_ms=300, ms_per_tok=3.0),
}

def call_llm(model, in_tokens, out_tokens):
    m = MODELS[model]
    latency_ms = m["base_ms"] + out_tokens * m["ms_per_tok"]
    cost_usd = (in_tokens / 1000) * m["usd_in"] + (out_tokens / 1000) * m["usd_out"]
    return latency_ms, cost_usd

# --- The workload: 1000 searches. 30% are "hard" (need reasoning). ---
random.seed(7)
N = 1000
IN_TOK = 300
OUT_EASY, OUT_HARD = 120, 300      # the hard ones ask for a longer response

queries = []
for _ in range(N):
    true_hard = random.random() < 0.30
    queries.append(true_hard)

# --- The cheap classifier: predicts difficulty and routes. Imperfect (92% accuracy). ---
CLASSIFIER_MS = 3          # a cheap heuristic: ~3 ms
CLASSIFIER_COST = 0.0      # rule/heuristic: no LLM call cost
def classify(true_hard):
    return true_hard if random.random() < 0.92 else (not true_hard)

def out_tokens_for(true_hard):
    return OUT_HARD if true_hard else OUT_EASY

def pct(sorted_vals, p):
    return sorted_vals[int(len(sorted_vals) * p)]

def measure(route_fn):
    """route_fn(true_hard) -> model name. Returns (total_cost, avg_lat, p95_lat, extra_ms)."""
    total_cost, lats = 0.0, []
    for true_hard in queries:
        extra_ms, extra_cost, model = route_fn(true_hard)
        lat, cost = call_llm(model, IN_TOK, out_tokens_for(true_hard))
        total_cost += cost + extra_cost
        lats.append(lat + extra_ms)
    lats.sort()
    return total_cost, sum(lats) / len(lats), pct(lats, 0.95)

# Baseline: EVERYTHING to the expensive model.
def all_strong(true_hard):
    return 0.0, 0.0, "strong"

# Cascade: the classifier routes easy->cheap, hard->strong.
def cascade(true_hard):
    pred_hard = classify(true_hard)
    return CLASSIFIER_MS, CLASSIFIER_COST, ("strong" if pred_hard else "cheap")

base_cost, base_avg, base_p95 = measure(all_strong)
casc_cost, casc_avg, casc_p95 = measure(cascade)

print(f"{'strategy':<14}{'total_cost':>12}{'avg_lat_ms':>12}{'p95_lat_ms':>12}")
print(f"{'all_strong':<14}{base_cost:>12.4f}{base_avg:>12.1f}{base_p95:>12.1f}")
print(f"{'cascade':<14}{casc_cost:>12.4f}{casc_avg:>12.1f}{casc_p95:>12.1f}")

print(f"\nCost saving:       {(1 - casc_cost/base_cost)*100:>5.1f}%  "
      f"(${base_cost:.4f} -> ${casc_cost:.4f} for {N} searches)")
print(f"Avg latency saving:{(1 - casc_avg/base_avg)*100:>5.1f}%  "
      f"({base_avg:.1f} ms -> {casc_avg:.1f} ms)")

# Monthly scale: 100k searches/day.
scale = 100_000 * 30 / N
print(f"\nAt scale ({100_000:,}/day): all_strong ${base_cost*scale:,.0f}/month -> "
      f"cascade ${casc_cost*scale:,.0f}/month")

What to expect. When you run it:

strategy        total_cost  avg_lat_ms  p95_lat_ms
all_strong          9.6192       841.4      1200.0
cascade             5.4137       506.7      1203.0

Cost saving:        43.7%  ($9.6192 -> $5.4137 for 1000 searches)
Avg latency saving: 39.8%  (841.4 ms -> 506.7 ms)

At scale (100,000/day): all_strong $28,858/month -> cascade $16,241/month

Here's the call center working, with its numbers. Read them carefully, because there's a big piece of good news and an honest nuance just as important.

The good news: the saving is enormous. The cascade costs 43.7% less than sending everything to the expensive one ($5.41 against $9.62 per 1000 searches) and its average latency drops 39.8% (507 ms against 841 ms). Where does it come from? From the fact that 70% of the queries are easy and now the cheap model handles them instead of the expensive one: each of those pays ~$0.00072 instead of ~$0.0027, and takes 150 ms instead of 750. Seven hundred queries that drop in price and time move the total hugely. At monthly scale, this is the difference between $28,858/month (everything to the expensive) and $16,241/month (cascade) —$12,600 a month saved with a single architecture decision, without touching the model or the hardware, just routing—.

The honest nuance: the p95 barely drops. Look at the p95_lat_ms column: the baseline has 1200 ms and the cascade 1203 ms —practically equal, even a hair worse because of the classifier's 3 ms—. Why doesn't the tail latency improve if the average dropped 40%? Because the p95 —the latency of the worst 5%— is dominated by the hard queries, and those still go to the strong model in both strategies. The cascade makes the easy ones cheaper and faster, but the hard ones still take what they took (300 + 300×3 = 1200 ms). This is crucial: the cascade fixes the cost and the average latency, but does NOT fix the tail latency. The user whose query is hard still waits 1.2 seconds. And remember from lesson 3 that the serious latency budget is measured on the tail: the cascade alone, then, isn't enough to respect a strict latency budget on the heavy queries. For that you need another technique —streaming (lesson 6)—, which shows the response while it's being generated. The cascade and streaming attack different problems: the cascade the cost and median latency, streaming the perceived latency on the tail.

And a warning lesson 3 anticipated: the cascade at scale costs $16,241/month, still above the $3,000 budget. The cascade alone doesn't bring the feature within its cost budget —it cuts almost half, but it's not enough—. The other big lever is missing: the cache (lesson 5), which eliminates the repeated traffic. No single technique is a silver bullet; they compose. That's what lesson 7 is going to demonstrate.

Going deeper: classify-and-route vs try-and-escalate, and the risk of mis-routing

The two forms of the cascade. The example uses classify-and-route: a cheap classifier predicts the difficulty and sends each query to a single model. Its cost is clean: each query pays for one model (plus the nearly free classifier). The other form is try-and-escalate: the query goes first to the cheap model, and if the cheap one "isn't sure" (low confidence in its own response), it's escalated to the strong one. Its advantage is that it doesn't need a separate classifier —the cheap model itself decides whether to escalate—. Its cost has a nuance: the queries that get escalated pay for two calls (cheap + strong), not one. If 30% escalates, that 30% costs more than in the classify-first approach. Which is better depends on how good your classifier is versus how cheap the cheap model is: if the classifier is expensive or unreliable, try-and-escalate may come out better; if the cheap model is cheap but the double call adds up, classify-first wins. Both are a cascade; both honor the "cheap first" principle.

The cost of mis-routing, and why it touches quality. The classifier is imperfect (92% in the example), and it errs in two ways, with different consequences:

  • Sends an easy one to the expensive one (false positive of difficulty): it wastes money —pays premium for something the cheap one would have resolved— but doesn't harm quality (the expensive one answers an easy one fine). It's an efficiency mistake, not a correctness one.
  • Sends a hard one to the cheap one (false negative): it saves money but can harm quality —the cheap model may answer worse a query that needed the strong one—. It's a correctness mistake, and it's the dangerous one.

Here it touches the boundary with module 3. This module measures the cascade's saving assuming the routing is correct; verifying that quality doesn't drop when you go cheaper —that the hard ones mis-routed to the cheap one don't ruin the experience— is the eval gate's job (module 3). The two go together: the cascade gives you the saving, the eval confirms the saving didn't come at the cost of bad responses. A cascade without an eval is optimizing cost blindly; with an eval, it's optimizing cost with a safety net. That's why, in practice, the classifier is calibrated so that when in doubt it escalates (prefers the cheap false positive cheap-to-expensive, which only costs money, over the false negative hard-to-cheap, which costs quality).

More than two levels. The cascade doesn't have to be cheap/expensive: it can be a ladder of three or more models (tiny → small → large), each handling its band of difficulty. The principle is the same —each query to the smallest model that resolves it well— and the saving compounds. In the real world, this maps to model families: something like Claude Haiku for the trivial, something like Claude Sonnet for the intermediate, something like Claude Opus for the hardest. How many levels to put is a trade-off between the extra saving and the complexity of maintaining the routing —each additional level is another threshold to tune—.

Common mistakes

Sending everything to the most expensive model "just in case" (over-engineering the cost). What happens: to guarantee the best quality, the team routes 100% of the traffic to the strong model. The quality is good but the cost skyrockets —you pay premium for every trivial query, which is most of them—. Why it happens: it's reasoned "the expensive one is better" and the "…but it's only needed where the cheap one falls short" is skipped. How to spot it: if a single model (the most expensive) handles all your traffic, you're in the call center where the seniors answer the hours questions. How to fix it: put in a classifier and route —the 70% easy to the cheap one lowers the cost by almost half without touching the quality of the hard ones—.

Trusting a perfect classifier (an unrealistic assumption). What happens: the design assumes the classifier always gets it right, and doesn't plan for routing errors. In production, the classifier sends hard ones to the cheap one and some responses come out worse, without anyone noticing until a user complains. Why it happens: in the design the classifier is drawn as a box that "decides the difficulty," and it's easy to forget it also errs. How to spot it: if your cascade has no eval verifying the quality of the queries routed to the cheap one, you're trusting a perfect classifier that doesn't exist. How to fix it: calibrate the classifier to escalate when in doubt (cheap false positive, not expensive false negative) and verify quality with an eval-set (module 3).

Expecting the cascade to fix the tail latency (a wrong expectation). What happens: the team puts in a cascade expecting the p95 to drop, and is surprised when the p95 stays the same —the hard queries still take the same in the strong model—. It wrongly concludes that "the cascade didn't help." Why it happens: the cascade lowers the average visibly, and it's intuitive (but false) to assume it also lowers the tail. How to spot it: if your success metric for the cascade was the p95 and it didn't drop, you measured the technique against the wrong problem. How to fix it: understand what each technique fixes —the cascade lowers cost and median latency; the tail latency is attacked by streaming (lesson 6)— and use each for its problem.

Exercises

Exercise 1 — Recompute with a better classifier. Suppose you improve the classifier from 92% to 100% accuracy (perfect). Does the cascade's cost saving go up or down? Reason about which queries change routing when going from 92% to 100% and in which direction they move the cost. (You don't need to run the code; reason with the logic.)

See solution

With 92%, the classifier mis-routes 8% of the queries in two directions:

  • Some easy ones it sends to the expensive one (false positive): those overpay. With a perfect classifier, they'd go back to the cheap one → it saves.
  • Some hard ones it sends to the cheap one (false negative): those underpay (the cheap one is cheaper). With a perfect classifier, they'd go up to the expensive one → it costs more.

The two effects oppose each other. But since in the workload there are more easy (70%) than hard (30%), and since the classifier errs in both directions, the net effect on cost is small and depends on the exact balance. The important lesson isn't the exact sign of the cost change, but this: the classifier's accuracy affects QUALITY more than cost. A perfect classifier doesn't dramatically change the saving (the bulk of the saving comes from the 70% easy going to the cheap one, and that happens with 92% or with 100%), but it does eliminate the false negatives —the hard ones mis-routed to the cheap one that came out with worse quality—. That's why investing in a better classifier is justified above all by quality, not by cost: the cost is already captured almost entirely by a mediocre classifier.

Exercise 2 — The try-and-escalate cascade. Instead of classifying first, you set up the other cascade: every query goes to the cheap model, and the 30% hard is escalated to the strong one (paying for both calls). With the stub rates, compute the cost per 1000 searches of this variant and compare it with the all_strong from the example ($9.6192). Use: easy = 120 output tokens, hard = 300 output tokens, input 300. (Assume the cheap model is right at detecting the hard ones.)

See solution
  • 700 easy, cheap only (in 300, out 120): cost of one = 0.3×0.0008 + 0.12×0.004 = 0.00024 + 0.00048 = $0.00072. Total: 700 × 0.00072 = $0.504.
  • 300 hard, cheap and strong (both paid):
    • cheap (in 300, out 120, the failed attempt): $0.00072 each.
    • strong (in 300, out 300, the final response): 0.3×0.008 + 0.3×0.040 = 0.0024 + 0.012 = $0.0144 each.
    • per hard one: 0.00072 + 0.0144 = $0.01512. Total: 300 × 0.01512 = $4.536.
  • Total try-and-escalate: 0.504 + 4.536 = $5.04 per 1000 searches.

Comparison: try-and-escalate costs $5.04, a bit less than the classify-and-route of the example ($5.41) and quite a bit less than all_strong ($9.62). Why does it come out a bit cheaper than classify-first here? Because in the example the imperfect classifier (92%) sent some easy ones to the expensive one (waste), while here the easy ones never touch the strong one. But mind the nuance: this variant pays the double call on the hard ones (the cheap attempt plus the strong), which would make it more expensive if the percentage of hard ones were high, or if the cheap model took a long time (the hard ones' latency sums the failed cheap's time + the strong's). The trade-off between the two variants lives in those numbers: fraction of hard ones, cost of the double call, and classifier reliability. There's no one that always wins —you have to measure with YOUR workload—.

Exercise 3 — Design the support agent's cascade. Mercado's support agent gets questions like "where's my order?" (easy, template), "how do I return this?" (easy, FAQ) and "it arrived broken, I paid with two cards, I want a partial refund to one of them and the rest to store credit" (hard, reasoning). Describe how you'd set up a cascade for this agent: what the classifier classifies, what goes to the cheap one, what to the expensive one, and a support-specific quality consideration that would make you calibrate the classifier to escalate when in doubt.

See solution

The support agent's cascade:

  • Classifier: looks at each customer message and estimates whether it's a routine query (order tracking, return policy, hours) or a complex one (multiple conditions, refunds with logic, complaints with context). It can be a heuristic (keywords, length, number of entities mentioned) or a tiny model —cheap, because it only routes—.
  • To the cheap model: the routine ones. "Where's my order?" is resolved by looking up the status and answering with a template —the small model does it perfectly—. These are most of the volume, and making them cheaper is where the bulk of the saving is (remember the agent's cost accumulates per turn).
  • To the strong model: the complex ones. The partial refund to two cards needs to understand several conditions and propose a coherent action —that warrants the big model—.

The quality consideration that pushes toward escalating when in doubt: in support, a bad response has direct and visible consequences —an angry customer, a badly computed refund, a promise the system can't keep—. The cost of a false negative (sending a complex query to the cheap one and it answering badly) is much greater than the cost of a false positive (sending an easy one to the expensive one and paying a few cents more). That's why the support agent's classifier is calibrated conservatively: when in doubt, it escalates to the strong. It's the same logic as the call center —a junior who isn't sure passes the call to the senior, doesn't risk it—. And all of this answers to the eval gate (module 3): the cascade's saving is only acceptable if an eval confirms that the queries routed to the cheap one are answered well.

Summary and next step

In this lesson you designed the model cascade, the highest-leverage technique on cost: route the easy queries to the cheap model and only the hard ones to the expensive one, with a cheap classifier as receptionist. With the junior→senior call center you saw the logic —don't put the senior to answer hours questions— and with the cascade executed over 1000 Mercado searches you measured the saving: 43.7% less cost and 39.8% less average latency versus sending everything to the expensive one ($28,858/month → $16,241/month at scale). And you learned the honest nuance almost nobody explains: the p95 barely drops, because the hard queries still go to the slow model —the cascade fixes the cost and the median latency, not the tail latency, which is streaming's problem—. You saw the two forms of the cascade (classify-and-route vs try-and-escalate), the cost of mis-routing (the hard-to-cheap false negative harms quality, and that's why it's calibrated to escalate when in doubt), and the boundary with module 3 (the eval confirms that going cheaper didn't break quality).

Before moving on you should be able to: design a cascade with a classifier for a feature; measure its cost and latency saving versus "everything to the expensive"; explain why the p95 doesn't drop and which technique does attack it; distinguish classify-and-route from try-and-escalate; and calibrate the classifier so the cheap error (false positive) is preferable to the expensive error (false negative).

What follows is the other big cost lever, complementary to the cascade. The cascade makes each call cheaper; lesson 5 eliminates entire calls: many queries repeat, and it makes no sense to pay again (even to the cheap one) for a response you already computed. You're going to build a cache, measure its hit-rate —what fraction of queries are served from the already-computed— and the saving it produces, and compare the exact cache (which catches the same literal query) with the semantic one (which catches the same intent written differently). And you're going to see that at high volume, not re-paying is the biggest saving of all.

Resources

← Previous3. The latency and cost budgetNext →5. Cache: many queries repeat