7. The cost-aware request path

Overview

By the end of this lesson you'll know how to compose the module's four pieces —budget, cascade, cache, streaming— into a single coherent design: the cost-aware request path. So far you saw each technique separately and against its own problem. Here you join them in the order a real request goes through them: first the cache (do I already have this response?), then the cascade (if not, cheap or expensive?), with the budget as the gate that verifies the result fits. You're going to apply this to Mercado's support agent —a conversational feature where the cost accumulates per turn— and you're going to measure, by executing, the most important lesson of the synthesis: the techniques accumulate but don't add up, because they overlap over the same cheap traffic, and no single one is enough —but together they bring the feature within its budget—.

This matters because it's the opposite mistake to sending everything to the expensive model: believing a single technique is the silver bullet. "We put in a cache, done" or "the cascade is enough" are tempting and false conclusions. Each technique attacks a part of the problem, and the budget is only respected when you combine them —in the right order, measuring the real combination, not adding the savings separately—. Composing well is what separates an AI feature that almost fits within its budget (and blows it when the traffic grows) from one that fits with room to spare (headroom to grow). This lesson is where the module stops being a box of loose tools and becomes a method: given a budget, this is how you assemble the path that respects it.

Connection with the module: this is the synthesis. Lesson 3 set the budget as the constraint that orders everything; lessons 4, 5, and 6 gave the techniques; this one assembles them. It's the second-to-last because you needed each piece before you could compose them, and it prepares the project (lesson 8), where you'll do the same with your own hands on a different feature. Here an open thread also closes: lesson 4 showed the cascade alone left search at $16,241/month (over budget), and lesson 5 showed the cache alone didn't reach it either —this lesson demonstrates, with numbers, why they're needed together and how much each yields on top of the other—.

The request path: a line with several gates

Think of it this way. A request that enters your AI feature isn't a single decision ("which model do I send it to?"); it's a journey through several stations, and at each one it can be resolved or move to the next —like a well-designed process that resolves most cases at the first window and only sends to the specialized window the ones that really need it—.

The first station is the cache: "did I already answer this?". If yes (a hit), the request ends here, free and instant —it doesn't even touch the LLM—. This is the most powerful filter, and that's why it goes first: it eliminates the bulk of the repeated traffic before spending a cent. Most requests in a real system don't even get past this window.

The second station, only for misses, is the cascade: "this request is new, is it resolved by the cheap one or does it need the expensive one?". The classifier routes —easy to the cheap, hard to the strong— and here the LLM is indeed called, but to the smallest model that suffices. Only the requests that survived the cache and that the classifier judges hard reach the expensive model. It's a small fraction of the total traffic.

And over the whole journey is the budget gate: at the end, does the result fit within the latency budget and the cost budget? If it generates text the user waits for, it's served with streaming to respect the perceived-latency budget. The budget is the inspector that verifies the complete process —cache + cascade + delivery— meets the two rules lesson 3 set.

The order matters, and it isn't arbitrary. The cache goes first because it's the one that eliminates the most traffic and is the cheapest to consult: it makes no sense to classify and route a query you already have answered. The cascade goes second because it operates on what the cache didn't catch. And the budget wraps everything because it's the reason for the journey. This lesson is learning to draw that path and to measure that, traveled in that order, it brings the feature within its budget.

Worked example: composing over the support agent

We're going to compose the techniques over Mercado's support agent —different from the search of previous lessons— and measure each combination against its budget. The agent is conversational: the frequently asked questions (FAQs) repeat a lot (good for cache), and their difficulty varies (good for cascade). The workload is 3000 turns, from 200 FAQs with Zipf popularity plus a 25% long tail (new and hard questions). 30% of the FAQs are hard. The classifier is right 92% of the time.

The agent's budget: $5,000/month at 30,000 turns/day. We measure four strategies —naive (everything to the expensive), cache only, cascade only, and cache+cascade— and see which fits.

# Lesson 07 — the cost-aware request path (cache + cascade + budget)
# Applied to Mercado's support agent. Everything SIMULATED. Fixed seed.
import random

MODELS = {
    "cheap":  dict(usd_in=0.0008, usd_out=0.004, base_ms=90,  ms_per_tok=0.4),
    "strong": dict(usd_in=0.008,  usd_out=0.040, base_ms=300, ms_per_tok=3.0),
}
def call_llm(model, in_tokens, out_tokens):
    m = MODELS[model]
    return m["base_ms"] + out_tokens * m["ms_per_tok"], \
           (in_tokens/1000)*m["usd_in"] + (out_tokens/1000)*m["usd_out"]

IN_TOK = 300
OUT_EASY, OUT_HARD = 120, 300
HIT_MS, HIT_COST = 2.0, 0.0
CLASSIFIER_MS = 3

# --- The support agent's BUDGET ---
MONTHLY_COST_BUDGET = 5000.0
TURNS_PER_DAY = 30_000

# --- The workload: 3000 turns. FAQs (they repeat) + unique long tail. ---
random.seed(23)
N = 3000
POOL = [f"faq_{i:03d}" for i in range(200)]
WEIGHTS = [1.0/(i+1) for i in range(200)]
DIFFICULTY = {faq: (random.random() < 0.30) for faq in POOL}   # 30% of the FAQs are hard

_uid = [0]
def make_turn():
    if random.random() < 0.25:                       # long tail: new and hard question
        _uid[0] += 1
        return (f"novel_{_uid[0]}", True)            # (text, is_hard)
    faq = random.choices(POOL, weights=WEIGHTS, k=1)[0]
    return (faq, DIFFICULTY[faq])
turns = [make_turn() for _ in range(N)]

def out_for(hard): return OUT_HARD if hard else OUT_EASY
def classify(hard): return hard if random.random() < 0.92 else (not hard)

def run(use_cache, use_cascade):
    cache = set()
    cost, lats = 0.0, []
    for text, hard in turns:
        if use_cache and text in cache:
            cost += HIT_COST; lats.append(HIT_MS); continue
        if use_cache:
            cache.add(text)
        if use_cascade:
            model = "strong" if classify(hard) else "cheap"
            extra = CLASSIFIER_MS
        else:
            model, extra = "strong", 0.0
        lat, c = call_llm(model, IN_TOK, out_for(hard))
        cost += c; lats.append(lat + extra)
    return cost, sum(lats)/len(lats)

def monthly(cost_per_run):
    return cost_per_run * (TURNS_PER_DAY * 30 / N)

def verdict(m): return "OK   " if m <= MONTHLY_COST_BUDGET else "OVER "

print(f"{'strategy':<20}{'total_cost':>12}{'avg_lat_ms':>12}{'monthly_usd':>13}{'budget':>8}")
for label, uc, ucas in (("all_strong", False, False),
                        ("cache_only", True, False),
                        ("cascade_only", False, True),
                        ("cache+cascade", True, True)):
    cost, avg = run(uc, ucas)
    mo = monthly(cost)
    print(f"{label:<20}{cost:>12.4f}{avg:>12.1f}{mo:>13,.0f}{verdict(mo):>8}")

print(f"\nmonthly_cost_budget = ${MONTHLY_COST_BUDGET:,.0f}/month   ({TURNS_PER_DAY:,} turns/day)")

What to expect. When you run it:

strategy              total_cost  avg_lat_ms  monthly_usd  budget
all_strong               30.9168       892.9        9,275   OVER 
cache_only               12.9888       364.8        3,897   OK   
cascade_only             19.4083       588.9        5,822   OVER 
cache+cascade            11.3818       323.7        3,415   OK   

monthly_cost_budget = $5,000/month   (30,000 turns/day)

This table is the whole module condensed. Read it row by row, because each one teaches something.

Naive (everything to the expensive): $9,275/month — OVER. The naive starting point, the one lesson 1 called "it's just another API call": every turn to the strong model. It costs almost double the $5,000 budget. It's the baseline against which everything else is measured.

Cache only: $3,897/month — OK. The cache alone, without a cascade, already brings the feature within budget. It cuts 58% of the cost ($30.92 → $12.99 per 3000 turns) because it eliminates the repeated FAQ traffic —the bulk of support conversations are frequently asked questions—. Notice: the cache alone fits ($3,897 < $5,000), but at the edge: it has little headroom left before the budget.

Cascade only: $5,822/month — OVER. The cascade alone, without a cache, cuts 37% ($30.92 → $19.41) by routing the easy to the cheap. It's a big improvement… but it's not enough: $5,822 is still above the $5,000 budget. This row is the most important of the table, because it's counterintuitive: a powerful technique, well applied, and it still violates the budget. The cascade alone isn't enough for the support agent. Anyone who concluded "we put in a cascade, done" would have left the feature over budget without noticing.

Cache + cascade: $3,415/month — OK, with room to spare. The two together cut 63% ($30.92 → $11.38) and leave the feature at $3,415/month —well under the $5,000, with real headroom to grow—. This is the architecture you take to production: not the one that barely fits, but the one that fits with margin.

Now, the lesson these numbers teach that almost nobody sees: the techniques accumulate but do NOT add up. Look: the cache alone saves 58%, the cascade alone saves 37%. If you added them, you'd expect 58 + 37 = 95% saving. But the combination saves only 63%, not 95%. Why? Because they overlap over the same traffic. The cache already eliminated 58% of the traffic (the repeated FAQs, which are also mostly easy); when the cascade arrives, only the misses are left —the hard long tail and the first occurrences—, which are more oriented to the strong model and where the cascade saves less. The cache ate the easy/repeated traffic the cascade would also have made cheaper, so the cascade's gain on top of the cache is modest ($12.99 → $11.38, ~12% more). The levers aren't independent: they attack overlapping traffic, and that's why you have to measure the real combination, not add the savings separately.

The budget as the orderer. Notice the budget's role in this whole story: it's the line that decides which combination is "enough." Without the $5,000 budget, you wouldn't know whether the cascade alone (which saves an impressive 37%) is acceptable —and it isn't—. The budget is what turns "I saved a lot" into "I meet it or I don't." And to have headroom against the traffic's growth (remember from lesson 3 that the verdict depends on the volume, and the volume rises with success), you choose the combination with the most margin: cache + cascade, not cache alone at the edge.

Going deeper: the order, the latency, and what's missing in the table

Why cache first, cascade after. The journey's order isn't casual. The cache goes first because (a) it's the one that eliminates the most traffic —the bulk, free— and (b) consulting it is nearly instant, so filtering early costs nothing. It would make no sense to classify a query's difficulty and choose a model for a request you already have answered in the cache: it would be wasted work before you realize you didn't need to call anyone. The cascade goes after because it operates on what the cache didn't catch —the genuinely new traffic—, and there it does decide cheap vs expensive. The general principle: put the cheapest, highest-yield filter first. The cache is that filter.

The latency also improved, and that's where streaming is. Look at the avg_lat_ms column: naive 893 ms, cache+cascade 324 ms —the average latency dropped along with the cost, because the hits come out in 2 ms and the easy ones in 150—. But remember lesson 4: the average latency isn't the tail one. The hard requests that reach the strong model still take ~1200 ms, and for those —if they generate text the user waits for live, as in the support agent— the path is completed with streaming (lesson 6): the user sees the first token at ~300 ms even though the complete response takes longer. So the cost-aware path respects both dimensions of the budget: the cost (cache + cascade) and the perceived latency (streaming on the tail). The budget gate verifies both.

What's missing in this table: quality. The table measures cost and latency —this module's two constraints— and shows that cache+cascade respects them. But it doesn't measure the third gate: are the responses still good? The cascade sends the easy ones to the cheap model and the cache serves stored responses; both decisions could, if badly calibrated, degrade the quality (a cheap model that answers worse, a cache that serves something stale or that doesn't correspond). Verifying that the saving didn't come at the cost of quality is the eval gate, all of module 3. The complete path of an AI feature in production has all three gates: cost budget (here), latency budget (here), and quality gate (module 3). This lesson composes the first two; the third is as mandatory as these.

Composing isn't optional at high volume. A pattern the table makes clear: as the volume and cost grow, no single technique is enough, and composing stops being an optimization to become a necessity. In the support agent, the cascade alone violated the budget; the cache on top was needed. In another feature it could be the reverse (the cache alone doesn't reach it and the cascade is needed). And in a third, not even the two together are enough and you have to add more —limit the response length, put in a third model level, a more aggressive TTL—. The rule isn't "apply this technique"; it's "measure against the budget and compose whatever it takes until you fit with room to spare." The budget says how much is enough; the techniques are the levers to get there.

Common mistakes

Believing a single technique is enough (a silver-bullet mistake). What happens: the team puts in a cascade (or a cache) alone, sees a big saving, and considers the budget solved —without verifying whether it really fits—. In the support agent, the cascade alone saves an impressive 37%… and still violates the budget. The feature goes to production over budget without anyone noticing. Why it happens: each technique gives a visible and satisfying saving, and it's easy to stop at the first. How to spot it: if you applied a technique but didn't weigh the result against the budget again, you don't know whether it fits. How to fix it: after each technique, run the budget gate; if it violates, compose the next —the budget, not the saving, says when you're done—.

Adding the savings separately (false arithmetic). What happens: the combined saving is estimated by adding each technique's —"the cache saves 58%, the cascade 37%, together 95%"— and the budget is planned on that inflated 95%. Reality (63%) falls short, and the feature doesn't fit where it was promised. Why it happens: adding is the intuitive operation, and it hides that the techniques overlap over the same cheap traffic. How to spot it: if your estimate of the combined saving is the sum of the individual savings, you overestimated. How to fix it: measure the combination by executing it, not by adding —cache+cascade together, over the real workload, give the true number, almost always less than the sum—.

Composing in the wrong order (a journey mistake). What happens: the cascade is put before the cache —you classify and choose a model, and then you check whether it was in the cache—, wasting the work of classifying requests that were already answered. It's correct in result but inefficient, and on a high-volume feature that inefficiency accumulates. Why it happens: the techniques are assembled in the order they were learned or implemented, not in the order in which it's best to execute them. How to spot it: if your path does work (classify, call) before consulting the cache, the order is inverted. How to fix it: put the cheapest, highest-yield filter first —the cache— and only what survives goes to the cascade.

Exercises

Exercise 1 — Why 63% and not 95%? Explain in your own words, without running code, why the cache (58% saving alone) and the cascade (37% alone) combined give 63% and not 95%. Use the idea of "overlapping traffic."

Exercise 2 — The agent grows. Mercado's support agent is a success and its traffic rises from 30,000 to 80,000 turns/day. With the same $5,000/month budget, which strategies still fit? (Use the total_cost per 3000 turns from the table: naive 30.9168, cache_only 12.9888, cascade_only 19.4083, cache+cascade 11.3818. The monthly is total_cost × (80000 × 30 / 3000).)

Exercise 3 — Draw the path. Draw (in ASCII or mermaid) the path of a support agent request going through cache → cascade → budget → delivery, marking where it can end early (hit) and where cheap vs expensive is decided. Then say at what point the module 3 eval gate would enter.

flowchart TD
    A[User request] --> B{Cache: already answered it?}
    B -- HIT --> C[Serve stored response<br/>~2 ms, $0]
    B -- MISS --> D{Classifier: easy or hard?}
    D -- easy --> E[cheap model<br/>~150 ms, cheap]
    D -- hard --> F[strong model<br/>~1200 ms, expensive]
    E --> G[Store in cache]
    F --> G
    G --> H{Budget gate<br/>fits latency + cost budget?}
    H -- generates live text --> I[Deliver with streaming<br/>TTFT ~300 ms]
    H -- direct result --> J[Deliver]

Summary and next step

In this lesson you composed the module's four pieces into a single cost-aware request path: cache first (do I already have it?), cascade after (cheap or expensive?), with the budget as gate and streaming for the tail latency. With the line of several windows you saw why the order matters —the cheapest, highest-yield filter (the cache) goes first—. And with the table executed over the support agent you measured the central lesson: naive $9,275/month (over), cache alone $3,897 (fits at the edge), cascade alone $5,822 (over —a powerful technique that still isn't enough), and cache+cascade $3,415 (fits with room to spare). You learned that the techniques accumulate but don't add up (58% + 37% give 63%, not 95%, because they overlap over the cheap traffic), that the budget is what decides when a combination is enough, that you choose the option with headroom to withstand growth, and that a third gate is missing —quality, module 3's eval gate—.

Before moving on you should be able to: draw a request's path (cache → cascade → budget → delivery) and explain why that order; compose the techniques and measure the real combination instead of adding savings; recognize that no single technique is enough at high volume; and locate where the eval gate enters in the complete design.

What follows is doing it yourself, from scratch. In lesson 8 —the project— you're going to design the budget of a Mercado AI feature different from this lesson's (the semantic search), and build its cost-aware path with your own hands: set the latency and cost budget, compose cascade + cache, measure against the naive, justify why the feature fits within its budget, and decide where streaming goes. It's the module's synthesis applied by you to a new case —the proof that you learned the method and didn't memorize a table—.

Resources

• martinfowler.com — "Emerging Patterns in Building GenAI Applications", Bharani Subramaniam and Martin Fowler — the catalog that treats routing, caching, and the request-path design as patterns that combine; the direct backing for this lesson's composition.
• Chip Huyen — AI Engineering (O'Reilly), cost, latency, and system design chapters — how the cost and latency levers are assembled into an AI system's architecture, and why they're measured combined.
• Anthropic — Reducing cost and prompt caching (Claude docs) — cost levers at the call level (choose a model, cache the prompt) that complement this module's cascade and response cache; conceptual, without fixing a version.