Module 2: Latency and Cost as Architecture

8. Project: design the budget for a Mercado AI feature

Overview

This is your graduation from the module. Over seven lessons you learned to measure an LLM's latency and cost, to bound them with a budget, and to respect it with three techniques —cascade, cache, streaming— that lesson 7 composed into a request path. You saw all of that applied to the support agent. Now it's your turn, from scratch, on a different Mercado feature: semantic search. The reason for changing feature is the usual one and it's hard: if I let you re-compose lesson 7's support agent, I wouldn't know whether you learned the method or memorized the table. With a new feature, the only way to solve it is to apply the method —set the budget, compose the techniques, measure against the naive, justify that it fits— and that is, exactly, the proof that the module worked.

Your deliverable is three artifacts for semantic search: (1) the budget (latency budget and cost budget) with its justification; (2) the cost-aware architecture (cascade + cache) measured against the naive, executed in Python; and (3) the justification of which technique solves what, why the feature ends up within its budget, and where streaming goes. No part requires building the LLM: the model is simulated with the usual stub. It's pure design work —budget, compose, measure, defend— which is exactly what separates an AI feature that fits within its budget from a surprise bill. Build it yourself first; reading the reference solution without having tried is like reading the score of a game you didn't play.

Connection with the module: this project closes the arc. Lessons 2 and 3 gave you the scale and the budget; lessons 4, 5, and 6 the techniques; lesson 7 composed them over the support agent. Here you produce the three artifacts with your own hands, from start to finish, over semantic search. And with this lesson the module closes: at the end is the summary of the eight lessons and the bridge to module 3 (the eval as a quality gate —the third constraint this module didn't cover) and to the AI Engineering ecosystem (where the pieces we only architected here are built).

The project's case: Mercado's semantic search

The feature you have to design —yours to solve— is this:

Mercado wants a semantic search: when a customer types "something to listen to music while running," the system uses an LLM to understand the intent and reorder the products by relevance. It's high-volume (every search on the site touches it) and lives in the critical path (if it's slow, the customer feels it). Design it so it fits within its budget.

It's a sibling feature to the support agent's —it also repeats, it also varies in difficulty— but with a different profile: it's a single-shot search (not conversational), of very high volume, and in the critical path of every search. We didn't solve it; it's your turn. Don't re-teach how an LLM works nor how embeddings are computed —that's AI Engineering—; your job is to architect the feature around the LLM so it respects its budget.

The facts the team gives you (measured and estimated, so you don't have to invent them): semantic search runs at 40,000 searches/day. The budget the business allocated: $5,000/month in cost, and 800 ms of latency per search. Users search for the same thing written in many ways (good for cache) and the searches vary in difficulty —30% are hard and ask for longer responses (350 tokens versus 150 for the easy ones)—. The long tail (rare and hard searches) is 22% of the traffic. The classifier is right 90% of the time.

What you have to deliver

Follow the steps in order; each one rests on the previous.

Part 1 — The budget

Write the latency budget and the cost budget of semantic search, and justify in one or two sentences why those numbers (why 800 ms for latency, why $5,000/month for cost, tied to the usage context and the volume). Don't invent: use the ones the team gave you, but explain why they make sense for THIS feature.

Part 2 — The cost-aware architecture, executed

Write and run the Python that measures, over a semantic-search workload, four strategies: naive (everything to the strong model), cache only, cascade only, and cache+cascade. Report for each one the monthly cost and whether it fits within the budget. Reuse the stub and the formulas from the module: the cache serves the hits free; the cascade routes easy→cheap, hard→strong; the monthly is cost_per_workload × (searches_day × 30 / N). Deliver the real table (not cited from memory).

Part 3 — The justification and streaming

With the table in hand, write: (a) which technique solves which part of the problem; (b) why the feature ends up within its budget (and which combination you choose, with what headroom); (c) what happens with the latency of the heavy searches and where streaming enters; and (d) a sentence about the third gate this design does NOT verify (quality) and which module it belongs to.

The rubric

This is how the project is evaluated. It's not by length nor by elegance: it's by whether the feature is well budgeted and the architecture respects the budget with evidence.

Criterion	Doesn't meet	Meets	Excels
Budget	No thresholds, or "make it fast/cheap"	Explicit latency and cost budget	Also justifies each number by the usage context and the volume
Executed architecture	Numbers cited from memory or invented	Table run in Python with the 4 strategies vs budget	Also states why each technique saves what it saves and notes the overlap
Composition	Applies a single technique and considers it solved	Composes cache + cascade and verifies it fits	Also chooses the combination with headroom and explains that the savings don't add up
Boundaries	Doesn't mention tail latency or quality	Locates streaming (tail) and the eval gate (quality)	Also says which module/ecosystem each boundary belongs to

The criterion that weighs most, and the one that separates a serious design from an opinion, is the executed architecture: if you deliver everything else but the numbers came out of your head and not from running the code, you didn't measure —you adjectivized with invented figures, which is worse, because it fakes rigor—.

The reference solution

Try the whole project before continuing. What follows is one correct solution, not the only one.

Part 1 — The budget

Latency budget: 800 ms per search. Justification: semantic search lives in the critical path of every search on the site —the customer types and waits for results immediately—. The UX evidence says that above ~800 ms an interactive search starts to cost abandonment, so 800 ms is the point where the feature stops feeling nimble. (Rigorously, on a high percentile —p95—, not the average, because the tail is where the bad experience lives.)
Cost budget: $5,000/month at 40,000 searches/day. Justification: it's the fraction of the margin the business allocated to the feature. The search helps close sales, and its cost has to be a small portion of that margin or the feature destroys value. At 40,000/day, $5,000/month is what was agreed with whoever knows the margin —not a number engineering invented—.

Part 2 — The executed architecture

# Lesson 08 (project) — design the budget for Mercado's semantic search
# Reference solution. Everything SIMULATED. Zero network, zero API, zero keys. Fixed seed.
import random

MODELS = {
    "cheap":  dict(usd_in=0.0008, usd_out=0.004, base_ms=90,  ms_per_tok=0.4),
    "strong": dict(usd_in=0.008,  usd_out=0.040, base_ms=300, ms_per_tok=3.0),
}
def call_llm(model, in_tokens, out_tokens):
    m = MODELS[model]
    return m["base_ms"] + out_tokens * m["ms_per_tok"], \
           (in_tokens/1000)*m["usd_in"] + (out_tokens/1000)*m["usd_out"]

# --- Step 1: THE BUDGET (the design constraint) ---
LATENCY_BUDGET_MS   = 800       # the user leaves if the search takes longer
MONTHLY_COST_BUDGET = 5000.0    # dollars/month allocated to the feature
SEARCHES_PER_DAY    = 40_000

IN_TOK = 300
OUT_EASY, OUT_HARD = 150, 350
HIT_MS, HIT_COST = 2.0, 0.0
CLASSIFIER_MS = 3

# --- The workload: 3000 searches. Repetition (cache) + difficulty (cascade). ---
random.seed(42)
N = 3000
POOL = [f"search_{i:03d}" for i in range(180)]
WEIGHTS = [1.0/(i+1) for i in range(180)]
DIFFICULTY = {q: (random.random() < 0.30) for q in POOL}
_uid = [0]
def make():
    if random.random() < 0.22:                     # long tail: new and hard search
        _uid[0] += 1; return (f"novel_{_uid[0]}", True)
    q = random.choices(POOL, weights=WEIGHTS, k=1)[0]
    return (q, DIFFICULTY[q])
searches = [make() for _ in range(N)]

def out_for(hard): return OUT_HARD if hard else OUT_EASY
def classify(hard): return hard if random.random() < 0.90 else (not hard)

def run(use_cache, use_cascade):
    cache, cost, lats = set(), 0.0, []
    for text, hard in searches:
        if use_cache and text in cache:
            cost += HIT_COST; lats.append(HIT_MS); continue
        if use_cache: cache.add(text)
        if use_cascade:
            model, extra = ("strong" if classify(hard) else "cheap"), CLASSIFIER_MS
        else:
            model, extra = "strong", 0.0
        lat, c = call_llm(model, IN_TOK, out_for(hard))
        cost += c; lats.append(lat + extra)
    return cost, sum(lats)/len(lats)

def monthly(c): return c * (SEARCHES_PER_DAY * 30 / N)
def v(m): return "OK   " if m <= MONTHLY_COST_BUDGET else "OVER "

# --- Step 2: measure naive vs the cost-aware architecture ---
print(f"{'strategy':<20}{'total_cost':>12}{'avg_lat_ms':>12}{'monthly_usd':>13}{'budget':>8}")
for label, uc, ucas in (("all_strong (naive)", False, False),
                        ("cache_only", True, False),
                        ("cascade_only", False, True),
                        ("cache+cascade", True, True)):
    cost, avg = run(uc, ucas)
    mo = monthly(cost)
    print(f"{label:<20}{cost:>12.4f}{avg:>12.1f}{mo:>13,.0f}{v(mo):>8}")

print(f"\nlatency_budget = {LATENCY_BUDGET_MS} ms   |   monthly_cost_budget = ${MONTHLY_COST_BUDGET:,.0f}/month"
      f"   ({SEARCHES_PER_DAY:,} searches/day)")

# --- Step 3: the heavy query and the latency budget (justifies streaming) ---
heavy_block = MODELS["strong"]["base_ms"] + OUT_HARD * MODELS["strong"]["ms_per_tok"]
heavy_ttft  = MODELS["strong"]["base_ms"] + 1 * MODELS["strong"]["ms_per_tok"]
print(f"\nheavy query to strong: blocking {heavy_block:.0f} ms ({v(0) if heavy_block<=LATENCY_BUDGET_MS else 'OVER'}"
      f" the latency budget), streaming TTFT {heavy_ttft:.0f} ms (OK)")

What to expect. When you run it:

strategy              total_cost  avg_lat_ms  monthly_usd  budget
all_strong (naive)       38.3840      1079.6       15,354   OVER 
cache_only               12.6508       350.8        5,060   OVER 
cascade_only             26.7924       778.7       10,717   OVER 
cache+cascade            10.8321       304.7        4,333   OK   

latency_budget = 800 ms   |   monthly_cost_budget = $5,000/month   (40,000 searches/day)

heavy query to strong: blocking 1350 ms (OVER the latency budget), streaming TTFT 303 ms (OK)

Read the table as you learned. The naive —everything to the strong model— costs $15,354/month, more than three times the $5,000 budget: the naive option isn't profitable. The cascade alone cuts to $10,717 (30% less) but is still well above. And here's the fine detail: the cache alone costs $5,060/month —it violates the budget by a hair (barely $60, 1.2% over)—. It's so close to fitting that it's tempting to approve it, but a design that barely grazes the limit has no headroom: as soon as the traffic rises a little, it goes over. Only cache + cascade really fits: $4,333/month, OK, with margin against the $5,000. It's the architecture you take to production, not because it's the only one that saves, but because it's the only one that fits with room to spare.

And as in lesson 7, the savings don't add up: the cache alone saves ~67% ($38.38 → $12.65) and the cascade alone ~30% ($38.38 → $26.79); if you added them, you'd expect 97%. But together they save 72% ($38.38 → $10.83), not 97% —because they overlap over the same easy/repeated traffic—. You have to measure the combination, not add.

An extra optional lever: trim the output. Semantic search doesn't need to generate 150-350 tokens of prose; it needs to reorder products, which can be requested as a compact list of IDs. If you limit the output (for example to 60 tokens for the easy ones, 120 for the hard ones), you lower the cost per call and the latency. Adding this block at the end of the script:

# --- Extra lever (optional): limit the output length (list of IDs, not prose) ---
OUT_EASY, OUT_HARD = 60, 120     # was 150, 350
cost_capped, avg_capped = run(True, True)
print(f"cache+cascade+short output: ${monthly(cost_capped):,.0f}/month  "
      f"avg {avg_capped:.0f} ms  {v(monthly(cost_capped))}")

produces:

cache+cascade+short output: $1,901/month  avg 152 ms  OK

With the trimmed output, cache+cascade drops to $1,901/month (much more headroom) and the average latency to 152 ms. It wasn't needed to fit —cache+cascade already fit at $4,333— but it's a legitimate lever when you want more margin or when the volume grows. And —a key point— it also solves the heavy query's latency in passing: with 120 output tokens instead of 350, the heavy one to the strong drops from 1350 ms to 300 + 120×3 = 660 ms, which now fits under 800 ms without even needing streaming. The output length is a double lever —cost and latency— exactly as lesson 2 taught.

Part 3 — The justification

(a) Which technique solves what. The cache eliminates the repeated traffic (users search for the same thing written differently) —from $15,354/month naive to $5,060 with just the cache, the biggest individual cut—. The cascade makes cheaper the new searches that do reach the LLM, routing the easy ones to the cheap model. And the output length (request IDs, not prose), as an optional lever, lowers the cost and latency per call. Each attacks a different portion: the cache the repeated volume, the cascade the price per call, the short output the size of each call.

(b) Why it fits, and with what headroom. I choose cache + cascade: it fits at $4,333/month, well under the $5,000 budget, with margin to grow. I discard the cache alone even though it seems to "almost fit" ($5,060) because it violates the budget —and even if it grazed it from below, it would have no headroom, ready to go over at the first traffic increase—. The rule: don't choose the one that barely fits; choose the one that fits with room to spare. And —lesson 7's lesson— the savings don't add up (67% + 30% give 72%, not 97%), because the cache already ate the easy/repeated traffic the cascade would also have made cheaper, so you have to measure the combination.

(c) The tail latency and streaming. Look at the avg_lat_ms: cache+cascade leaves it at 305 ms, under the 800 budget. But that's the average; the heavy searches that reach the strong model take longer. The final block of the code measures it: a blocking heavy query takes 1350 ms (with the original 350-token output) —it violates the latency budget—, while with streaming the first token appears at 303 ms (OK). So for the heavy searches, the path is completed with streaming: the user sees results appearing at ~300 ms even though the complete list takes longer. (Alternative we saw above: trimming the output to 120 tokens drops the heavy one to 660 ms, which already fits blocking —two ways of respecting the latency budget on the tail—.)

(d) The gate this design does NOT verify. The design respects the cost budget and the latency budget —this module's two constraints— but it doesn't verify quality: does the cheap model reorder as well as the strong one? does the cache serve results that are still relevant? did requesting IDs instead of prose not degrade the response? That third gate is the eval gate, and it's all of this guide's module 3. A production-ready semantic search passes all three: it fits in cost, it fits in latency, and it passes the quality threshold. This project delivers the first two.

Exercises

These exercises transfer the method to other Mercado decisions, so you confirm you learned to design budgets and not to repeat a table.

Exercise 1 — The budget that blows up with success. Your semantic search fits within $5,000/month at 40,000 searches/day with cache+cascade ($4,333/month). Mercado grows and the traffic doubles to 80,000/day. Without running code, reason about what happens to the budget verdict and list three ways to react.

See solution

What happens: the monthly cost scales linearly with the volume (lesson 2's taximeter), so at 80,000/day the cost of cache+cascade doubles —from $4,333 to ~$8,666/month— and violates the $5,000 budget. The verdict flips with the growth, exactly as in lesson 7's exercise 2: an architecture that fits today blows up when the feature becomes more popular. (That's exactly why you chose cache+cascade and not the cache alone: the cache alone already grazed the limit at 40k/day, so it would have gone over with much less growth.)

Three ways to react:

Compose more techniques. Raise the cache's hit-rate (semantic key instead of exact, longer TTL where the data allows), trim the output even more, or add a third model level (tiny → small → large) to make more of the traffic cheaper. Each extra lever gives some margin.
Renegotiate the cost budget with the business. If the feature at double the volume generates double the sales, maybe it deserves more budget. The budget isn't sacred if whoever knows the margin revises it —but that's a business decision, not an engineering one—.
Accept a controlled quality trade-off. Move more traffic to the cheap model (lower the classifier's threshold so more searches count as "easy"), verifying with the eval gate that the relevance doesn't drop too much. Go cheaper at the cost of a bit of quality, measured.

The moral: the budget and the volume are tied, the volume grows with success, and that's why the design needs headroom and a plan for when the headroom runs out —fitting once isn't enough—.

Exercise 2 — Another feature, same method. Mercado wants recommendations on the homepage: an LLM that, given the user's history, suggests products. It's high-volume (every visit), but —unlike search— the response is personalized per user. Apply the method: what latency and cost budget would you propose, and —key— what happens to the cache when the response is personalized?

See solution

Budget: the homepage recommendations are in the critical path of every visit, so a latency budget similar to search's (on the order of hundreds of ms to ~1 s) makes sense —the user shouldn't wait for the home—. The cost budget is set by the business according to how much the sales the recommendations generate are worth; at high volume (every visit), the number matters a lot.

What happens to the cache —the point of the exercise: the cache works much worse when the response is personalized. In search, "iphone" gives the same response for everyone, so a hit serves many users (high hit-rate). In recommendations, the response for Ana (based on her history) isn't useful for Beto —each user has their own response—. Caching per user only makes sense if the same user comes back often before their recommendations need refreshing, and even then the hit-rate is much lower than in a shared search. Personalization is the enemy of the cache.

Consequence for the design: since the cache yields little here, you have to lean more on the other levers —aggressive cascade (most recommendations may be resolved by a cheap model), async (do you really have to generate the recommendations live on every visit, or can they be precomputed in the background and served from a table?), and trimming the output—. In fact, for recommendations, async/precomputation is usually the main lever: instead of calling the LLM in the critical path of every visit, each user's recommendations are generated in a background job (when their history changes) and the home reads them from a table —moving the expensive AI work off the critical path entirely—. The lesson: the feature's profile (shared vs personalized, single-shot vs precomputable) decides which levers yield, and the method is the same —budget, compose the techniques that do apply, measure— even though the mix changes.

Exercise 3 — Defend the naive (once). The naive design —everything to the strong model, no cache or cascade— violated the budget by more than three times ($15,354 against $5,000). But there's a Mercado context where the naive is the correct choice. Find it and explain why, tying it to the module's lessons.

See solution

The context where the naive wins: a very low-volume, very short-lived prototype. Imagine Mercado wants to validate whether semantic search is even worth it, with a two-week experiment for 50 internal users. Volume: maybe 200 searches a day. At that volume, the naive (strong model for everything, ~$0.0128 per search of the mixed workload) costs 0.0128 × 200 × 30 ≈ $77/month —nothing—, and it fits comfortably within any reasonable budget. And the latency of an occasional search in an internal experiment doesn't scare away anyone who already knows it's a test.

Why it's correct there, tied to the module:

The taximeter (lesson 2) depends on the volume. The cost that makes the naive unviable at 40,000/day ($15,354/month) is trivial at 200/day (~$77/month). The same expensive call is a problem at high volume and a non-problem at low volume —the budget verdict depends on the volume (lesson 3)—.
Composing has a complexity cost (lesson 7). Setting up cache, cascade, classifier, and streaming is real work —pieces to build, tune, and operate—. For a two-week prototype that might be discarded, that work is over-engineering: you spend days optimizing something you don't yet know will live. The naive lets you validate the idea today, with minimal architecture, and only if the experiment works and goes to production at real volume, then you compose the techniques.

The lesson, the same as the whole module applied in reverse: the cost-aware architecture isn't free, and it's only justified when the budget demands it. The naive wasn't "bad"; it was premature to optimize it. Designing well isn't applying all the techniques always —it's applying the ones THIS feature's budget and volume need, no more, no less—.

Module summary and where you go next

With this project you close module 2, where you learned to treat an LLM's latency and cost as first-class design constraints. You started with the thesis and the three analogies —the taximeter (the cost that scales with each call), the junior→senior call center (the cascade), the express lane (the cache), and the kitchen that notifies (streaming/async)— (lesson 1). You measured the two new physics: the LLM is slow (hundreds of ms to seconds, and more with the output tokens) and costs per token, with the taximeter that turns $0.008/call into $25,200/month at scale (lesson 2). You bounded them with a budget —latency budget and cost budget—, and saw that "everything to the expensive" violates both (lesson 3). You learned the techniques: the cascade (cheap first, 44% less cost, but the p95 doesn't drop) (lesson 4); the cache (don't re-pay, hit-rate = saving, exact vs semantic) (lesson 5); the async and streaming (perceived latency, pull the work off the critical path) (lesson 6). You composed them into the request path and saw they accumulate but don't add up, and that no single one is enough (lesson 7). And here, in the project, you did it all yourself over semantic search: you budgeted, composed cache+cascade (which fits with room to spare where the cache alone barely grazed the limit), measured against the naive, saw the output length as an extra cost and latency lever, and located streaming and the eval gate.

The capability you take away: faced with any AI feature, you can set it a latency and cost budget, estimate its taximeter at real volume, compose cascade + cache + streaming (and the output length, and async) to bring it within the budget, measure the real combination instead of adding savings, and know which levers yield according to the feature's profile —without falling into either "everything to the expensive model just in case" or "a single technique and done"—. And always with the boundary clear: this is architecture (how you surround the model), not inference optimization (how the model runs internally), which is AI Engineering/infra.

Where you go next, within this guide:

Module 3 — The eval as a fitness function. This module gave you two of the three gates of an AI feature (cost and latency); module 3 gives you the third: quality. How do you "test" a probabilistic component? With an eval-set that works as a gate —a prompt or model change that lowers the score blocks the deploy—. It's what verifies that this module's cascade and cache didn't go cheaper at the cost of bad responses. The three gates together —cost budget, latency budget, quality gate— are what makes an AI feature safe.

And toward the rest of the ecosystem: every time this module simulated the LLM with a stub, it relied on the real pieces —RAG, agents, embeddings, the model itself— being built in the AI Engineering ecosystem. This module taught you to architect those pieces around a budget; building them is the next step. And the pattern of pulling the work off the critical path (async), applied here to AI, is taught in depth in the ecosystem's event-driven architecture and resilience guides. You now have the method to put an AI feature into a real system respecting its latency and its cost; those guides are the pieces on which you apply it.

Resources

martinfowler.com — "Emerging Patterns in Building GenAI Applications", Bharani Subramaniam and Martin Fowler — the module's natural close: the catalog of architecture patterns for LLM features (routing, caching, request path, streaming) treated as a system that composes; the reference for taking this method to real features.
Chip Huyen — AI Engineering (O'Reilly) — the go-to book for the next step: building the pieces (RAG, agents, embeddings) this module only architected, and going deeper into AI-system design by cost, latency, and quality.
Anthropic — Claude docs (models, pricing, prompt caching, streaming, batch processing) — the panorama of model families and the real levers (choose a model, cache the prompt, streaming, batch processing) that correspond to this module's simulated techniques; conceptual, without fixing a model version.

← Previous7. The cost-aware request path Next →1. Module introduction: the eval as a fitness function