Module 2: Latency and Cost as Architecture

3. The latency and cost budget

Next →

Overview

By the end of this lesson you'll have the tool that turns the two measurable constraints of lesson 2 into a design rule: a feature's latency budget and cost budget. A budget is an explicit threshold —"a semantic search must respond in under 800 ms" and "must cost less than X a month"— that the architecture has to respect, just as a bridge has a load limit that isn't negotiable. Without a budget, "fast" and "cheap" are opinions: everyone tolerates a different latency and estimates a different cost. With a budget, they become a gate: a design option passes or violates the threshold, and that can be verified by running a function. You're going to write that gate —fits(value, budget)— and pass through it the naive option ("everything to the expensive model"), which you're going to see violate both budgets of Mercado's semantic search: the latency one on the heavy queries and the cost one at scale.

This matters because the budget is what gives direction to the whole design. The three techniques that come after —cascade, cache, async— aren't ornaments applied "because they're good practice"; they're the mechanisms for meeting a budget the naive option doesn't meet. Without the threshold, you don't know whether you need a cascade or not, nor how much cache, nor whether streaming is optional or mandatory: you're optimizing blindly, with no goal. With the threshold, each technique has a clear and measurable job: lower the cost from $25,200 to below the budget, lower the heavy query's latency below 800 ms. The budget is the constraint that orders the whole module, and this lesson is where you set it and make it executable.

Connection with the module: this lesson is the hinge. Lesson 2 gave you the scale (measure latency and cost); this one draws the line you must not cross. What comes after are the techniques for respecting that line when the obvious option crosses it: lesson 4 (cascade) attacks the cost budget by routing the easy to the cheap; lesson 5 (cache) attacks it by not re-paying the repeated; lesson 6 (async/streaming) attacks the latency budget by showing the response earlier or pulling the work off the critical path. Lesson 7 composes them and verifies —with the same fits gate— that the feature ends up within its budget. Everything that follows answers to the line you draw here.

The bridge's load limit

Think of it this way. A bridge has a sign: load limit, 10 tons. It isn't a suggestion nor an opinion about "how heavy" a truck feels; it's a physical threshold, and there are two ways to treat it. The first is the careless engineer's: "let the trucks through, and if the bridge holds, it holds." The second is the serious engineer's: before letting a truck through, weigh it and compare against the limit. Nine tons: passes. Eleven tons: doesn't pass, no matter how urgent the load. The limit turns a fuzzy question ("will it hold?") into a binary, verifiable gate ("does it weigh less than 10?").

An AI feature has two of those signs. The first is latency: "a search must respond in under 800 ms." It's not a whim —it's the point where the evidence says the user perceives the search as slow and starts to leave—. The second is cost: "this feature must not cost more than X a month." It's not a whim either —it's what the business allocated, above which the feature stops being profitable—. And with both signs, the architect's job is the serious engineer's: before accepting a design, weigh it against the two limits. If an option takes 1500 ms, it violates the latency sign; if it costs $25,200 against a $3,000 budget, it violates the cost one. It doesn't matter how good it is at other things: a design that violates its budget is an eleven-ton truck on a ten-ton bridge. This lesson is learning to put up the signs and to weigh the trucks.

Worked example: Mercado's budget gate

We're going to define the budget for Mercado's semantic search and weigh the naive option against it. Two budgets, two gates.

The latency budget is per call: 800 ms. Above that, the user perceives the search as slow. We test it with two types of query: a typical one (150 output tokens) and a heavy one (400 output tokens), because —from lesson 2— latency grows with the output, and a real feature receives both.

The cost budget is monthly: $3,000/month, what the business allocated to the feature, at a volume of 100,000 searches a day. We weigh it with the typical query as the cost unit.

The gate is a one-line function: fits(value, budget) returns "OK" if the value fits and "OVER" if it exceeds. With it we walk through the options.

# Lesson 03 — a feature's latency budget and cost budget
# LLM STUB: everything SIMULATED. Zero network, zero API, zero keys.

MODELS = {
    "cheap":  dict(usd_in=0.0008, usd_out=0.004, base_ms=90,  ms_per_tok=0.4),
    "strong": dict(usd_in=0.008,  usd_out=0.040, base_ms=300, ms_per_tok=3.0),
}

def call_llm(model, in_tokens, out_tokens):
    m = MODELS[model]
    latency_ms = m["base_ms"] + out_tokens * m["ms_per_tok"]
    cost_usd = (in_tokens / 1000) * m["usd_in"] + (out_tokens / 1000) * m["usd_out"]
    return latency_ms, cost_usd

# --- THE BUDGET for Mercado's semantic search (the design constraint) ---
LATENCY_BUDGET_MS   = 800        # above this the user perceives slowness and leaves
MONTHLY_COST_BUDGET = 3000.0     # dollars/month the business assigned to the feature
SEARCHES_PER_DAY    = 100_000
IN_TOK = 300

def fits(value, budget):
    return "OK   " if value <= budget else "OVER "

# --- Gate 1: the LATENCY BUDGET (per call) ---
# Tested with a typical query (150 output tok) and a heavy one (400 tok).
print("== Latency budget (<= 800 ms per search) ==")
print(f"{'option':<14}{'query':<9}{'latency_ms':>12}{'verdict':>9}")
for model in ("cheap", "strong"):
    for label, out_tok in (("typical", 150), ("heavy", 400)):
        lat, _ = call_llm(model, IN_TOK, out_tok)
        print(f"all_{model:<10}{label:<9}{lat:>12.1f}{fits(lat, LATENCY_BUDGET_MS):>9}")

# --- Gate 2: the COST BUDGET (monthly, with the typical query as the unit) ---
print(f"\n== Cost budget (<= ${MONTHLY_COST_BUDGET:,.0f}/month, {SEARCHES_PER_DAY:,} searches/day) ==")
print(f"{'option':<14}{'unit_usd':>10}{'monthly_usd':>13}{'verdict':>9}")
for model in ("cheap", "strong"):
    _, unit = call_llm(model, IN_TOK, 150)
    monthly = unit * SEARCHES_PER_DAY * 30
    print(f"all_{model:<10}{unit:>10.6f}{monthly:>13,.0f}{fits(monthly, MONTHLY_COST_BUDGET):>9}")

What to expect. When you run it:

== Latency budget (<= 800 ms per search) ==
option        query      latency_ms  verdict
all_cheap     typical         150.0    OK   
all_cheap     heavy           250.0    OK   
all_strong    typical         750.0    OK   
all_strong    heavy          1500.0    OVER 

== Cost budget (<= $3,000/month, 100,000 searches/day) ==
option          unit_usd  monthly_usd  verdict
all_cheap       0.000840        2,520    OK   
all_strong      0.008400       25,200    OVER

Here's the bridge with its signs, and here's why the naive option doesn't fit. Read it gate by gate.

The latency budget. The cheap model always passes: 150 ms the typical, 250 ms the heavy, both under 800. The strong model passes barely on the typical (750 ms, just under the sign) and violates on the heavy: 1500 ms, almost double the budget. This already tells you something important: sending everything to the strong model isn't just expensive, it's that on the queries generating long responses it breaks the latency budget. The user searching for something that produces a long response waits a second and a half staring at an unresponsive screen. The cheap model doesn't have that problem —but (a preview of the boundary with module 3) the cheap model may give worse quality, and that's another constraint a latency budget doesn't see—.

The cost budget. Here the naive option falls flat. The cheap model fits: $2,520/month, under the $3,000 budget. But the strong model —the "use the best model for everything, just in case" option— costs $25,200/month, more than eight times the budget. That's lesson 2's number, now with a verdict: OVER. The feature, as you naively designed it, isn't profitable: it eats more than eight times the money the business gave it.

Put the two gates together and you have the module's problem stated precisely: "everything to the strong model" violates both budgets —latency on the heavy queries, cost at scale—. But "everything to the cheap model" isn't the answer either: it fits both budgets, yes, but at the cost of quality (which we don't measure here, that's module 3). Neither extreme works. The architecture that respects the budget without dropping quality is one that uses the cheap where it suffices and the expensive only where needed, and that doesn't pay again for what it already computed. That's exactly the cascade (lesson 4) and the cache (lesson 5). The budget is what makes those techniques mandatory.

Going deeper: how a budget is set (and what it is NOT)

Where the latency number comes from. The 800 ms isn't arbitrary, but it isn't a universal law either: it comes from context. There's plenty of evidence that, in an interactive search, the perceived latency above a certain threshold (on the order of hundreds of milliseconds to a second) starts to cost abandonment and satisfaction. The exact number depends on your feature: a search in the critical path of every keystroke has a stricter budget than a "describe your product" the seller requests once. The rule: set the latency budget by what the user tolerates in THAT feature, not by what the model happens to take. The budget rules; the design adapts to it, not the other way around.

Where the cost number comes from. The cost budget comes from the business, not from engineering: it's the fraction of the margin (or of the product budget) the feature can consume without ceasing to be worth it. If each semantic search helps close sales that leave a certain margin, the search's cost has to be a small portion of that margin, or the feature destroys value instead of creating it. The rule: the cost budget is negotiated with whoever knows the margin, and then the architecture has to fit within it. A budget engineering invents without talking to the business is a sign with no authority.

Latency budget: average, or tail. An honest nuance about the latency budget: does the threshold apply to the average, or to the worst case? The serious answer is a high percentile (p95 or p99): it's not enough for the average search to take 300 ms if one in twenty takes 2 seconds, because that one in twenty is a furious user. In this module we simplify and measure per-query latencies, but remember it: a serious latency budget is set on the tail, not on the average, because the tail is where the bad experience lives. (In-depth percentile analysis is system-design; here it's enough to know the budget is measured where it hurts.)

What a budget is NOT. It's not an aspirational goal ("I hope it's fast"): it's a limit that's verified and that, if violated, blocks the design —like the bridge that doesn't let the eleven-ton truck through—. And it's not a substitute for quality: the latency budget and the cost budget say the feature is fast and cheap, but not that its responses are good. That third gate —the eval gate— is all of module 3, and it's as mandatory as these two. A well-architected AI feature passes three gates: it fits its latency budget, it fits its cost budget, and it fits its quality threshold. This module gives you the first two.

Common mistakes

Not setting a budget (an omission mistake). What happens: the team designs the feature with no explicit latency or cost threshold, trusting that "it'll turn out fine." The latency and cost only become visible when a user complains or the bill arrives, and by then the architecture is already assembled and expensive to change. Why it happens: setting the budget requires talking to the business (cost) and deciding what the user tolerates (latency), and it's easier to skip that step and start coding. How to spot it: if you can't say in a number how long your feature can take and how much it can cost, you don't have a budget —you have a hope—. How to fix it: set both thresholds before choosing the architecture, so the architecture has something to answer to.

Designing for the average and ignoring the tail (a latency mistake). What happens: the budget is verified against the typical query (150 tokens, 750 ms, "passes") and the heavy one is ignored (400 tokens, 1500 ms, violates). The average looks fine and the feature still frustrates the users whose queries generate long responses. Why it happens: the typical query is the one you test, and the heavy one shows up in production with real volume. How to spot it: if your budget check used only one query size, you didn't test the tail. How to fix it: weigh the budget against the heavy query (and seriously, against a high percentile), not just against the typical one —the tail is where the budget breaks—.

Confusing the budget with quality (a scope mistake). What happens: the team sees that the cheap model fits both budgets and concludes "done, let's use the cheap for everything." It passes the latency and cost gates, but it ignores that the cheap one may give worse responses —it violates a quality gate nobody set—. Why it happens: latency and cost are easy to measure and the budget makes them visible; quality is harder and doesn't (yet) have its gate. How to spot it: if your model decision was made only with latency and cost, you're missing the third gate. How to fix it: remember that this module's budget is two of the three constraints; quality (the eval gate, module 3) is the third, and "cheap and fast but bad" isn't an acceptable feature.

Exercises

Exercise 1 — Set a budget for another feature. The "describe your product" generator for Mercado's sellers isn't in anyone's critical path: the seller requests it once, when creating a product, and can wait. It generates long responses (600 tokens). Propose a reasonable latency budget and cost budget for this feature, different from search's, and justify why they differ.

See solution

Latency budget: looser, for example 5,000 ms (or even async, with no synchronous latency budget). Justification: unlike search —which lives in the critical path of every keystroke and where 800 ms is already a lot—, "describe your product" is a deliberate and occasional action. The seller understands that "generating a description" takes a few seconds, just as they understand that uploading a photo takes a while. With 600 output tokens, the strong model would take 300 + 600×3 = 1900 ms, which fits in a loose budget. (Better yet, it's a perfect candidate for async —lesson 6—: it's pulled off the critical path and the seller doesn't even wait.)

Cost budget: more expensive per call, but low total volume. Justification: sellers create products far less often than customers search, so the volume is orders of magnitude lower. A description costs more per call (600 output tokens to the strong = 0.5×0.008 + 0.6×0.040 = $0.028), but if, say, 2,000 are generated a day, the monthly is 0.028 × 2,000 × 30 = $1,680/month —manageable—. The key difference: the budget is set by the usage context (critical vs occasional, high vs low volume), not by the model. The same expensive call is a problem in high-volume search and a non-problem in the low-volume generator.

Exercise 2 — Adjust the volume and review the verdict. Mercado's semantic search starts with just 8,000 searches a day (it's a pilot). With the strong model (unit_cost $0.008400 per typical search) and the same $3,000/month budget, does it pass or violate the cost budget? At what daily volume does the strong model start to violate it?

See solution
  • At 8,000/day: monthly = 0.008400 × 8,000 × 30 = $2,016/month. Since $2,016 ≤ $3,000, it passes (OK).
  • Threshold volume: we look for the volume V where the monthly equals the budget: 0.008400 × V × 30 = 3,000, from which V = 3,000 / (0.008400 × 30) = 3,000 / 0.252 ≈ 11,905 searches/day.

That is: with the strong model, the feature fits within the cost budget up to about ~11,900 daily searches, and beyond that it violates it. The moral for the module: the verdict of a cost budget depends on the volume, and the volume grows with the feature's success. A naive architecture that "fits" in the pilot (8,000/day) blows the budget exactly when the feature becomes popular (100,000/day). That's why fitting today isn't enough: you have to leave headroom for growth —and that's where the cascade and the cache come in, giving margin instead of leaving you at the edge of the budget—.

Exercise 3 — The gate that was missing. A team presents its semantic search: "it runs on the cheap model, takes 150 ms (passes the latency budget) and costs $2,520/month (passes the cost budget). It's ready for production." A critical check is missing. Which one, and why aren't this module's two gates enough to approve the feature?

See solution

The quality gate (the eval gate) is missing: are the cheap model's responses good enough? This module's two gates —latency and cost— confirm that the feature is fast and cheap, but they say nothing about whether the semantic search returns relevant results. A cheap model could reorder the products badly —put the irrelevant on top— and still pass this module's two gates with flying colors: it's lightning-fast and dirt-cheap, simply because it's worse.

Why the two gates aren't enough: latency and cost are operational properties (how fast, how cheap), not quality ones (how well). An AI feature needs all three:

  1. Latency budget (this module): does it respond on time?
  2. Cost budget (this module): does it fit the margin?
  3. Eval gate (module 3): is the response good?

Approving with only the first two is the exercise's mistake: latency and cost were optimized to the extreme (cheap for everything) without verifying that quality survived. The team's correct answer would be: "it passes latency and cost; now we have to run the eval-set to confirm the cheap model keeps the relevance —and if not, escalate the hard queries to the strong model (cascade), which is how you respect all three gates at once."

Summary and next step

In this lesson you turned the two measurable constraints into a design rule: the latency budget and the cost budget, explicit thresholds the architecture must respect. With the bridge analogy you saw that a budget isn't an opinion but a limit that's verified: you weigh the design and it passes or violates, like the truck against the 10-ton sign. You wrote the fits(value, budget) gate and passed Mercado's naive option through it: "everything to the strong model" violates both budgets —latency on the heavy queries (1500 ms > 800) and cost at scale ($25,200 > $3,000)—, while "everything to the cheap model" fits both but (a preview) sacrifices quality. You learned where the numbers come from (latency from the usage context, cost from the business's margin), that a serious latency budget is set on the tail and not the average, and that these two gates are two of the three every AI feature needs —the third, quality, is module 3—.

Before moving on you should be able to: define a feature's latency budget and cost budget as explicit thresholds; write and run a gate that verifies whether an option fits or violates; explain why the naive option violates both of Mercado's search budgets; and recognize that latency and cost don't cover quality, which has its own gate.

What follows is the first technique for respecting the cost budget without dropping quality. In lesson 4 you're going to design the model cascade: a cheap classifier looks at each query, sends the easy ones to the cheap model and only the hard ones to the expensive one —the call center that handles with the junior and escalates the complicated to the senior—. You're going to measure the saving versus "everything to the expensive": how much the cost and latency drop, and —the honest detail— why the p95 barely drops, because the hard queries still go to the slow model. It's the first big mechanism for bringing the feature within its budget.

Resources

← Previous2. The LLM is slow and costs per callNext →4. The model cascade: cheap first, escalate only if needed