Module 2: Latency and Cost as Architecture

1. Module introduction: latency and cost as architecture

Overview

By the end of this lesson you'll understand the idea that holds up the whole module, and that most teams learn late and expensively: an LLM's latency and cost are not an infrastructure detail, they're first-class design constraints. In module 1 you saw that an LLM is not a normal function because it's probabilistic —you can't assert its exact output—. Here we add the other two properties that separate it from a classic function, and they're just as architectural: the LLM is slow (it takes hundreds of milliseconds or seconds, not microseconds) and costs money per call (you pay per token, every time). These two properties change how you design, where you place the component, which model you use for what, and what you store so you don't pay for it again. If you ignore them until the end, you end up with two surprises: a feature that feels slow and a monthly bill that eats the margin.

This matters because the most dangerous phrase when putting AI into a system is "it's just another API call." It sounds harmless, and it's exactly the one that precedes disaster. A normal API call —your database, another microservice— responds in single-digit milliseconds and costs essentially zero, so calling it a thousand or a million times doesn't change your architecture. A call to an LLM breaks both assumptions at once: it takes hundreds of times longer and costs real money that multiplies with every call. Putting it in the critical path of a search without thinking about its latency, or letting it handle all the traffic without thinking about its cost, isn't a minor oversight: it's a design mistake paid in users who leave and money that leaks. And —the good news— it's fixed with architecture, not hardware: by deciding which model handles what, what gets cached, what gets pulled off the critical path, and how long and how much each operation can take before you consider the design broken.

Connection with the module: this lesson is the map, not the territory. Here you design nothing yet; you understand why the six lessons that follow go in the order they go. First the two new physics: lesson 2 measures what "slow" and "costs per call" mean —the latency that grows with tokens, the price per token, the taximeter of cost at scale—. Then the constraint that turns them into design: lesson 3 defines a feature's latency budget and cost budget, the threshold the architecture must respect. With the budget in hand, the three techniques that enforce it: lesson 4, the model cascade (cheap first, expensive only if needed); lesson 5, the cache (don't pay again for what's already computed); lesson 6, async and streaming (when the user can't wait for the complete block). And lesson 7 composes them into a single request path and demonstrates, by executing, that together they bring the feature within its budget. Lesson 8 —the project— puts you to designing the budget of a Mercado feature with your own hands.

Three analogies: the taximeter, the call center, and the express lane

Before dropping down to the code, three everyday images you'll recognize in every lesson of the module. Each captures one of the central ideas.

The taximeter. When you get into a taxi, there's a device that runs from the moment you start: every block you advance, the number goes up. You don't pay a flat fare for "using the taxi"; you pay for each unit of the trip, and at the end the sum is what the device shows. An LLM works the same way: each call has its taximeter. You don't pay a fixed monthly license for "having AI"; you pay for each token that goes in and each token that goes out, on each call. A single call costs a fraction of a cent and you don't feel it —like one block in a taxi—. But a marketplace does hundreds of thousands of searches a day, and there the taximeter becomes the bill: the same cheap call, multiplied by the real volume, is thousands of dollars a month. Lesson 2 puts you to watching that taximeter run.

The junior→senior call center. A well-run call center doesn't put its best agent —the senior, expensive, with years of experience— to handle every call. It would be a waste: most calls are easy ("what are your hours?", "how do I reset my password?") and a junior agent resolves them in seconds, at a fraction of the cost. The senior is reserved for the hard ones: the complex complaint, the case the junior couldn't handle. The rule is handle with the cheap one, escalate to the expensive one only when needed. That's exactly the model cascade of lesson 4: a cheap classifier looks at the query, sends the easy ones to the cheap model and only the hard ones to the expensive one. Putting the expensive model to handle everything "just in case" is like putting your senior agent to answer what time you open: it works, but you overpay for every trivial call.

The supermarket's express lane. The supermarket doesn't send everyone through the same line. It has an express lane for the one with few items, precisely so they don't wait behind the one with the full cart. It's a design decision about who waits how long: it recognizes that not every operation deserves the same latency, and organizes the flow so the fast one is fast. In AI, that idea shows up in two places. In the cache (lesson 5): the response you already computed doesn't go back in line, it's served instantly —like the customer who already paid and just comes to pick up—. And in async/streaming (lesson 6): when an operation takes seconds, either you show the user what's coming out (streaming, so they don't stare at a stalled line) or you pull it out of the main line entirely (async, it's processed separately and you notify when it's ready). The opposite mistake —a single slow line for everything— is the synchronous one that makes the user wait three seconds staring at a blank screen.

Keep the three. The taximeter is why cost matters; the call center is how you lower it without losing quality; the express lane is how you organize latency so the user doesn't suffer. The whole module is learning to assemble those three things into a real feature.

The case: Mercado's AI features

Let's drop down to Mercado, the ecosystem's marketplace. Mercado added AI features, and two of them accompany us through the whole module because they exhibit the two constraints differently:

Semantic search: when a customer types "something to listen to music while running," the system doesn't do a LIKE '%music%'; it uses an LLM to understand the intent and reorder the products by relevance. It's high-volume (every search on the site touches it) and lives in the critical path: if it's slow, the customer feels it instantly. Both constraints bite here: latency (the user waits) and cost (multiplied by each search).
The support agent: a conversational assistant that answers customer questions ("where's my order?", "how do I return this?"). It's conversational —several turns per conversation, each turn a call to the LLM— so the cost accumulates per turn, and the latency per turn defines whether the conversation feels nimble or sluggish.

Throughout the module we're going to give each one its budget for latency and cost, and design the architecture that respects it: which model handles each query, what gets cached, what gets pulled off the critical path. We're not going to build the LLM nor train anything —that's AI Engineering—; we're going to architect the feature around the LLM so it fits within its budget.

And to start, let's see the whole module condensed into an executed table. Look closely, because here's the thesis in figures.

# Lesson 01 (intro) — the module in miniature: two new physics + the budget
# Everything SIMULATED. Zero network, zero API, zero keys. Fixed seed.
import random

MODELS = {
    "cheap":  dict(usd_in=0.0008, usd_out=0.004, base_ms=90,  ms_per_tok=0.4),
    "strong": dict(usd_in=0.008,  usd_out=0.040, base_ms=300, ms_per_tok=3.0),
}
def call_llm(model, in_tokens, out_tokens):
    m = MODELS[model]
    return m["base_ms"] + out_tokens * m["ms_per_tok"], \
           (in_tokens/1000)*m["usd_in"] + (out_tokens/1000)*m["usd_out"]

# The budget for Mercado's semantic search.
MONTHLY_COST_BUDGET = 3000.0
SEARCHES_PER_DAY = 30_000

# A small workload: 1000 searches, 30% hard, many repeated.
random.seed(1)
N = 1000
POOL = [f"q_{i:02d}" for i in range(60)]
W = [1.0/(i+1) for i in range(60)]
DIFF = {q: (random.random() < 0.30) for q in POOL}
_uid = [0]
def make():
    if random.random() < 0.20:
        _uid[0] += 1; return (f"novel_{_uid[0]}", True)
    q = random.choices(POOL, weights=W, k=1)[0]
    return (q, DIFF[q])
qs = [make() for _ in range(N)]
def out_for(h): return 300 if h else 150

def run(smart):
    cache, cost = set(), 0.0
    for text, hard in qs:
        if smart and text in cache:      # cache: free
            continue
        if smart: cache.add(text)
        model = ("strong" if hard else "cheap") if smart else "strong"  # cascade
        _, c = call_llm(model, 300, out_for(hard))
        cost += c
    return cost

def monthly(c): return c * (SEARCHES_PER_DAY * 30 / N)

naive = monthly(run(smart=False))     # everything to the expensive model, no cache
smart = monthly(run(smart=True))      # cascade + cache

print(f"{'strategy':<24}{'usd_month':>12}{'budget':>13}")
print(f"{'all-to-expensive (naive)':<24}{naive:>12,.0f}{('OVER' if naive>MONTHLY_COST_BUDGET else 'OK'):>13}")
print(f"{'cascade + cache':<24}{smart:>12,.0f}{('OVER' if smart>MONTHLY_COST_BUDGET else 'OK'):>13}")
print(f"\nbudget = ${MONTHLY_COST_BUDGET:,.0f}/month   ({SEARCHES_PER_DAY:,} searches/day)")
print(f"the same feature, {(1-smart/naive)*100:.0f}% less cost — without touching GPUs or the model, just architecture")

What to expect. When you run it:

strategy                   usd_month       budget
all-to-expensive (naive)      10,535         OVER
cascade + cache                2,700           OK

budget = $3,000/month   (30,000 searches/day)
the same feature, 74% less cost — without touching GPUs or the model, just architecture

Stop at those two rows, because they're the whole module in miniature. The same feature —the same semantic search, answering the same queries— costs $10,535 a month if you design it the naive way ("everything to the expensive model, storing nothing") and $2,700 a month if you design it with two architecture techniques: the cascade (easy ones to the cheap one) and the cache (don't re-pay the repeated). The feature's budget is $3,000 a month. The naive version exceeds it by more than triple; the architected version respects it comfortably. And —read the last line— the difference didn't come from buying faster GPUs nor from switching the model for a more efficient one. It came from design decisions: which model handles what, and what doesn't get paid for again. That's exactly what "latency and cost as architecture" means: the saving is in how you surround the model, not in the model.

Don't understand yet how each piece works —that's what the lessons are for—. Keep the shape of the result: two designs of the same feature, one that breaks the budget and another that fits within it, just by changing the architecture around the LLM.

The map of the six lessons

The six lessons that follow go in this order because each assembles the piece the next one needs.

Lesson	What it gives you	Why it goes here
2	The two physics measured: latency by tokens, cost per token, the taximeter at scale	You can't budget what you can't measure; first you quantify
3	The budget (latency + cost budget) as an explicit threshold, with the gate that verifies it	The budget is the constraint the whole rest of the module has to meet
4	The model cascade: classify and route, cheap first	The highest-leverage technique on cost without sacrificing quality where it matters
5	The cache (exact and semantic), the hit-rate and its saving	The second big lever: at high volume, not re-paying is the biggest saving
6	Async and streaming: perceived vs real latency	When the operation takes seconds, the design of when matters as much as how much
7	The synthesis: compose cache + cascade + budget into a request path	It closes the thesis: the techniques accumulate and bring the feature within its budget

The arc is: first you learn to measure the two constraints (2), then to bound them with a budget (3), then the three techniques that enforce it —route (4), don't re-pay (5), don't block (6)— and at the end you compose them into a single design (7). Lesson 8 —the project— brings it all together on a Mercado feature you design from scratch, so you confirm you learned the method and didn't memorize a table.

What this module does NOT touch

It's worth marking the boundary from now, because there's a neighboring topic that looks like it belongs here and belongs to another part of the ecosystem.

Inference optimization is AI Engineering / infrastructure, not here. This module lowers cost and latency with architecture: which model you use, what you cache, what you pull off the critical path. There's another whole family of techniques for making the same model run faster and cheaper on the hardware —quantization (using fewer bits per weight), batching requests on the GPU, distillation, serving the model with an optimized runtime, choosing the GPU size—. All of that is real and valuable, but it's model infrastructure, not systems architecture, and it's taught in the AI Engineering and infra ecosystems. The mental rule: if the technique changes how the model runs internally, it's not from this module; if it changes how your system surrounds it (when you call it, with which model, what you store), it is. Throughout the module the model is a black box that takes and costs what it takes and costs; our work is the design around that box.

The model's quality —whether its answer is good— is module 3. When in lesson 4 we send an easy query to the cheap model, the obvious question will arise: "what if the cheap one answers worse?". That's a quality question, and it's answered with an eval —the gate that measures whether an answer is good enough—, which is all of module 3. Here we assume the routing is correct and measure the saving; verifying that quality doesn't drop when you go cheaper is the next module. The two go hand in hand: the cost budget (this module) and the quality gate (the next) are the two gates every AI feature needs.

In-depth capacity planning is system-design. We're going to use numbers —latency, cost, volume— to bound a budget. But the serious calculation of how many requests per second the system withstands, how it scales horizontally, how it's sized for five years, is system-design-fundamentals and system-design-scaling. Here the numbers exist to set and verify a feature budget, not to size Mercado's infrastructure.

Common mistakes

Treating the LLM as "just another API call" (a mental-model mistake). What happens: the team puts the LLM in the critical path as if it were just another internal microservice, without budgeting its latency or its cost. Weeks later, two surprises: the feature feels slow (seconds where the user expected milliseconds) and the monthly bill is ten times the estimate. Why it happens: the word "API" evokes something cheap and fast, and the LLM looks that way when you try it once with a query. How to spot it: if the design of an AI feature has no expected-latency number nor a projection of monthly cost at real volume, you didn't budget it. How to fix it: it's the whole module —measure (lesson 2), budget (lesson 3), and design within the budget (lessons 4–7)—.

Sending everything to the most expensive model "just in case" (over-engineering the cost). What happens: to ensure the best quality, the team routes all queries to the most powerful model. It works, but it pays the premium price for every trivial query —most of them—, and the cost skyrockets without the extra quality being of any use on the easy queries. Why it happens: "the expensive model is better" is confused with "the expensive model is better for everything," and it's forgotten that most traffic doesn't need that power. How to spot it: if a single model handles 100% of the traffic and it's the most expensive one, you're overpaying. How to fix it: the model cascade of lesson 4 —classify and route, like the call center—.

Ignoring the cost until the bill arrives (financial myopia). What happens: nobody looks at the cost during design because "one call costs cents"; the cost only becomes visible when the provider's charge arrives at the end of the month, and by then the architecture is already assembled and expensive to change. Why it happens: the cost per call is invisible and tiny; the aggregate cost is enormous but only shows up late, summed. How to spot it: if you can't say today how much your AI feature will cost per month at the expected volume, the cost will surprise you. How to fix it: put the taximeter to run in the design —project the monthly cost at real volume from day one (lesson 2)— and put an explicit budget on it (lesson 3).

Exercises

Exercise 1 — Translate the analogy to design. For each analogy of the module, say what architectural technique it represents and give a concrete example in a Mercado feature. (a) The taximeter that runs on each block. (b) The call center that handles with the junior and escalates to the senior. (c) The supermarket's express lane for few items.

See solution

(a) The taximeter → the cost per call that scales with volume. It represents the cost constraint, not a saving technique: each call to the LLM costs, and the total is that sum. Example in Mercado: each semantic search is a "block" with its cost; at 30,000 searches a day, the taximeter shows thousands of dollars a month. Recognizing it is the first step to budgeting it.
(b) The junior→senior call center → the model cascade. Classify the difficulty and route: easy to the cheap model, hard to the expensive one. Example in Mercado: the search "iphone" is trivial and the cheap model handles it; "something elegant but casual for a beach wedding" needs the strong model. Only the hard one pays the premium price.
(c) The express lane → the cache (and, in its don't-block version, async/streaming). Don't make wait what can go through a faster route. Example in Mercado: the search "laptop," already done a thousand times today, is served from the cache instantly (express lane) instead of recomputing it; and the long generation of "describe your product" is pulled off the critical path (async) so it doesn't stop the line.

The important thing: each analogy is a different idea —the taximeter is why it matters, the call center and the express lane are how you solve it—.

Exercise 2 — The dangerous phrase. A colleague proposes semantic search like this: "It's easy, it's just another API call: you send the query to the most powerful model and it returns the products sorted. We connect it to the search box and done." Identify the two hidden assumptions this plan makes about latency and cost, and why each is a problem.

See solution

The two hidden assumptions, one for each new physics of the module:

Latency assumption: "it responds fast like any API." A powerful LLM takes hundreds of milliseconds to seconds, not the few milliseconds of an internal service. Placed in the critical path of every search —and "connect it to the search box and done" puts it right there, synchronous— it makes the search box feel slow. The problem: the model's latency becomes the product's perceived latency, and the user leaves.
Cost assumption: "it costs little, it's just one call." Each call to the most powerful model costs the premium price per token, and "the search box" is tens of thousands of searches a day. The problem: "just one call" multiplied by the real volume is a bill of thousands of dollars a month, that nobody budgeted because the cost per call seemed insignificant.

The plan isn't wrong as a starting point —the powerful model gives good quality—; it's incomplete: it lacks the budget and lacks the architecture that respects it (cascade, cache, streaming). It's exactly the "naive" $10,535/month design from the table at the start.

Exercise 3 — Architecture or infrastructure? For each technique, say whether it's within this module's scope (architecture: how you surround the model) or on the AI Engineering/infra boundary (inference optimization: how the model runs internally), and why. (a) Caching the responses of repeated searches. (b) Quantizing the model to 4 bits so it takes less memory on the GPU. (c) Sending the easy queries to a smaller model. (d) Batching several requests into a single GPU pass.

See solution

(a) Cache responses → architecture (this module). It doesn't change how the model runs; it changes when you call it (you don't call it if you already have the answer). It's design around the model. Lesson 5.
(b) Quantize to 4 bits → infrastructure (boundary, AI Eng/infra). It changes how the model is represented internally so it runs faster/cheaper on the hardware. It's not a systems-architecture decision; it's inference optimization. Outside this module.
(c) Route the easy ones to a small model → architecture (this module). It doesn't change how any model runs; it decides which model handles which query. It's the model cascade. Lesson 4.
(d) Batching on the GPU → infrastructure (boundary). Grouping requests to make better use of the hardware is a model serving technique, internal to how inference is executed. Outside this module.

The rule that separates: if the technique changes how the model runs internally (b, d), it's infrastructure; if it changes how your system surrounds it —when you call it, with which one, what you store— (a, c), it's architecture, and it belongs here.

Summary and next step

In this lesson you met the module's thesis: the LLM's latency and cost are first-class design constraints, not an infrastructure detail. You saw the two new physics that separate the LLM from a classic function —it's slow (hundreds of ms to seconds) and costs per call (per token, every time)— and why "it's just another API call" is the phrase that precedes the surprise bill and the loading screen. The three analogies gave you the frame: the taximeter (why cost matters, it scales with the calls), the junior→senior call center (how to lower it with the cascade without losing quality), and the express lane (how to organize latency with cache and async). And in the executed table you saw the thesis in figures: the same Mercado semantic search costs $10,535/month in its naive version and $2,700/month with cascade + cache —74% less, just with architecture, without touching the model or the hardware—.

Before moving on you should be able to: explain why an LLM breaks the two assumptions of a normal function (cost ~0 and latency ~0); recognize the phrase "it's just another API call" as an incomplete design that lacks a budget and architecture; and separate an architecture technique (how you surround the model) from an infrastructure one (how the model runs), which is outside this module.

What follows is measuring the two constraints precisely, because you can't budget what you can't quantify. In lesson 2 you're going to put a number on "slow" —the latency that grows with the output tokens— and on "costs per call" —the price per token, and why the expensive model costs close to ten times the cheap one—, and you're going to see the taximeter running: how the same cheap call becomes $252, $2,520, or $25,200 a month depending on the volume. It's the step from "I know the LLM is slow and expensive" to "I know exactly how much, and I can project it to Mercado's scale."

Resources

Anthropic — Models overview (Claude docs) — the panorama of models by family (from fastest/cheapest to slowest/most capable); the conceptual basis of why there's a "cheap model" and a "strong model" that this module routes, without fixing a version.
martinfowler.com — "Emerging Patterns in Building GenAI Applications", Bharani Subramaniam and Martin Fowler — the catalog of architecture patterns for LLM apps (including the approach of treating cost and latency as design constraints); the essayistic reference that frames the whole module.
Chip Huyen — AI Engineering (O'Reilly), cost and latency chapters — the systematic treatment of how cost and latency shape the design of an AI system, including the model trade-off by size; the go-to book for the budgets and cascade part.

← PreviousMini-project: place an AI feature into Mercado Next →2. The LLM is slow and costs per call