Module 2: Latency and Cost as Architecture

2. The LLM is slow and costs per call

Next →

Overview

By the end of this lesson you'll be able to put a number on the two phrases lesson 1 left as intuition: the LLM is slow and costs per call. "Slow" stops being an adjective when you measure it: an LLM response takes hundreds of milliseconds to seconds, and that latency grows with the number of tokens it generates —a long response takes longer than a short one, proportionally—. "Costs per call" stops being vague when you see the price: you pay per token, with one rate per thousand input tokens and another per thousand output tokens, and the expensive model costs close to ten times the cheap one for the same work. You're going to measure both in the simulated stub, and then you're going to see the taximeter running: how that call costing a fraction of a cent becomes hundreds or thousands of dollars a month when you multiply it by Mercado's real volume.

This matters because the two constraints are invisible in testing and brutal in production. When you test semantic search once, with one query, the latency feels acceptable ("half a second, fine") and the cost is imperceptible ("$0.008, you don't even notice"). The problem is that no feature lives on a single call: it lives on tens of thousands a day. And there the two figures you ignored grow until they become the main problem: the latency, multiplied by every user who waits, defines whether the feature feels fast or sluggish; the cost, multiplied by every call, defines whether the feature is profitable or whether it eats the margin. Measuring both per call and projecting them at scale is the step that separates "I proved it works" from "I know what it will cost and how long it will take when they actually turn it on."

Connection with the module: this lesson is the module's scale. Lesson 1 gave you the intuition and the analogies; here you turn them into figures you can reproduce. You can't budget (lesson 3) what you can't measure, nor decide whether a cascade (lesson 4) or a cache (lesson 5) is worth it without knowing how much a call costs and takes. Everything that follows in the module rests on the stub you build here —call_llm(model, in_tokens, out_tokens), which returns the simulated latency and cost of a call— and on the taximeter, the idea that cost scales linearly with the number of calls. It's the most arithmetic lesson of the module, and it's the basis of all the others.

The taximeter that runs on each call

Think of it this way. You get into a taxi to cross the city. As soon as it starts, a device on the dashboard begins to run: every stretch you advance, the number goes up a few cents. You don't pay a flat fare for "using the taxi that day"; you pay for each unit of the trip, and when you get off, what you owe is exactly what the device shows. A short trip costs little and you don't even think about it. But if you made a hundred short trips in a day —something a person never does, but a system does—, the sum of all those "little bits" would be a huge bill.

Now swap the taxi for a call to the LLM. Each call has its own taximeter, and it runs on two axes at once: time (it takes hundreds of ms to seconds, and it takes longer the longer the response, just as a longer trip costs more) and money (you pay per token, input and output). A single call is like a short trip: the latency is tolerated and the cost isn't felt. But your feature doesn't make one trip: it makes tens of thousands a day. The taximeter that on one call shows $0.008 and 750 ms, multiplied by 100,000 daily calls, shows thousands of dollars a month in cost and defines the experience of every user who waits.

The difference from a normal function is exactly this. A classic function —a lookup in an index, a simple database query— has no taximeter: it costs essentially zero and responds in microseconds, so calling it a thousand times or a million changes nothing. The LLM does have one, on both axes. That's why "it's just another API call" is so dangerous: it treats the taxi as if it were the flat-fare subway. This lesson is learning to read the taximeter —measure the call and project the bill— before getting on the trip.

Worked example: measure a call and watch the taximeter

We're going to build the stub that simulates the LLM and measure with it. The stub is a deterministic function: it calls no API, uses no network, has no keys. You give it fixed numbers for latency and price, and it returns the latency and cost of a call. With that we measure Mercado's semantic search and project its monthly cost at different volumes.

The stub's fixed facts, declared once: each model has a price per 1000 input tokens (usd_in) and another per 1000 output tokens (usd_out), a base latency per call (base_ms), and an extra latency per output token (ms_per_tok, because the model generates the response token by token, and more tokens = more time). The cheap model is fast and cheap; the strong model is slower and costs close to ten times more. These are numbers chosen for the example —not from any real provider—, but the geometry is that of the real world: a small model responds in fractions of the time and cost of a large one.

# Lesson 02 — the LLM is slow and costs per call
# LLM STUB: everything SIMULATED. Zero network, zero API, zero keys.

MODELS = {
    "cheap":  dict(usd_in=0.0008, usd_out=0.004, base_ms=90,  ms_per_tok=0.4),
    "strong": dict(usd_in=0.008,  usd_out=0.040, base_ms=300, ms_per_tok=3.0),
}

def call_llm(model, in_tokens, out_tokens):
    """Simulates ONE call to the LLM. Returns (latency_ms, cost_usd). Deterministic."""
    m = MODELS[model]
    latency_ms = m["base_ms"] + out_tokens * m["ms_per_tok"]
    cost_usd = (in_tokens / 1000) * m["usd_in"] + (out_tokens / 1000) * m["usd_out"]
    return latency_ms, cost_usd

# A typical call from Mercado's semantic search:
# the prompt carries the user's query + instructions (~300 input tokens)
# and the model returns a reordered list (~150 output tokens).
IN_TOK, OUT_TOK = 300, 150

print(f"{'model':<8}{'latency_ms':>12}{'cost_usd':>14}")
for name in ("cheap", "strong"):
    lat, cost = call_llm(name, IN_TOK, OUT_TOK)
    print(f"{name:<8}{lat:>12.1f}{cost:>14.6f}")

# Contrast: a normal function (a lookup in an index) — the "classic component".
# A lookup is around a microsecond (~0.001 ms) and costs no money. Representative number.
normal_ms = 0.001
print(f"\n{'lookup':<8}{normal_ms:>12.3f}{0.0:>14.6f}   (normal function: ~0 ms, $0)")

# The taximeter: cost scales LINEARLY with the number of calls.
print("\nThe taximeter — monthly cost by volume (strong model only):")
_, unit_cost = call_llm("strong", IN_TOK, OUT_TOK)
for calls_per_day in (1_000, 10_000, 100_000):
    monthly = unit_cost * calls_per_day * 30
    print(f"  {calls_per_day:>7,} calls/day  ->  ${monthly:>10,.2f}/month")

What to expect. When you run it:

model     latency_ms      cost_usd
cheap          150.0      0.000840
strong         750.0      0.008400

lookup         0.001      0.000000   (normal function: ~0 ms, $0)

The taximeter — monthly cost by volume (strong model only):
    1,000 calls/day  ->  $    252.00/month
   10,000 calls/day  ->  $  2,520.00/month
  100,000 calls/day  ->  $ 25,200.00/month

Read these numbers slowly, because each is one of the two constraints made a figure.

The latency. The cheap model responds in 150 ms; the strong in 750 ms —five times slower for the same query—. Compare them with the lookup, which is around 0.001 ms: the strong model is about 750,000 times slower than a normal function. That's the jump "it's just another API call" ignores. And notice where the strong's latency comes from: base_ms (300) plus 150 output tokens × ms_per_tok (3.0) = 300 + 450 = 750. Most of the time goes into generating the response token by token. That's why a long response takes longer: if the strong had to generate 400 tokens instead of 150, it would take 300 + 400×3 = 1500 ms. Latency isn't a constant of the model; it depends on how much text it produces. Remember it, because it's the key to streaming (lesson 6).

The cost. The cheap model costs $0.000840 per call; the strong $0.008400 —ten times more—. On one call, both are imperceptible: less than a cent. This is where instinct fails: "it costs cents, doesn't matter." But then comes the taximeter.

The taximeter. The same call to the strong model, multiplied by volume:

  • 1,000 calls/day → $252/month. A small pilot; tolerable.
  • 10,000 calls/day → $2,520/month. Starts to hurt.
  • 100,000 calls/day → $25,200/month. That's a bill a CFO notices, for a single feature.

The cost scales linearly with the calls: ten times more traffic, ten times more bill, exactly. There's no economy of scale to save you —each call pays its full taximeter—. And this number, $25,200/month, is the one that's going to chase semantic search all module long: it's what the naive version costs ("everything to the strong model"), and it's what the cascade and the cache will have to lower for the feature to fit within its budget.

Going deeper: tokens, input/output, and why the expensive one costs more

It's worth understanding where these figures come from, because the rest of the module manipulates them.

What a token is. An LLM doesn't process characters or whole words: it processes tokens, which are fragments of text (approximately 3-4 characters, or about ¾ of an English word). "Semantic search" is a few tokens; a paragraph is dozens; a long document, thousands. Everything you send the model (the prompt: the user's query plus your instructions plus any context) are input tokens, and everything the model generates are output tokens. The price is charged for both, separately, typically per million (or thousand) tokens.

Why output costs more than input. In almost all models, the price per output token is several times that of input —in the stub, usd_out (0.004 / 0.040) is 5× and 5× the usd_in (0.0008 / 0.008)—. The reason is technical and we don't teach it here (it's from the inference boundary): generating each output token is more computationally expensive than reading an input token. What does matter at the architecture level is the consequence: a long response is expensive twice over —it costs more money (more output tokens at the high price) and more time (more ms_per_tok)—. That's why a design decision as simple as "ask for concise responses" or "limit the output length" is, literally, a cost and latency decision.

Why the expensive model costs ~10x. The strong costs ten times the cheap because it's a larger and more capable model. In the real world this gap exists and is large: a small model in a family (something like Claude Haiku) versus a large one (something like Claude Sonnet or Opus) has a price difference of several times —the stub's order of magnitude is realistic, though the exact numbers change with the provider and version, which is why we don't fix them—. This gap is the whole reason for the model cascade (lesson 4): if the cheap one can resolve a query well, sending it to the expensive one is overpaying 10× for nothing. And it's the reason for the cache (lesson 5): if you already computed a response, paying for it again —even to the cheap one— is throwing money away.

How a feature's cost is estimated. The pattern is always the same: monthly_cost = cost_per_call × calls_per_day × 30. And cost_per_call depends on the tokens: (in_tokens/1000)×usd_in + (out_tokens/1000)×usd_out. With those two formulas you can estimate, before writing the feature, how much it will cost at the expected volume. That's the taximeter put into the design, and it's what lesson 3 turns into a budget.

Common mistakes

Measuring latency and cost with a single call (a sampling mistake). What happens: you test the feature once, see "750 ms, $0.008" and conclude it's fine. But one call tells you neither the tail latency (some queries generate long responses and take twice as long) nor the aggregate cost (a cheap call × 100,000 = expensive). Why it happens: in development you always test with volume one, and volume one hides both constraints. How to spot it: if your cost or latency estimate comes from "I tried it and it felt fine" instead of "I measured the call and multiplied it by the expected volume," you sampled, you didn't measure. How to fix it: measure the typical call and the heavy one (more output tokens), and project the cost at real volume —the taximeter, not the single trip—.

Ignoring the output tokens (a wrong-focus mistake). What happens: the team optimizes the input prompt to save tokens, but lets the model generate extremely long responses with no limit. Since the output costs more per token and adds latency per token, the long response is where the money and the time really go. Why it happens: the input prompt is visible and editable, so it's optimized; the output length feels like "whatever the model decides" and is ignored. How to spot it: if you have no limit nor instruction about the response length, you're not controlling the more expensive half of the taximeter. How to fix it: treat the output length as a design lever —ask for concision, limit max_tokens, and on high-volume features this alone can cut the cost notably—.

Treating the cost per call as the total cost (a scale mistake). What happens: someone says "one call costs less than a cent, the cost is negligible" and closes the topic. The mistake is confusing the unit cost (tiny) with the aggregate cost (enormous), which only appears when you multiply by the volume. Why it happens: the per-call number is reassuringly small, and the brain doesn't multiply by 3 million (100,000/day × 30) intuitively. How to spot it: if your argument about cost doesn't include the monthly volume, you're looking at the trip and not the taximeter. How to fix it: never cite the cost per call without citing, in the same sentence, the monthly cost at expected volume —$0.008/call is $25,200/month at 100k/day, and that second half is the one that decides—.

Exercises

Exercise 1 — The heavy query. With the example's stub, compute by hand the latency and cost of a heavy query (300 input tokens, 400 output) for the two models, and say how much longer and more expensive the heavy one is than the typical one (150 output) on the strong model.

See solution

Stub formulas: latency = base_ms + out_tokens × ms_per_tok; cost = (in/1000)×usd_in + (out/1000)×usd_out.

  • cheap, heavy (in 300, out 400): latency = 90 + 400×0.4 = 250 ms; cost = 0.3×0.0008 + 0.4×0.004 = 0.00024 + 0.0016 = $0.001840.
  • strong, heavy: latency = 300 + 400×3.0 = 1500 ms; cost = 0.3×0.008 + 0.4×0.040 = 0.0024 + 0.016 = $0.018400.

The heavy one versus the typical one on the strong model: the typical one (out 150) takes 750 ms and costs $0.0084; the heavy one (out 400) takes 1500 ms (double) and costs $0.0184 (2.2×). Almost all the increase comes from the output tokens: 250 more output tokens add 250×3 = 750 ms and 0.25×0.040 = $0.010. Moral: the response length dominates both latency and cost, and that's why it's a first-order design lever.

Exercise 2 — The support agent's taximeter. Mercado's support agent has conversations of, on average, 4 turns, and each turn is a call to the strong model with 500 input tokens and 200 output. If Mercado has 5,000 conversations a day, how much does the agent cost per month? (Use the strong rates: usd_in=0.008, usd_out=0.040.)

See solution
  • Cost per turn: (500/1000)×0.008 + (200/1000)×0.040 = 0.004 + 0.008 = $0.012.
  • Cost per conversation: 4 turns × $0.012 = $0.048.
  • Calls/day: 5,000 conversations × 4 turns = 20,000 calls/day.
  • Monthly cost: $0.048 × 5,000 conversations × 30 = $7,200/month (equivalent: $0.012 × 20,000 × 30 = $7,200).

$7,200/month for the support agent, if everything goes to the strong model. The key lesson for the rest of the module: the agent's cost accumulates per turn —a conversation isn't one call, it's four—, so the cost constraint bites harder on conversational features than on a single-shot search. That's exactly why lesson 7 applies the cascade and the cache to the support agent.

Exercise 3 — When is an LLM "just another API call"? A colleague argues: "deep down, calling the LLM is like calling our inventory microservice: you send it a request, it returns a response. I don't see why it needs special treatment." Answer with the two measurable differences you saw in this lesson, and give an example of an architecture decision that changes because of each.

See solution

The two differences, with their number:

  1. Latency. The inventory microservice responds in a few milliseconds; the LLM takes hundreds of ms to seconds (750 ms the strong in the example, and more if the response is long). Architecture decision that changes: where you put it in the flow. A millisecond service can go synchronous in the critical path without anyone noticing; an LLM of seconds in the critical path makes the feature feel slow, so you have to decide whether to use streaming, pull it off the path with async, or bound its latency with a budget.

  2. Cost. The inventory microservice is your own infrastructure: its marginal cost per call is essentially zero. The LLM charges per token, every time: $0.008 per call, which at 100k/day is $25,200/month. Architecture decision that changes: which model handles what and what you cache. With a free service it doesn't matter to call it extra; with an LLM, every unnecessary call is money, so the cascade (cheap for the easy) and the cache (don't re-pay) appear —decisions that wouldn't make sense for the inventory microservice—.

The synthesis: the LLM is called like an API, but it behaves like a taxi with a two-axis taximeter, not like the flat-fare subway. That behavior is what demands the redesign, and it's everything this module teaches.

Summary and next step

In this lesson you measured the two constraints lesson 1 only named. With the call_llm stub you saw that the cheap model responds in 150 ms and the strong in 750 ms —hundreds of thousands of times slower than a lookup—, and that latency grows with the output tokens (300 base + 3 ms per token in the strong), which makes the response length a design lever. You saw that you pay per token, input and output separately, with the output more expensive, and that the strong costs ~10× the cheap. And you saw the taximeter: the same $0.008 call becomes $252, $2,520, or $25,200 a month depending on the volume, because the cost scales linearly with the calls. That $25,200/month of semantic search on the strong model is the number the rest of the module is going to attack.

Before moving on you should be able to: estimate the latency and cost of an LLM call from its input and output tokens; project a feature's monthly cost at real volume (the taximeter); explain why the response length dominates both cost and latency; and answer "it's just another API call" with the two measurable differences.

What follows is turning these numbers into a design constraint. In lesson 3 you're going to define a feature's latency budget and cost budget —how long a search can take before the user leaves, how much it can cost per month before the margin disappears— and you're going to write the gate that verifies whether an option respects or violates them. You're going to see, by executing, that sending everything to the strong model violates both budgets of Mercado's semantic search. With that, the three lessons that follow (cascade, cache, async) stop being loose tricks and become what they are: the techniques for making the design fit within its budget.

Resources

← Previous1. Module introduction: latency and cost as architectureNext →3. The latency and cost budget