Module 2: Latency and Cost as Architecture

5. Cache: many queries repeat

Next →

Overview

By the end of this lesson you'll know how to design the other big cost lever of an AI feature, complementary to the cascade: the cache. The idea is as old as computing and here it pays off more than ever: don't recompute what you already computed. Many queries in a real system repeat —"iphone", "laptop", "how do I return a product" are asked thousands of times a day— and calling the LLM again to answer something you already answered is throwing away money and time. The cache stores the response the first time and serves it instantly the following times, at zero cost and millisecond latency. You're going to measure the metric that governs all its saving —the hit-rate, the fraction of queries served from the stored— and you're going to compare two ways of caching: the exact (same literal query) and the semantic (same intent written differently), which catches repetitions the exact one lets through.

This matters because, at high volume, not re-paying is the biggest saving of all —bigger than the cascade—. The cascade makes each call cheaper (from expensive to cheap); the cache eliminates the call entirely (from cheap to zero). If half your traffic is repetitions, caching well cuts your cost in half in one stroke, no matter which model you use for what does reach the LLM. And latency improves in the same proportion: a cached response arrives in 2 ms instead of 750. The cache is the first technique to consider in any high-volume AI feature, because the repeated traffic is money you're paying twice —and many times, many more than twice—.

Connection with the module: this lesson closes the pair of cost levers. Lesson 3 gave you the budget; lesson 4 attacked it by making each call cheaper (cascade); this one attacks it by eliminating repeated calls (cache). They're complementary and they accumulate: remember the cascade alone left Mercado's search at $16,241/month, still over the $3,000 budget —the cache is what's missing to go lower—. Lesson 6 (async/streaming) will attack the tail latency that neither cascade nor cache fixes. And lesson 7 composes the three and demonstrates —with the numbers— that together they bring the feature within its budget. The cache also touches an idea from lessons 3 onward: it serves "old" data (the stored response can become outdated), which introduces a freshness trade-off we'll see in the deep-dive.

The supermarket's express lane and the customer who already paid

Think of it this way. At the supermarket, not everyone waits in the same line. There's an express lane for the one with few items, and there's something even faster: the customer who already paid and just comes back to pick up something they set aside —that one doesn't even wait in line, it's handed over instantly—. The cache is that second customer. The first time someone searches "iphone", the system does the full work: it calls the LLM, pays its tokens, waits its 750 ms, and —crucially— stores the response. The second time someone searches "iphone", it doesn't repeat the work: it takes the stored response and hands it over instantly, at zero cost, like the set-aside package. Only the new queries wait in the full line (the LLM call); the repeated ones are served from what's already done.

The word that measures how well this works is the hit-rate: of all the queries that come in, what fraction finds its response already stored (a "hit") instead of having to compute it (a "miss")? A 50% hit-rate means half the queries are served free and instantly; a 70% hit-rate, that only three in ten reach the LLM to bother it. Since the saving is directly proportional to the hit-rate —each hit is a call you didn't pay for—, raising the hit-rate is raising the saving, point by point.

And here comes the interesting question, which is where the design lives: when are two queries "the same"? If one customer searches "iphone" and another searches "iPhone" (with a capital) or "iphone " (with a space) or "iphone phone" (different order), are they the same search or three different ones? An exact cache says "only if the text is identical letter by letter" —and lets the variants through as new—. A semantic cache says "if they have the same intent, even if written differently" —and catches the variants as repetitions—. The looser the definition of "the same," the more hits, the more saving. This lesson is learning to set up the express lane and to decide how generous it is at recognizing the customer who already paid.

Worked example: hit-rate and saving, exact versus semantic

We're going to set up a cache over a realistic workload and measure its hit-rate and its saving, comparing three ways of deciding "the same query." The scenario is real search life: users search for the same thing written a thousand different ways. We generate 3000 searches over a catalog of 256 intents (adjective + noun combinations, like "wireless earbuds") with Zipf popularity —a few concentrate almost all the traffic— plus a 20% "long tail": rare searches that never repeat. And each search is rendered with a surface variant of its intent: different word order, different capitalization, different spacing. Thus the same intent rarely appears twice identical —exactly the case that separates the exact cache from the semantic one—.

A miss costs a model call (we use the strong with a typical query: 750 ms, $0.0084). A hit costs almost nothing: read from cache, ~2 ms and $0. The three cache keys, from strictest to loosest:

  • raw: the query as it was written (only literal repetitions count).
  • exact: normalizes case and spaces (" ".join(q.lower().split())) —catches the case and spacing variants—.
  • semantic: also sorts the words (tuple(sorted(...))) —also catches the different word order, near-duplicates of the same intent—.
# Lesson 05 — response cache (many queries repeat)
# LLM STUB: everything SIMULATED. Zero network, zero API, zero keys. Fixed seed.
import random

MODELS = {
    "cheap":  dict(usd_in=0.0008, usd_out=0.004, base_ms=90,  ms_per_tok=0.4),
    "strong": dict(usd_in=0.008,  usd_out=0.040, base_ms=300, ms_per_tok=3.0),
}
def call_llm(model, in_tokens, out_tokens):
    m = MODELS[model]
    return m["base_ms"] + out_tokens * m["ms_per_tok"], \
           (in_tokens/1000)*m["usd_in"] + (out_tokens/1000)*m["usd_out"]

# A MISS costs a model call (strong, typical query). A HIT is nearly free.
MISS_MS, MISS_COST = call_llm("strong", 300, 150)   # (750.0, 0.0084)
HIT_MS, HIT_COST = 2.0, 0.0                          # read from cache: ~2 ms, $0

# --- The workload: 3000 searches. Users search for the SAME thing written differently. ---
# 256 intents (adjective x noun) with Zipf popularity, + 20% unique long tail.
ADJ = ["wireless","gaming","leather","stainless","portable","waterproof","ergonomic","compact",
       "premium","budget","vintage","smart","foldable","insulated","rechargeable","adjustable"]
NOUN = ["earbuds","laptop","case","bottle","chair","lamp","backpack","speaker","keyboard","desk",
        "jacket","kettle","boots","watch","monitor","scale"]
INTENTS = [f"{a} {n}" for a in ADJ for n in NOUN]
WEIGHTS = [1.0/(i+1) for i in range(len(INTENTS))]

random.seed(11)
N, P_UNIQUE = 3000, 0.20
_uid = [0]
def render(intent):
    """The same intent, written each time with different order, spacing, and case."""
    words = intent.split()
    words = random.choice([words, words[::-1]])          # word order
    sep   = random.choice([" ", "   "])                  # 1 or 3 spaces
    case  = random.choice([str.lower, str.upper, str.title])
    return case(sep.join(words))
def one_query():
    if random.random() < P_UNIQUE:                       # long tail: never repeats
        _uid[0] += 1
        return f"rare oneoff {_uid[0]}"
    return render(random.choices(INTENTS, weights=WEIGHTS, k=1)[0])
queries = [one_query() for _ in range(N)]

# --- Three cache keys, from strictest to loosest ---
def key_raw(q):      return q                                    # as it was written
def key_exact(q):    return " ".join(q.lower().split())          # normalizes case + spacing
def key_semantic(q): return tuple(sorted(key_exact(q).split()))  # + order-invariant

def run_cache(key_fn):
    cache, hits, cost, ms = set(), 0, 0.0, 0.0
    for q in queries:
        k = key_fn(q)
        if k in cache:
            hits += 1; cost += HIT_COST; ms += HIT_MS
        else:
            cache.add(k); cost += MISS_COST; ms += MISS_MS
    return hits/N, cost, ms

base_cost, base_ms = N*MISS_COST, N*MISS_MS
print(f"{'strategy':<16}{'hit_rate':>10}{'total_cost':>12}{'cost_saved':>12}{'avg_lat_ms':>12}")
print(f"{'no_cache':<16}{'0.0%':>10}{base_cost:>12.4f}{'0.0%':>12}{base_ms/N:>12.1f}")
for name, fn in (("cache_raw", key_raw), ("cache_exact", key_exact), ("cache_semantic", key_semantic)):
    hr, cost, ms = run_cache(fn)
    print(f"{name:<16}{hr*100:>9.1f}%{cost:>12.4f}{(1-cost/base_cost)*100:>11.1f}%{ms/N:>12.1f}")

What to expect. When you run it:

strategy          hit_rate  total_cost  cost_saved  avg_lat_ms
no_cache              0.0%     25.2000        0.0%       750.0
cache_raw            47.7%     13.1796       47.7%       393.2
cache_exact          66.9%      8.3328       66.9%       249.3
cache_semantic       72.5%      6.9300       72.5%       207.7

Here's the express lane working, and here's the story of how "what counts as the same query" moves the saving. Read it row by row.

Caching, plainly, is already enormous. Going from no-cache (0% hit, $25.20, 750 ms average) to the dumbest cache, raw (which only catches literal repetitions), already gives you 47.7% hit-rate: almost half the searches are served free. The cost is cut in half ($13.18) and the average latency too (393 ms, because half now arrives in 2 ms). And notice: the cost saving is exactly equal to the hit-rate —47.7% hits = 47.7% saving—, because each hit is a call you didn't pay for. The hit-rate is the saving. That's the metric that governs everything.

Normalizing (exact) raises the hit-rate a lot. The exact cache —which collapses case and spaces, treating "iphone", "IPHONE", and "iphone " as the same— jumps to 66.9% hit-rate: almost 20 percentage points more than raw. Why so much? Because in the real world people write the same thing with different casing and spacing all the time, and the raw cache counted them as new searches —an unnecessary miss each time—. Normalizing is cheap (a string transformation) and recovers all that traffic: the cost drops from $13.18 to $8.33. It's the best benefit/effort ratio improvement of the lesson.

Semantic (near-duplicates) raises it a bit more. The semantic cache —which also ignores the word order, treating "wireless earbuds" and "earbuds wireless" as the same— reaches 72.5% hit-rate: almost 6 points more than exact. It catches the repetitions that differ only in order, which the exact one let through. The cost drops to $6.93 and the average latency to 208 ms. The gain over the exact one is smaller than that of the exact over the raw —because the order variants are less common than the case/spacing ones— but it's real, and on very high-volume features those 6 points are money.

Put the three rows together and you have the lesson: the cache is the biggest cost lever at high volume —from 0% to 72.5% saving just by not re-paying— and how loose the definition of "the same query" is decides how much you save. The big gain is caching plainly (0 → 47.7%); normalizing enlarges it a lot (→ 66.9%); the semantic tops it off (→ 72.5%). And notice what it doesn't cache: the long tail, that 20% of rare searches that never repeat, is always a miss, no matter the technique —that's why no cache reaches 100%—. The hit-rate ceiling is set by how much your traffic repeats, not by how clever your cache is.

Going deeper: real semantic cache, TTL, and the danger of stale data

How a real semantic cache works. In the example, the "semantic" cache is an honest but simple approximation: it normalizes and sorts words. A production semantic cache goes further: it turns each query into an embedding (a vector that represents its meaning) and considers "the same" two queries whose vectors are very close —so it catches not just "earbuds wireless" but "audífonos inalámbricos", "wireless headphones", synonyms and paraphrases—. How those embeddings are computed and how the nearest one is searched for is AI Engineering mechanics (vector stores, cosine similarity) and is outside this module. What is from here is the architecture decision: a semantic cache raises the hit-rate (catches more repetitions) but introduces a risk —two "close" queries may not really be the same, and serving one's response for the other would be a mistake—. The looser the cache, the more hits but the more risk of serving a response that doesn't correspond. That's the trade-off you decide when choosing the similarity threshold.

The TTL and the danger of stale data. A cached response is, by definition, old: it was computed in the past, and the world may have changed since. If you cache the search "iphone" and the iphone's price changes, the cache keeps serving the old price until you invalidate it. That's why every cache has a TTL (time-to-live): how long an entry is considered fresh before recomputing it. A long TTL maximizes the hit-rate (fewer recomputations) but serves older data; a short TTL keeps the freshness but lowers the hit-rate. It's the same performance↔freshness trade-off as the classic cache —caching buys speed and cost in exchange for freshness—, and the decision depends on how fast the data changes: a product's description tolerates a long TTL (it rarely changes); the price or the stock, a short one (they change often). Serving stale data by accident is the classic cache mistake, and the TTL is the knob to control it.

What to cache and what not. Not everything is cached the same. Read-only and stable queries (catalog searches, support FAQs) are ideal: they repeat a lot and their response changes little. Personalized queries (which depend on the specific user, their history, their cart) cache poorly —the answer to "recommend me something" for Ana isn't useful for Beto—, so they're either not cached, or cached per user (fewer hits). And queries that trigger an action (they don't just read, they do something: create an order, send an email) are never cached —serving a cached response for an action would be executing the action with old data, or not executing it believing it was already done—. The cache is for reading, not for acting; the boundary between proposing and disposing (which the guide's module 6 covers in depth) also applies here.

Common mistakes

Not caching repeated queries (an omission mistake). What happens: the feature calls the LLM on every query, even for those that repeat a thousand times a day. The full work is paid for (and waited on) for responses that were already computed. At high volume, this is throwing away half the budget or more. Why it happens: caching is seen as "premature optimization" and postponed, or it's not noticed how much traffic is repeated until it's measured. How to spot it: if you don't know your feature's potential hit-rate (what fraction of the traffic repeats), you don't know how much you're overpaying. How to fix it: measure your traffic's repetition and put in a cache —starting with the simplest (raw or exact) already recovers most of the saving—.

Caching with the key too strict (a low hit-rate mistake). What happens: a raw cache is put in (only literal repetitions) and the hit-rate comes out low, because people write the same query with different case, spacing, and order, and each variant is a miss. The team concludes "the cache didn't help much" when the problem was the key. Why it happens: the raw key is the one-line implementation, the obvious one, and the surface variants aren't thought of. How to spot it: if your hit-rate is suspiciously low for traffic you know repeats, the key is being too literal. How to fix it: normalize the key (case + spacing) to jump from raw to exact —in the example, almost 20 points of hit-rate for free—; and consider semantic if the language/order varies a lot.

Caching data that changes, with no TTL (a freshness mistake). What happens: everything is cached with a long TTL (or no invalidation), including data that changes —prices, stock, order statuses—, and the feature starts serving old information: outdated prices, "in stock" when it's already sold out. The bug is invisible until a customer complains. Why it happens: the cache is set up thinking about the saving (high hit-rate = long TTL) and it's forgotten that the response ages. How to spot it: if you cache data that changes and can't say how long it can stay stale, you didn't control the freshness. How to fix it: set a TTL according to how fast the data changes (short for price/stock, long for descriptions), or invalidate the entry when the underlying data changes. And never cache a query that triggers an action.

Exercises

Exercise 1 — Translate the hit-rate to money. Mercado's semantic search, without a cache, costs $25,200/month (100k searches/day to the strong model). If you put in a cache with 66.9% hit-rate (the exact from the example), how much does it cost per month? And if you go up to 72.5% (the semantic)? Is the jump from exact to semantic worth it?

See solution

Since the cost saving equals the hit-rate (each hit is an unpaid call), the monthly cost with a cache is cost_without_cache × (1 − hit_rate):

  • With exact (66.9%): $25,200 × (1 − 0.669) = $25,200 × 0.331 = $8,341/month.
  • With semantic (72.5%): $25,200 × (1 − 0.725) = $25,200 × 0.275 = $6,930/month.

The jump from exact to semantic saves $8,341 − $6,930 = $1,411/month more. Is it worth it? It depends on the cost of setting up and operating the semantic cache —which is more complex: it needs embeddings, a vector store, and tuning a similarity threshold, with the risk of serving a "close" response that doesn't correspond—. $1,411/month is real and grows with more volume, so on a high-traffic feature it's usually justified; but the biggest gain was already captured by the exact cache (from $25,200 to $8,341), which is much cheaper to set up. The rule: start with the exact (big saving, little effort) and go up to semantic only if the incremental saving justifies its complexity and its risk. Don't jump straight to the complex.

Exercise 2 — Cache plus cascade, together. So far the cache serves the misses with the strong model. But what if you combine cache with cascade? The hits go free (0%), and the misses go to the cascade (which makes them cheaper by difficulty). Reason why the incremental gain of the cascade shrinks when you already have a cache with a high hit-rate, and what that implies for the order in which you apply the techniques.

See solution

With a cache of, say, 70% hit-rate, only 30% of the traffic (the misses) reaches the LLM. The cascade operates only on that 30% —the hits no longer touch any model, they go free—. So the cascade's saving, which over 100% of the traffic cut ~44% of the cost, now cuts ~44% but only of the 30% that's left: its absolute impact on the total is much smaller, because the cache already ate the 70% cheap.

Put another way: the cache and the cascade attack the same easy/repeated traffic, so they overlap —they don't add up independently—. The cache eliminates the repetitions (which tend to be easy); the cascade makes cheaper what's left (which tends to be more unique, harder, more oriented to the strong model). That's why the cascade's gain on top of a good cache is modest.

Implication for the order: it's best to cache first (eliminates the bulk of the traffic free) and then apply the cascade over the misses (makes cheaper what does reach the LLM). It's the order lesson 7 is going to set up: cache → cascade. And the general moral: the techniques compose but don't add up, because they overlap over the cheap traffic —you have to measure the combination, not add the savings separately—.

Exercise 3 — What to cache in the support agent? Mercado's support agent answers many types of message. For each one, say whether you'd cache it, with what TTL (short/long/never), and why: (a) "what's your return policy?"; (b) "where's my order #48213?"; (c) "I want to cancel my order #48213".

See solution
  • (a) Return policy → cache, long TTL. It's a very frequent question (a lot of potential hit-rate) and the answer rarely changes (the policy is stable). Caching it is ideal: it repeats a lot, ages slowly. A long TTL (hours or days) maximizes the saving with minimal risk of serving something stale; and when the policy really changes, the entry is invalidated.
  • (b) "Where's my order #48213?" → cache carefully, short TTL (or cache the pattern, not the response). An order's status changes (in transit → delivered), so a cached response ages fast. If you cache it, the TTL must be short (minutes) so as not to say "in transit" when it's already been delivered. Better yet: here the response depends on live data (the order status), so instead of caching the LLM's response, the current status is looked up each time —the cache is for stable responses, not for data that changes by the second—.
  • (c) "I want to cancel my order #48213" → NEVER cache. This isn't a read query: it's an action (canceling an order). Serving a cached response here would be disastrous —either it would believe it was already canceled when it wasn't, or it would trigger the cancellation with old data—. Queries that act on the system's state are never cached; the cache is for reading, not for doing.

The rule that emerges: cache the stable and read-only (a), handle carefully and with a short TTL what changes (b), and never cache an action (c). The data's freshness and the read-vs-act distinction decide what enters the express lane.

Summary and next step

In this lesson you set up the other big cost lever: the cache, which doesn't recompute the already-computed. With the supermarket's express lane you saw that the first search does the full work and stores it, and the following ones are served instantly at zero cost. You measured the metric that governs the saving —the hit-rate, equal to the saving percentage because each hit is an unpaid call— and you compared three keys over 3000 searches where users write the same thing in different ways: raw 47.7% → exact 66.9% → semantic 72.5%, with the cost dropping from $25.20 to $6.93 and the average latency from 750 to 208 ms. You learned that caching plainly is already enormous, that normalizing (exact) is the best benefit/effort improvement, and that the semantic tops it off by catching near-duplicates. And you saw the limits: the long tail never caches (the hit-rate ceiling is set by your traffic), the cached data ages (TTL, freshness trade-off), and actions are never cached (the cache reads, it doesn't act).

Before moving on you should be able to: design a cache for a feature and choose its key (raw/exact/semantic) according to how the traffic varies; measure its hit-rate and translate it to cost and latency saving; explain why the cache and the cascade compose but don't add up (they overlap over the cheap traffic); and decide what to cache, with what TTL, and what never to cache (data that changes, actions).

What follows attacks the problem neither the cascade nor the cache solves: the perceived latency on the queries that do reach the LLM and take seconds. In lesson 6 you're going to see two techniques: streaming, which shows the response token by token so the user sees text almost immediately instead of waiting for the complete block; and async, which pulls the heavy work off the critical path entirely, for the operations where the user shouldn't even wait. You're going to measure how streaming lowers the perceived latency even though the real work still takes the same, and when each technique applies. It's the latency lever that completes the set.

Resources

← Previous4. The model cascade: cheap first, escalate only if neededNext →6. Async and streaming: when the user can't wait