Module 8: Project — Design Enlace End to End

6. Step 4b — The read path and the cache

Next →

Description

If the write path was the easy one, this is the one that matters. Step 2 said it with a number that governs Enlace's entire design: ~4,000 reads/s, a hundred times the writes. Enlace is read-heavy, and all the design energy goes here, to the resolve path. In this lesson you deep-dive into that path end to end over the distributed architecture: the cache-aside pattern (look at the cache first, go to the database only if it misses), the target hit ratio and the average latency it produces (the formula L = h·L_cache + (1−h)·L_db, executed: hit ratio 0.90 → 5.90 ms), the working set that makes the cache dirt cheap (~333 MB, not the 6 TB), and the residual load that the cache does not absorb (386 reads/s) —the number lesson 7 will have to distribute among replicas—.

You are going to execute the latency formula to see the real table and the counterintuitive lesson it brings (raising the hit ratio does not lower the latency proportionally, but at an accelerating rate), compute the working set with the 80/20 rule, and —crucial for the capstone— deliver the cache sizing sheet: target hit ratio, memory, resulting latency, residual load, eviction policy and TTL. It is the component where Enlace's design proves it understood its own nature: a system whose challenge is not the data nor the network, but serving the same read, 4,000 times per second, in under 100 ms.

Connection to the module: this is the second "deep dive" lesson (5: write, 6: read, 7: scaling). It takes the box of the diagram that says "read → cache → (miss) → replica" and opens it from the inside. It leans on all of module 4 (cache-aside, hit ratio, working set, TTL). And its output —the 386 residual reads/s— is the input of lesson 7: the load the cache lets through is exactly what the database scaling must absorb. The cache is the first line of defense of the read path; the replicas are the second.

The receptionist with the notebook of regulars

Think of it this way. In a very busy office building, the receptionist has to tell each visitor which floor the company they are looking for is on. They could, each time, call the central directory by phone, wait for the answer, and pass it on —slow, and the phone always busy—. But the smart receptionist does something else: they have a notebook at hand with the companies asked about most often. When a visitor arrives, they look at the notebook first: if the company is there (the most frequent, because a few companies concentrate almost all the visits), they answer instantly, without touching the phone. Only if the company is not in the notebook —a rare visit— do they call the central directory, and in passing they jot it down in the notebook, in case it is asked about again.

Notice the economy of this. The notebook does not have all the companies in the building —it would be a huge and useless tome—; it has only the hot ones, the ones asked about often, which are a handful. And that handful resolves the vast majority of the queries, because the visits concentrate on few companies (a few popular businesses receive almost all the traffic; the long tail of small offices receives almost no visits). The result: the receptionist answers 90% of the time instantly from the notebook, and only 10% costs the slow call. The notebook is tiny, cheap to maintain, and still catches almost all the traffic.

That receptionist with their notebook is Enlace's read path with its cache. The notebook is the cache (Redis); the central directory is the database (the sharded replicas); "look at the notebook first, call the directory only if it is missing, and jot down what was missing" is the cache-aside pattern. The small notebook that catches 90% is the working set (~333 MB of hot links, not the 6 TB of the complete directory). And the receptionist's lesson —few companies concentrate the visits— is the 80/20 rule that makes the cache cheap. resolve is, exactly, a receptionist answering "where is this short_code?" 4,000 times per second, and the notebook is what lets them do it in 1 ms 90% of the time.

It is worth spelling it out:

Enlace's read path is cache-aside: look at the cache first (hit → 1 ms), go to the database only if it misses (miss → 50 ms) and populate the cache back. The cache stores only the hot working set (~333 MB, not the 6 TB), thanks to the 80/20 rule. With hit ratio 0.90, the average latency is 5.90 ms and the database gets only 386 reads/s instead of 4,000.

The path of resolve, end to end

Let us go through the read over the distributed architecture, with the cache-aside drawn:

sequenceDiagram
    participant U as Browser
    participant LB as Load balancer
    participant S as Server (stateless)
    participant CA as Cache (Redis)
    participant R as Shard router
    participant DB as Shard replica
    U->>LB: GET /aX9kR2q
    LB->>S: route to a free server
    S->>CA: get("aX9kR2q")
    alt HIT (90%)
        CA-->>S: long_url  (~1 ms)
    else MISS (10%)
        CA-->>S: (empty)
        S->>R: where does "aX9kR2q" live?
        R-->>S: shard k
        S->>DB: SELECT long_url WHERE short_code='aX9kR2q'
        DB-->>S: long_url  (~50 ms, PK index)
        S->>CA: set("aX9kR2q", long_url)  (populate)
    end
    S-->>U: 302 Location: https://...long-url

Read it step by step: (1) the browser asks for the code; (2) the load balancer routes it to any stateless server; (3) the server looks up the cache first. There it forks: on the hit (90% of the time) the long_url is in the cache and comes back in ~1 ms —end of the path, the database is not even touched—. On the miss (10%), the server asks the shard router which shard the code belongs to, reads from a replica of that shard (~50 ms, primary-key index lookup), and populates the cache with the result before responding (so that the next one to ask for that code gets a hit). In both cases, the server responds with a 302 redirect. That hit/miss fork is the cache-aside, and it is all of Enlace's read path.

Notice the cost asymmetry: the hit touches only the cache (1 ms); the miss touches cache + router + replica (50 ms) and on top writes back to the cache. The miss is ~50× more expensive than the hit. That is why the whole design pushes to maximize the hits: every read you resolve in the notebook is one that does not pay the slow call to the directory.

The latency, executed

How long does an Enlace user wait, on average? It depends on how many reads hit (fast) and how many miss (slow), weighted by the hit ratio. It is the central formula of module 4, and we run it for Enlace's design:

# read_latency.py — the average latency of Enlace's read path.
L_cache = 1.0     # ms — a hit (RAM)
L_db = 50.0       # ms — a miss (database replica)
qps_read = 3858   # reads/s (capacity table)

def avg_latency(h):
    return h * L_cache + (1 - h) * L_db

print(f"{'hit ratio':>10} | {'latency':>9} | {'vs 50ms':>7} | {'to DB/s':>10}")
print("-" * 48)
for h in (0.0, 0.5, 0.8, 0.9, 0.95):
    L = avg_latency(h)
    to_db = (1 - h) * qps_read
    print(f"{h:>10.2f} | {L:>7.2f}ms | {L_db/L:>5.1f}x | {to_db:>10,.0f}")

What to expect. Running python read_latency.py with Python 3.14.0:

 hit ratio |   latency | vs 50ms |    to DB/s
------------------------------------------------
      0.00 |   50.00ms |   1.0x |      3,858
      0.50 |   25.50ms |   2.0x |      1,929
      0.80 |   10.80ms |   4.6x |        772
      0.90 |    5.90ms |   8.5x |        386
      0.95 |    3.45ms |  14.5x |        193

There is the row that governs the sizing of Enlace's cache: hit ratio 0.90 → 5.90 ms, 8.5× faster than without a cache, and with only 386 reads/s reaching the database instead of 3,858. The by-hand calculation confirms: 0.90 × 1 + 0.10 × 50 = 0.9 + 5 = 5.9. If your calculation does not give 5.9, something moved.

And the counterintuitive lesson you have to understand to choose the target hit ratio: the latency does not drop proportionally to the hit ratio, but at an accelerating rate. Notice the jump from 0.90 to 0.95: the hit ratio rises five points, but the latency plummets from 5.90 to 3.45 ms —almost in half—, and the load to the database too (from 386 to 193/s). Why? Because what rules is the fraction of misses (1 − h), and going from 0.90 to 0.95 reduces it from 0.10 to 0.05 —in half—; since each miss costs 50×, eliminating half the misses lowers the latency almost in half. The last points of hit ratio are the most valuable (and the most expensive to get, because they require caching the coldest tail). For Enlace, 0.90 is the sweet spot: single-digit latency with minimum memory; you can aim for 0.95 if the database suffers with 386/s, in exchange for a bit more RAM.

The working set: why the cache is dirt cheap

The previous formula leaves a question: for a hit ratio of 0.90, how much RAM is needed? Do you have to cache the 6,000 million records (6 TB)? The answer —and the reason the cache is the first tool of the read-heavy— is no: thanks to the 80/20 rule, it is enough to cache the working set, the hot data in a window, which is tiny. Let us compute it:

# working_set.py — how much RAM Enlace's cache needs.
new_links_per_day = 100_000_000 / 30      # new links/day
hot_fraction = 0.20                        # 80/20 rule: the hottest 20%
entry_bytes = 500                          # short_code + long_url + overhead

working_set = new_links_per_day * hot_fraction
mem = working_set * entry_bytes

print(f"new links/day    = {new_links_per_day:,.0f}")
print(f"working set (20%)= {working_set:,.0f} entries")
print(f"memory @500B     = {mem/1e6:,.0f} MB ({mem/1e9:.2f} GB)")
print(f"vs database      = {mem / 6e12 * 100:.4f}% of the 6 TB")

What to expect. With python working_set.py:

new links/day    = 3,333,333
working set (20%)= 666,667 entries
memory @500B     = 333 MB (0.33 GB)
vs database      = 0.0056% of the 6 TB

Enlace's hot working set for a day is ~666,667 entries ≈ 333 MB —about 0.005% of the total 6 TB—. With less than a hundredth of a percent of the storage, in RAM, you resolve 90% of the traffic. A Redis of 1 or 2 GB —cheap, common— is more than enough. That is the magic of the 80/20 rule made a number: you do not cache everything, you cache the hot little piece (the recent and viral links), and with that you catch the vast majority of the reads. That is why the cache goes first on the read path: it is the cheapest and fastest layer.

The cache sizing sheet

Putting the latency and the working set together, this is the artifact that goes into the deliverable —the defensible sizing of Enlace's cache—:

MetricValueJustification
Target hit ratio0.90Sweet spot: 5.90 ms with minimum memory; raising to 0.95 costs RAM and buys 3.45 ms
Working set666,667 entries20% of the ~3.33 M new links/day (80/20 rule)
Memory~333 MB666,667 × 500 B; ~0.005% of the 6 TB
RAM to provision~1 GB~2× the working set: margin for peaks and Redis overhead
Average latency5.90 ms0.9·1 + 0.1·50; 8.5× faster than without a cache
Residual reads to the DB386/s (average), ~1,157/s (peak)(1 − 0.9) × 4,000; what the data scaling must absorb
Eviction policyallkeys-lruPure cache; LRU exploits the locality of recent/viral links
TTLLong (24 h) + jitterAlmost immutable data (the mapping does not change); jitter against the stampede

Two decisions deserve a note. The allkeys-lru eviction: Enlace's cache is a pure cache (all its content is reconstructed from the database), so LRU can evict any key without losing anything, and evicting "the least recently used" keeps the notebook full of the hot ones. The long TTL with jitter: since the short_code → long_url mapping is almost immutable (a link does not change destination), a long TTL almost never serves old data and maximizes the hit ratio; the jitter (24 h ± 10%) prevents many entries from expiring at once and causing an avalanche of misses (cache stampede). The only explicit invalidation is when deleting a link (expiration): database first, then delete from the cache.

The plan for the bad day

A good sizing anticipates the day the cache fails. The scenario: Redis restarts or a deploy empties it, and the hit ratio drops to 0 for a few seconds. Then:

  • The average latency jumps from 5.90 ms to 50 ms (everything is a miss).
  • The database goes from 386 reads/s to 3,858 reads/s at once —ten times its usual load—, and if it does not hold that peak, it saturates and the real latency exceeds 50 ms.

The mitigations that go into the deliverable: warm up the cache before sending it traffic after a restart; jitter in the TTLs so that the expirations do not synchronize; and —the bridge to lesson 7— size the database (the replicas) to survive the peak, not just the steady state. The cache protects the database 99% of the time; the 1% it fails, the database has to hold alone. Replicas and sharding are the net below the net, and that is exactly the topic of the next lesson.

Common mistakes

Sizing the cache by the total database size, not by the working set. What happens: someone sees Enlace's 6 TB and asks for a gigantic Redis (or discards the cache because "it all does not fit"). The design comes out super expensive or nonexistent. Why it happens: "all the data" is confused with "the hot data". How to spot it: if your proposed memory approaches the 6 TB, you did not use the 80/20 rule. How to fix it: size by the working set (~333 MB), not by the total. The cache catches the concentrated traffic —the recent and viral links—, which is a tiny fraction (0.005%) of the data. More RAM than the working set needs barely raises the hit ratio (the cold tail contributes little), so you would pay for memory that does not buy hits.

Promising a latency without saying at what hit ratio. What happens: someone writes "the cache gives 5.9 ms" without clarifying that it assumes a sustained hit ratio of 0.90. When the real hit ratio turns out to be 0.80 (because the working set was larger than estimated, or the cache too small), the real latency is 10.8 ms and the promise breaks. Why it happens: the pretty number is reported without its condition. How to spot it: if your promised latency does not come tied to a concrete hit ratio and RAM, it is a loose figure. How to fix it: every latency promise carries its hit ratio and its memory: "5.9 ms at hit ratio 0.90, with ~1 GB of RAM". And verify the real hit ratio in production, not just the estimated one —the formula describes the steady state; the engineering work is keeping the system within it—.

Forgetting the residual load the cache lets through. What happens: someone celebrates that the cache absorbs 90% and considers the read path solved, without noticing that the remaining 10% —386 reads/s on average, ~1,157/s at the peak— keeps falling on the database, plus the writes, and that this is a problem the cache does not solve. Why it happens: the 90% relief feels like "solved". How to spot it: if your design does not say how many reads/s reach the database after the cache, you left a loose end. How to fix it: always compute the residual load (1 − h) × qps_read and treat it as the input of the data scaling (lesson 7). The cache is the first line; the 386/s residual (and the peak) are exactly what the replicas have to absorb. A complete read design says what the cache catches and what it lets through.

Exercises

Exercise 1 — Compute the latency and the residual load at three hit ratios. With L_cache = 1 ms, L_db = 50 ms and qps_read = 4,000, compute the average latency and the reads/s that reach the database for (a) h = 0.80, (b) h = 0.90, (c) h = 0.95. Which would you choose for Enlace and why?

See solution
  • (a) h = 0.80: L = 0.80×1 + 0.20×50 = 0.80 + 10 = 10.80 ms. To the DB: 0.20 × 4,000 = 800/s.
  • (b) h = 0.90: L = 0.90×1 + 0.10×50 = 0.90 + 5 = 5.90 ms. To the DB: 0.10 × 4,000 = 400/s (≈386 with the exact qps of 3,858).
  • (c) h = 0.95: L = 0.95×1 + 0.05×50 = 0.95 + 2.50 = 3.45 ms. To the DB: 0.05 × 4,000 = 200/s (≈193 exact).

Choice for Enlace: 0.90 as the base target. It gives 5.90 ms —well below the <100 ms requirement— with the minimum memory (~333 MB). Raising to 0.95 lowers the latency to 3.45 ms and the load to the DB by half (200/s), but it costs more RAM (you have to cache part of the coldest tail). It is justified if the database suffers with 400/s or if you want more margin for the peak; if 5.90 ms and 400/s already suffice, 0.90 saves memory. Both are defensible with numbers —that is the key: the choice is not "the highest wins", it is "the point where the benefit stops compensating the RAM cost"—.

Exercise 2 — The day of the mass miss. Enlace runs stable with hit ratio 0.90 (the database sees 386 reads/s). A deploy empties the cache and the hit ratio drops to 0 for 30 seconds. (a) How many reads/s does the database receive in those 30 seconds? (b) What happens to the average latency? (c) Name two mitigations and say which connects with lesson 7.

See solution
  • (a) With hit ratio 0, all the reads go to the database: (1 − 0) × 3,858 = 3,858 reads/s. The database goes from 386/s to 3,858/s at once —ten times its usual load—.
  • (b) The average latency jumps to L(0) = 0×1 + 1×50 = 50 ms, the worst case. And if the database saturates with 3,858/s, those 50 ms can balloon to hundreds (the queue at the well), so the real latency is worse than 50 ms while the avalanche lasts.
  • (c) Two mitigations: (i) warm up the cache before sending it traffic after the deploy (populate it in a staggered way with the known hottest links), so as not to expose the database to the cold avalanche; and (ii) size the replicas to survive the peak of 3,858/s, not just the steady state of 386/s —this is the one that connects with lesson 7: the cache protects the database 99% of the time, but the 1% it fails, the replicas have to hold the avalanche alone—. (A third, the jitter in the TTLs, prevents periodic mini-stampedes from synchronized expirations.)

The lesson: the formula describes the steady state (stable hit ratio); the robust design anticipates when that regime breaks. The cache does not only speed up —it also protects the database—, and when it disappears at once, that protection vanishes exactly when it is most needed. That is why the data scaling (lesson 7) is not sized for 386/s, but with margin for the peak.

Exercise 3 — Why not cache the 6 TB? A colleague proposes: "Let's put Enlace's whole database (6 TB) in RAM to have a 100% hit ratio and never go to disk". Give three reasons why it is a bad idea, with numbers from this lesson.

See solution

Three reasons:

  1. Disproportionate cost. 6 TB of RAM cost orders of magnitude more than the ~333 MB the working set needs for hit ratio 0.90. You would be paying to put the very long tail of cold links that almost nobody asks for into expensive RAM —99.99% of the cache wasted on data that generates no traffic—.

  2. Diminishing returns. Going from hit ratio 0.90 (with ~333 MB) to 1.00 (with 6 TB) lowers the average latency from 5.90 ms to 1.00 ms —an improvement of ~5 ms— in exchange for multiplying the RAM by ~18,000. The last points of hit ratio, which require caching the cold tail, are super expensive and buy very little. It is not worth it: 5.90 ms already meets the <100 ms requirement with plenty to spare.

  3. The cache stops being a cache. A cache is a partial and disposable copy of the hot data; if you put all the data, it is no longer a cache, it is a second database in RAM —volatile, expensive and without persistence—. You lose the advantage of the cache being small and cheap, which is exactly what the 80/20 rule makes possible.

The moral: 100% hit ratio is not the goal —it is an expensive mirage—. The goal is a high hit ratio (0.90–0.95) at the lowest RAM cost, and the 80/20 rule says that is achieved by caching only the working set (~333 MB). Chasing 100% is fighting against the long tail, which gives less and less per TB. A good cache design accepts that there will always be some misses and is content to catch the cheap bulk of the traffic —and lets the replicas (lesson 7) absorb that rest—.

Summary and next step

In this lesson you deep-dived into Enlace's read path, where all the design effort concentrates because it is where the ~4,000 reads/s pinch. You went through resolve with the cache-aside pattern —look at the cache first (hit → 1 ms), go to the replica only if it misses (miss → 50 ms) and populate back—, like the receptionist with their notebook of regulars. You executed the latency formula: hit ratio 0.90 → 5.90 ms (8.5× faster), with the counterintuitive lesson that the last points of hit ratio are worth more (0.90 → 0.95 cuts the latency almost in half). You computed the working set —~666,667 entries, ~333 MB, 0.005% of the 6 TB— that makes the cache dirt cheap thanks to the 80/20 rule. And you delivered the sizing sheet: hit ratio 0.90, ~1 GB of RAM, 5.90 ms, allkeys-lru, long TTL with jitter, plus the plan for the day of the mass miss.

The essential thing for what comes next: the cache absorbs 90%, but lets through 386 reads/s (and ~1,157/s at the peak) to the database, plus the ~40 writes/s. That residual load is exactly the problem of lesson 7.

Before moving on you should be able to: go through resolve with cache-aside over the distributed architecture; execute the latency formula and reproduce the 5.90 ms; compute the working set and justify why the cache is cheap; and name the residual load the cache lets through.

What comes next is the second line of defense. In lesson 7 you deep-dive into scaling out: how the replicas absorb those 386/s (and the peak), how the sharding by short_code with consistent hashing distributes the 6 TB and grows without a storm (13.1% remap, not 88.9%), how the load balancing over stateless servers scales the compute, and —the capstone's finishing touch— the consistency tradeoff (eventual for resolve), justified with the numbers and with CAP/PACELC. It is where the distributed design closes.

Resources

← Previous5. Step 4a — The write pathNext →7. Step 4c — Scaling out: replicas, sharding, load balancing and consistency