Module 4: Cache — The Read-Heavy Path

5. Hit ratio and average latency

Next →

Description

We reach the numeric heart of the module. Up to now we talked about the hit ratio as "the fraction of reads the cache did have", and about latency as "1 ms if it hits, 50 ms if it misses". This lesson brings the two together into a single formula, runs it with real numbers, and draws from it the conclusion that governs all caching design: the average latency the user feels is a weighted average between the fast hit and the slow miss, and the weight is set by the hit ratio. The formula is the one you've been seeing since lesson 1:

L = h · L_cache + (1 − h) · L_db

A fraction h of the reads are hits (they cost L_cache), and the remaining fraction (1 − h) are misses (they cost L_db). The average of the two, weighted by how many are of each type, is the average latency L. You'll run this formula in Python for hit ratios of 0.5, 0.8, 0.9, and 0.95, with L_cache = 1 ms and L_db = 50 ms, and you'll get a real table. The module's anchor result —5.90 ms at a hit ratio of 0.9— comes from that table, and with it the most counterintuitive lesson of caching: raising the hit ratio doesn't lower the latency proportionally, but in an accelerated way. The last points of hit ratio are worth gold.

Connection to the module: lessons 3 and 4 built the cache (the pattern and the eviction); this one measures its effect. It takes the latency gap from lesson 2 (1 ms vs 50 ms) and turns it into the metric that really matters: how long the user waits. Lesson 6 will tell you how much RAM is needed to reach the hit ratio this lesson proves you want. And the project (lesson 8) will use this formula as its central calculator: "I choose hit ratio 0.9 → I get 5.9 ms". Without this lesson, the hit ratio is an abstract number; with it, it's a lever with measurable consequences.

The average of the fast line and the slow line

Think of it this way. You arrive at a bank with two lines. The fast line is an ATM: you go in, you go out, thirty seconds. The slow line is the window with a human teller: paperwork, questions, five minutes. Not all customers use the same line: it depends on what they came to do. If you're asked "how long does a customer of this bank take on average?", the answer isn't "thirty seconds" or "five minutes" —it's a weighted average that depends on what fraction of customers goes to each line.

If 90% of the customers only withdraw cash (fast line, 30 s) and 10% do transactions (slow line, 5 min), the average time is 0.9 × 30 s + 0.1 × 300 s = 27 s + 30 s = 57 seconds. Notice something curious: the average (57 s) is much closer to the fast line (30 s) than to the slow one (300 s), because most go through the fast one. But notice also the other side: that 10% who go to the slow line contribute 30 of the 57 seconds —more than half the average— even though they're a minority. The few slow customers weigh a lot in the average.

That's exactly the average latency of a cache, and the analogy is literal. The fast line is a cache hit (L_cache, 1 ms); the slow line is a miss that goes to the database (L_db, 50 ms). The hit ratio (h) is the fraction of customers who go through the fast line. And the average latency L is the average time any given customer waits. The formula L = h·L_cache + (1−h)·L_db is, word for word, "the fast line's time times how many use it, plus the slow line's time times how many use it". And the bank's lesson —that the few slow ones weigh a lot— is what makes gaining hit ratio matter so much: each customer you move from the slow line to the fast one saves you the full difference, 49 ms.

The average latency is a weighted average between the fast hit and the slow miss, with the hit ratio as the weight. Since the miss is ~50× more expensive than the hit, the few misses dominate the average —and that's why reducing the fraction of misses (raising the hit ratio) is the most powerful lever you have.

Running the formula

Enough theory: let's run the formula. We're going to compute L for a table of hit ratios, with L_cache = 1 ms and L_db = 50 ms, and we're going to break down the two parts of the average so you see where each millisecond comes from:

# avg_latency.py — the module's central formula, run
L_cache = 1.0     # ms — a hit (RAM)
L_db = 50.0       # ms — a miss (database)

def avg_latency(h):
    return h * L_cache + (1 - h) * L_db

print(f"{'h':>5} | {'h*L_cache':>10} | {'(1-h)*L_db':>11} | {'L (ms)':>8} | {'vs 50ms':>8}")
print("-" * 55)
for h in (0.0, 0.5, 0.8, 0.9, 0.95, 0.99):
    part_cache = h * L_cache
    part_db = (1 - h) * L_db
    L = avg_latency(h)
    speedup = L_db / L
    print(f"{h:>5.2f} | {part_cache:>10.2f} | {part_db:>11.2f} | {L:>8.2f} | {speedup:>6.1f}x")

Each row computes the two halves of the average separately —h·L_cache (what the hits contribute) and (1−h)·L_db (what the misses contribute)— and sums them in L. The last column, L_db / L, says how many times faster the system is with a cache compared to having no cache (where every read would cost the 50 ms of L_db).

What to expect. With python avg_latency.py:

    h |  h*L_cache |  (1-h)*L_db |   L (ms) |  vs 50ms
-------------------------------------------------------
 0.00 |       0.00 |       50.00 |    50.00 |    1.0x
 0.50 |       0.50 |       25.00 |    25.50 |    2.0x
 0.80 |       0.80 |       10.00 |    10.80 |    4.6x
 0.90 |       0.90 |        5.00 |     5.90 |    8.5x
 0.95 |       0.95 |        2.50 |     3.45 |   14.5x
 0.99 |       0.99 |        0.50 |     1.49 |   33.6x

There's the table that anchors the module. Let's read it slowly, row by row, because each one teaches something:

  • h = 0.00 (no cache): 50.00 ms. Every read goes to the database. It's the starting point, the world of lesson 2.
  • h = 0.50: 25.50 ms. Half the reads hit, and the latency drops by half (plus a hair). A 50% hit ratio already doubles the speed. But 50% is a bad hit ratio for a cache —it means half the reads still go to the database—.
  • h = 0.80: 10.80 ms. With 80% hits, the latency falls to 10.8 ms: almost 5× faster than without a cache.
  • h = 0.90: 5.90 ms. The module's anchor number. With 90% hit ratio, the average latency is 5.90 ms, 8.5× faster than the initial 50 ms. 0.9 × 1 + 0.1 × 50 = 0.9 + 5 = 5.9. If your calculation doesn't give 5.9, something moved.
  • h = 0.95: 3.45 ms. Here's the surprise: going from 0.90 to 0.95 —only five points of hit ratio— lowers the latency from 5.90 to 3.45 ms, almost by half.
  • h = 0.99: 1.49 ms. With 99% hits, you almost touch the floor of L_cache (1 ms).

The counterintuitive lesson: the hit ratio isn't linear

Pause on the jump from 0.90 to 0.95. The hit ratio rose five points (from 90% to 95%), but the latency didn't drop "a little": it plummeted from 5.90 to 3.45, almost by half. How can that be, if I only improved five points? The answer is in the formula, and it's the most important idea of the lesson.

The average latency is dominated by the misses term: (1 − h) · L_db. The hits term (h · L_cache) is almost negligible —it never exceeds 1 ms, because L_cache is 1—. So what moves the latency is how many misses remain, and the misses are (1 − h). Look at what happens to (1 − h) when you raise the hit ratio:

  • From h = 0.90 to h = 0.95: the misses go from 0.10 to 0.05. They're cut in half.
  • From h = 0.95 to h = 0.99: the misses go from 0.05 to 0.01. They're cut to a fifth.

There's the trick: what matters isn't how much the hit ratio rises, but how much the fraction of misses drops, and that fraction drops in an accelerated way near 100%. Going from 90% to 95% hit ratio looks like a modest advance (five points), but it actually eliminates half the remaining misses, and since each miss costs 50×, eliminating half the misses lowers the latency almost by half. That's why in the real world people fight tooth and nail for the last points of hit ratio: the difference between 90% and 99% isn't "9% better", it's the difference between 5.9 ms and 1.49 ms —the system four times faster—. The last points of hit ratio are the most expensive to get and the most valuable.

Seeing it drawn helps the shape stick. Each bar is the average latency L at that hit ratio (one █ ≈ 2 ms), taken from the table you just ran:

 h=0.00  █████████████████████████   50.00 ms
 h=0.50  █████████████               25.50 ms
 h=0.80  █████                       10.80 ms
 h=0.90  ███                          5.90 ms   <- the elbow of the curve
 h=0.95  ██                           3.45 ms
 h=0.99  █                            1.49 ms

Notice the shape: the latency plummets at the start (from 0.00 to 0.80 it falls from 50 to 10.8 ms) and then it lies down —from 0.90 to 0.99 it barely drops from 5.9 to 1.5 ms on the bar's scale—. But don't be fooled by how "small" the final stretch looks: remember that going from 0.90 to 0.95 splits the latency almost in half (5.90 → 3.45). The "elbow" around 0.90 is where most systems settle: they've already captured almost all the benefit, and each extra point costs caching more cold tail (lesson 6). The curve is steep first and flat after, and knowing where you stand on it —before or after the elbow— is what tells you whether it's worth chasing more hit ratio or you've already reached the land of diminishing returns.

The other effect: the load you take off the database

Latency is what the user feels, but the hit ratio has a second effect just as important: how much load reaches the database. You already touched it in lessons 1 and 2; let's formalize it with the same table, because it's the other half of the cache's value.

# db_relief.py — how many reads/s reach the DB according to the hit ratio
qps_read = 3858

print(f"{'h':>5} | {'to the DB/s':>12} | {'to the cache/s':>15}")
print("-" * 38)
for h in (0.0, 0.5, 0.8, 0.9, 0.95):
    to_db = (1 - h) * qps_read
    to_cache = h * qps_read
    print(f"{h:>5.2f} | {to_db:>12,.0f} | {to_cache:>15,.0f}")

What to expect. With python db_relief.py:

    h |  to the DB/s |  to the cache/s
--------------------------------------
 0.00 |        3,858 |               0
 0.50 |        1,929 |           1,929
 0.80 |          772 |           3,086
 0.90 |          386 |           3,472
 0.95 |          193 |           3,665

The same non-linearity, seen from the database. At hit ratio 0.90, the database receives 386 reads/s instead of 3,858 —the cache took 90% of the load off it—. And rising to 0.95 halves that load again: 193/s. So the last points of hit ratio not only lower the user's latency, they also double the database's relief. It's the same (1 − h) ruling both metrics: the fraction of misses is at once what weighs in the average latency and what hits the database. Lowering it is the lever, and from here on the whole module pushes in that direction.

A warning: the average hides the tail

The formula gives the average latency, and the average is useful but tells only part of the story. Think of the 10% of misses at hit ratio 0.90: those individual reads still cost 50 ms each, even though the average is 5.9. For the user who falls into a miss, their experience is 50 ms, not 5.9. That's why, besides the average, serious systems look at the tail latency —the high percentiles, the p99, "the slowest 1% of requests"—, because a nice average can hide a minority of suffering users. The cache improves the average spectacularly, but the tail is still marked by L_db: as long as there are misses, there will be slow requests.

There's a related danger worth naming even though we'll develop it with TTL in lesson 7: the mass miss. If for some reason many hot entries disappear from the cache at once —a Redis restart, or a bunch of TTLs expiring in the same instant—, suddenly an avalanche of reads that were hits become misses and fall all together on the database. The hit ratio plummets for a few seconds, the average latency shoots toward L_db, and the database —which was comfortable with 386/s— suddenly receives thousands/s and can saturate. This phenomenon has names (cache stampede, thundering herd) and mitigations, and we touch it in lesson 7. For now, keep in mind that the formula describes the steady state: it assumes a stable hit ratio. When the hit ratio collapses suddenly, the latency goes toward the worst case, and that's exactly what you have to avoid.

Common mistakes

Believing that latency drops proportionally to the hit ratio. What happens: someone reasons "if I raise the hit ratio from 90% to 95%, I improve 5%, no big deal" and doesn't bother chasing those points. But the table shows that going from 90% to 95% lowers the latency from 5.90 to 3.45 ms —almost by half—, not 5%. Why it happens: the change in the hit ratio (linear) is confused with its effect on the latency (accelerated), because what rules is the fraction of misses, which is cut in half. How to detect it: if you treat all points of hit ratio as equal, you didn't understand the formula. How to fix it: always think in terms of (1 − h), the fraction of misses. Lowering the misses from 10% to 5% cuts them in half, and since each miss costs 50×, that lowers the latency almost by half. The last points of hit ratio are the most valuable, not the least.

Optimizing the average latency and ignoring the tail. What happens: someone celebrates that the average latency dropped to 5.9 ms and calls the work done, without noticing that the 10% of users who fall into a miss still wait 50 ms each. If those misses coincide with the most important users or with critical moments, the real experience is bad even though the average is good. Why it happens: the average is the easy number to report, and it hides the slow minority. How to detect it: look at the p99, not just the average. If the p99 is close to L_db, you have users suffering the tail. How to fix it: use the average for the general design, but watch the high percentiles; and raise the hit ratio to reduce the number of requests that fall into the 50 ms tail. The cache doesn't eliminate the tail, it thins it.

Using the numbers L_cache = 1 and L_db = 50 as absolute truths. What happens: someone puts these values into a calculation of their real system without measuring theirs, and their estimates come out wrong because in their case the database responds in 5 ms (everything in its memory) or the cache in 3 ms (it's far). Why it happens: it's convenient to borrow the guide's numbers. How to detect it: if you never measured L_cache and L_db in your system, your latencies are fake. How to fix it: the shape of the formula is universal —weighted average by the hit ratio, dominated by the misses—; the values are yours to measure. Use 1 ms and 50 ms to learn the shape and do napkins; measure the real ones when you design for real. What doesn't change is that the miss is much more expensive than the hit, and that's why the hit ratio rules.

Exercises

Exercise 1 — Compute three latencies by hand. With L_cache = 1 ms and L_db = 50 ms, compute the average latency L = h·L_cache + (1−h)·L_db for (a) h = 0.70, (b) h = 0.90, (c) h = 0.98. Show the two parts of the average in each case.

See solution
  • (a) h = 0.70: hits part = 0.70 × 1 = 0.70; misses part = 0.30 × 50 = 15.00; L = 0.70 + 15.00 = 15.70 ms.
  • (b) h = 0.90: hits part = 0.90 × 1 = 0.90; misses part = 0.10 × 50 = 5.00; L = 0.90 + 5.00 = 5.90 ms. (The module's anchor number.)
  • (c) h = 0.98: hits part = 0.98 × 1 = 0.98; misses part = 0.02 × 50 = 1.00; L = 0.98 + 1.00 = 1.98 ms.

Notice two things. First: in all three cases, the misses part dominates the result (15.00 of 15.70; 5.00 of 5.90; 1.00 of 1.98) —the hits contribute almost nothing, because L_cache is tiny—. Second: from (b) to (c), raising the hit ratio 8 points (from 0.90 to 0.98) lowers the latency from 5.90 to 1.98, to a third, because the misses fell from 10% to 2%, to a fifth. All the movement lives in (1 − h).

Exercise 2 — How much is each point of hit ratio worth? Using the table from the worked example, compare two improvements of the same size in points of hit ratio: (a) going from 0.50 to 0.55, and (b) going from 0.90 to 0.95. Both are "five points". Compute how many milliseconds of latency each one saves and explain why they aren't worth the same.

See solution
  • (a) From 0.50 to 0.55: L(0.50) = 0.50×1 + 0.50×50 = 25.50 ms. L(0.55) = 0.55×1 + 0.45×50 = 0.55 + 22.50 = 23.05 ms. Savings = 25.50 − 23.05 = **2.45 ms**.
  • (b) From 0.90 to 0.95: L(0.90) = 5.90 ms. L(0.95) = 0.95×1 + 0.05×50 = 0.95 + 2.50 = 3.45 ms. Savings = 5.90 − 3.45 = **2.45 ms**.

Surprise: they save the same, 2.45 ms! Five points of hit ratio in any stretch lower the misses part by 0.05 × 50 = 2.5 ms (minus the hair on the hits side). So why is it said that the last points are worth more? Because the relative savings is what changes: 2.45 ms over a base of 25.50 is a 10% improvement; the same 2.45 ms over a base of 5.90 is a 42% improvement. Near the ceiling, each point of hit ratio represents an ever-larger fraction of the remaining latency —and on top of that those last points are the hardest to get, because you have to cache the long tail of rarely requested data—. In absolute terms they save the same; in relative terms, and in difficulty, the last ones are much more expensive.

Exercise 3 — The mass miss. Enlace runs with a stable hit ratio of 0.90 (the database sees 386 reads/s, comfortable). Suddenly, Redis restarts and the cache starts empty: for a few seconds, the hit ratio is 0. (a) How many reads/s does the database receive in those seconds? (b) What happens to the average latency? (c) Why is this "mass miss" dangerous even though the steady state is healthy?

See solution
  • (a) With hit ratio 0, all the reads go to the database: (1 − 0) × 3858 = **3,858 reads/s**. The database goes from 386/s to 3,858/s suddenly —ten times its usual load—.
  • (b) The average latency shoots up to L(0) = 0×1 + 1×50 = **50 ms**, the worst case. Furthermore, if the database saturates with 3,858/s, those 50 ms can inflate to hundreds (the well's line), so the real latency is worse than 50 ms while the avalanche lasts.
  • (c) It's dangerous because the system is sized for the steady state (386/s to the database), not for the avalanche. The database that comfortably handles 386/s may not handle 3,858/s, and if it goes down under the peak, it can't respond to the misses, so the cache can't repopulate, and the system enters a vicious circle. The healthy steady state (hit ratio 0.90) hides this fragility: the cache not only accelerates, it also protects the database, and when the cache disappears suddenly, that protection vanishes exactly when it's needed most.

The moral connects with lesson 7: you have to prevent many hot entries from disappearing at once. It's achieved, among other things, by warming up the cache before sending it traffic, and by spreading the TTLs so they don't all expire together (jitter). The formula describes the stable regime; the engineering work is keeping the system within that regime.

Summary and next step

In this lesson you ran the module's central formula: L = h·L_cache + (1−h)·L_db, the average latency as a weighted average between the fast hit (1 ms) and the slow miss (50 ms), with the hit ratio as the weight —the two bank lines—. You ran the table and drew the anchor number: hit ratio 0.90 → 5.90 ms, 8.5× faster than without a cache. And you discovered the counterintuitive lesson: the latency does not drop proportionally to the hit ratio, but in an accelerated way, because what rules is the fraction of misses (1 − h), which is cut in half when going from 0.90 to 0.95 —and that's why the latency also almost halves, and the load on the database too (from 386/s to 193/s)—.

You also saw the limits of the average: the average latency hides the tail (the individual misses still cost 50 ms, you have to look at the p99), and the danger of the mass miss —when many hot entries disappear at once, the hit ratio plummets and the database receives an avalanche—, which the steady-state formula doesn't capture.

Before moving on you should be able to: write and apply L = h·L_cache + (1−h)·L_db; reproduce the 5.90 ms at hit ratio 0.9; explain why the last points of hit ratio are (relatively) worth more; and name why the average hides the tail and what a mass miss is.

What comes next is the question this lesson leaves open: if I want a hit ratio of 0.90, how much RAM do I need? Do I have to cache Enlace's 6 billion records? In lesson 6 you'll see why not —the 80/20 rule says a few hot links concentrate most of the reads— and you'll compute the real size of Enlace's working set: about 666,667 entries, ~333 MB, which fit easily in RAM.

Resources

← Previous4. Eviction policies and LRUNext →6. The working set and the 80/20 rule