4. Replication lag and reading from replicas

Description

By the end of this lesson you'll understand the price paid for spreading the reads among replicas: the replication lag, the small delay with which each replica follows the primary. You'll see why it exists (asynchronous replication confirms the write before the replicas receive it), what concrete problem it causes —the classic "read your own write", where just-created data appears not to exist because you requested it from a replica that doesn't have it yet—, and how that problem looks in Enlace: a short_code you just generated that, resolved an instant later from a behind replica, returns a fleeting 404. You'll learn why for the vast majority of Enlace's reads the lag doesn't matter —the system is, by nature, tolerant of eventual consistency—, in which few cases it does, and the basic mitigations for those cases.

This matters because the lag is replication's silent trap: you set up primary/replica, the reads are spread, everything looks perfect in the tests... and in production, every so often, a user reports that "the link I just created doesn't work" and then, magically, it does. That intermittent and hard-to-reproduce bug is almost always replication lag, and whoever doesn't know the concept loses days looking for it in the wrong place. Understanding the lag lets you do two things: pay the price with your eyes open —accept it where it doesn't hurt— and mitigate it precisely where it does, without over-designing. It's the difference between a replica that helps you and one that gives you phantom bugs.

Connection to the module: this lesson is the honest asterisk of lesson 3. There I presented replication in its ideal form —one chef, many cooks, all with the same menu—; here I admit that the menu takes a moment to reach the last cook's station. It's also the guide's first contact with eventual consistency, a topic module 7 formalizes (CAP, PACELC, strong vs. eventual consistency as a design decision). Here you touch it with Enlace's concrete case and a practical mitigation; there you turn it into a general framework. The boundary: the consistency guarantees in depth —read-your-writes, monotonic reads, consistent prefix as formal models— are from module 7; here it's enough to recognize the problem and apply the simplest mitigation.

The letter that takes a while to reach the branches

Think of it this way. A store chain has a central office that decides the prices and several branches that serve the public. When the central lowers the coffee price, it doesn't appear by magic at all the registers instantly: a notice is sent to each branch, and that notice takes a while —a few seconds, sometimes more if the connection is slow—. During that little while, a customer who asks the price at the farthest branch will hear the old price, while at the central the new one already applies. It's not that the branch is broken: it's that the notice hasn't arrived yet. When it arrives, the branch catches up and everyone sees the same price.

The central is the primary; the branches are the replicas; the time the notice takes to arrive is the replication lag. The concrete inconvenience appears in a special case: imagine you yourself requested the price change at the central and, zero point two seconds later, you cross the street to the branch to check it. You still see the old price. You know it changed —you requested it an instant ago— but the branch you're checking hasn't found out yet. That feeling of "I just made a change and I don't see it reflected" is the problem of reading your own write (read-your-own-writes), and it's the most common and most confusing face of the lag.

In Enlace it translates directly: the central registers your new short_code (shorten), but if an instant later someone —or you— resolves it (resolve) against a replica that hasn't received the notice yet, the replica responds "that short_code doesn't exist" and the user sees a 404. Half a second later, the notice arrives, the replica catches up, and the same link works. The link was never broken; you simply checked it at the wrong branch an instant too soon.

Why the lag exists and how long it lasts

The lag is the direct consequence of the asynchronous replication we saw in lesson 3. The primary confirms the write to the client without waiting for the replicas —that's why it's fast—, and then it transmits the change through the replication log. Between "the primary confirmed" and "the replica applied the change" there's a window in which the replica is behind. That window is the lag.

How long does it last? On a healthy network and with up-to-date replicas, the typical lag is milliseconds to a few seconds. But it's not constant: it grows when the network between primary and replica gets congested, when the replica is busy with a heavy query, or when a flood of writes arrives that the replica applies slower than they come. In the worst case —a replica that disconnected and is catching up— the lag can be minutes. The design key: the lag is variable and unpredictable in the worst case, so a correct system doesn't assume "the lag is always small"; it assumes there may be lag and decides, for each read, whether it tolerates it.

Worked example: the stale-read window

Let's put numbers on the problem's window. Let's simulate the timeline of a write followed by a read, with a given lag, to see exactly when a read from a replica returns the old data and when the new:

# Timeline: shorten on the primary, then resolve on a replica
# t is measured in milliseconds from the shorten

replication_lag_ms = 800     # the replica applies the change 800 ms later

def replica_sees_short_code(read_time_ms):
    """The replica knows the short_code only after applying the change."""
    return read_time_ms >= replication_lag_ms

for read_at in (100, 500, 799, 800, 1500):
    seen = replica_sees_short_code(read_at)
    result = "200 OK (redirects)" if seen else "404 (doesn't have it yet)"
    print(f"resolve at t={read_at:>4} ms  ->  {result}")

print(f"\nstale-read window: 0 .. {replication_lag_ms} ms after the shorten")

What to expect. When you run it:

resolve at t= 100 ms  ->  404 (doesn't have it yet)
resolve at t= 500 ms  ->  404 (doesn't have it yet)
resolve at t= 799 ms  ->  404 (doesn't have it yet)
resolve at t= 800 ms  ->  200 OK (redirects)
resolve at t=1500 ms  ->  200 OK (redirects)

Here's the problem, quantified. There's a stale-read window that goes from the instant of the shorten (t=0) until the replica applies the change (t=800 ms in this example). Any resolve of that same short_code that falls in that window, and that hits a behind replica, returns 404. Outside the window, everything works. And here's the timeline drawn:

   shorten                    the replica applies the change
   (primary)                       (t = lag)
      │                                │
  ────●────────────────────────────────●─────────────────►  time
      │◄──── stale window ────────────►│      all OK
      │   resolve here -> fleeting 404 │   resolve here -> 200
      t=0                          t=800 ms

The design question isn't "how do I eliminate this window?" —eliminating it entirely costs dearly (synchronous replication, or always reading from the primary, which gives up scaling)—. The question is "does Enlace care that this window exists?". And the answer, for almost all of Enlace's reads, is no.

Why Enlace almost doesn't care (and when it does)

Enlace is, by its nature, a system tolerant of eventual consistency, and it's worth understanding why, because not all systems are.

Why it almost doesn't matter. The bulk of Enlace's ~4,000 reads/s are resolve of links that have existed for a while —minutes, hours, months—. For all of those, the lag is completely irrelevant: the short_code has been replicated everywhere for an eternity, and any replica has it. The stale window only affects a link in the first seconds of its life, and only if someone resolves it at that exact instant against a replica that hasn't received it yet. It's a tiny fraction of the traffic. And even when it happens, the "damage" is a 404 that cures itself in under a second: annoying, but not data corruption or loss of money. A shortener can live with that.

When it does matter. There's a concrete case: the user who just created the link and immediately tests it. It's the read-your-own-writes of the analogy —you changed the price and cross over to check it—. If a user does shorten, sees their enla.ce/aX9kR2q, clicks to verify it works, and gets a 404, the experience is bad: they know the link should exist because they just created it. That case —and only that— deserves a mitigation.

The simplest and most direct mitigation: for the immediate read that follows a write by the same user, read from the primary, not from a replica. The primary always has the most recent data (it's where it was written), so there's no stale window. In Enlace: when shorten's response shows the user their new link, and if the interface resolves it to preview it, that resolution goes to the primary. All the other resolutions —everyone else's, on links they didn't just create— go to the replicas. That way you reserve the primary for the handful of reads that really demand freshness and leave the 99.9% on the replicas.

Other mitigations exist —waiting for the replica to confirm the short_code, or "monotonic reads" so a user doesn't see the data come and go— but they're consistency models that module 7 treats as a framework. For this module, the practical rule is enough: read from the replica by default; read from the primary only when you need your own just-made write.

Common mistakes

Assuming the lag is always small (assumption mistake). What happens: you design taking for granted that the replica is at most a few milliseconds behind, and the system breaks the day a replica falls seconds or minutes behind due to congestion or catching up. Why it happens: in tests and normal operation the lag is small, and it's easy to confuse "almost always small" with "always small". How to detect it: ask yourself "what happens if this replica is 30 seconds behind?". If your design assumes that can't happen, you have a time bomb. How to fix it: treat the lag as variable and unpredictable in the worst case; for any read that doesn't tolerate old data, don't trust the replica to be up to date —read it from the primary or check the lag—.

Reading everything from the primary to "avoid the lag" (over-correction mistake). What happens: someone gets scared of the lag and routes all the reads to the primary, which completely cancels the benefit of having replicas —the primary saturates and the replicas sit idle—. Why it happens: the lag is scary and reading from the primary eliminates it, so it seems the "safe" option. How to detect it: if your replicas are almost idle while the primary suffers, you over-corrected. How to fix it: only a handful of reads need the freshest data (the user's own just-made write); those go to the primary, the rest to the replicas. Applying the freshness rule selectively is what makes replication useful.

Confusing the lag 404 with a data bug (diagnostic mistake). What happens: an intermittent report appears of "just-created link gives 404 and then works", and the team looks for a bug in the short_code generation or in the database, when the short_code is perfectly stored —only in the primary and not yet in the replica that responded—. Why it happens: the symptom (a 404 that cures itself) doesn't scream "replication", and the concept is invisible if you don't know it. How to detect it: if the bug only happens in the first seconds of a link's life and disappears on its own, and if reading from the primary makes it disappear, it's lag, not corruption. How to fix it: knowing the concept is half the cure; the other half is the mitigation of reading the own recent write from the primary.

Exercises

Exercise 1 — Does it tolerate the lag? For each Enlace read, say whether it tolerates the replication lag (can go to a replica) or demands the freshest data (must go to the primary). (a) A visitor clicks enla.ce/aX9kR2q, a link created three months ago. (b) A user just did shorten and the interface previews the link by resolving it immediately. (c) The admin panel shows the total of links created today. (d) A third-party bot resolves millions of varied links per hour.

Exercise 2 — Draw the window. A link is created on the primary at t=0. Replica A has a lag of 300 ms; replica B, which was congested, a lag of 4,000 ms. A resolve of that link arrives at t=1,000 ms. What does it return if A serves it? And if B serves it? What does this tell you about trusting "the lag is small"?

Exercise 3 — Design the routing. Write, in pseudocode or prose, Enlace's read-routing rule that resolves the read-your-own-writes problem without sacrificing scaling. Indicate what fraction of the traffic goes to each place and why.

def route_read(short_code, session):
    if short_code in session.recently_created:   # this user created it seconds ago
        return primary
    return pick_replica()                        # everything else

Summary and next step

In this lesson you added the honest asterisk to replication: the replicas are behind the primary, and that delay is the replication lag. With the store chain you saw that a change made at the central takes a while to reach the branches, and that the most confusing case is reading your own write —you changed the price and cross over to check it before the notice arrives—. You quantified the stale-read window: from the shorten (t=0) until the replica applies the change (t=lag), any resolve that falls there returns a fleeting 404. And you saw why Enlace almost doesn't care —the bulk of the reads are of old links, and the lag is milliseconds— except in one case, read-your-own-writes, mitigated by reading only that read from the primary.

Before moving on you should be able to: explain why the lag exists from asynchronous replication; draw the stale-read window; decide for a given read whether it tolerates the lag or demands the primary; and write the routing rule that resolves read-your-own-writes without giving up scaling.

With this you close the reads axis. What comes next is crossing to the other axis —the data and writes one—, the one replicas can't scale. In lesson 5 you'll see sharding: when neither the 6 TB nor the writes fit on one machine, the data is split among several shards. You'll learn what the shard key is, why in Enlace it's the short_code, and how a bad key concentrates the load on a single node (a hotspot) while a good one spreads it evenly. It's the start of the module's hardest —and most interesting— part.

Resources

• Designing Data-Intensive Applications, Martin Kleppmann — Chapter 5, "Problems with Replication Lag" — the reference section on the lag and its guarantees (read-your-writes, monotonic reads, consistent prefix); read it as a bridge toward module 7, where these models are formalized.
• PostgreSQL — Hot Standby and lag monitoring (pg_stat_replication) — how a real engine exposes and measures the replication lag; seeing the concrete metric anchors this lesson's abstract concept.
• System Design Primer — Consistency patterns (weak, eventual, strong) — a brief summary of the consistency patterns that frame why Enlace tolerates the lag; a preview of module 7.