3. Replication: primary and replica

Description

By the end of this lesson you'll understand the first horizontal-scaling tool and the one Enlace needs first: replication in its most common form, the primary/replica pattern. You'll see how a single machine —the primary— accepts all the writes and copies each change to other machines —the replicas— that store a complete copy of the data and dedicate themselves to serving reads. You'll understand what the replication log is (the ordered list of changes the primary transmits), how that design fits like a glove with Enlace's 100:1 —many reads, few writes—, and you'll compute how many replicas you need to spread the ~4,000 reads/s. You'll also name, without going in depth, the difference between synchronous and asynchronous replication and what failover is, to make clear the boundary with the resilience guide.

This matters because replication is, for most read-heavy systems, the scaling lever with the best benefit/complexity ratio. It's simpler than sharding —each replica is an identical copy, there's no deciding "which data lives where"—, it doesn't change the data model, and it resolves at once the limit that squeezes first in Enlace. Furthermore, replication gives you, as a bonus, the first brick of redundancy: if the primary goes down, a replica already has all the data and can take over. Understanding primary/replica well is understanding half of database scaling; the other half, sharding, comes after and rests on this.

Connection to the module: this lesson opens the reads axis that lesson 2 identified as the most urgent for Enlace. It's the direct answer to the read-throughput limit. Lesson 4 will add the indispensable asterisk —the replicas are a bit behind, and that has consequences—, so here I present the pattern in its ideal form and there the honest one. And it marks its boundary: here I mention failover (a replica that replaces the downed primary) but I don't teach it; that's territory of resilience-and-reliability-patterns-guide. Replication as a scaling tool is from here; replication as an availability tool, with all its detection and promotion machinery, is from the sibling guide.

The chef and the line cooks

Think of it this way. In a high-demand restaurant there's an executive chef who is the only one authorized to change the menu: they create the new dish, adjust the recipe, decide the risotto now has saffron. When they make a change, they don't keep it in their head: they write it in the recipe book and that change is copied to the stations of all the line cooks. The line cooks don't invent or modify dishes —they have no authority for that—; their job is to execute the menu as it is, hundreds of times a night, in parallel. Ten line cooks serve ten times more dishes than one, because they all work with the same faithful copy of the recipe book.

The executive chef is the primary: the only one who accepts changes (writes). The recipe book that's copied is the replication log: the ordered list of changes the chef transmits. The line cooks are the replicas: each one has a complete and up-to-date copy of the menu, and they all serve dishes (reads) in parallel. The brilliance of the arrangement is that it separates two jobs that don't have to compete: deciding what the menu is (infrequent, centralized, one chef is enough) and serving the menu (very frequent, parallelizable, the more cooks the better). Enlace has exactly that shape: deciding which long_url corresponds to a new short_code happens ~40 times per second; serving that correspondence happens ~4,000 times per second. One chef who decides, many cooks who serve.

There's an honest detail in the analogy that the next lesson squeezes: when the chef changes a recipe, a moment passes —seconds, sometimes— until the change reaches the last cook's station. In that moment, a cook could be serving the old recipe. That delay is the replication lag, and for now we only name it.

How it flows: the replication log

The heart of replication is a simple idea: the primary doesn't share its entire database each time, it shares the list of changes. Every time the primary executes a write —an INSERT of a new Link, an UPDATE that increments clicks—, it notes that change in an ordered log and transmits it to the replicas, which apply it in the same order. Since they all start from the same initial copy and apply the same changes in the same order, they all converge to the same state.

flowchart TD
    client_w[Client: shorten<br/>~40 writes/s] -->|write| primary[(PRIMARY<br/>accepts writes)]
    primary -->|replication log| replica1[(REPLICA 1<br/>reads only)]
    primary -->|replication log| replica2[(REPLICA 2<br/>reads only)]
    primary -->|replication log| replica3[(REPLICA 3<br/>reads only)]
    client_r[Clients: resolve<br/>~4,000 reads/s] -->|spread reads| replica1
    client_r -->|spread reads| replica2
    client_r -->|spread reads| replica3

Notice the diagram's asymmetry, which is the whole idea: a single write arrow enters the primary, and many read arrows are spread among the replicas. The primary doesn't saturate because it only receives ~40 writes/s. The replicas don't saturate because the ~4,000 reads/s are divided among them. And they all have the same data because they all apply the same replication log.

An important consequence for Enlace's design: the application has to know how to route each query to the correct place. The writes (shorten, or incrementing clicks) go to the primary. The reads (resolve) go to a replica. In practice this is handled by the database client, a proxy, or a routing layer; the conceptual point is that write and read no longer go to the same place, and your code —or your infrastructure— has to decide it for each query.

Worked example: how many replicas Enlace needs

Let's compute, not from memory, how many replicas are needed to spread Enlace's reads. We're going to do it in two scenarios: without a cache (the pessimistic case) and with the module 4 cache (the real case):

import math

qps_read = 3_858            # reads/s (module 2)
qps_write = 38.6            # writes/s
reads_per_replica = 2_000   # reads/s a replica handles (budget)

# Scenario A: no cache, all the reads fall on the replicas
replicas_no_cache = math.ceil(qps_read / reads_per_replica)

# Scenario B: with a cache at 90% hit ratio, only 10% reaches the database
hit_ratio = 0.90
reads_to_db = qps_read * (1 - hit_ratio)
replicas_with_cache = math.ceil(reads_to_db / reads_per_replica)

print(f"writes to the primary: {qps_write:.0f}/s  (one primary is plenty)")
print(f"A) no cache:  {qps_read:,} r/s  -> {replicas_no_cache} replicas")
print(f"B) with cache:  {reads_to_db:,.0f} r/s -> {replicas_with_cache} replica(s)")
print(f"   per replica in A: {qps_read / max(replicas_no_cache,1):,.0f} r/s")

What to expect. When you run it:

writes to the primary: 39/s  (one primary is plenty)
A) no cache:  3,858 r/s  -> 2 replicas
B) with cache:  386 r/s -> 1 replica(s)
   per replica in A: 1,929 r/s

Read the result. A single primary is enough for the writes in both scenarios —39/s is child's play—. For the reads, without a cache you need 2 replicas (each serving ~1,929 r/s, just under the budget), and with the cache from module 4, 1 replica is enough, because the cache already absorbs 90%. In practice you'd deploy more replicas than strictly necessary —2 or 3— for two reasons the minimal arithmetic doesn't capture: redundancy (if a replica goes down, the others absorb its load) and spikes (real traffic isn't flat; a viral link or peak hour multiplies the reads for a while). The napkin number is the floor, not the final design.

Synchronous vs. asynchronous (the boundary with resilience)

When the primary transmits a change to the replicas, there's a design decision with an important tradeoff, which we only name here because its in-depth treatment —and its recovery patterns— belong to the resilience guide.

• Asynchronous replication: the primary confirms the write to the client without waiting for the replicas to receive it. It's fast (the client waits for no one) and it's the most common. The price: if the primary goes down right after confirming but before transmitting, that last change can be lost, and the replicas are, by design, a bit behind. This delay is the replication lag of the next lesson.
• Synchronous replication: the primary waits for at least one replica to confirm it received the change before confirming to the client. It's safer (the change is already in two places) but slower (the client waits for the round trip to the replica), and if the replica doesn't respond, the primary blocks.

For Enlace, asynchronous is almost always the correct choice: the writes are few, the data (a short_code → long_url mapping) tolerates a millisecond delay without drama, and speed matters. The consequence —the lag— is the topic of lesson 4.

And here's the explicit boundary: what happens when the primary goes down —how it's detected, how a replica is chosen and promoted to the new primary without splitting the system in two (the "split brain" problem), how the old primary is reintegrated when it comes back— is failover, and it's a topic of resilience-and-reliability-patterns-guide. In this module replication is a tool of scale (more reads); in the sibling guide it's a tool of availability (surviving crashes). That the same technology serves both purposes is exactly why it's so valuable, but they're two different lenses.

Common mistakes

Sending reads to the primary "just in case" (routing mistake). What happens: replication is set up but the application keeps reading from the primary for everything, "to have the freshest data", and the replicas sit idle while the primary saturates. Why it happens: the lag is scary, and reading from the primary avoids it. How to detect it: if your replicas are at 5% CPU and the primary at 95%, you're not spreading anything. How to fix it: most of Enlace's reads (resolve of a link that's existed for a while) tolerate the lag perfectly; send them to the replicas. Reserve the primary for the writes and for the handful of reads that really demand the most recent data (the next lesson teaches how to distinguish them).

Believing the replicas scale the writes (concept mistake). What happens: someone hits the write limit and adds replicas expecting relief, but the writes still all go to the single primary and nothing improves. Why it happens: "more machines" is confused with "more capacity for everything". How to detect it: if your bottleneck is write throughput and you added replicas, you won't see improvement in the writes. How to fix it: remember the asymmetry —a single primary accepts writes—; scaling writes is sharding (lesson 5), not replication. For Enlace it's not a problem (~40/s), but it's the conceptual mistake that separates the two axes.

Forgetting redundancy when counting replicas (operational mistake). What happens: the napkin calculation says "2 replicas are enough" and exactly 2 are deployed, with no margin. The day one goes down for maintenance, the other receives double the load and also goes down. Why it happens: the arithmetic floor is confused with the production design. How to detect it: ask yourself "if a replica goes down, do the others handle its load?". If the answer is no, you have no margin. How to fix it: size so the system holds up with N−1 replicas, not N; add at least one extra for redundancy and spikes. The napkin number is the minimum, not the goal.

Exercises

Exercise 1 — Route each operation. For each Enlace operation, say whether it goes to the primary or to a replica, and why. (a) shorten(long_url) creates a new Link. (b) resolve(short_code) of a link created a year ago. (c) Incrementing a Link's clicks counter. (d) A report that counts how many Links were created last month.

Exercise 2 — Size the replicas. Enlace grows and the reads rise to 10,000/s. The cache maintains a hit ratio of 85%. With a budget of 2,000 reads/s per replica, compute how many replicas you need for the reads that reach the database, and how many you'd deploy in production taking redundancy into account.

reads to the database = 10,000 × (1 − 0.85) = 1,500 r/s
minimum replicas = ceil(1,500 / 2,000) = 1 replica

Exercise 3 — Why the cache goes before the replicas. Enlace already has a cache (module 4). A colleague proposes removing the cache and "resolving everything with many replicas". With the numbers —4,000 reads/s, hit ratio 90%, 2,000 r/s per node—, compare how many nodes each approach calls for and explain why the cache is still the first line.

Summary and next step

In this lesson you met the first horizontal-scaling tool: primary/replica replication. With the executive chef and the line cooks you saw the asymmetry that makes it work —a single place decides the changes (the primary, ~40 writes/s), many places serve the menu (the replicas, ~4,000 reads/s spread)— and how the replication log copies each change from the primary to the replicas so they all converge to the same state. You computed, with numbers, that Enlace only needs one primary and one or two replicas (plus the redundancy ones), and you saw how the module 4 cache reduces how many you need. You named the difference between synchronous and asynchronous replication, and marked the boundary: failover in depth is from the resilience guide.

Before moving on you should be able to: draw the primary → replication log → replicas flow; correctly route an Enlace read and write; compute how many replicas given reads/s call for; and explain why replicas don't scale the writes.

What comes next is the honest asterisk this lesson left pending. The replicas are a bit behind the primary —that's the replication lag—, and it has a concrete consequence: a just-created short_code resolved from a replica that hasn't received it yet returns a fleeting 404. In lesson 4 you'll see why it happens, when it matters for Enlace (almost never) and when it does, and the basic mitigations. It's the price of spreading the reads, and you have to know it to pay it with your eyes open.

Resources

• Designing Data-Intensive Applications, Martin Kleppmann — Chapter 5, "Leaders and Followers" — the reference explanation of the primary/replica pattern (the book uses "leader/follower"), including the replication log and the synchronous/asynchronous contrast we only name here.
• PostgreSQL — Streaming Replication — how a real engine implements streaming the primary's log to the replicas; the concrete grounding of this lesson's "replication log".
• System Design Primer — Replication (master-slave / master-master) — a practical summary of the replication variants and their tradeoffs, useful as an overview before going deep into the lag.