2. When one database is not enough

Description

By the end of this lesson you'll be able to look at a system and say, with a number in hand, whether a single database is still enough or whether the time has come to scale —and which of three different limits ran out—. It's a diagnostic skill, not a construction one: before adding a single machine you have to know which resource ran out, because each exhausted resource has a different cure and applying the wrong cure fixes nothing and adds complexity. You'll see the three limits that push you to scale —storage runs out, the read saturates, the write saturates—, each with the Enlace number that triggers it; the difference between scaling up (a bigger machine) and out (more machines), and why the first has a hard ceiling; and the two independent axes along which you scale out.

This matters because scaling badly costs dearly in both directions. Scaling too late leaves you with a slow or downed system at the worst moment —the traffic peak, the marketing campaign, the link that went viral—. But scaling too early, or on the wrong axis, leaves you carrying the complexity of a distributed system —several machines, spread data, queries that cross nodes, remappings— without any of its benefits, and that also makes the system go down, only from coordination bugs instead of saturation. The correct diagnosis is what stops you at the right point: neither before nor after, and on the axis that really ran out.

Connection to the module: this lesson is the diagnosis that enables everything that follows. Lesson 1 gave you the map —two axes, three pressures—; this one sharpens it down to the number that triggers each decision. When in lesson 3 you start adding replicas, it'll be because here you established that the ~4,000 reads/s are the limit that squeezes first in Enlace. When in lesson 5 you shard, it'll be because here you established that the ~6 TB don't fit comfortably on one machine. Without this diagnosis, the tools of the following lessons would be solutions in search of a problem.

The single-lane bridge

Think of it this way. A town has a single-lane bridge over the river, and for years it's enough. One day it starts getting small, and I want you to see that "getting small" can mean three completely different things, each with a different construction job.

First: so many cars pass that a line forms. The bridge holds the weight, but not the number of cars per minute. The job that fixes this is widening the bridge: more lanes, same ends, same river. More cars pass in parallel without the line growing.

Second: ever-heavier trucks start crossing, and the bridge, designed for cars, can't support that weight. No matter how many lanes you add: the problem isn't the flow, it's that each crossing exceeds what the structure can bear. Widening doesn't help; you have to spread the load over several bridges or build a different one.

Third: the bridge is the river's only crossing, and the day it's closed for maintenance, the town is split in two. The problem isn't capacity, it's that there's no redundancy. The job is a second bridge to serve as backup.

Widen (more lanes), spread (several bridges), back up (an extra bridge): three jobs for three problems that from afar look the same —"the bridge got small"—. In a database it's identical. Widening the bridge is scaling vertically (a more powerful machine). Spreading over several bridges is scaling horizontally (more machines). Backing up is the redundancy the resilience guide handles. And the flow of cars versus the weight of the trucks are two different limits, just as in Enlace the read throughput and the data volume are two different limits.

The three limits, each with its number

A single database stops being enough when one of these three resources runs out. For each one, Enlace's number and the symptom you'd see.

1. Storage runs out. There's a physical cap on how much data fits. Enlace's number: ~6 TB at five years. The symptom: the disk fills, or —before filling completely— the backups take hours, restoring from a backup becomes a half-day event, and adding a new index requires temporary space you no longer have. It's not a matter of speed: it's that 6 TB of data plus its indexes plus the working space are more than you want to have on a single node if you value being able to operate it. This limit is cured by spreading the data: sharding.

2. The read saturates. There's a cap on how many read queries per second a node can handle before the latency shoots up. Enlace's number: ~4,000 reads/s. The symptom: at peak hour, the resolve queries that normally take 5 ms start taking hundreds of milliseconds, the node's CPU and I/O go to 100%, and the requests queue up. Watch the sequence: the cache from module 4 already absorbs most of these reads; this limit is about what escapes the cache (the long tail, the cold cache on restart, the spikes). This limit is cured by spreading the reads among copies: replication.

3. The write saturates. There's a cap on how many writes per second the node that accepts them can handle. Enlace's number in normal operation: ~40 writes/s —comfortable, a single node digests it without a problem—. This is the limit Enlace does not hit in its normal operation, and that's why sharding arrives after replicas. You'd hit this limit only in an extreme scenario: a campaign that creates a billion links in days, shooting the writes to thousands per second. The symptom would be the same (latency and queuing) but on the write path. This limit is cured by spreading the writes among several nodes that accept writes: sharding again, because replicas do not help here —all the writes still go to a single primary per copy—.

Note Enlace's asymmetry: the three limits don't squeeze at the same time. The write (~40/s) is roomy, the read (~4,000/s) squeezes soon, and the storage (~6 TB) squeezes over time. That profile —read-heavy, light-write, data that grows— is typical of many real systems, and it dictates the module's order.

Worked example: put each limit in front of its number

Let's make Enlace's diagnosis explicit, with reasonable per-node budgets (round and conservative numbers, the kind you'd use on a napkin). Let's say a database node comfortably handles 2,000 reads/s and ~1 TB of operable data before you start suffering. They're not sacred numbers —they depend on the hardware— but they serve for the diagnosis:

# Diagnosis: how many machines each Enlace limit calls for
qps_read = 3_858          # reads/s (from module 2; what the cache absorbs is separate)
qps_write = 38.6          # writes/s
storage_tb = 6.14         # TB at 5 years

# Per-node budgets (conservative, depend on the hardware)
reads_per_node = 2_000    # reads/s a node handles
writes_per_node = 2_000   # writes/s a node handles
tb_per_node = 1.0         # operable TB per node

import math
replicas_needed = math.ceil(qps_read / reads_per_node)
write_nodes_needed = math.ceil(qps_write / writes_per_node)
shards_needed = math.ceil(storage_tb / tb_per_node)

print(f"reads: {qps_read:,} r/s  -> {replicas_needed} nodes to read (replicas)")
print(f"writes: {qps_write:,.0f} w/s -> {write_nodes_needed} node to write (plenty)")
print(f"storage: {storage_tb} TB -> {shards_needed} shards to spread the data")

What to expect. When you run it:

reads: 3,858 r/s  -> 2 nodes to read (replicas)
writes: 39 w/s -> 1 node to write (plenty)
storage: 6.14 TB -> 7 shards to spread the data

Read the result as the diagnosis it is. The write fits on 1 node with plenty of margin (39 of a 2,000 budget): there's no write problem, don't shard for that. The read calls for 2 nodes to serve reads —and here's where the module 4 cache changes the number: if the cache absorbs 90%, the reads that reach the database are ~386/s and a single node is enough; without a cache, you need to spread—. The storage calls for 7 shards to not carry 6 TB on a single disk. Three numbers, three different conclusions. That's diagnosing before building: not "the database can't handle it, let's add machines", but "the write is plenty, the read is resolved by the cache plus maybe a replica, and the storage at five years will call for ~7 shards".

Vertical vs. horizontal: why "a bigger machine" has a ceiling

Faced with any of the three limits, the first temptation is the simplest: scaling vertically, buying a bigger machine —more CPU, more RAM, more disk—. And sometimes it's the right thing: it's simple, it doesn't change the code, it doesn't introduce distributed systems. The problem is that it has a hard ceiling for three reasons.

• There's a maximum machine size. You can buy the largest instance your cloud provider sells, and that's where it ends. There's no infinitely large machine.
• The price isn't linear. Doubling the power of a large machine costs much more than double; the last portion of performance is the most expensive. There comes a point where two medium machines cost less than an equivalent giant one.
• It's still a single point of failure. However large it is, if that machine goes down, everything goes down. Vertical scaling doesn't give redundancy.

Scaling horizontally —more machines, not bigger ones— doesn't have that ceiling: you can keep adding nodes. The price is complexity: now you have a distributed system, with spread data, coordination between nodes, and the hard questions of this module (which data lives where, what happens when you add a node). The practical rule: scale vertically while it's simple and cheap; cross to horizontal when you hit the ceiling of price, of size, or when you need redundancy. Enlace, with 6 TB and 4,000 reads/s, crosses to horizontal for sure —6 TB don't fit comfortably in any instance you'd want to back up—, but many smaller systems live happily their whole life on a single vertical machine with a backup replica.

   VERTICAL SCALING               HORIZONTAL SCALING
   (a bigger machine)             (more machines)

     ┌───────────┐                ┌───┐ ┌───┐ ┌───┐
     │           │                │db │ │db │ │db │
     │   db      │  ── ceiling ─►  └───┘ └───┘ └───┘
     │  (large)  │                no ceiling, but
     │           │                distributed system
     └───────────┘
   simple, expensive at the end,  complex, no ceiling,
   a single point of failure      redundant

The two axes, formalized

Scaling horizontally isn't a single direction. You already drew it in lesson 1; now we fix it precisely, because it's the distinction that governs the rest of the module.

• Reads axis — replication. Each new machine is a complete copy of the data that only serves reads. It adds read capacity, not storage (each copy stores the same) or write (the writes still go to a single node). It resolves limit 2.
• Data and writes axis — sharding. Each new machine stores a different piece of the data and accepts its reads and writes. It adds storage capacity and write capacity (each piece receives a fraction). It resolves limits 1 and 3.

They're independent: you can move on one, on the other, or on both. A mature system moves on both —it shards the data and replicates each shard—, which is the topology you'll build in the project. But you get there in stages, resolving the limit that squeezes first. In Enlace, that's the reads one, and that's why lesson 3 starts with replication.

Common mistakes

Scaling without measuring which resource ran out (diagnostic mistake). What happens: the system is slow, someone says "the database can't handle it" and adds replicas... when the problem was the full disk, which replicas don't fix. Why it happens: "it's slow" is a symptom common to all three limits, and without metrics you guess. How to detect it: if you can't point to the saturated metric —disk at X%, CPU at Y%, reads/s at Z—, you're guessing. How to fix it: measure before scaling. A dashboard with disk usage, read throughput, and write throughput separately turns panic into diagnosis.

Believing the cache replaces the replicas (sequence mistake). What happens: someone adds a cache and considers the read limit solved forever. Why it happens: the cache does absorb most of the hot reads, and for a while it's enough. How to detect it: look at what happens when the cache restarts (cold cache) or when a spike of links no one had cached arrives: all those reads fall on the database at once. How to fix it: cache and replicas are complementary layers, not substitutes. The cache reduces the average; the replicas handle the peak and the long tail the cache doesn't cover. Enlace needs both.

Scaling vertically to the end (cost mistake). What happens: a team buys ever-bigger machines to avoid dealing with the distributed, until the bill shoots up or they hit the maximum instance at the worst moment. Why it happens: vertical is tempting because it doesn't change the code. How to detect it: if you're on your provider's largest instance, or if doubling the power already costs more than triple, you've hit the ceiling. How to fix it: plan the cross to horizontal before hitting the ceiling, not when you've already hit it with the system down. Scaling horizontally is a project, not a button; do it with margin.

Exercises

Exercise 1 — The number that triggers. For each of the three limits, write the napkin formula that would tell you how many machines you need, using Enlace's anchor numbers and a per-node budget of your choice. Then say which limit squeezes first in time for Enlace and why.

Exercise 2 — Vertical or horizontal. For each situation, say whether you'd scale vertically or horizontally and why. (a) A personal blog with a 20 GB database and 50 reads/s that sometimes gets slow at peaks. (b) Enlace, with 6 TB projected. (c) A service whose database already runs on the provider's largest instance and is still saturated with reads.

Exercise 3 — The cache's effect on the diagnosis. In the worked example, without a cache the ~4,000 reads/s called for 2 read nodes. Recompute how many read nodes Enlace calls for with the module 4 cache, assuming a hit ratio of 90% (the cache serves 90% of the reads and only 10% reaches the database). Does it change the urgency of adding replicas? Explain how the two tools relate.

reads that reach the database = 3,858 × (1 − 0.90) = 385.8 r/s
read nodes = ceil(385.8 / 2,000) = 1 node

Summary and next step

In this lesson you learned to diagnose before building. With the single-lane bridge you saw that "it got small" is actually three problems —car flow, truck weight, lack of backup— and three different jobs: widen (vertical), spread over several bridges (horizontal), and a backup bridge (redundancy). You put a database's three limits in front of Enlace's numbers: storage (~6 TB → sharding), read (~4,000 r/s → replication, attenuated by the cache), write (~40 w/s → comfortable, doesn't squeeze). You saw why vertical scaling has a hard ceiling —maximum size, non-linear price, a single point of failure— and when to cross to horizontal. And you formalized the two axes: replicas add reads, shards add data and writes.

Before moving on you should be able to: name the three limits and the Enlace number that triggers each; write the napkin formula that estimates how many machines each limit calls for; explain why vertical has a ceiling; and say which axis a replica scales and which a shard.

What comes next is the first tool, for the limit that squeezes first in Enlace: the reads. In lesson 3 you'll see the primary/replica pattern in detail —a machine that accepts the writes and several that copy its data and serve reads—, how the replication log flows, and you'll compute how many replicas Enlace needs to spread its ~4,000 reads/s. It's the horizontal axis in action.

Resources

• Designing Data-Intensive Applications, Martin Kleppmann — Chapter 5 (Replication) — the chapter that underpins the replication we start in the next lesson; its introduction explains precisely why you replicate and what you gain.
• System Design Primer — Scaling: vertical vs horizontal — a concise review of vertical versus horizontal and of why vertical hits a ceiling, with the same ideas from this lesson.
• PostgreSQL — Chapter 27: High Availability, Load Balancing, and Replication — the official documentation that grounds these concepts in a real engine; its comparison table of scaling solutions is an excellent map of the practical options.