Module 8: Project — Design Enlace End to End

7. Step 4c — Scaling out: replicas, sharding, load balancing and consistency

Next →

Description

This is the densest lesson of the capstone, because it closes step 4 by bringing the three distributed scale modules together into a single deep dive: the data scaling (module 5: replicas and sharding with consistent hashing), the compute scaling (module 6: load balancing over stateless services), and the consistency tradeoff (module 7: CAP/PACELC, eventual vs strong, and which one Enlace gets). It is where the design stops being "a fast box" and becomes a real distributed system —many machines collaborating, with distributed data, balanced traffic, and an explicit and justified decision about what consistency guarantees it gives and which it sacrifices—.

You are going to pick up the loose end lesson 6 left —the 386 residual reads/s (and ~1,157/s at the peak) the cache does not absorb— and see it absorbed by the replicas; you are going to distribute the 6 TB among shards by short_code, and execute the number that makes sharding safe to grow: consistent hashing remaps 130,623 keys (13.1%) when adding a node, not 888,920 (88.9%) like mod-N. You are going to load-balance the traffic among the stateless servers with health checks, and —the capstone's finishing touch— you are going to choose and justify Enlace's consistency model: eventual for resolve, with the numbers that prove why that choice is correct and what it buys. When you finish, you will have the last piece of the deliverable: the list of justified tradeoffs.

Connection to the module: it is the third and last "deep dive" lesson, and the one that turns Enlace's design into a distributed one. It takes the residual load of lesson 6 as input, and produces the list of tradeoffs that lesson 8 puts into the final deliverable. It brings three modules together because, in the capstone, replicas + load balancing + consistency are a single story: scaling out and reasoning about what guarantees are preserved when doing so. The border with the sibling guides —failover in depth (resilience), event sourcing/CQRS (events), architectural styles— is marked in each stretch.

The restaurant chain and its shared menu

Think of it this way. A restaurant that succeeds wants to grow, and there are three different ways to grow that solve three different problems —and it is worth not confusing them—.

The first: open more identical branches of the same popular kitchen. The recipe of the star dish is a single one, decided at headquarters, but it is copied to each branch, and each branch serves the diners of its neighborhood. Ten branches serve ten times more diners of the same dish. That is replication: a complete copy of the data on each node, many nodes serving reads in parallel. It solves the problem of too many diners (too many reads).

The second: when the menu becomes enormous —a thousand dishes, impossible for a single kitchen to prepare them all well—, distribute the menu among specialized kitchens: the Italian kitchen, the Asian one, the desserts one, each with its part of the menu. No diner finds everything in one kitchen; a coordinator sends them to the one that has their dish. That is sharding: distributing the data among nodes when it no longer fits (neither the writes nor the volume) on a single one. It solves the problem of too much menu (too much data).

The third: a host at the door who distributes the diners among the free tables, without sending anyone to an occupied table or to a sick waiter. That is load balancing: distributing the traffic among the available servers and avoiding the downed ones. It solves the problem of distributing well whoever arrives.

And there is an underlying tension in every chain: when headquarters changes a recipe, a while passes until the new one reaches the last branch. During that while, one branch serves the old recipe. Is it a problem? It depends on the dish. For the dish of the day, a lag of minutes does not matter to anyone (eventual consistency, and that is fine). For the price charged at the register, it does matter that all the branches coincide instantly (strong consistency). Choosing which guarantee to give each datum —and accepting what you sacrifice— is the consistency tradeoff. Enlace is, almost entirely, "dish of the day": the short_code → long_url mapping does not change, so a lag of milliseconds between replicas bothers nobody, and that tolerance is what buys the read scale.

It is worth spelling it out:

Scaling out is three different decisions for three different problems: replicate (copy the data to serve many reads), shard (distribute the data when it does not fit on a node) and load-balance (distribute the traffic among healthy servers). And a cross-cutting decision: what consistency to give each datum. For Enlace, eventual consistency is enough —the data is almost immutable—, and that tolerance is what makes the scale possible.

Scaling the data: replicas for the reads

The cache lets through 386 reads/s on average (and ~1,157/s at the peak), plus the ~40 writes/s. Where do they go? To the primary/replica pattern: one node (the primary) accepts all the writes and copies each change, via the replication log, to other nodes (the replicas) that serve reads. It fits Enlace's 100:1 like a glove: few writes to the primary, many reads distributed among replicas. Let us compute how many replicas are needed:

# replicas.py — how many replicas for Enlace's residual load.
import math
reads_per_replica = 2_000     # reads/s a replica holds (budget)

for label, reads in [("average (386/s)", 386), ("peak (1,157/s)", 1157)]:
    n = math.ceil(reads / reads_per_replica)
    print(f"{label:>18}: ceil({reads}/{reads_per_replica}) = {n} minimum replica(s)")

print("\nIn production: size for N-1 (tolerate the loss of one)")
print("  -> with the peak of 1,157/s: 1 minimum -> deploy 2-3 for redundancy")

What to expect. With python replicas.py:

   average (386/s): ceil(386/2000) = 1 minimum replica(s)
    peak (1,157/s): ceil(1157/2000) = 1 minimum replica(s)

In production: size for N-1 (tolerate the loss of one)
  -> with the peak of 1,157/s: 1 minimum -> deploy 2-3 for redundancy

The arithmetic floor is 1 replica —even at the peak, 1,157/s fits under the 2,000/s budget of a replica—. But the napkin number is the minimum, not the design: in production you deploy 2 or 3 replicas per shard, for two reasons the minimum arithmetic does not capture. Redundancy: if a replica goes down (or is pulled for maintenance), the others absorb its load —you size so that the system holds with N−1—. And margin for the peak and the bad day: if the cache empties (the mass miss of lesson 6), the database gets 3,858/s at once, and you want enough replicas not to collapse while the cache repopulates. The rule: count the minimum, deploy with margin.

An honest asterisk the design must name: the replicas are a bit behind the primary (the replication lag). Concrete consequence for Enlace: a freshly created short_code resolved from a replica that has not yet received it returns a fleeting 404 of milliseconds until the replication log arrives. For Enlace this almost never matters (whoever just created a link does not visit it instantly from another machine), and when it matters it is mitigated by reading from the primary right after writing. That Enlace tolerates that lag is, precisely, the consistency decision we will see at the end —and the one that allows distributing the reads—.

Scaling the data: sharding, and why consistent hashing

Replicas scale the reads, but not the data: each replica is a complete copy of the 6 TB, and the day comes when 6 TB do not fit comfortably on one machine (nor its writes on a single primary). The answer is sharding: splitting the data among several shards, each a primary with its replicas. Enlace's shard key is the short_code, and it is distributed by hash (not by range) to avoid hotspots —a distribution by range or by date would concentrate all the new writes on one shard; the hash spreads them evenly—.

But how to hash matters enormously, and it is the decision that makes sharding safe or dangerous to grow. The naive way, shard = hash(short_code) % N, distributes perfectly... until you add a node, and then N changes and almost all the keys remap. The correct way, consistent hashing (the ring), remaps only the arc of the new node. Let us run the two, side by side, over a million keys when going from 8 to 9 shards:

# sharding_growth.py — mod-N versus consistent hashing, measured.
import bisect, hashlib

def ring_point(t): return int(hashlib.md5(t.encode()).hexdigest(), 16) % (2**32)
def node_modn(sc, n): return int(hashlib.md5(sc.encode()).hexdigest(), 16) % n

def build_ring(nodes, vnodes=100):
    r = {}
    for nd in nodes:
        for i in range(vnodes):
            r[ring_point(f"{nd}#{i}")] = nd
    return r

def node_ring(sc, ring, pts):
    p = ring_point(sc); idx = bisect.bisect(pts, p)
    return ring[pts[0 if idx == len(pts) else idx]]

def make_keys(k):
    ab = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
    out = []
    for i in range(k):
        num, s = 100_000_000 + i, ""
        while num: s, num = ab[num % 62] + s, num // 62
        out.append(s.rjust(7, "0"))
    return out

K, N = 1_000_000, 8
keys = make_keys(K)

moved_modn = sum(node_modn(k, N) != node_modn(k, N + 1) for k in keys)

nodes = [f"db-{i}" for i in range(N)]
ring = build_ring(nodes); pts = sorted(ring)
before = {k: node_ring(k, ring, pts) for k in keys}
for i in range(100): ring[ring_point(f"db-{N}#{i}")] = f"db-{N}"
pts = sorted(ring)
moved_ring = sum(node_ring(k, ring, pts) != before[k] for k in keys)

print(f"K = {K:,} keys,  from N={N} to N={N+1} shards\n")
print(f"  hash(key) % N       {moved_modn:>9,}  ({100*moved_modn/K:4.1f}%)  <- almost all")
print(f"  consistent hashing  {moved_ring:>9,}  ({100*moved_ring/K:4.1f}%)  <- ~K/(N+1)={K//(N+1):,}")

What to expect. With python sharding_growth.py:

K = 1,000,000 keys,  from N=8 to N=9 shards

  hash(key) % N         888,920  (88.9%)  <- almost all
  consistent hashing    130,623  (13.1%)  <- ~K/(N+1)=111,111

There is the number that makes Enlace's sharding safe to grow. Adding the ninth shard with mod-N remaps 888,920 keys (88.9%) —translated to Enlace, moving ~5.3 TB between machines, emptying almost all the cache, and risking inconsistency for hours: a storm—. With consistent hashing it remaps 130,623 (13.1%), close to the ideal K/(N+1) = 111,111 —moving only ~0.8 TB, cooling only the arc of the new node—. Almost 7 times less. The same operation —growing— goes from a major-risk event to routine maintenance. And the vnodes (each shard placed at ~100 points of the ring) keep the distribution even, preventing one shard from receiving 10× more data than another. For Enlace: sharding by short_code, with consistent hashing and vnodes —distributes the 6 TB evenly, routes each read to a shard in O(log N), and grows without a storm—.

Scaling the compute: load balancing over stateless services

The data already scales; the compute is missing. A single application server is a bottleneck and single point of failure, so Enlace runs several stateless servers behind a load balancer. That they are stateless (the state lives in the cache and the database, not in the server) is what makes the load balancing possible: any server handles any request, so the load balancer distributes freely, adding capacity is plugging in another identical server, and if one goes down, it is taken out of rotation without anyone losing anything.

The load balancer distributes with some strategy, and there are three common ones, each with its logic:

  • Round-robin: distributes by turns, one to each server in order. Simple and fair when all the requests cost about the same —Enlace's case, where one resolve costs almost the same as another—.
  • Least-connections: sends each request to the server with the fewest active connections. Better when the requests last very different times (some long, some short), so as not to load a server that already has heavy work.
  • By hash (of the IP or of the key): always sends the same client (or the same key) to the same server. Useful if there is a local cache per server —not Enlace's case, whose cache is shared (Redis)—.

For Enlace, round-robin is enough: the requests are homogeneous and the cache is shared, so there is no need for the sophistication of least-connections or the affinity of the hash. And the piece that makes the load balancing fault-tolerant are the health checks: the load balancer periodically asks each server "are you alive?", and takes out of rotation the one that does not respond, sending its traffic to the healthy ones. This plugs the last crack of the single box of module 1 —the compute's single point of failure— and is what sustains the 99.9% availability on the application-server side. The border: the advanced resilience patterns (circuit breaker, bulkhead, retry with backoff) are from resilience-and-reliability-patterns-guide; here load balancing with health checks is enough.

The consistency tradeoff: why eventual is enough for Enlace

We reach the capstone's finishing touch, the decision that crowns the list of tradeoffs. When you have data distributed and replicated across many machines, an unavoidable question appears: if two replicas can be momentarily out of sync (due to the lag), what consistency guarantee do you give the user? The theory that frames this is CAP: facing a network partition (P), a distributed system must choose between consistency (C: all the replicas show the same datum) and availability (A: the system responds even if it cannot guarantee the datum is the most recent). You cannot have both during a partition. And PACELC extends it with what happens when there is no partition (E, else): even in normal operation, you choose between latency (L) and consistency (C) —waiting for all the replicas to confirm is more consistent but slower—.

For Enlace, the choice is clear and is justified by the nature of the data, not by a preference:

  • The data is almost immutable. A short_code → long_url, once created, does not change. There are no UPDATEs that can diverge between replicas; only INSERTs of new records. A datum that does not change cannot become "inconsistent" in the serious sense —two replicas will never show different destinations for the same code—.
  • The only possible inconsistency is benign and fleeting. A freshly created short_code resolved from a replica that is behind gives a 404 of milliseconds until the replication log arrives. It does not return the wrong datum; it returns "I do not have it yet". And in Enlace that almost never occurs (the creator does not visit their link instantly from another machine) and it recovers on its own as soon as the replica catches up.
  • The tolerance buys scale. Accepting eventual consistency is what allows distributing the reads among replicas that are behind —if you demanded that every read see the most recent write instantly, you would have to read them all from the primary, and goodbye to the replicas and to the read scale—. Eventual consistency is not a concession: it is the decision that makes the entire design possible.

That is why Enlace chooses eventual consistency for resolve (reading from replicas that are behind is fine), with asynchronous replication (the primary confirms without waiting for the replicas, prioritizing latency —the "L" of PACELC—). In CAP terms, Enlace leans toward availability: it prefers to respond fast (even if a replica is a hair behind) over blocking waiting for perfect consistency. It is the correct choice for a redirect service, and it is justified by the number (the data does not change) and the consequence (a fleeting 404 nobody notices), not by taste.

And to close the reliability side, the non-functional requirement is set as a number: the SLO (service level objective) of availability is 99.9%, which means a maximum of ~8.76 hours of downtime a year (0.001 × 365 × 24). That number is what justifies the redundancy (several servers, several replicas per shard, failover): a single machine fails more than that, so copies are needed. Enlace's distributed design is, in good part, the answer to that 99.9%.

The list of tradeoffs (the last artifact)

Everything above condenses into Enlace's list of tradeoffs —the artifact that proves the design understood the problem, not just that it drew boxes—. Each one with its "I gain / I pay", justified with a number:

DecisionI gainI payNumeric justification
Cache in front of the DB5.90 ms vs 50 ms; DB sees 386/s vs 4,000/sPossible old datum (mitigated: almost immutable data + TTL)Hit ratio 0.90; working set 333 MB
Read replicasDistribute the reads; redundancyReplication lag (fleeting 404)386/s residual; N−1 tolerance
Sharding by short_codeDistributes 6 TB and writes; no hotspotsCross-shard queries more complex (Enlace has none)6 TB / N shards; hash avoids hotspots
Consistent hashing (not mod-N)Growing moves ~13.1%, not 88.9%Ring table (vnodes) in memory130,623 vs 888,920 when going from 8 to 9
Stateless servers + load balancingHorizontal scale; tolerates failuresThe shared state (cache/DB) is the bottleneckHealth checks; 99.9% availability
Eventual consistency (async)Low latency; read scaleReads may see a datum from ms agoAlmost immutable data; benign fleeting 404
302 redirect (not 301)Leaves the door to future analyticsEvery visit passes through Enlace (browser does not cache)Scope v1: analytics deferred

That table is the heart of the deliverable. Notice that no row says "I chose X because it is the best"; each one says "I gain this, I pay that, and the number justifies it". Designing is not choosing the optimum (there is none); it is choosing tradeoffs for concrete requirements, and knowing how to defend each one.

Common mistakes

Confusing replication with sharding (believing that replicas scale the writes or the data). What happens: someone hits the storage limit (6 TB) or the write limit and adds replicas expecting relief, but replicas are complete copies (they do not distribute the data) and the writes still all go to the single primary (they do not distribute them). Nothing improves. Why it happens: "more machines" is confused with "more capacity of everything". How to spot it: if your bottleneck is the data volume or the write throughput and you added replicas, you are not going to see improvement. How to fix it: remember the two axes —replicating scales reads (copies that serve in parallel); sharding scales data and writes (distributing among nodes)—. Enlace uses both: replicas for the 386 residual reads/s, sharding for the 6 TB. They are different tools for different problems.

Sharding with mod-N and discovering the remap when growing. What happens: someone chooses hash(short_code) % N because it distributes perfectly in the tests (~125k per shard with N=8), and considers the decision good. The day they add a shard, 888,920 of 1,000,000 keys remap —moving ~5.3 TB, emptying the cache, risking inconsistency—, exactly when the system is big and that hurts most. Why it happens: the good distribution is visible immediately; the bad remap is invisible until you grow. How to spot it: ask yourself "what happens when I add node N+1?". If you did not test it, you did not test what matters. How to fix it: evaluate the sharding by two properties —balance (distributes evenly) and stability (little remap when N changes)—; mod-N passes the first and fails the second. Use consistent hashing with vnodes: distributes evenly and remaps only ~K/N (13.1%).

Choosing strong consistency by default "to be safe". What happens: someone, out of prudence, demands that every resolve read from the primary (strong consistency, always-fresh datum), and with that wastes the replicas —the primary saturates with the 4,000 reads/s, and the replicas sit idle—. The "safety" cost all the read scale. Why it happens: strong consistency sounds more responsible, and the lag is scary. How to spot it: if your replicas have 5% CPU and the primary 95%, you are not distributing anything out of fear of the lag. How to fix it: choose the consistency the data needs, not the maximum available. Enlace's data is almost immutable and its only inconsistency (a fleeting 404) is benign, so eventual is correct and buys the scale. Strong consistency has a cost (latency, or saturating the primary); it is only paid where the data requires it —and in Enlace, it almost never requires it—.

Exercises

Exercise 1 — The two axes of scale. For each of Enlace's growth problems, say whether it is solved with replicas, with sharding, or with compute load balancing, and why: (a) the residual reads rise from 386/s to 4,000/s (the cache gets worse); (b) the data grows from 6 TB to 20 TB and does not fit on a node; (c) the application server saturates its CPU processing requests; (d) a single primary does not hold the write volume.

See solution
  • (a) Residual reads rise → replicas. More reads are resolved with more replicas that distribute them (the reads axis). ceil(4,000/2,000) = 2 minimum replicas, 3 in production.
  • (b) Data from 6 to 20 TB → sharding. The data does not fit on a node; it is distributed among more shards (the data axis). Replicas do not help here —each replica is a complete copy, it would still not fit—.
  • (c) Application server saturated on CPU → compute load balancing. More stateless servers behind the load balancer distribute the CPU load. It is not a data problem (replicas/sharding), it is a compute one.
  • (d) A primary does not hold the writes → sharding. Scaling writes is sharding (more primaries, each with its part of the data), not replication (the replicas do not accept writes). For Enlace this does not happen (~40/s), but if it did, more shards = more primaries = more write capacity.

The lesson: there are three axes of scale —reads (replicas), data and writes (sharding), compute (load balancing)— and each problem belongs to one. Confusing them leads to adding the wrong tool (replicas for a data problem, for example) and not seeing improvement.

Exercise 2 — Translate the remap to Enlace. Enlace has 8 shards with 6 TB distributed, and adds the ninth. With the measured numbers (mod-N: 888,920 remaps; consistent hashing: 130,623; over 1,000,000 keys), compute: (a) what fraction of the 6 TB each strategy moves; (b) how many times less consistent hashing moves; (c) why the difference grows if Enlace had 100 shards instead of 8.

See solution
  • (a) mod-N moves 88.9% × 6 TB ≈ 5.33 TB. Consistent hashing moves 13.1% × 6 TB ≈ 0.78 TB. For the same operation —adding a shard— one moves ~5.3 TB and the other ~0.8 TB.
  • (b) 888,920 / 130,623 ≈ 6.8 times less remap with consistent hashing (almost 7×).
  • (c) Because mod-N remaps N/(N+1), which grows toward 100% with more nodes (from 8 to 9: 88.9%; from 100 to 101: 99.0%), while consistent hashing remaps ~1/(N+1), which decreases toward 0% (from 100 to 101: ~1%). The bigger the system, the worse mod-N gets and the better consistent hashing gets, so the gap widens. Exactly when Enlace is bigger —and moving its data is more expensive—, mod-N punishes more and consistent hashing helps more. That is why the design chooses consistent hashing: it turns "growing" from a storm into routine, and the advantage increases with scale.

Exercise 3 — Justify (or reject) strong consistency. A colleague proposes that Enlace use strong consistency: every resolve reads from the shard's primary to guarantee the most recent datum. (a) What does it gain with that? (b) What does it lose, with numbers? (c) Is it worth it for Enlace? Answer with the nature of the data.

See solution
  • (a) It gains: the guarantee that every read sees the most recent write instantly —zero replication lag, zero fleeting 404s from a freshly created link—.
  • (b) It loses, with numbers: all the reads would go to the shard's primary, not to the replicas. The primary, which only had to hold ~40 writes/s distributed, would now also receive the 386 residual reads/s (and ~1,157/s at the peak, and 3,858/s on the day of the mass miss) —it would become the bottleneck, and the replicas would sit idle—. All the read scale the replicas gave is lost. Moreover, the "L" of PACELC: reading with strong confirmation adds latency.
  • (c) It is not worth it for Enlace, and the reason is the nature of the data: the short_code → long_url mapping is almost immutable (there are no UPDATEs that diverge), so the only possible inconsistency is a fleeting 404 of milliseconds for a freshly created link —benign (it is not the wrong datum, it is "I do not have it yet") and very rare (the creator does not visit their link instantly from another machine)—. Paying all the read scale to avoid a 404 of milliseconds that almost never occurs is a terrible tradeoff. Eventual consistency is correct because the data does not change: there is nothing serious for strong consistency to protect. Where the data does change (a price, a balance), the conversation would be another; in Enlace, no.

The lesson: strong consistency is not "more responsible", it is more expensive, and it is only paid where the data requires it. Choosing consistency according to the nature of the data —and not out of generic prudence— is what separates a design that scales from one that strangles itself out of fear of the lag.

Summary and next step

In this lesson you closed step 4 by bringing together Enlace's three distributed-scale decisions. You scaled the data on two axes: replicas to absorb the 386 residual reads/s (1–2 minimum, 2–3 in production for redundancy and the peak), with the asterisk of the replication lag; and sharding by short_code with consistent hashing, which you executed —remaps 130,623 keys (13.1%) when growing, not 888,920 (88.9%) like mod-N—, plus the vnodes for the even distribution. You scaled the compute with load balancing (round-robin is enough for Enlace) over stateless servers, with health checks that plug the last single point of failure. And you crowned the capstone with the consistency tradeoff: eventual for resolve, justified by the almost immutable nature of the data (CAP → availability, PACELC → latency), with the SLO of 99.9% (~8.76 h/year) as the number that requires the redundancy. Everything condensed into the list of tradeoffs, the artifact that proves the design understood the problem.

With this, the four artifacts of the deliverable are complete: requirements (lesson 2), capacity table (lesson 3), high-level design and diagram (lesson 4), and the deep dive with the list of tradeoffs (lessons 5, 6, 7). The journey back reached the end of the map.

Before moving on you should be able to: distinguish the three axes of scale (replicas, sharding, load balancing); execute the consistent hashing vs mod-N contrast and explain why the gap grows with N; choose the load-balancing strategy for Enlace; and justify eventual consistency with the nature of the data and the numbers.

What comes next is the final project. In lesson 8 you put the four artifacts together into a single deliverable: the project prompt, the rubric it is evaluated with, and a complete reference solution —the diagram, the executed capacity table, and the list of justified tradeoffs— so you can compare your design with a defensible one. And you close the guide: the summary of the eight modules, and where to continue in the ecosystem of sibling guides.

Resources

← Previous6. Step 4b — The read path and the cacheNext →8. Project: design Enlace end to end