8. Project: design Enlace's load balancing layer

Description

The moment has come to bring the seven lessons together into a single deliverable. In this project you're going to do what an engineer does when it's time to put a balancing layer in front of a read-heavy service: design it with judgment and with numbers. Not "let's add an nginx and see what happens", but a defensible design that answers six concrete questions and backs them with arithmetic that runs:

1. What balancing algorithm do I choose, and why that one?
2. Do I confirm the servers are stateless, and where does any state live?
3. How many instances do I provision? (with slack to survive one going down)
4. How do I configure the health checks? (type, thresholds, what they check)
5. How do I add and remove capacity? (readiness, connection draining, autoscaling)
6. How do I keep the balancer from being a single point of failure?

The deliverable is a topology diagram, a sizing sheet (a table with the numbers, produced by a calculator you run), and a list of justified tradeoffs, plus the plan for the day an instance dies. By the end you'll have a template that serves to design the balancing layer of any stateless service, not just Enlace's.

Connection to the module: this is the capstone. It uses the balancer from lesson 2, the algorithms from lesson 3, the statelessness from lesson 4, the session from lesson 5, the health checks from lesson 6, and the scaling from lesson 7 —all at once— to produce a real artifact. It's also the bridge to module 7: when you finish designing the balancing layer, you'll see there are unclosed reliability questions —how the balancer is made redundant, what a single point of failure is, how consistency is reasoned about—, and those are exactly module 7's territory. And with this layer ready, Enlace finally has its four scaled layers (cache, database, balancing, and the reliability that's coming), which module 8 will assemble end to end.

The project brief

You're the engineer in charge of Enlace's balancing layer. The team gives you the anchor numbers —the same ones from the whole guide— and asks you for a proposal. These are the input data, fixed:

Input data	Value	Where it comes from
Reads per second (qps_read)	~3,858	100:1 ratio (module 2)
Writes per second (qps_write)	~39	module 2
Total load on the app tier	~3,897 req/s	qps_read + qps_write
Capacity per instance	~1,000 req/s	measured under load (one app instance)
Nature of the resolve	anonymous redirect, uniform	module 3 and lesson 4
Daytime peak factor	~2× the average	a shortener's traffic pattern

And these are the decisions you make (the design levers): the balancing algorithm, the state handling, the utilization ceiling and the number of instances, the health checks configuration, the capacity add/remove strategy, and the balancer's redundancy. The project consists of choosing each one, justifying it, and computing its consequences.

The design steps

Before seeing the reference solution, here's the process. Do it yourself first; the solution is after so you can compare.

Step 1 — Choose the balancing algorithm

Look at lesson 3's decision table. Enlace's resolve is a uniform redirect (all requests cost about the same) and the traffic can concentrate in viral links (a dominant key). Which of the three algorithms fits, and which do you discard why?

Step 2 — Confirm the statelessness and place the state

Go through Enlace's operations and confirm they're stateless (lesson 4). Decide what happens if Enlace adds a dashboard with accounts: where does that session live (lesson 5)? Make clear that the hot path touches none of the three session options.

Step 3 — Size the instances with slack

Apply lesson 7: not the ones at the limit, but the ones that keep the utilization under a ceiling and survive one instance going down. Choose the utilization ceiling and compute N. Verify the "one dies" scenario.

Step 4 — Configure the health checks

From lesson 6, decide: active, passive, or both; the thresholds (down_threshold, up_threshold) to avoid flapping; and what the /healthz checks (light liveness, not a deep check that brings down the pool).

Step 5 — Define the capacity add and remove

From lesson 7, describe how an instance comes in (readiness before rotation) and how it goes out (connection draining), and whether you use autoscaling for the daytime peak.

Step 6 — Make the balancer redundant

From lesson 2, close the single point of failure: the balancer can't be a single machine. Sketch the redundancy (the failover detail is module 7).

Reference solution

Here's a complete and defensible design. It's not the only correct answer —another utilization ceiling or least-connections instead of round-robin are also defensible—, but it's a solid proposal with each decision justified. First the calculator that produces the sizing sheet:

# enlace_lb_sizing.py — size Enlace's balancing layer, run
import math

# --- Fixed inputs (anchor numbers) ---
qps_read = 3858
qps_write = 39
qps = qps_read + qps_write

# --- Design decisions ---
capacity = 1000            # req/s an app instance handles (measured under load)
target_util = 0.70         # utilization ceiling in normal operation
peak_factor = 2.0          # the daily peak doubles the average

# Step 1: instances for the steady state, with slack
N_steady = math.ceil(qps / (capacity * target_util))
util_steady = qps / (N_steady * capacity)
util_one_dead = qps / ((N_steady - 1) * capacity)

# Step 2: instances for the daily peak (autoscaling goes up to here)
qps_peak = qps * peak_factor
N_peak = math.ceil(qps_peak / (capacity * target_util))

print("=== Sizing sheet — Enlace's balancing layer ===\n")
print(f"total load (average)    : {qps:,} req/s  ({qps_read:,} read + {qps_write} write)")
print(f"capacity per instance   : {capacity:,} req/s")
print(f"utilization ceiling     : {target_util:.0%}\n")
print(f"instances (steady state) : {N_steady}")
print(f"  utilization            : {util_steady:.0%}  ({qps/N_steady:,.0f} req/s per instance)")
print(f"  if one dies ({N_steady-1} alive)  : {util_one_dead:.0%}  -> "
      f"{'holds' if util_one_dead < 1 else 'saturates'}\n")
print(f"instances at the daily peak ({peak_factor:.0f}x = {qps_peak:,.0f} req/s) : {N_peak}")
print(f"  the autoscaling goes from {N_steady} (night) to {N_peak} (peak)")

What to expect. With python enlace_lb_sizing.py:

=== Sizing sheet — Enlace's balancing layer ===

total load (average)    : 3,897 req/s  (3,858 read + 39 write)
capacity per instance   : 1,000 req/s
utilization ceiling     : 70%

instances (steady state) : 6
  utilization            : 65%  (650 req/s per instance)
  if one dies (5 alive)  : 78%  -> holds

instances at the daily peak (2x = 7,794 req/s) : 12
  the autoscaling goes from 6 (night) to 12 (peak)

The topology diagram

The architecture you'd deliver to the team, with the decisions marked:

flowchart TD
    C[clients] --> DNS[enla.ce · DNS]
    DNS --> LB1[active LB]
    DNS -.failover.-> LB2[passive LB]
    LB1 -->|round-robin| S0[app-0]
    LB1 -->|round-robin| S1[app-1]
    LB1 -->|round-robin| S2[app-2 ... app-5]
    LB1 -.GET /healthz.-> S0
    S0 --> Cache[(cache · M4)]
    S1 --> Cache
    S2 --> Cache
    Cache --> DB[(primary + replicas + shards · M5)]

Read it: the clients arrive at enla.ce, which resolves by DNS to an active balancer (with a passive one ready for failover). The balancer spreads by round-robin among 6 stateless app instances, which it probes with /healthz. The instances share the cache (module 4) and the scaled database (module 5) below. All of module 6 is in the band between the DNS and the app instances.

The sizing sheet

The numbers and the decisions, presented as the deliverable:

Decision	Choice	Justification
Algorithm	Round-robin	resolve is uniform (least-conn adds nothing) and there are viral links (the hash would concentrate them); round-robin spreads blindly and evenly (lesson 3)
Statelessness	Confirmed; no session on the hot path	resolve/shorten keep no client state; the state lives in the shared cache+DB (lesson 4)
Session (if there's a dashboard)	Signed token + Redis if revocation is needed	Never sticky (imbalances 48.7%) or server RAM (lesson 5)
Instances (steady)	6	65% utilization; survives one going down (rises to 78%, holds), not the 4 of the limit (lesson 7)
Utilization ceiling	70%	Margin for spikes and to absorb one instance going down without saturating
Health checks	Active + passive; down=3, up=2; light liveness /healthz	Thresholds that avoid flapping (6 changes with threshold 1 → 2 with threshold 3); light check to not bring down the pool (lesson 6)
Add/remove	Readiness before rotation; connection draining on exit	No user sees errors when scaling or deploying (lesson 7)
Autoscaling	6 (night) ↔ 12 (daytime peak 2×)	Elasticity: pay for what you use; possible for being stateless (lesson 7)
LB redundancy	Two active-passive balancers, floating IP	The balancer can't be a SPOF; failover in depth → module 7 (lesson 2)

How each decision is defended

• Round-robin and not least-connections or hash. The resolve is uniform, so least-connections adds nothing (there are no "long" requests to avoid) and adds complexity. The hash would worsen things: a viral link is a dominant key the hash would send all to the same server (lesson 3). Round-robin spreads blindly and evenly, and immunizes against viral links by diluting their traffic among the 6 instances. It's the textbook choice for uniform requests with possible hot keys.
• 6 instances and not 4. The 4 of the limit cover the ~3,897 req/s at 97%, but one going down raises them to 130% and everything falls in cascade. With 6, the normal utilization is 65% and one going down rises to 78% —it holds—. The margin between 65% and 97% is what turns one going down from a catastrophe into a non-event (lesson 7). We size so that N-1 hold the whole load.
• Light health check with down=3, up=2. A threshold of 3 failures to expel avoids flapping (a flapping server would cause 6 state changes with threshold 1, only 2 with threshold 3). The /healthz checks only that the process responds (liveness), not that it can reach the database —a deep check would bring down the whole pool if the database had a hiccup— (lesson 6). The slow-dependency protection is left to the resilience guide's circuit breakers.
• Redundant balancer. A single balancer would be the single point of failure: if it goes down, the 6 healthy servers become unreachable. Two in active-passive with a floating IP that jumps to the healthy one close that hole (lesson 2). The detection and failover mechanism is module 7's; the decision to not have just one is made here.

What happens the day an instance dies

A good design doesn't only describe the good day; it anticipates the bad one. Include this section in your delivery, because it's what distinguishes a naive proposal from a robust one. The scenario: one of the 6 instances dies suddenly (a crash, a hardware failure).

• The health check (lesson 6) detects it after 3 failed probes (a few seconds with probing every 2 s) and the balancer takes the instance out of the pool.
• The 5 remaining absorb its load: they go from 650 to 780 req/s each, from 65% to 78% utilization —they hold, because you sized with slack (lesson 7)—.
• No user sees a sustained error: the requests that were going to the dead instance are re-routed to the live ones, which have margin. This only works because the instances are stateless (lesson 4): any of the 5 serves any request that was going to the 6th, without needing to recover a session or state.
• The autoscaler (lesson 7), seeing the utilization rise, can launch a replacement instance, which enters rotation after passing its readiness, returning the pool to 6.

And the day the active balancer dies: the passive takes its place (floating IP), and the service continues. The detail of how it's detected and how the failover is done without split-brain is module 7 —here it's enough that the redundancy is in the design—.

Common mistakes

Choosing the "most sophisticated" algorithm instead of the right one (design mistake). What happens: someone chooses least-connections or hash "because they sound more advanced than round-robin", and ends up with more complexity and no benefit (least-conn) or with an imbalance due to viral links (hash). Why it happens: sophistication is confused with suitability. How to detect it: if you can't name what concrete problem your algorithm resolves in this service, you chose by fashion. How to fix it: for Enlace's uniform resolve, round-robin is the correct answer precisely for being simple and blind; the sophistication is justified only when the traffic demands it (unequal-cost requests → least-conn; locality → hash).

Sizing the balancing layer by the average, with no slack or peak (capacity mistake). What happens: someone provisions 4 instances "because they cover the 3,897 req/s average" and considers neither one going down nor the daytime peak. The first failure brings down the system, and the 3 p.m. peak saturates it. Why it happens: it's sized for the average in the happy case. How to detect it: if your design doesn't say what happens when an instance dies and what happens at the peak, it's half-done. How to fix it: size so that N-1 hold the average (6 instances at 65%) and have the autoscaling that goes up at the peak (to 12). The average in the happy case isn't a sizing, it's an illusion.

Forgetting the balancer's redundancy (reliability mistake). What happens: someone designs 6 perfect instances behind a balancer, feels safe for the 6, and the day the balancer restarts, all of Enlace goes down despite the 6 healthy instances. Why it happens: attention is concentrated on the pool and the piece everything passes through is forgotten. How to detect it: ask yourself "which machine, if it dies alone, brings everything down?". If it's the balancer, you have a SPOF. How to fix it: the balancer is deployed redundantly (active-passive with a floating IP), just as you made the instances redundant. A design with a single balancer is a design with a single point of failure, however many instances it has behind it.

Exercises

Exercise 1 — Resize for double the traffic. A campaign takes Enlace from ~3,897 to ~7,800 req/s sustainedly (not a passing peak, but the new average). With a capacity of 1,000 req/s per instance and a 70% ceiling: (a) how many instances do you need now? (b) Do they survive one going down? (c) What other piece of the design has to be reviewed with double the traffic?

See solution

• (a) ceil(7,800 / (1,000 × 0.70)) = ceil(11.1) = 12 instances. Utilization = 7,800 / 12,000 = 65%, the same comfortable margin as with 6 at the original load (the design scales linearly).
• (b) Yes. If one dies, 11 remain with 7,800 / 11,000 = 71% —they hold below 100%—. In fact, with more instances one going down hurts less (65%→71%), as you saw in lesson 7: large pools tolerate individual crashes better.
• (c) The balancer. With double the traffic you have to verify that the balancer (and its passive partner) handles ~7,800 req/s passing through it —unlike the app instances, the balancer doesn't multiply as easily, so its capacity is a limit to watch—. It's also worth reviewing that the cache and the replicas (modules 4-5) absorb the downstream load. Scaling the app layer is useless if the bottleneck moves to the balancer or the database.

Exercise 2 — Defend the design against three objections. A colleague questions your proposal. Answer each objection in a couple of sentences, with the module's material. (a) "Why round-robin and not least-connections, which is smarter?" (b) "Why 6 instances if 4 are enough for the 3,897 req/s?" (c) "Why doesn't the /healthz check the database, to be sure the server really works?"

See solution

• (a) Round-robin and not least-connections: because Enlace's resolve is uniform —all requests cost about the same—, so there are no "long" requests for least-connections to avoid; its intelligence buys nothing here and adds complexity. Round-robin, moreover, is immune to viral links (it spreads each visit by turns, without concentrating), which the hash would worsen. Simple is correct when the traffic is uniform.
• (b) 6 and not 4: because 4 cover the average at 97%, but one going down raises them to 130% and the system falls in cascade. With 6 (65%), one going down rises to 78% and it holds. We don't size for "all alive", but for "N-1 hold the whole load" —the slack is what survives the bad day—.
• (c) The /healthz doesn't check the database: because if each server checked the database, a transient hiccup of it would make the health check of all of them fail at once, and the balancer would take out the whole pool —a total crash over a minor problem of a shared dependency—. The health check verifies that the server responds (liveness); the database's health is handled with timeouts and circuit breakers (resilience guide), not by putting it in each instance's check.

Exercise 3 — The plan for the bad day. Write, in five steps, what happens from the moment one of Enlace's 6 instances dies until the pool returns to normal, naming which module piece acts in each step.

See solution

1. The instance dies (crash or hardware failure). Its in-flight requests are lost; the new ones would keep being routed to it until the balancer finds out.
2. The health check detects it (lesson 6): after 3 consecutive failed probes (down_threshold=3, a few seconds with probing every 2 s), the balancer takes the instance out of the pool.
3. The 5 remaining absorb the load (lesson 7): they go from 65% to 78% utilization (650 → 780 req/s each). They hold, because the sizing with slack foresaw it.
4. No user sees a sustained error (lesson 4): since the instances are stateless, any of the 5 serves any request that was going to the dead one, without recovering a session or state. The re-routing is transparent.
5. The autoscaler launches a replacement (lesson 7): seeing the utilization rise, it starts a new instance that, after passing its readiness check, enters rotation and returns the pool to 6 at 65%. Normality restored.

(And if the one that dies is the active balancer, the passive takes its place by floating IP —lesson 2—, with the failover detailed in module 7.)

Summary and next step

In this project you designed Enlace's complete balancing layer, the module's capstone. You started from the anchor numbers and produced a defensible design: round-robin (the resolve is uniform, with viral links the hash would worsen), confirmed stateless servers (no session on the hot path; token+Redis only if a dashboard arrives), 6 instances at 65% utilization (surviving one going down by rising to 78%, not the 4 of the limit that fall in cascade), light health checks with anti-flapping thresholds (down=3, up=2), add with readiness and remove with connection draining, autoscaling from 6 to 12 at the peak, and a redundant balancer so it's not a single point of failure —each decision justified with the arithmetic and the experiments of the seven lessons—. And you added what separates a naive proposal from a robust one: the step-by-step plan for the day an instance dies.

With this you close the balancing and statelessness module. You know what a balancer is (lesson 2), how it spreads (lesson 3), why the servers must be stateless (lesson 4), where the session lives (lesson 5), how the balancer knows who's alive (lesson 6), and how capacity is added and removed (lesson 7). And you know how to design the whole layer, which is the skill that ties everything else together.

What comes next are the questions this design left open, and they're all about reliability. You made the balancer redundant, but you didn't say how it's detected that the active one died or how the passive takes over without both believing they're the active. You sized to survive one instance going down, but you didn't formalize what a single point of failure is or how consistency is reasoned about when there are replicas that are behind. In module 7 —reliability and the consistency tradeoff— you'll close those questions: redundancy and failover, CAP and PACELC, strong vs. eventual consistency (and why eventual is enough for Enlace's redirects), and the SLA/SLO as a number. It's where the distributed design you built over six modules becomes a reliable design.

Resources

• System Design Primer — Load balancer + Application layer (comprehensive design) — the comprehensive review of everything you designed here (balancer, algorithms, statelessness, health checks, horizontal scaling) in the context of a complete system design. Ideal for reviewing the whole module at a glance before moving on to reliability.
• nginx / HAProxy — reference configuration (upstream, balance, check, drain) — the documentation of a real balancer where each decision in this sheet has its directive: the algorithm (round-robin/least_conn/ip_hash), the health checks (max_fails, fail_timeout), and the draining (drain). The bridge from your design on paper to a deployable configuration.
• Designing Data-Intensive Applications, Kleppmann — Chapter 1 (Reliability, Scalability) and a preview of Chapter 9 (Consistency) — chapter 1 backs all the load and scaling reasoning of this project; chapter 9 (consistency and consensus) is the direct bridge to module 7, where you'll formalize the reliability and the consistency tradeoff this design left open.