Module 6: Load Balancing and Statelessness

7. Scaling horizontally by adding instances

Description

This lesson harvests the whole module. You have stateless servers (lessons 4-5), a balancer with spreading algorithms (lessons 2-3), and health checks that know who's alive (lesson 6). With those three pieces, you can finally do what the module promised from the title: scale horizontally by adding and removing instances at will. You'll see the contrast between vertical scaling (a bigger machine, with its ceiling) and horizontal (more machines, almost linear) applied to the app layer; the exact flow of adding an instance (it starts up → passes the readiness → the balancer puts it in rotation) and of removing one gracefully (connection draining: it stops receiving new requests, finishes the ones in flight, and only then turns off); and why statelessness is what makes all this routine instead of risk.

You'll compute, by running, how many instances Enlace needs: not the 4 of the absolute limit, but 6, to not exceed 70% utilization and —crucially— survive one instance going down without the others saturating in cascade. You'll see the table that shows why sizing "at the limit" is fragile: with 4 instances, losing one raises the load to 130% of the rest and everything goes down; with 6, losing one raises it to 78% and it holds. And you'll see the autoscaling: bringing instances up in the daytime peak and down in the early morning, paying only for what you use.

Connection to the module: this lesson is where the previous six become an operational capability. The number of instances you compute here is the one the project (lesson 8) will put in the sizing sheet. The statelessness (lessons 4-5) is what lets a new instance serve any request instantly; the health checks (lesson 6) are what put it in and take it out of the pool safely. The boundary: what the system does when an instance fails in the middle of a request —retry safely, idempotency— is the resilience guide; here we see how capacity is added and removed in a planned way, not how you react to a failure in flight.

A higher floor versus more buildings

Think of it this way. Your company grew and the office isn't enough. You have two ways to get more space, and they're the two ways to scale.

The first: build another floor on top of the building you have —vertical scaling—. It's comfortable: everything stays in one place, without changing anything about how you work. But it has a literal ceiling: you can't keep stacking floors forever —there's a structural limit, and each extra floor is more expensive and harder than the previous one—. And there's still a single building: if it burns down, it's all over. In computing, scaling vertically is buying a bigger machine (more CPU, more RAM). It works for a while, it's simple, but it hits the biggest hardware that exists, costs disproportionately at the high end, and leaves a single point of failure.

The second: open more identical buildings —horizontal scaling—. Each new building is the same as the others, and you add capacity almost without limit: need double? Open double the buildings. If one burns down, the others keep operating. The cost: you need something that distributes people among the buildings (a balancer) and for the buildings to be interchangeable (stateless) —no one can depend on being in a specific building—. In computing, scaling horizontally is adding more identical instances behind the balancer. It's what this module built, piece by piece, and it's why we demand statelessness: only interchangeable buildings can be multiplied freely.

The analogy's lesson: vertical is the easy reflex (a bigger machine) but it hits a ceiling and leaves a single point of failure; horizontal scales without a ceiling and tolerates failures, in exchange for the complexity —balancer, statelessness, health checks— you already paid in the previous lessons. For Enlace, with ~4,000 req/s and the need to tolerate crashes, horizontal is the answer, and now you have everything needed to use it.

How many instances Enlace needs, run

"At the limit" is no longer enough. Sizing the number of instances has two requirements: not exceeding a utilization ceiling in normal operation (to leave margin for spikes and variability), and surviving one instance going down without the rest saturating. Let's compute it with the anchor numbers:

# horizontal_scaling.py — how many instances Enlace needs, and what happens if one dies
import math

qps_read = 3858
qps_write = 39
qps = qps_read + qps_write                 # total load on the app tier
capacity = 1000                            # one instance handles ~1000 req/s
target_util = 0.70                         # don't exceed 70% in normal operation

# instances to not exceed the utilization target
N = math.ceil(qps / (capacity * target_util))
util = qps / (N * capacity)
util_if_one_dies = qps / ((N - 1) * capacity)

print(f"total load (qps)       = {qps:,} req/s")
print(f"capacity per instance  = {capacity:,} req/s")
print(f"instances (util<=70%)  = {N}")
print(f"utilization with {N}     = {util:.0%} per instance ({qps/N:,.0f} req/s each)")
print(f"if ONE dies -> {N-1} alive = {util_if_one_dies:.0%}  "
      f"({'HOLDS' if util_if_one_dies < 1 else 'SATURATES'})\n")

# Comparison: sizing at the limit (no slack) is fragile
print("sizing at the limit (packed, no slack):")
for n in (4, 5, 6, 7):
    u = qps / (n * capacity)
    u_dead = qps / ((n - 1) * capacity)
    verdict = "HOLDS" if u_dead < 1 else "SATURATES"
    print(f"  N={n}: util={u:4.0%}   if one dies: {u_dead:4.0%} -> {verdict}")

What to expect. With python horizontal_scaling.py:

total load (qps)       = 3,897 req/s
capacity per instance  = 1,000 req/s
instances (util<=70%)  = 6
utilization with 6     = 65% per instance (650 req/s each)
if ONE dies -> 5 alive = 78%  (HOLDS)

sizing at the limit (packed, no slack):
  N=4: util= 97%   if one dies: 130% -> SATURATES
  N=5: util= 78%   if one dies:  97% -> HOLDS
  N=6: util= 65%   if one dies:  78% -> HOLDS
  N=7: util= 56%   if one dies:  65% -> HOLDS

Here's the difference between a naive sizing and a robust one. The ~3,897 req/s over 1,000 per instance give 4 instances at the absolute limit (97% utilization). But look at the table below: with N=4, if one instance dies, the three remaining ones receive 130% of their capacity —they saturate, and since they saturate equally, they fall in cascade and all of Enlace goes down—. Sizing at the limit means the first crash brings everything down.

That's why the real number is 6, not 4. With 6 instances, the normal utilization is 65% (650 req/s each, comfortable margin for spikes), and —the important part— if one dies, the five remaining rise to 78%: they hold without saturating. The rule the table reveals: size so that N-1 instances hold the whole load, not just N. That margin —the one that separates 65% from 97%— is what turns one instance going down from a catastrophe into a non-event. The balancer, thanks to lesson 6's health checks, takes the dead instance out of the pool, and the five remaining absorb its load without breaking a sweat, precisely because you left slack.

Notice that this calculation depends on statelessness. That "the five remaining absorb the sixth's load" is only possible if any of the five can serve any request that was going to the sixth —that is, if they're interchangeable—. In a stateful system, the dead instance's sessions would be lost, and "absorbing its load" would also mean re-logging in all its users. The numeric slack only works over stateless servers. The whole module converges on this number.

The flow of adding and removing an instance

Scaling horizontally isn't just "how many"; it's how each instance comes in and out without any user seeing an error. Here's the flow, which uses the pieces from the previous lessons:

Adding an instance (scale-out):

1. the new instance starts up (app-6)         process alive, but not ready yet
2. app-6 opens connections to cache and database, warms what it needs
3. the balancer probes it: GET /healthz        -> readiness (lesson 6)
4. when the readiness passes, the balancer     puts it in rotation
5. app-6 starts receiving its share of the traffic

Step 3-4 is why lesson 6 matters here: the balancer does not send traffic to app-6 as soon as it starts, but when it confirms it's ready (readiness). If it sent traffic before, the first users would see errors while the instance is still warming. With statelessness, app-6 can serve any request from the first instant it's ready —it doesn't need to "recover" any session or prior state, because there's none that belongs to it—.

Removing an instance (scale-in) with connection draining:

1. you decide to remove app-6 (traffic drops, or maintenance)
2. the balancer STOPS SENDING it new requests   (takes it out of rotation)
3. app-6 FINISHES the requests it already had in flight   (connection draining)
4. when it has no active requests left, app-6 turns off

Step 3 —connection draining— is what makes the removal graceful. If you simply turned off app-6, the requests it had half-served would fail (the user would see an error). Instead, the balancer first stops sending it new requests, waits for the in-flight ones to finish, and only then the instance turns off. No user sees an error: the ones already being served by app-6 finish normally, and the new ones go to other instances. It's the difference between firing someone in the middle of a call and letting them finish the call before they leave.

These two flows are the basis of downtime-free deployment (rolling deployment): to update Enlace, you add instances with the new version, drain and remove the ones with the old version, one by one, and the service is never interrupted. All resting on the module's three pieces: statelessness (the instances are interchangeable), balancer (spreads during the transition), and health checks (it knows when the new one is ready).

Autoscaling: paying for what you use

The last gift of horizontal scaling is the elasticity: since adding and removing instances is routine, you can do it automatically according to demand. An autoscaler watches a metric (CPU utilization, requests per second, latency) and adjusts the number of instances: it goes up when the load grows, down when it eases.

For Enlace this is money directly. A URL shortener's traffic isn't flat: it has a daytime peak (when people browse and share links) and a nighttime valley (in the early morning almost no one visits). Without autoscaling, you'd have to provision for the peak 24 hours a day —paying in the early morning for a capacity no one uses—. With autoscaling, the number of instances follows the demand:

   req/s
   8000 ┤            ╭────╮              <- daytime peak: the autoscaler goes up to ~12 instances
   6000 ┤         ╭──╯    ╰──╮
   4000 ┤      ╭──╯          ╰──╮        <- average: 6 instances
   2000 ┤  ╭───╯                ╰───╮
      0 ┼──┴────────────────────────┴──  <- nighttime valley: goes down to ~3 instances
        0h    6h    12h    18h    24h

If the daytime peak doubles the average (~7,800 req/s), the autoscaler goes up to ~12 instances to keep the same 65% utilization; in the early morning, with a fraction of the traffic, it goes down to 3. You pay for the capacity you really use, not for the permanent peak. This is only possible because the instances are stateless: the autoscaler can turn off three instances in the early morning without losing anything, because none kept its own state; and turn them on at noon without "warming" any session, because any of them serves any request. The elasticity is statelessness turned into savings.

How exactly the autoscaler is configured —which metric, what up and down thresholds, how long to wait between adjustments to not oscillate (an instance flapping, cousin of lesson 6's)— is an operational topic of specific platforms (AWS Auto Scaling, Kubernetes HPA). The conceptual, which is this guide's, is this: with stateless servers behind a balancer with health checks, the number of instances can follow the demand automatically.

Common mistakes

Sizing at the limit, with no slack for one instance going down (capacity mistake). What happens: someone computes "4 instances at 1,000 req/s cover the ~3,900 req/s" and deploys 4. The day one instance dies, the three remaining receive 130% of their capacity, all three saturate at once, and Enlace goes down in cascade —the first crash brings everything down—. Why it happens: it's sized for the happy case (all alive) and the real case (one dies) is forgotten. How to detect it: if your normal utilization is above ~80%, you have no margin to lose an instance —the table showed it: N=5 at 78% barely holds one going down; N=4 at 97% doesn't—. How to fix it: size so that N-1 instances hold the whole load (65% with N=6), not just N. The slack isn't waste: it's what turns a crash into a non-event.

Turning off an instance without draining its connections (operational mistake). What happens: someone removes an instance (to scale down or deploy) by turning it off suddenly, and the requests it had half-served fail —the users see 502 errors during each scale-in and each deployment—. Why it happens: "removing an instance" is treated as "turning it off", skipping the draining. How to detect it: if you see a spike of errors every time you reduce instances or deploy, you're not draining. How to fix it: use connection draining: the balancer stops sending it new requests first, the instance finishes the in-flight ones, and only then it turns off. The graceful removal is what makes the scale-in invisible.

Believing you can scale a stateful service horizontally (precondition mistake). What happens: someone tries to autoscale a service that keeps sessions in each instance's RAM. When adding instances, the new ones don't have the sessions and fail requests; when removing instances, the sessions that lived in them are lost (massive logout). The autoscaling causes errors instead of resolving them. Why it happens: someone tries to harvest horizontal scaling without having paid its precondition, statelessness. How to detect it: if scaling up or down produces lost sessions or failed requests, the service isn't stateless. How to fix it: make the service stateless first (token or shared store, lessons 4-5), and then autoscale. Horizontal scaling is built on statelessness, not the reverse.

Exercises

Exercise 1 — Size with slack. A service receives 12,000 req/s and each instance handles 1,000 req/s. You want to not exceed 70% utilization in normal operation. (a) How many instances at the absolute limit? (b) How many for 70%? (c) With the 70% ones, if one dies, to what utilization do the rest rise and do they hold?

See solution

(a) At the limit: ceil(12,000 / 1,000) = 12 instances, at 100% utilization. No margin for anything.
(b) At 70%: ceil(12,000 / (1,000 × 0.70)) = ceil(17.1) = 18 instances. Real utilization = 12,000 / 18,000 = 67%.
(c) If one dies: 17 remain, receiving 12,000 / 17,000 = 71% utilization. They hold (below 100%), with margin still. Note that the margin to survive one instance going down is more comfortable the more instances you have: losing 1 of 18 raises the load much less (67%→71%) than losing 1 of 4 (97%→130%). Large pools tolerate individual crashes better.

Exercise 2 — Add and remove gracefully. Order the correct steps to (a) add a new instance and (b) remove an instance, using these pieces: "the balancer takes it out of rotation", "passes the readiness check", "finishes the in-flight requests", "the process starts up", "the balancer puts it in rotation", "turns off".

See solution

(a) Add an instance:

The process starts up (the instance is alive, but not ready yet).
Passes the readiness check (confirms it can serve: cache and database connections ready).
The balancer puts it in rotation (only now, when it's ready, it starts receiving traffic).

(b) Remove an instance:

The balancer takes it out of rotation (stops sending it new requests).
Finishes the in-flight requests (connection draining: the ones it had half-served are completed).
Turns off (when it has no active request left, without any user seeing an error).

The key in both: the traffic is turned on after confirming readiness (add) and cut before turning off (remove). You never send traffic to a not-ready instance, or turn off one with requests in flight.

Exercise 3 — Vertical or horizontal. For each situation, say whether it's best to scale vertically (bigger machine) or horizontally (more instances) and why. (a) Enlace goes from 4,000 to 40,000 req/s due to a campaign, and needs to tolerate instances going down without interrupting the service. (b) A database that needs more RAM for its internal cache and can't be sharded easily yet. (c) A stateless app service whose traffic varies 10× between day and night.

See solution

(a) Horizontal. 10× more load and fault tolerance: vertical hits the ceiling of the biggest machine and leaves a single point of failure. Horizontal adds instances almost without limit and survives crashes —and Enlace is already stateless with a balancer, so it's trivial—. It's the central case of this module.
(b) Vertical (for now). A database that needs more RAM and can't yet be sharded is scaled vertically (a bigger machine) as an intermediate step —until the module 5 sharding is inevitable—. Not everything is scaled horizontally immediately; state (databases) is harder to spread than the stateless app layer.
(c) Horizontal, with autoscaling. A stateless service whose traffic varies 10× is the ideal case of elasticity: you bring instances up by day, down by night, and pay for what you use. Vertical would force you to pay the big machine 24 hours a day. Statelessness makes the savings possible.

Summary and next step

In this lesson you harvested the module: with statelessness, a balancer, and health checks, you scaled Enlace horizontally by adding and removing instances at will. With the higher-floor-versus-more-buildings analogy you saw why vertical scaling hits a ceiling and leaves a single point of failure, while horizontal scales without limit and tolerates crashes —in exchange for the complexity you already paid—. You computed, by running, that Enlace needs 6 instances, not the 4 of the limit: sizing at the limit (97% utilization) makes one instance going down saturate the rest in cascade (130%), while 6 instances (65%) survive one going down (78%) without breaking a sweat. You saw the flow of adding (starts up → readiness → rotation) and removing with connection draining (out of rotation → drains in-flight → turns off), the basis of downtime-free deployment. And you saw the autoscaling: following demand automatically —up by day, down by night—, savings that only statelessness makes possible.

Before moving on you should be able to: contrast vertical and horizontal scaling with their tradeoffs; explain why you size so that N-1 instances hold the load; describe the flow of adding (readiness) and removing (draining) an instance; and explain why autoscaling depends on statelessness.

What comes next is putting the six lessons together into a single deliverable. In lesson 8 —the module's project— you'll design Enlace's complete balancing layer: you choose the algorithm, confirm the stateless design, size the instances with slack, configure the health checks, define how you add and remove capacity, and make the balancer redundant so it's not a single point of failure. You deliver the topology diagram, the run sizing sheet, and the list of tradeoffs, with the plan for the day an instance dies. It's where the six pieces become a defensible design.

Resources

System Design Primer — "Horizontal scaling" and "Application layer" — the summary of horizontal versus vertical scaling and their tradeoffs (vertical's ceiling, the need for statelessness and a balancer for horizontal), in the context of a complete design. This lesson's conceptual review.
AWS — Auto Scaling groups and health checks — how a production system implements this lesson's autoscaling: groups of instances behind a balancer, health checks that replace sick instances, and policies that bring capacity up and down according to demand. The bridge from the concepts to the real operation.
Designing Data-Intensive Applications, Martin Kleppmann — Chapter 1, "Scaling out" (shared-nothing) and "Coping with load" — the framework of scaling out with shared-nothing architectures (no shared state on the nodes), which is exactly what this module's statelessness enables. The guide's canonical reference to close the scaling arc.

← Previous6. Health checks Next →8. Project: design Enlace's load balancing layer