1. Module introduction: the last bottleneck

Description

By the end of this lesson you'll know exactly which assumption this module breaks and why it's the last one still standing. Over five modules you scaled almost all of Enlace. In module 1 you drew it as a service with a database beside it; in module 2 you put numbers on it —100 million new URLs a month, ~40 writes per second, ~4,000 reads per second—; in module 3 you modeled the Link record and generated the short_code; in module 4 you put a cache in front to absorb the hot reads; and in module 5 you scaled the database with replicas and sharding. But in all those modules there was a box you never questioned: the application server. That server —the one that receives each HTTP request, looks at the cache, talks to the database, and responds— was always a single one. This module is where that last assumption breaks: a single app server has a ceiling, and Enlace receives more requests than that ceiling can handle.

The solution has two inseparable halves, and that's why the module is called "load balancing and statelessness". The first half is obvious: if one server isn't enough, put several and spread the requests among them with a load balancer. The second half is the one almost nobody explains and the one that separates a design that works from one that goes down: to be able to spread the requests at will —send this one to one server and the next to another—, the servers have to be stateless, that is, they can't keep in their memory anything about the user that they can't reconstruct. If server A remembers that "Ana logged in" and Ana's next request falls on server B, which knows nothing about Ana, everything breaks. Balancing and statelessness are two sides of the same coin: you can't have one without the other. This module gives you both.

Connection to the module: this lesson is the map, not the territory. Here you don't balance anything yet; you understand the order of the six lessons that follow and why they go that way. Lesson 2 defines what a balancer is and why it's the single point of entry to the pool. Lesson 3 runs the three spreading algorithms (round-robin, least-connections, by hash) and measures which wins when. Lesson 4 attacks statelessness head-on: why state in the server prevents scaling, run. Lesson 5 answers where the state should live instead: the session, sticky vs. token. Lesson 6 shows how the balancer knows which server is alive: the health checks. And lesson 7 puts it all together —stateless + LB + health checks— to add and remove instances without drama. Lesson 8, the project, asks you to design Enlace's complete balancing layer.

The supermarket with a single register

Think of it this way. Imagine a supermarket that opened with a single cash register. At first it's perfect: a customer comes in, pays, leaves; the next comes in. The single cashier serves everyone without a line forming.

One day the supermarket becomes popular and four times more customers arrive. Now the single register is the bottleneck: the line goes around the aisle, and it doesn't matter how much product you have on the shelves or how many stockers work in the back —the funnel is at the register—. The solution is evident: open more registers. You put four identical cash registers and hire four cashiers. But opening registers isn't enough: if customers choose a line at random, some registers saturate while others are empty. You need someone standing at the entrance to the registers —the one who says "you to register 3, which is free; you to register 1"—. That person who distributes customers among the registers is the load balancer.

And here comes the subtle part, the statelessness one. For that distribution to work, any register has to be able to serve any customer. If register 2 were the only one with the price list, or the only one that remembers a customer's cart, then you couldn't send that customer to register 4 —they'd always have to go back to 2, and 2 would saturate just like before—. The registers work because they're interchangeable: they all have the same scanner, they all charge the same, none keeps anything special about a specific customer. That interchangeability is exactly what "stateless" means for a server. A stateless server is a register that can serve anyone, and that's why you can open and close registers at will according to the line. A stateful server is a register with something unique inside, and that's why you can't distribute freely. The whole module fits in that image: to spread the load among servers, the servers have to be interchangeable.

The case: Enlace's numbers that force the decision

We don't start from scratch: we start from the anchor numbers you computed in module 2. And as throughout the guide, we don't quote them from memory —we reproduce them— and compute with them how many app servers are needed:

# How many app servers Enlace needs, reproduced (not quoted)
import math

writes_per_month = 100_000_000
seconds_per_month = 30 * 24 * 3600            # 2,592,000 s

qps_write = writes_per_month / seconds_per_month
qps_read = qps_write * 100                     # read:write ratio = 100:1
qps_total = qps_read + qps_write

capacity_per_server = 1000                     # one instance handles ~1,000 req/s
servers_packed = math.ceil(qps_total / capacity_per_server)

print(f"qps_write = {qps_write:.0f} writes/s")
print(f"qps_read  = {qps_read:,.0f} reads/s")
print(f"qps_total = {qps_total:,.0f} req/s")
print(f"servers at the limit = {servers_packed}  (at {capacity_per_server}/s each)")

What to expect. When you run it:

qps_write = 39 writes/s
qps_read  = 3,858 reads/s
qps_total = 3,897 req/s
servers at the limit = 4

Four servers at the absolute limit, each digesting 1,000 requests per second. But "at the absolute limit" is a trap: if one of the four goes down, the other three receive ~1,300 req/s each —above their capacity— and also go down, in cascade. That's why in lesson 7 you'll see that the real number, with slack to survive one instance going down, is 6, not 4. What matters now is the conceptual leap: we go from "one app server" to "a pool of app servers with a balancer in front". That pool is the heart of this module.

Notice also what the cache and the replicas did not resolve. The module 4 cache protects the database, not the app server: each of those ~4,000 reads still reaches an app server, which then looks at the cache. The module 5 replicas scale the database, not the app server. The app server is a different layer, with its own bottleneck, and this module is the one that scales it.

The two layers that spread load (and not confusing them)

Here's a classic confusion worth disarming from the start. In module 5 you "spread load" among databases (replicas, shards). In this module you "spread load" among app servers. It sounds the same, but they're two different layers, with two different distributors:

                 client
                   │
                   ▼
          ┌─────────────────┐
          │  LOAD BALANCER  │   <── this module (M6): spreads HTTP requests
          └─────────────────┘
             │      │      │
             ▼      ▼      ▼
          app-0  app-1  app-2   <── pool of STATELESS app servers
             │      │      │
             └──────┼──────┘
                    ▼
          ┌─────────────────┐
          │   cache (M4)    │
          └─────────────────┘
                    │
                    ▼
          primary + replicas + shards   <── module 5: spreads data and reads

• The load balancer (this module) spreads incoming HTTP requests among interchangeable app servers. It lives between the client and the app servers.
• The replicas and shards (module 5) spread data and queries among databases. They live below the app servers.

Both "spread load", but at different layers and for different reasons. Confusing them leads to mistakes like "I already have replicas, why balance?" (the replicas don't serve HTTP requests) or "I put a balancer in front of the database" (which is sometimes done, but it's not what this module is about). Here, "balancer" always means the one in front of the app servers.

The order of the six lessons, and why it's that

Lesson	Piece	The problem it solves, in one sentence
2	The balancer	A point of entry that spreads requests among N servers
3	The algorithms	How the balancer chooses which server to send each request to
4	Statelessness	Why state in the server prevents spreading freely
5	The session	Where the user's state lives if not in the server
6	Health checks	How the balancer knows which server is alive
7	Scaling horizontally	Adding and removing instances at will, now that it's possible

We start with what a balancer is (lesson 2), because it's the new physical piece. Then, how it spreads (lesson 3): the three algorithms, run, to know which to choose. Then we turn to the condition that makes all this possible: statelessness (lesson 4), and its corollary, where the session lives (lesson 5) —because the state has to live somewhere, and choosing that somewhere well is half the battle—. Lesson 6 gives the balancer the eyes it needs to not send traffic to a dead server: the health checks. And lesson 7 harvests all of the above: with stateless servers, a balancer, and health checks, adding or removing instances stops being a risky event and becomes routine. Lesson 8 —the project— puts the six pieces together into the design of Enlace's balancing layer.

What this module does NOT touch

It's good to mark the boundary from now, because there are neighboring topics that seem to belong here and belong to another guide in the ecosystem.

The resilience patterns in depth are not this module's. Here you'll mention that a server can go down and that the balancer takes it out of rotation, because it's impossible to talk about a pool without naming it. But how a server protects itself from a slow dependency (circuit breaker), how it isolates one type of traffic so it doesn't sink the rest (bulkhead), how it retries a failed request without amplifying the problem (retry with backoff), and idempotency in depth —all of that is the sibling guide resilience-and-reliability-patterns-guide. When in lesson 6 I say "the health check takes out the sick server", that sentence is a door to that guide; I point it out to you, I don't cross it here.

The database replicas and sharding were already module 5. In this module, "scaling" means scaling the app server (more instances behind the balancer). Scaling the database —read replicas, partitioning the data, consistent hashing— was module 5. You'll see them together in the project's diagram, but they're different layers and we don't repeat module 5 here.

The consistency models and formal reliability are module 7. In this module the balancer and the redundancy appear as concrete mechanisms. But the complete framework —what a single point of failure is, CAP, PACELC, strong vs. eventual consistency, SLA/SLO as a number— is module 7. Here you touch the balancer's redundancy in passing; there you formalize it.

Common mistakes

Believing that "adding more servers" is enough (mental-model mistake). What happens: someone says "the server can't handle it, I'll put four" and deploys four copies without a balancer in front and without making them stateless. Without a balancer, there's no one to spread the traffic —DNS sends everything to one IP—; without statelessness, spreading breaks the sessions. Why it happens: "scaling horizontally" sounds like a single action (cloning), when it's three (clone, put a balancer, make stateless). How to detect it: if you can't name who spreads the requests and where the state lives, you haven't designed the layer yet, you just copied the binary. How to fix it: it's exactly what the six lessons of this module do —the balancer (2-3), the statelessness (4-5), the operation (6-7)—.

Confusing app balancing with database scaling (concept mistake). What happens: someone believes the module 5 replicas already "balance" and nothing else is needed, or puts a balancer expecting it to resolve a bottleneck that's actually in the database. Why it happens: both "spread load" and it's easy to blur them. How to detect it: ask yourself "what's saturated, the app server —CPU at 100% serving requests— or the database —slow queries, full disk—?". If it's the first, it's this module; if it's the second, it was module 5. How to fix it: the two-layers diagram from this lesson; the balancer lives in front of the app servers, the replicas below.

Scaling the app tier before needing it (sequence mistake). What happens: a team deploys ten app servers and a balancer on launch day, when the real traffic is 5 req/s and a single server would have been enough for years. Why it happens: "designing for scale" is confused with "building for scale" from minute zero. How to detect it: if your single server is at 5% CPU, adding a pool and a balancer is premature complexity —more machines to operate, more things that can break—. How to fix it: the module's rule is the same as module 5's: scale when the number demands it. Enlace does demand it (~4,000 req/s against ~1,000 per server), and that's why this module applies; a small project would wait.

Exercises

Exercise 1 — Which layer is saturated? For each Enlace symptom, say whether the bottleneck is in the app server (this module, M6, scales it) or in the database (module 5 scaled it), and which tool it calls for. (a) The single app server's CPU is at 100% and the requests queue up, even though the database queries are still fast. (b) The resolve queries take 400 ms because the database's disk is saturated, even though the app server is at 30% CPU. (c) The traffic quadrupled and you want to be able to add and remove app servers according to the time of day.

See solution

• (a) App server. CPU at 100% serving requests, with a healthy database, is the bottleneck of the app layer. The tool is a pool of app servers behind a balancer (this module): spread the requests among several instances.
• (b) Database. The app server is idle (30%) and the bottleneck is the database's disk: it's the axis module 5 scaled (read replicas, sharding). Adding app servers here wouldn't help —they'd all hit the same saturated database—.
• (c) App server (elasticity). Being able to add and remove instances according to demand is horizontal scaling of the app tier, and it requires a balancer with health checks and stateless servers: exactly what this module builds, with lesson 7's autoscaling.

Exercise 2 — Interchangeable or not. For each server, say whether it's stateless (interchangeable, can be cloned and distributed freely) or stateful (keeps something unique, not freely distributed), and a sentence why. (a) A resolve server that, for each short_code, looks at the cache and the database and returns the long_url. (b) A server that keeps in its RAM the shopping cart of each user who visited it. (c) A server that keeps the login sessions in a shared Redis, not in its own RAM.

See solution

• (a) Stateless. It keeps nothing about the user between requests: each resolve is self-sufficient (the short_code arrives, it's looked up, it's answered). Any replica of this server gives the same answer, so the balancer can send each request to any of them. It's exactly Enlace's case.
• (b) Stateful. The cart lives in that server's RAM. If the user's next request falls on another server, the cart "disappears". It can't be distributed freely —the user would always have to go back to the same server—, and that's what breaks the scaling (lesson 4).
• (c) Stateless. Watch this one: the state exists (the sessions), but it doesn't live in the server, it lives in a shared Redis all the servers query. That's why the server is still interchangeable: any of them can serve any request because they all read the session from the same place. It's the "shared store" option of lesson 5.

Exercise 3 — The order matters. A colleague proposes: "For Enlace, let's first put a balancer with ten app servers, and later we'll see if we make them stateless". With what you know from this module, explain why that order is a mistake and what the correct one would be.

See solution

The mistake: balancing without statelessness doesn't work. The moment the balancer spreads requests among ten servers, any state kept in one server's RAM becomes inaccessible to the other nine. If a user logs in on server 3 and their next request falls on 7, 7 doesn't know who they are —the session "is lost"—. The balancer causes the problem statelessness prevents; you can't add the first without having resolved the second. Balancing and statelessness aren't two sequential steps, they're two simultaneous requirements.

The correct order: first design the servers to be stateless (they don't keep user state in their RAM: the state goes to a signed token or a shared store), and then put the balancer that spreads them. In Enlace this is almost free, because resolve and shorten are already stateless by nature —there's no session to lose in an anonymous redirect—. That's exactly why Enlace clones at will, and why this module uses it as a case: it's the clean example of a service that was born interchangeable.

Summary and next step

In this lesson you understood the last assumption still standing: that a single application server serves all of Enlace. With the single-register supermarket you saw that the solution has two inseparable halves —opening more registers (balancing) and any register being able to serve any customer (statelessness)—. You reproduced the anchor numbers and computed that Enlace needs on the order of half a dozen app servers (4 at the limit, 6 with slack to survive one going down, as you'll see in lesson 7). And you separated the two layers that "spread load": the balancer, in front of the app servers (this module), against the replicas and shards, below (module 5).

Before moving on you should be able to: explain why a single app server is a bottleneck even though the database is scaled; say what it means for a server to be "interchangeable" (stateless) and why it's needed to balance; and distinguish app-layer balancing from database scaling.

What comes next is the new physical piece. In lesson 2 you'll see what exactly a load balancer is: how it sits between the client and the pool as the single point of entry, what the system gains by having it (spreading, adding and removing servers without the client noticing, taking the downed ones out of rotation), and the difference between balancing at layer 4 and at layer 7. It's the step from "I know I need to spread" to "I know which machine does the spreading and how".

Resources

• System Design Primer — Load balancing and Application layer — the practical and much-cited summary of this whole module: what a balancer does, the algorithms, layer 4 vs. layer 7, and why the application layer is designed without state. A good map to keep alongside during the six lessons.
• Designing Data-Intensive Applications, Martin Kleppmann — Chapter 1 (Reliability, Scalability, Maintainability) — the chapter that gives the "scaling" framework the whole module works with: what load is, how it's measured, and why horizontal scaling (more machines) and vertical (a bigger one) are different decisions. The guide's canonical reference.
• nginx — What Is Load Balancing? — a short and concrete introduction to balancing from the documentation of one of the most widely used balancers in the world. It anchors this lesson's abstract concepts in a real product you could deploy.