Module 6: Load Balancing and Statelessness

5. Where the session lives

Next →

Description

Lesson 4 left a question open: the state has to live somewhere, which one? This lesson answers it for the most common and most delicate case —the login session, that "memory of who you are" that a web app needs to keep between requests—. There are three ways to handle it, and choosing well is half the design battle. Sticky sessions (session affinity): the balancer pins each client to a server —by cookie or by hash of their IP— and the session stays in that server's RAM; it's the solution that "works" but drags the stateful problem of lesson 4. Signed token (JWT): the session travels with the client, any server validates it, zero state in the server. Shared store (Redis/database): the session lives outside, in a place all the servers read, and the server stays stateless.

You'll measure why sticky sessions, although they're the easy way out, are the bad one: you'll simulate a thousand normal clients and three high-traffic "whales", and you'll see that with sticky one server ends up with 48.7% of the traffic while others are at 7.6% —an imbalance of 6.4×—, while the stateless spread ends up perfect (20% each). With that you'll be able to decide where to put the session of any service, and understand why Enlace, which has no session on the hot path, saves itself the whole problem.

Connection to the module: this lesson develops the three answers to the question lesson 4 posed. The sticky imbalance you measure here is the same mechanism as the "hash balancing with a dominant key" of lesson 3 —pinning by hash to a heavy client is hashing a dominant key—. And it prepares lesson 7: only with the state outside the server (token or store) can you add and remove instances without losing sessions. The boundary: the tokens' security in depth (key rotation, revocation, refresh tokens) is from the API design and security guide; here we treat them as a statelessness mechanism, not as an auth topic.

Three ways to remember the café customer

Think of it this way. You're a regular customer of a café chain with several branches, and they want to remember "your usual order" (an oat latte). There are three ways for them to remember it, and they're exactly the three ways to handle a session:

  • Each branch writes you down in its notebook (sticky). The first branch you visit writes you down: "Ana has an oat latte". It works… if you always go back to that branch. For it to work, the system has to force you to always go back to the same one —they give you a card that says "you're a customer of the Downtown branch"—. Problem: if that branch is full and the one next door empty, you still have to line up at yours. And if your branch closes, your order is lost. This is sticky sessions: they pin you to a server.
  • You bring a card with your order written on it (token). They give you a laminated card, signed by the chain, that says "Ana: oat latte". Any branch reads it and knows your order, without looking in any notebook. You enter anywhere, show the card, they prepare it. The chain keeps nothing of yours: you bring the information. This is the signed token.
  • There's a central notebook all the branches consult (shared store). There's a central system where "Ana: oat latte" is, and each branch consults it when you arrive (identifying yourself with a member number). You enter anywhere; the branch asks the central, gets your order, prepares it. The branch keeps nothing; the central does. This is the shared store (Redis).

All three "remember" your order, but only two let you enter any branch without penalty. The local-notebook one (sticky) ties you to a branch, with everything that drags. The rest of the lesson measures why that tie hurts.

Option A: sticky sessions (and why they imbalance)

Sticky sessions —or session affinity— is the easy way out when you already have stateful servers and don't want to rewrite them. Instead of fixing the state, you fix the spreading: the balancer commits to sending all of a client's requests always to the same server, where their session lives. It does it by pinning the client by their IP (hash of the IP) or by a cookie the balancer injects. With that, lesson 4's problem "disappears": Ana's session is in app-0, and since Ana always goes to app-0, she always finds it.

The problem is that it breaks exactly what the balancer was supposed to give you: the even spread. Because now you don't spread requests, you spread whole clients —and clients don't generate the same traffic—. A client who makes 100 requests and one who makes 100,000 count the same "by head", but weigh 1,000× differently. When a heavy client ("whale") gets pinned to a server, that server carries all their traffic and there's no way to spread it. We measure it: a thousand normal clients (~100 requests each) and three whales (50,000 requests each), pinned by hash to five servers, against the stateless per-request spread:

# sticky_breaks_balance.py — why sticky sessions imbalance
import hashlib, random
from collections import Counter

def sticky_server(client_id, n):
    h = int(hashlib.md5(client_id.encode()).hexdigest(), 16)
    return h % n                            # the client gets pinned to a server

random.seed(7)
N = 5
# 1000 normal clients (~100 requests) + 3 "whales" (50,000 each)
clients = {}
for i in range(1000):
    clients[f"user-{i}"] = random.randint(80, 120)
for w in range(3):
    clients[f"whale-{w}"] = 50_000

total = sum(clients.values())

# --- STICKY: all of a client's requests go to their pinned server ---
sticky = Counter()
for client, reqs in clients.items():
    sticky[sticky_server(client, N)] += reqs

# --- STATELESS round-robin: each request is spread separately ---
stateless = Counter()
i = 0
for client, reqs in clients.items():
    for _ in range(reqs):
        stateless[i % N] += 1
        i += 1

def report(name, load):
    per = [load[s] for s in range(N)]
    print(f"{name}")
    print(f"  load per server: {per}")
    print(f"  max/min = {max(per)/max(min(per),1):.1f}x   "
          f"(worst server = {100*max(per)/total:.1f}% of the traffic)\n")

print(f"total requests = {total:,}\n")
report("STICKY (client pinned to a server):", sticky)
report("STATELESS round-robin (request by request):", stateless)

What to expect. With python sticky_breaks_balance.py:

total requests = 249,384

STICKY (client pinned to a server):
  load per server: [18918, 19056, 70205, 121500, 19705]
  max/min = 6.4x   (worst server = 48.7% of the traffic)

STATELESS round-robin (request by request):
  load per server: [49877, 49877, 49877, 49877, 49876]
  max/min = 1.0x   (worst server = 20.0% of the traffic)

Look at sticky's disaster. app-3 receives 121,500 requests —48.7% of all the traffic— while app-0 receives 18,918, an imbalance of 6.4×. Why? Because two of the three whales got pinned to app-3 (and one to app-2, which is why it's also high with 70,205), and there's no way to move their traffic: sticky nailed them there. The balancer is powerless —it promised affinity, and the affinity prevents it from rebalancing—. Server app-3 is on the verge of collapse while three servers idle at 7.6%.

Now look at the stateless spread: each request is spread separately, no matter which client it's for, and the result is almost perfect —49,877 on each server, 20.0% each, imbalance of 1.0×—. The whales' requests dilute in the pool instead of piling up. The difference between 48.7% and 20.0% is the difference between a server on fire and five calm servers, and it all comes from one decision: spread clients (sticky) or spread requests (stateless).

The lesson: sticky sessions "work" but imbalance, and the imbalance worsens the more unequal the traffic among clients is. Sticky also drags two other ills: if the pinned server dies, all its sessions are lost (the client has to log in again); and you can't rebalance after adding a new server (the old sessions stay pinned to the old ones, the new one starts idle). Sticky is a patch over lesson 4's stateful problem, not a solution.

Option B: signed token (state in the client)

The first real solution: don't store the session on any server, make it travel with the client in a signed token (a JWT, JSON Web Token). When the client logs in, the server hands them a token that contains their identity and small data (who they are, their roles, when it expires), all signed with a secret only the servers know. On each following request, the client sends the token; any server validates it by recomputing the signature —if it matches, the token is authentic and the server knows who the client is without querying anything—. It's the café's "laminated card".

  login:   client --> app-0 --> returns signed token: {user: ana, exp: ...}.<signature>
  request: client --> [round-robin] --> app-2  (validates the signature, knows it's ana)
           client --> [round-robin] --> app-1  (validates the signature, knows it's ana)
           no server stores anything: the session travels in the token

The advantage is maximum scalability: zero queries to validate the session (the signature is verified with pure CPU), and any server serves anyone. The disadvantage, which you already saw in lesson 4: the token can't be revoked easily before it expires (once signed, it's valid until expiry; if you want to invalidate it earlier —for example, an immediate logout or a ban—, you need a revocation list, which is… shared state); and it only carries small state (kilobytes traveling on each request is expensive). That's why tokens are used for the identity —the small and stable— and are given a short expiration (minutes to hours) to bound the damage of not being able to revoke them. The tokens' fine security (secret rotation, refresh tokens, revocation) is a topic of the APIs and security guide; here it's enough for us that the token makes the server stateless.

Option C: shared store (state outside)

The second real solution: store the session in a shared store —typically Redis— all the servers query. The client carries in a cookie only a session_id (an opaque identifier); on each request, the server takes that id, looks up the session in Redis, and gets the state. The server keeps nothing of its own: it's interchangeable, because they all read from the same Redis. It's the café's "central notebook".

  login:   client --> app-0 --> stores session in Redis, returns cookie: session_id=abc
  request: client(abc) --> [round-robin] --> app-2 --> Redis.get(abc) -> {user: ana}
           client(abc) --> [round-robin] --> app-1 --> Redis.get(abc) -> {user: ana}
           the session lives in Redis; the servers just query it

The advantage over the token: the session can be revoked instantly (you delete the Redis entry and the client is logged out on the next request) and it can store more state (the token doesn't grow, because only the id travels). The disadvantage: a query to Redis per request (fast —submillisecond— but not free), and Redis becomes a piece that has to be kept available (if Redis goes down, the sessions go down; that's why Redis itself is replicated). In practice, token and store combine: the token carries the identity (cheap, no query) and the store keeps what has to be revocable or the heavy stuff. For Enlace, if it added accounts, a Redis of sessions is the robust and simple option.

The decision table and the Enlace case

Let's put the three options together:

OptionWhere the session livesServerSpreadRevocableCost per request
Sticky sessionsPinned server's RAMStateful (patch)Imbalanced (48.7% on one)Lost if the server diesNone, but imbalances
Signed tokenIn the clientStatelessEvenNo (until expiry)None (just CPU)
Shared storeShared Redis/DBStatelessEvenYes (delete from the store)One query to Redis

The practical rule: avoid sticky if you can; use a token for the identity (fast, no query) and a shared store when you need to revoke or store more state. Sticky is only justified as a temporary patch over stateful servers you can't rewrite yet —and knowing that it imbalances—.

And Enlace? Here's the good news that closes lesson 4's arc: Enlace's hot path has no session. A resolve is an anonymous redirect —no one logs in to visit a short link—, so there's no session to place in any of the three options. The ~4,000 resolve/s are spread with pure round-robin, no affinity, no token, no session Redis, because there's no client state to remember. Enlace skips the whole problem on its critical path. Only if it adds a dashboard with accounts does a session appear, and then the choice is clear: signed token for the dashboard's identity, plus a shared Redis if it wants immediate logout —never sticky, never in the server's RAM—. The mass-read path stays stateless forever.

Common mistakes

Using sticky sessions as a permanent solution (architecture mistake). What happens: a team enables sticky sessions so the in-RAM sessions "work" and leaves it that way indefinitely. Over time, the unequal traffic imbalances the pool (one server at 48.7%, others at 7.6%), the deployments drop user sessions, and adding servers doesn't rebalance. Why it happens: sticky is the least-effort way out and postpones the real work. How to detect it: if your pool is imbalanced despite the balancer, and you use sticky, that's the reason —you measured it—. How to fix it: move the session outside the server (token or shared store) and turn off sticky. The balancer only spreads evenly if it spreads requests, not pinned clients.

Putting revocable or large data in the token (option B mistake). What happens: someone stores in the JWT permissions that change, or a cart, or state they need to be able to invalidate (a logout, a ban). When the permission changes or the user must be kicked out, the old token is still valid until expiry —the ban has no effect, the logout closes nothing—. Why it happens: it's forgotten that a signed token is immutable and irrevocable until expiry. How to detect it: if you need to invalidate a session now and can't, you put in the token what should have gone in a store. How to fix it: the token carries only stable identity with a short expiration; the revocable (sessions you close, permissions that change) goes to a shared store you can delete instantly.

Storing the session in Redis but without making Redis redundant (option C mistake). What happens: someone moves the sessions to a shared Redis —good, the servers end up stateless— but puts a single Redis. The day that Redis restarts, all the sessions of all the users disappear at once: massive logout, and the system is left unable to authenticate anyone. Why it happens: the server's statelessness is resolved and a new single point of failure is created in the store. How to detect it: ask yourself "if the session Redis goes down, what happens?". If the answer is "all the sessions go down", you have a SPOF. How to fix it: replicate the session Redis (just as you replicated the database in module 5), or combine it with tokens so the identity doesn't depend only on the store. The state you take out of the server has to live in something as available as the whole system.

Exercises

Exercise 1 — Read the imbalance. In the simulation, with sticky the load per server was [18918, 19056, 70205, 121500, 19705] over a total of 249,384 requests. (a) What percentage of the traffic does the worst server carry? (b) Why are app-3 and app-2 so far above the others? (c) What would have to be done to spread that traffic evenly?

See solution
  • (a) The worst is app-3 with 121,500 of 249,384 = 48.7%. Almost half the traffic on a single server of five (the ideal would be 20%).
  • (b) Because the whales (50,000 requests each) got pinned by hash: two fell on app-3 (2 × 50,000 = 100,000, plus its share of normal clients ≈ 121,500) and one on app-2 (≈ 70,205). The hash pinned the heavy clients to those two servers, and sticky prevents moving their traffic. It's the "hash balancing with a dominant key" of lesson 3, with the client as the key.
  • (c) Take the session out of the server (token or shared store) and turn off sticky, so the balancer spreads each request by round-robin instead of whole clients. Then the whales' requests dilute and the spread ends up ~20% each, as the stateless block showed.

Exercise 2 — Choose where the session lives. For each requirement, say which option (sticky / token / shared store) fits best and why. (a) A dashboard where "log out" must invalidate the access instantly. (b) A high-traffic internal API where zero extra latency for validating the identity matters and the sessions last only 15 minutes. (c) A legacy system with in-RAM sessions you can't rewrite this week but you need to put a second server now.

See solution
  • (a) Shared store. The immediate logout requires being able to revoke the session, and that only comes from deleting a store (Redis) entry. A signed token can't be invalidated before expiry, so "log out" wouldn't close anything until the token expires.
  • (b) Signed token. Zero extra latency (the signature is validated with CPU, without querying anything) and a short expiration (15 min) that bounds the risk of not being able to revoke. It's the token's ideal case: small identity, very cheap validation, short window.
  • (c) Sticky sessions, as a temporary patch. It's the only thing that lets you put a second server without rewriting the state this week: you pin each client to their server and the in-RAM sessions keep working. Knowing that it imbalances and that you have to migrate to token/store soon —sticky buys time, it doesn't resolve—.

Exercise 3 — Enlace and its dashboard. Enlace launches a dashboard with accounts. The hot path (resolve) still receives ~4,000 requests/s; the dashboard, about 10 requests/s. (a) Does the resolve need any of the three session options? (b) What do you recommend for the dashboard session? (c) Why does the dashboard decision not affect the hot path's performance?

See solution
  • (a) No. The resolve is an anonymous redirect: there's no login or session to place. It's spread with pure round-robin, no sticky, no token, no session Redis. The session options don't apply to the hot path because there's no client state there.
  • (b) For the dashboard, a signed token for the identity (validation with no query, fits its modest traffic), and optionally a shared Redis if Enlace wants immediate logout or to store more session state. Never sticky (imbalances) or the server's RAM (breaks the balancing). The dashboard thus inherits the rest of the system's statelessness.
  • (c) Because they're separate paths: the resolve (4,000/s) doesn't touch the dashboard session at all, so giving the dashboard a token or Redis doesn't add a millisecond to the resolve. The login state lives only in the dashboard's flow (10/s), where its cost is irrelevant. Separating the stateless hot path from the session path is exactly what keeps the mass spread cheap.

Summary and next step

In this lesson you resolved where to put the state that lesson 4 left placeless, for the login-session case. With the café chain you saw the three ways to remember the client: the local notebook that ties you to a branch (sticky), the card you bring with you (token), and the central notebook all consult (shared store). You measured why sticky, the easy way out, is the bad one: with unequal traffic, one server ends up with 48.7% of the load (6.4× imbalance) while the stateless spread ends up perfect (20% each). You compared the two real solutions —signed token (zero queries, but not revocable, small state) and shared store (revocable, more state, one query per request)— and saw that they combine: identity in the token, the revocable or heavy in the store. And you closed Enlace's arc: the resolve has no session, so it skips the whole problem; only the dashboard, if it comes, would use a token plus Redis, never sticky.

Before moving on you should be able to: name the three options and where the session lives in each; explain why sticky imbalances (spreading clients, not requests); distinguish token from store by its ability to revoke; and say why Enlace's hot path needs none of them.

What comes next is giving the balancer a sense we've been taking for granted so far: knowing which server is alive. In lesson 6 you'll see the health checks —the probes with which the balancer checks each server's health and takes the failing one out of rotation—, and you'll measure why the thresholds matter: with threshold 1, a flapping server causes 6 state changes (it enters and leaves nonstop); with threshold 3, only 2. It's what makes "taking the downed ones out of rotation" reliable instead of chaotic.

Resources

← Previous4. Stateless versus stateful servicesNext →6. Health checks