Module 6: Load Balancing and Statelessness

4. Stateless versus stateful services

Description

The three algorithms of lesson 3 share an assumption they never said out loud: that any server can serve any request. Round-robin sends Ana's first request to app-0 and the second to app-1 without asking, taking for granted that app-1 lacks nothing to serve it. This lesson examines that assumption head-on. A server is stateless when it keeps no client state between requests —each request is self-sufficient, or the state it needs lives in a shared place all the servers can read—. A server is stateful when it remembers something about the client in its own memory: Ana's session, her cart, a counter. The difference seems like an implementation detail; in reality it decides whether you can scale horizontally or not.

You'll see, run, why state in the server breaks the balancing. You'll simulate a login: in the stateful version, Ana logs in on app-0, which stores the session in its RAM; when round-robin sends her next requests to app-1 and app-2, which know nothing about Ana, 4 of 6 requests fail. In the stateless version, Ana's token carries her signed identity, any server validates it without storing anything, and 6 of 6 pass. With that you'll understand the three places state can live (the server —bad—, a shared store, or the client) and why Enlace is already stateless from birth.

Connection to the module: this is the lesson that justifies the word "statelessness" in the title. Lesson 1 announced it with the supermarket (the interchangeable registers); here you demonstrate it with code. Lesson 5 takes the problem you pose here —"the state has to live somewhere"— and develops the three answers (sticky, token, shared store) with their tradeoffs, measuring why sticky imbalances. And lesson 7 uses the statelessness you establish here as the precondition to add and remove instances at will: only interchangeable servers clone without drama. Without this lesson, the balancing of lessons 2-3 wouldn't work in a service with sessions.

The gym locker desk versus the chip wristband

Think of it this way. You go to a gym and there are two ways to handle your locker. In the old gym, when you enter, the receptionist at that desk writes you down in a notebook: "locker 47 is Ana's". If tomorrow you enter through another door, with another receptionist, that notebook isn't there —the new receptionist doesn't know which locker is yours—. Your information lives in a specific desk's notebook. That's a stateful server: the state (your locker) is tied to a machine, and if another one serves you, it's lost.

In the modern gym they give you a chip wristband that carries encoded, signed, and forgery-proof, "this wristband is Ana's, locker 47". Now any desk, any turnstile, any door reads your wristband and knows who you are and which locker is yours —without consulting any local notebook, because the information travels with you, it doesn't live in the machine—. You can enter anywhere; all the doors are interchangeable. That's a stateless server: it keeps nothing of yours; you bring what's needed, or it's in a central place all the doors query.

The difference decides whether the gym can open more doors. With the local notebook (stateful), opening a new door is useless —your notebook isn't there, you'd always have to go back to your original desk—. With the wristband (stateless), opening new doors is trivial: each door reads any wristband. The local state is what prevents multiplying the doors. And multiplying the doors —adding servers— is exactly what this module wants to do.

A stateless service keeps no client state in the server's memory: each request brings what it needs (a token) or reads it from a shared store. That's why any server serves any request, and that's why it can be scaled horizontally. State in the server —stateful— ties the client to a machine and breaks the spreading.

Why local state breaks the balancing, run

Let's bring the analogy down to code. We simulate a login with three servers behind a round-robin balancer. In the stateful version, each server stores the sessions in a local dictionary (its "notebook"). In the stateless one, the token carries the signed identity (the "chip wristband") and the server validates it without storing anything:

# stateful_vs_stateless.py — why state in the server breaks scaling
import hmac, hashlib

# ---------- STATEFUL version: the session lives in each server's RAM ----------
class StatefulServer:
    def __init__(self, name):
        self.name = name
        self.sessions = {}                 # token -> user, LOCAL to this server

    def login(self, token, user):
        self.sessions[token] = user        # stored only here

    def whoami(self, token):
        return self.sessions.get(token)    # None if this server doesn't have it


# ---------- STATELESS version: the token is signed, nothing is stored ----------
SECRET = b"enlace-secret"
def issue_token(user):
    sig = hmac.new(SECRET, user.encode(), hashlib.sha256).hexdigest()[:8]
    return f"{user}.{sig}"                  # the token CARRIES the user + its signature

class StatelessServer:
    def __init__(self, name):
        self.name = name                   # no sessions: it stores nothing

    def whoami(self, token):
        user, sig = token.rsplit(".", 1)
        good = hmac.new(SECRET, user.encode(), hashlib.sha256).hexdigest()[:8]
        return user if hmac.compare_digest(sig, good) else None


def round_robin(servers):
    i = 0
    while True:
        yield servers[i % len(servers)]
        i += 1

# --- Stateful scenario: login on one server, following requests round-robin ---
sf = [StatefulServer("app-0"), StatefulServer("app-1"), StatefulServer("app-2")]
rr = round_robin(sf)
login_server = sf[0]
login_server.login("tok-123", "ana")       # ana enters through app-0
print("STATEFUL (session in the server's RAM):")
ok = 0
for req in range(6):
    s = next(rr)
    who = s.whoami("tok-123")
    print(f"  request {req}: {s.name}.whoami() -> {who}")
    ok += who is not None
print(f"  successful: {ok}/6\n")

# --- Stateless scenario: signed token, any server validates it ---
sl = [StatelessServer("app-0"), StatelessServer("app-1"), StatelessServer("app-2")]
rr = round_robin(sl)
token = issue_token("ana")                  # ana receives a signed token
print("STATELESS (signed token, zero state in the server):")
ok = 0
for req in range(6):
    s = next(rr)
    who = s.whoami(token)
    print(f"  request {req}: {s.name}.whoami() -> {who}")
    ok += who is not None
print(f"  successful: {ok}/6")

What to expect. With python stateful_vs_stateless.py:

STATEFUL (session in the server's RAM):
  request 0: app-0.whoami() -> ana
  request 1: app-1.whoami() -> None
  request 2: app-2.whoami() -> None
  request 3: app-0.whoami() -> ana
  request 4: app-1.whoami() -> None
  request 5: app-2.whoami() -> None
  successful: 2/6

STATELESS (signed token, zero state in the server):
  request 0: app-0.whoami() -> ana
  request 1: app-1.whoami() -> ana
  request 2: app-2.whoami() -> ana
  request 3: app-0.whoami() -> ana
  request 4: app-1.whoami() -> ana
  request 5: app-2.whoami() -> ana
  successful: 6/6

There's the whole lesson in two blocks. In the stateful version, Ana logged in on app-0, whose notebook now says tok-123 → ana. But the round-robin spreads her next requests among the three servers, and only app-0 recognizes her: when the request falls on app-1 or app-2, their whoami returns None —"I don't know who you are"—. Four of the six requests fail, not because the system is broken, but because the state lives on a single server and the balancing spreads among all. The round-robin and the local session are incompatible: one spreads, the other ties.

In the stateless version, Ana's token carries her identity plus a signature (ana.<signature>), and any server validates it by recomputing the signature with the shared secret —without consulting any local notebook, because there's no notebook—. The six requests pass, no matter which server the round-robin sends them to. The servers are interchangeable because they keep nothing of Ana's: she brings her chip wristband and all the doors read it.

Notice what didn't change: the balancer is identical in both cases (the same round-robin). The only thing that changed is where the state lives. That's the point: the problem isn't the balancing, it's the local state. And statelessness isn't an optimization, it's the requirement that makes the balancing work.

The three places state can live

Every real service has some state —who logged in, what's in the cart, how many requests they've made—. The question isn't "how do I eliminate the state?" but "where do I put it so the server stays interchangeable?". There are exactly three places:

   (1) IN THE SERVER            (2) IN A SHARED STORE          (3) IN THE CLIENT
   sessions = {} in app-0's      Redis / database                signed token (JWT)
   RAM                           that all servers read           that travels on each request

   app-0 [ana]  app-1 [ ]        app-0 --\                       app-0  <- token(ana)
   app-2 [ ]                     app-1 ---> [ Redis: ana ]       app-1  <- token(ana)
                                 app-2 --/                       app-2  <- token(ana)

   STATEFUL: breaks balancing    STATELESS: server interchangeable  STATELESS: server interchangeable

In the server (stateful). The state lives in the RAM of the server that served the first request. It's what you just saw fail: it ties the client to a machine, breaks the balancing, and if that machine dies the state is lost. It's the option this module teaches you to avoid.
In a shared store (stateless). The state lives in a Redis or a database all the servers query. The server keeps a session identifier in a cookie, but the session content is in the central store. The server keeps nothing of its own: it stays interchangeable, because any of them reads the session from the same place. The cost: a query to the store per request (fast, but it exists).
In the client (stateless). The state travels with the client, in a signed token (a JWT). The server keeps or queries nothing: it validates the token's signature —which carries the identity— and responds. It's what you saw in the stateless block. The cost: the token can't be "revoked" easily before it expires (once signed, it's valid until expiry), and it only serves for small state.

Options 2 and 3 are the two ways to be stateless, and lesson 5 compares them in depth (along with the sticky option, which is a patch over option 1). What unites them: the server keeps no client state of its own. The state either travels with the client (token) or lives outside, shared (store). In both cases, any server serves anyone, which is what the balancing needs.

Why Enlace is already stateless

Here comes the easy part, and it's why Enlace is this module's textbook case. Look at Enlace's two operations:

resolve(short_code): a short_code arrives, the server looks at the cache and the database, returns the long_url. There's no client state. It doesn't matter who requests the redirect or what they requested before; the request is completely self-sufficient. Any server serves it identically.
shorten(long_url): a long_url arrives, the server generates a short_code, stores it in the database, returns it. The state created (the Link record) goes to the shared database, not to the server's RAM. The server remembers nothing between requests.

Enlace is stateless from birth. A redirect is anonymous —no login, no cart, no session—, so there's no client state that could tie a request to a server. All the state that exists (the links) lives in the database and the cache, below the app servers, shared. That's why the resolve servers clone at will: they're identical boxes with nothing of their own inside, and the balancer spreads among them without anything breaking. It's the ideal other services chase with effort, and Enlace has it for free.

The only day Enlace would have to think about this is if it added user accounts (a dashboard where you see your links, your statistics). There a login session appears, and then lesson 5's decision becomes real: does that session live in the server (no), in a token (yes, option 3), or in a shared Redis (yes, option 2)? But the hot path —the resolve that serves the ~4,000 reads/s— never stops being stateless, and that's what makes it so cheap to scale.

Common mistakes

Confusing "has state" with "is stateful" (definition mistake). What happens: someone hears "the servers must be stateless" and concludes the system can't have sessions, carts, or anything persistent —which is absurd, every real service has state—. Why it happens: that state exists is confused with where it lives. How to detect it: if you believe stateless means "no state anywhere", you have the definition wrong. How to fix it: stateless means "the state doesn't live in the server's RAM"; it can (and must) live in a shared store or in the client. The system has all the state it needs; it's just not tied to a specific machine.

Storing the session in the server's RAM "because it's faster" (architecture mistake). What happens: someone stores the sessions in an in-memory dictionary of the server because it's faster than querying Redis, and it works perfectly in tests with a single server. When putting the balancer with three servers, the sessions fail intermittently —you measured it: 2 of 6— and the bug is hard to reproduce because sometimes the request falls on the correct server. Why it happens: with one server there's no difference; the problem only appears when scaling, which is exactly when it's already too late. How to detect it: if your sessions fail intermittently when adding servers, and they worked with one, the state is in the local RAM. How to fix it: move the session to a shared store (option 2) or a token (option 3) from the design, before scaling. Design stateless from the first server, not when you add the second.

Putting too much in the client's token (option 3 mistake). What happens: someone, to be stateless, puts all the state in the client's token —the whole cart, the history, permissions— and the token grows to kilobytes that travel on each request, plus it can't be revoked if something changes. Why it happens: option 3 is taken to the extreme without seeing its limits. How to detect it: if your token weighs more than a few hundred bytes or you need to invalidate it before it expires, you're overloading it. How to fix it: the token carries only the identity and small, stable data (who you are, your roles); the large or revocable state (the cart, sessions you need to be able to close) goes to a shared store (option 2). Options 2 and 3 combine: identity in the token, heavy data in the store.

Exercises

Exercise 1 — Count the failures. In the stateful simulation, Ana logged in on app-0 and the round-robin spreads her 6 requests among [app-0, app-1, app-2]. (a) Which requests succeed and which fail? (b) Why exactly 2 of 6? (c) If there were 5 servers instead of 3, how many of 6 would succeed?

See solution

(a) Round-robin spreads app-0, app-1, app-2, app-0, app-1, app-2. Only app-0 has Ana's session. Success on the requests that fall on app-0: 0 and 3. The ones that fall on app-1 and app-2 fail: 1, 2, 4, and 5.
(b) Because of every 3 requests, only 1 falls on app-0 (where the session lives). In 6 requests, that's 2 hits (requests 0 and 3) and 4 failures. The success rate is 1/3 = the spreading sends only a third of the requests to the single server that knows who Ana is.
(c) With 5 servers, round-robin sends to app-0 only 1 of every 5 requests. In 6 requests, app-0 receives the ones that fall on its turn: requests 0 and 5 (indices 0 and 5, both ≡ 0 mod 5). It would be 2 of 6 again by coincidence of the number, but the rate drops to 1/5: the more servers, the worse the stateful problem, because fewer requests hit the single server with the session. Local state scales inversely to the system.

Exercise 2 — Where does the state live? For each design, say in which of the three places the state lives (server / shared store / client) and whether the server ends up stateless or stateful. (a) The sessions are stored in a dict in each server's memory. (b) The sessions are stored in a Redis the three servers query; the client's cookie carries only the session_id. (c) The client carries a JWT signed with their identity and roles, and the server validates it without querying anything.

See solution

(a) In the server. Stateful. The local dict ties each session to a server's RAM. It breaks the balancing (what you measured). It's the option to avoid.
(b) In a shared store. Stateless. The state lives in Redis, not in the server; the cookie only carries the identifier to look it up there. Any server reads the session from the same Redis, so it's interchangeable. Cost: a query to Redis per request.
(c) In the client. Stateless. The state (identity, roles) travels in the signed JWT; the server validates it with the secret and queries nothing. Maximum scalability (zero queries), with the limit that the token isn't easily revoked before expiry and only carries small data.

Options (b) and (c) are the two correct ways to be stateless; (a) is the incorrect one.

Exercise 3 — Enlace adds accounts. Enlace decides to add a dashboard with login, where each user sees their links. A colleague proposes storing the login session in the server's RAM "because it's fast and we already have balancing". (a) What will happen to the login with the round-robin? (b) Is the resolve hot path affected by this decision? (c) What do you recommend for the dashboard session?

See solution

(a) The login will fail intermittently. The session will live in the server where the user logged in, and the round-robin will spread their next requests among all —two thirds (or more) will fall on servers that don't know the session and will return "not authenticated"—. It's the same 2-of-6 you measured, now in production.
(b) No. The resolve is and stays stateless —it's an anonymous redirect, no session—. The decision of where to put the dashboard session doesn't touch the hot path; the resolve servers clone just as freely. The login state is a problem apart from the mass-read path.
(c) Make the dashboard session stateless: a signed token (JWT) if the identity and roles are enough (option 3, zero queries), or a shared store (Redis, option 2) if you need to be able to close sessions or store more state. Never in the server's RAM. That way the dashboard inherits the same property as the resolve: any server serves any user, and the balancing works.

Summary and next step

In this lesson you demonstrated the hidden requirement of balancing: statelessness. With the gym locker desk (local notebook) against the chip wristband you saw that state in the server ties the client to a machine, and that state that travels with the client frees it. You ran it: a session in app-0's RAM makes 4 of 6 requests fail when the round-robin spreads them, because only app-0 knows it; the signed token, which any server validates, passes 6 of 6. And you placed the three places state can live —the server (stateful, bad), a shared store (stateless), and the client (stateless)— understanding that stateless doesn't mean "no state" but "no state in the server's RAM". You closed by seeing why Enlace is stateless from birth: resolve is an anonymous redirect with no client state, and everything that persists lives in the shared database and cache, below the servers.

Before moving on you should be able to: define stateless precisely (not "no state", but "no local state"); explain why state in the server breaks the round-robin; name the three places state can live; and say why Enlace's resolve is stateless.

What comes next is going deeper into the most common and most delicate case of state: the login session. In lesson 5 you'll compare the three ways to handle it —sticky sessions (pin the client to a server, a patch over the stateful option), signed token (option 3), and shared store (option 2)—, and you'll measure why sticky sessions, although they "work", imbalance the pool: one server ends up with 48.7% of the traffic while the others idle. It's the lesson that tells you, with numbers, why the easy solution isn't the good one.

Resources

Designing Data-Intensive Applications, Martin Kleppmann — Chapter 1, "Scalability" (stateless services and shared state) — the framework of why application services are designed without state and the state is pushed to the shared data systems (databases, caches). The conceptual basis of this lesson in the guide's canonical reference.
The Twelve-Factor App — VI. Processes (execute the app as stateless processes) — the principle, formulated as a design rule: "the app's processes are stateless and share nothing; any data that must persist is stored in a stateful backing service". The classic formulation of what you ran here.
System Design Primer — "Application layer" (microservices, statelessness) — the summary of why the application layer is kept stateless to be able to scale horizontally behind a balancer, with the same interchangeable-servers reasoning. A good bridge to the session lesson.

← Previous3. Load balancing algorithms Next →5. Where the session lives