6. Health checks

Description

This whole module took for granted something the balancer doesn't know by magic: which server is alive. Lesson 2 said the balancer "takes the downed ones out of rotation", but how does it find out one went down? The answer is the health checks: the balancer periodically probes each server in the pool —it sends it a light request, typically a GET /healthz— and observes the response. If the server responds well, it stays in rotation; if it stops responding, the balancer takes it out of the pool and spreads only among the healthy ones; when it responds again, it readmits it. Without health checks, the balancer would keep sending traffic to a dead server, and a fraction of the requests would fail —exactly what the balancer was supposed to prevent—.

You'll see the two types of health check (active, the balancer probes; passive, it observes the real failures), and above all the thresholds that avoid chaos: how many consecutive failures are needed to expel a server and how many successes to readmit it. You'll measure why they matter: with threshold 1, a server that flaps (fails and responds alternately) enters and leaves the pool nonstop —6 state changes—, a phenomenon called flapping that destabilizes the spreading; with threshold 3, the same server only changes state 2 times, only under a sustained crash. You'll also see the difference between liveness and readiness, and the danger of the deep health check that can expel the whole pool at once.

Connection to the module: the health checks are lesson 2's balancer's eyes —the concrete mechanism behind "taking the downed ones out of rotation"—. And they're the operational precondition of lesson 7: to add an instance, the balancer waits for it to pass the health check before sending it traffic; to remove it gracefully, it stops sending to it before turning it off. The boundary: what a server does when its dependency (the database) is slow or down —protect itself with a circuit breaker, retry with backoff— is the resilience guide. Here we see what the balancer does when a server doesn't respond: take it out of the pool. They're two complementary mechanisms at different layers.

The foreman who checks his crew

Think of it this way. A foreman directs a crew of workers and spreads the work among them. Every so often he passes by and asks each one: "all good? can you keep going?". If a worker responds "yes", they keep receiving work. If a worker doesn't answer —they fainted, they left, they're exhausted—, the foreman stops giving them tasks and spreads them among the others, until the worker recovers and responds again. That round of questions is the health check, and the foreman is the balancer.

But a sensible foreman doesn't expel a worker for one weak response. If you ask "all good?" right when the worker is drinking water and takes a while to answer, you don't send them home —you wait to see if they fail several times in a row—. And the reverse: when a worker who was resting comes back, you don't dump all the work on them for a single "I'm ready"; you wait to confirm they really recovered. That "several times in a row" is the threshold, and it's what distinguishes a firm foreman from a hysterical one who expels and readmits people every two minutes according to the mood of the moment. A hysterical foreman —threshold of 1— destabilizes the whole crew: no one knows who's working. A firm foreman —threshold of 3— only takes out whoever really can't keep going.

The analogy brings the whole lesson: the health check isn't just "does it respond?", it's "does it respond consistently?". A single response —good or bad— shouldn't change the pool. A sustained pattern is needed, and the thresholds are what encode "sustained". The rest of the lesson measures them.

Active vs. passive: two ways to monitor the health

There are two ways for the balancer to know about a server's health, and real systems usually use both:

• Active health check. The balancer probes each server at regular intervals (every few seconds) with a dedicated request —a GET /healthz— and looks at the response. It's proactive: it detects a sick server before sending it real traffic, because it's testing it separately. The cost: constant probe traffic, and a detection window (if you probe every 5 s, you take up to 5 s to notice a crash).
• Passive health check. The balancer observes the real requests it's already routing: if a server starts returning errors or not responding to the users' requests, it takes it out. It's reactive, with no extra traffic, but it only detects the problem when it already affected real requests —some users already saw the error—.

The two complement each other: the active one catches the server that went down while idle (before traffic reaches it), and the passive one catches the one that fails under real load even though it responds well to the light probe. In this lesson we model the active one, which has the clearest threshold logic, but remember that in production both run together.

The thresholds, run: why they avoid flapping

Let's go to the heart of the lesson. A health check doesn't change the pool with a single response: it accumulates. It expels a server after down_threshold consecutive failures, and readmits it after up_threshold consecutive successes. Let's implement that tracker and run two scenarios: a clean crash with recovery, and a server that flaps:

# health_checks.py — the LB takes the sick server out of rotation and returns it when it heals
class HealthTracker:
    """Tracks a backend: takes it out after `down` consecutive failures, returns it after `up` successes."""
    def __init__(self, up_threshold, down_threshold):
        self.up_th = up_threshold
        self.down_th = down_threshold
        self.in_pool = True
        self.consec_fail = 0
        self.consec_ok = 0

    def observe(self, healthy):
        if healthy:
            self.consec_ok += 1
            self.consec_fail = 0
            if not self.in_pool and self.consec_ok >= self.up_th:
                self.in_pool = True            # readmitted
        else:
            self.consec_fail += 1
            self.consec_ok = 0
            if self.in_pool and self.consec_fail >= self.down_th:
                self.in_pool = False           # expelled

# Scenario 1: app-1 goes down, recovers. Threshold down=3, up=2.
# Timeline of the health probes to app-1 (True=responds 200, False=fails):
timeline = [True, True, False, False, False, False, False, True, True, True]
t = HealthTracker(up_threshold=2, down_threshold=3)
print("Scenario 1 — crash and recovery (down=3, up=2):")
print("  probe | health | in_pool")
for i, healthy in enumerate(timeline):
    t.observe(healthy)
    print(f"    {i:>2}  |  {'OK ' if healthy else 'FAIL'}  |  {'YES' if t.in_pool else 'NO'}")

# Scenario 2: flapping server. Threshold=1 takes it out/in nonstop;
# threshold=3 tolerates it except a sustained crash.
flaky = [True, False, True, False, True, False, False, False, False, True]
for down_th in (1, 3):
    t = HealthTracker(up_threshold=1, down_threshold=down_th)
    trace = []
    for healthy in flaky:
        t.observe(healthy)
        trace.append("O" if t.in_pool else "x")   # O=in pool, x=out
    flips = sum(trace[i] != trace[i-1] for i in range(1, len(trace)))
    print(f"\nScenario 2 — flapping, down_threshold={down_th}:")
    print(f"  pool state: {' '.join(trace)}   (state changes: {flips})")

What to expect. With python health_checks.py:

Scenario 1 — crash and recovery (down=3, up=2):
  probe | health | in_pool
     0  |  OK   |  YES
     1  |  OK   |  YES
     2  |  FAIL  |  YES
     3  |  FAIL  |  YES
     4  |  FAIL  |  NO
     5  |  FAIL  |  NO
     6  |  FAIL  |  NO
     7  |  OK   |  NO
     8  |  OK   |  YES
     9  |  OK   |  YES

Scenario 2 — flapping, down_threshold=1:
  pool state: O x O x O x x x x O   (state changes: 6)

Scenario 2 — flapping, down_threshold=3:
  pool state: O O O O O O O x x O   (state changes: 2)

Read scenario 1 first. app-1 responds well (probes 0-1), then starts failing (probe 2). Notice it does not leave the pool on the first failure: it stays at YES during probes 2 and 3, because the threshold is 3 consecutive failures. Only at probe 4 —the third consecutive failure— is the threshold met and it leaves (NO). When it recovers (probe 7, first OK), it doesn't come back immediately either: it waits for the second consecutive success (probe 8) to readmit itself. The thresholds introduce that deliberate hysteresis: they cost a bit of late reaction in exchange for not reacting to passing noise.

Now scenario 2, which is the reason thresholds exist. The server flaps: it responds, fails, responds, fails… With down_threshold=1 (expels on the first failure), the pool takes it out and in nonstop —O x O x O x x x x O, 6 state changes—. That's flapping: a server that enters and leaves rotation constantly, and each change is disruptive (it reconfigures the spreading, moves connections, dirties the metrics). With down_threshold=3, the same flapping produces only 2 changes: the server stays in the pool during the isolated failures and only leaves when it really fails sustainedly (the four consecutive failures in the middle). The threshold filters the noise and reacts only to the signal.

The lesson: the threshold is what turns the health checks from a hysterical switch into a reliable judgment. Threshold 1 reacts to every hiccup; threshold 3 waits for a pattern. The price of a high threshold is detecting the real crashes a bit later (three probes instead of one); the price of a low one is the flapping. The typical sweet spot —2 or 3 failures to expel, 2 successes to readmit, probing every few seconds— balances quick reaction with stability.

Liveness vs. readiness: what /healthz should check

It's not enough to have a health check; what it checks matters. There are two different questions a server can answer, and confusing them causes serious problems:

• Liveness ("are you alive?"). Is the server's process running and able to respond? A liveness /healthz returns 200 if the server is up, nothing more. If it fails, the server is dead or hung and it has to be restarted.
• Readiness ("are you ready to receive traffic?"). Can the server really serve a useful request right now? A readiness /healthz checks that the dependencies it needs —the cache, the database— are reachable. If it fails, the server is alive but shouldn't receive traffic yet (starting up, or with a dependency down), so it's taken out of the pool without restarting it.

The distinction matters when adding instances (lesson 7): a just-started server is alive (liveness OK) but maybe not yet ready (it's still warming its local cache, opening its database connections). The balancer must wait for the readiness before sending it traffic, or the first users who fall on it will see errors.

But there's a dangerous trap in deep readiness checks, and it's worth seeing because it's a classic design failure:

The danger of the deep health check. If each server's /healthz checks "can I reach the database?", and the database has a two-second hiccup, then all the servers fail their health check at once. The balancer, obediently, takes them all out of the pool —and now no server is left in rotation, so all of Enlace goes down completely, over a database hiccup that maybe wasn't even affecting the real requests! A minor and transient problem in a shared dependency is amplified into a total crash.

The mitigation: deep readiness checks must be conservative —don't expel the whole pool over a transient failure of a shared dependency—. In practice, a light liveness check is preferred for the balancer's health check (which only looks at whether the process responds), and the dependencies' health is handled with other mechanisms: the server itself protects from the slow dependency with timeouts and circuit breakers —which is exactly the topic of the resilience guide, not this one—. The rule: the balancer's health check verifies that the server is healthy, not that the whole downstream system is; mixing the two layers is how a database hiccup brings down the whole pool.

Common mistakes

Threshold of 1: expel on the first failure (flapping mistake). What happens: someone configures the health check to take out a server as soon as one probe fails, seeking to "react fast". A server with transient failures (a GC spike, a probe that arrived at a bad moment) enters and leaves the pool nonstop, destabilizing the spreading —you measured it: 6 state changes with a flapping server—. Why it happens: "react fast" is confused with "react to every hiccup". How to detect it: if your servers enter and leave rotation frequently and the pool metrics oscillate, you have flapping. How to fix it: raise the down_threshold to 2 or 3 consecutive failures, to expel only under sustained crashes. The health check should react to a pattern, not a point.

Sending traffic to a server that isn't ready yet (readiness mistake). What happens: a new instance is added and the balancer sends it traffic as soon as the process starts (liveness OK), but the instance hasn't yet opened its database connections or warmed anything. The first users who fall on it see errors or enormous latencies. Why it happens: liveness (is the process alive?) is checked when readiness (can it serve yet?) should have been. How to detect it: if every time you scale there's a spike of errors in the first seconds of the new instances, you didn't wait for the readiness. How to fix it: the balancer must wait for the readiness health check to pass before routing traffic to a new instance —give traffic only to whoever confirms they're ready, not just alive—.

Deep health check that brings down the whole pool (amplification mistake). What happens: each server's /healthz checks the database; the database has a transient hiccup; all the servers fail the health check at once; the balancer takes them all out; all of Enlace goes down. Why it happens: the health of a shared dependency is put into each server's health check, turning a minor problem into a total one. How to detect it: if a brief database hiccup empties the whole pool, your health check is too deep. How to fix it: keep the balancer's health check light (liveness, or a conservative readiness that doesn't expel everything over a transient shared failure), and protect the server from its dependency with timeouts and circuit breakers (resilience guide). The balancer checks the server's health, not that of everything below it.

Exercises

Exercise 1 — Trace the pool. A server has down_threshold=2 and up_threshold=2. The probes give: OK, OK, FAIL, FAIL, FAIL, OK, FAIL, OK, OK. Trace when it's in the pool (YES/NO), starting in the pool. In which probe does it leave and in which probe does it come back?

See solution

With down_threshold=2 (2 consecutive failures to leave) and up_threshold=2 (2 consecutive successes to come back), starting in the pool:

Probe	Health	Consec. fail	Consec. success	In pool
0	OK	0	1	YES
1	OK	0	2	YES
2	FAIL	1	0	YES
3	FAIL	2	0	NO (2nd consecutive failure → leaves)
4	FAIL	3	0	NO
5	OK	0	1	NO (only 1 success, needs 2)
6	FAIL	1	0	NO (the isolated success reset)
7	OK	0	1	NO
8	OK	0	2	YES (2nd consecutive success → comes back)

It leaves at probe 3 (second consecutive failure) and comes back at probe 8 (second consecutive success). Notice how the isolated OK at probe 5 does not readmit it: the FAIL at probe 6 resets the success counter, and two in a row are needed, which are only met at 8. The hysteresis in action.

Exercise 2 — Choose the threshold. For each situation, say whether a low down_threshold (1, quick reaction) or a high one (3+, stable) is best and why. (a) Servers that occasionally have half-second GC pauses that make an isolated probe fail. (b) A server that, when it really dies, starts corrupting data, so you want to take it out as soon as possible. (c) A pool on an unstable network where the probes fail every so often due to packet loss, not due to sick servers.

See solution

• (a) High threshold (3+). The GC pauses cause isolated failures that don't mean "dead server". A high threshold filters them: the server only leaves if it fails sustainedly, not over a GC hiccup. Threshold 1 would cause flapping on every GC.
• (b) Low threshold (1-2). If a dead server does active harm (corrupts data), the cost of leaving it a moment longer exceeds the cost of an occasional flapping. Here you do want quick reaction, accepting a bit more sensitivity. (In practice data corruption is prevented in other ways, but the principle of "quick reaction when the failure is expensive" holds.)
• (c) High threshold (3+). If the probes fail due to the network, not the servers, a low threshold would expel healthy servers over packet loss —flapping caused by the medium, not the backends—. A high threshold demands a sustained pattern before believing the server (and not the network) is the problem.

Exercise 3 — The deep health check. Enlace configures each app server's /healthz to check "can I do a SELECT to the database?". One day, the database has a 3-second latency spike. (a) What happens to the app server pool? (b) Why is it worse than not having a health check for that dependency? (c) How should the health check be designed instead?

See solution

• (a) All the app servers fail their /healthz at once (they all share the same database, which is slow), so the balancer takes them all out of the pool. No server is left in rotation: all of Enlace goes down completely, over a transient database spike.
• (b) Because it amplifies the problem instead of containing it. Without that deep health check, a 3 s database spike would cause, at most, some slow requests —but the system would stay up, and the cache (module 4) would absorb most of the reads—. With the deep health check, a minor and transient problem of a shared dependency becomes a total crash: the cure is worse than the disease.
• (c) The balancer's health check should be light —check that the server's process responds (liveness), not that all the downstream infrastructure is perfect—. The database's health is handled where it belongs: the server protects from the slow dependency with timeouts and circuit breakers (resilience guide), and the cache dampens the spikes. The balancer checks the server's health, not that of the whole system; putting the database in each server's health check is how a hiccup brings down the pool.

Summary and next step

In this lesson you gave the balancer the eyes you took for granted: the health checks, the probes with which it knows which server is alive. With the foreman who checks his crew you saw that the key isn't "does it respond?" but "does it respond consistently?", and that expelling or readmitting over a single response is a hysterical foreman. You distinguished the active health check (the balancer probes, proactive) from the passive one (observes the real failures, reactive), and you ran the thresholds logic: a server leaves after N consecutive failures and comes back after M consecutive successes, with the hysteresis that implies. You measured why they matter: a flapping server causes 6 state changes with threshold 1 (flapping) and only 2 with threshold 3 (which filters the noise). And you separated liveness (is it alive?) from readiness (is it ready?), with the warning of the deep health check that expels the whole pool over a database hiccup —a problem contained by keeping the check light and leaving the dependency protection to the resilience guide's circuit breakers—.

Before moving on you should be able to: explain how a balancer knows a server went down; distinguish active from passive health check; justify the thresholds from the flapping they avoid; distinguish liveness from readiness; and explain why a deep health check can bring down the whole pool.

What comes next is harvesting the whole module. With stateless servers (lessons 4-5), a balancer with spreading algorithms (lessons 2-3), and health checks that know who's alive (this lesson), you finally have the three pieces to do what the module promised: add and remove instances at will. In lesson 7 you'll see horizontal scaling in action —how many instances Enlace needs, how a new one comes in (starts up, passes the readiness, enters rotation) and how one leaves gracefully (connection draining)—, and why statelessness is what makes all this routine instead of risk.

Resources

• HAProxy — Health checks (check, rise, fall, inter) — HAProxy's documentation with this lesson's real parameters: fall (failures to expel, the down_threshold), rise (successes to readmit, the up_threshold), and inter (probe interval). The direct bridge from the threshold theory to the production configuration.
• Kubernetes — Liveness, Readiness and Startup Probes — the liveness/readiness distinction formalized in the most widely used orchestrator, with the same concepts (thresholds, periods, what to restart vs. what to take out of rotation). It shows how this lesson's health checks are configured in a real system.
• nginx — Health checks (upstream, max_fails, fail_timeout) — the configuration of passive and active health checks in nginx, with the failure thresholds and the slow start (slow_start) that avoids sending full traffic to a just-readmitted instance. A third angle on the same mechanisms.