3. Failover basics

Description

By the end of this lesson you will understand failover: the mechanism by which a system detects that a component has died and hands the work to a backup, to take advantage of the redundancy you built in lesson 2. You will see that failover has three parts —detect (how does the system know the primary went down?), decide (who takes over?) and switch (how is traffic redirected to the new one?)— and that each part takes time, and that time counts as downtime. You will distinguish the two basic configurations: active-passive (a standby backup that only enters when the primary goes down) and active-active (several active at once, and when one goes down the others absorb its load). You will connect detection with the health checks and heartbeats you brushed against in module 6, apply everything to Enlace —promoting a replica when the primary dies (the redundancy of module 5) and why the failover of the stateless app is almost free—, and you will precisely mark the border with the resilience guide: split-brain, fencing, leader election by consensus, and idempotency in depth are named here but taught there.

This matters because redundancy without failover is an ambulance without a driver: you have the resource, but nobody sets it in motion when it is needed. The formula of lesson 2 (1 - (1-a)^N) assumes the backup enters instantly and without failing; failover is everything that separates that theory from reality. And it is a treacherous topic, because the hard parts of failover are not the obvious ones: detecting an outage sounds easy until you discover that "not responding" can mean "it is dead" or can mean "it is alive but the network between us was cut" —and confusing those two cases is how split-brain is born, the most dangerous failure of distributed systems—. This lesson gives you the complete map of failover, shows you where the trap is, and tells you honestly which part is solved here (the concept) and which part needs the tools of the resilience guide.

Connection to the module: this lesson closes the reliability block and is the bridge to the consistency one. It closes reliability because it completes the pair from lesson 2: there you learned to have redundancy and to compute its benefit; here you learn the mechanism that activates it when something goes down. And it is the bridge to consistency because the central trap of failover —"not responding" can be "dead" or can be "network cut"— is exactly a network partition, which is the protagonist of lessons 4, 5 and 6. When we say here "the system cannot distinguish a dead primary from a primary isolated by the network", we will be planting the seed of the CAP theorem: under that uncertainty, you have to choose between continuing to serve and risking a split-brain, or stopping and sacrificing availability. Lesson 4 formalizes that choice.

The backup goalkeeper

Think of it this way. A soccer team has a starting goalkeeper and, on the bench, a backup goalkeeper. While the starter plays well, the backup does nothing: he is ready, warming up, but does not touch the ball. That is having redundancy —a backup on standby—. But having a backup on the bench is not enough for the team not to be left without a goalkeeper: someone has to notice that the starter got injured, the coach has to decide to bring in the backup, and the backup has to enter the field and put on the gloves. That process —noticing, deciding, entering— is the failover, and it is not instant: there are a few minutes in which the team plays with an empty goal or with the starter limping. Those minutes are the failover's downtime.

Now imagine the scary scenario. The starter is not injured —he is perfectly fine— but the coach, through a misunderstanding, thinks he got injured and brings in the backup. Now there are two goalkeepers on the field at the same time, both convinced they are the starter, getting in each other's way and leaving gaps. In soccer that is a funny anecdote; in a distributed system, two machines that believe they are "the primary" at the same time —each accepting writes, diverging from one another— is the split-brain, and it can corrupt data irreparably. The cause of the misunderstanding is almost always the same: the coach did not see the starter well (communication failed) and confused "I do not see him" with "he is injured".

The lesson I want you to take from the backup goalkeeper is this: failover is a process with parts and with time, not a magic switch, and its most dangerous part is detection —because distinguishing "the starter went down" from "I cannot see the starter" is genuinely hard, and getting it wrong puts two goalkeepers on the field—. Everything easy about failover we see here. Everything hard —how to really avoid split-brain— we point at and send to the resilience guide.

The three parts of failover

Let us bring the analogy down to mechanics. A failover has three steps, and each one has its cost and its trap.

1. Detect the outage. The system needs to know that the primary died, and it does so by asking it "are you still alive?" every so often. Two ways, which you already brushed against in module 6:

• Health check: an external watcher (the load balancer, a supervisor) makes a periodic request to the component —"do you respond?"— and if it fails several times in a row, it marks it as down.
• Heartbeat: the component itself sends a periodic signal —"I am still here"— to a supervisor, and if the supervisor stops receiving heartbeats for a while, it gives it up for dead.

The trap is in the threshold. If you make it too sensitive (give it up for dead after a single failure), a half-second network hiccup triggers an unnecessary failover. If you make it too tolerant (wait 60 seconds of silence), the failover takes a minute and that minute is downtime. And whatever you do, you can never be 100% sure: "I did not receive heartbeats" can be "it is dead" or "it is alive but the network was cut". That irreducible uncertainty is what leads to split-brain.

2. Decide the replacement. Once the outage is detected, who takes over? If there is a single backup (active-passive), it is trivial: it enters. If there are several candidates (several replicas that could be promoted to primary), you have to choose only one, and have everyone agree on which one —because if two replicas are promoted at the same time, you have a split-brain—. Agreeing on "who is the new boss" when some machines cannot communicate is the problem of leader election, and solving it well requires a consensus algorithm (Raft, Paxos). That is a border: it is named here, taught in the resilience guide and in the distributed systems one.

3. Switch the traffic. With the dead one detected and the replacement chosen, you have to redirect the traffic to the new one. This can be the load balancer that stops sending requests to the downed server (module 6), or a virtual address (VIP) that is reassigned to the new primary, or a DNS update. Each mechanism takes a different amount of time —reassigning a VIP is fast; propagating a DNS change can take minutes because of DNS caching—, and that time is also downtime.

       starter alive                   starter falls               failover complete
    ┌──────────────┐              ┌──────────────┐              ┌──────────────┐
    │  ● primary   │◄─ heartbeat  │  ✗ primary   │   silence    │  ✗ primary   │
    │              │              │   (dead or   │  ───────────►│              │
    │  ○ replica   │              │   isolated?) │   detect     │  ● replica   │◄─ now
    │  (standby)   │              │  ○ replica   │   decide     │  (promoted)  │   primary
    └──────────────┘              └──────────────┘   switch     └──────────────┘
         time ─────────────────────────────────────────────────────────────►
                                  └──── failover window = DOWNTIME ──────────┘

The total failover time —detect + decide + switch— is downtime that the formula of lesson 2 did not count. That is why a system with redundancy and failover does not reach the theoretical 99.99% of the calculation: each outage of the primary costs the failover window. Reducing that window (faster detection without false positives, automatic promotion, quick rerouting) is a good part of the work of real reliability.

Active-passive versus active-active

There are two ways to organize redundancy, and they change how failover looks.

Active-passive. One active component does all the work; one or more passive ones wait on the bench, synchronized but not serving traffic. When the active one goes down, a passive one is promoted to active. It is the model of the database primary/replica you saw in module 5: the primary serves, the replicas wait (well, they serve reads, but only the primary accepts writes), and if the primary dies a replica is promoted. Advantage: simple, no ambiguity about who is in charge. Disadvantage: the passive one is idle capacity (you pay for it to sit on the bench), and there is a failover window while it is promoted.

Active-active. Several components serve traffic at the same time, and when one goes down, the others simply absorb its share —there is no "promotion", only load redistribution—. It is the model of the stateless app servers of module 6: the three apps serve in parallel behind the load balancer, and if one goes down, the load balancer distributes its load between the other two, without ceremony. Advantage: there is no idle capacity (everyone works) and the failover is almost instant (the load balancer just stops sending traffic to the downed one). Disadvantage: it only works if the components are interchangeable, that is, stateless or with the state shared outside —that is why module 6 insisted so much on statelessness—.

The practical rule: the stateless component does active-active and its failover is trivial; the stateful component (like the database) usually does active-passive and its failover is the one you have to design carefully. This explains why in module 6 we worked so hard to make the app stateless: not only does it scale better, it also fails better.

Failover in Enlace

Let us apply the three parts and the two configurations to our topology.

Enlace's app: active-active, trivial failover. Thanks to module 6, Enlace's apps are stateless: any one can handle any request, because they do not store session state (the state lives in the database and the cache). Failover: the load balancer does health checks; if an app stops responding, it takes it out of the rotation and distributes its traffic among the others. The failover window is on the order of the health check interval (seconds), and no in-flight request is lost irreparably —the client retries and lands on another app—. Here the N+1 rule of lesson 2 does its job: if you have one extra app, losing one does not leave you short.

Enlace's database: active-passive, failover to design. The primary accepts the writes; the replicas (module 5) serve reads and wait. If the primary dies:

• Detect: a supervisor notices that the primary stopped beating.
• Decide: an up-to-date replica is chosen (the most current one, with the least lag) and it is promoted to primary.
• Switch: the apps start sending writes to the new primary (by reassigning a virtual address or updating the connection configuration).

During that window, Enlace cannot accept writes (shorten fails or waits), but —watch this, because it is a gift of Enlace's asymmetry— it can keep serving reads from the replicas. Since the 100:1 says that 99% of the traffic is resolve (reads), a failover of the primary degrades only 1% of the traffic (the writes) while it lasts. A shortener that for 20 seconds does not let you create new links but keeps redirecting the existing ones is a minor problem. That is, once again, Enlace's asymmetry turning a serious failure into a tolerable one.

Enlace's load balancer: it cannot be active-passive with slow failover. As we saw in lesson 2, the single load balancer is the SPOF at the door. Its failover is solved with network-level redundancy —two load balancers with a floating virtual IP address that jumps to whichever is alive, or a load balancer managed by the cloud provider that is already redundant internally—. The point: the component at the door needs the fastest failover of all, because while it is down, nothing enters.

The border: where this lesson ends

Here is the line, drawn precisely, because failover is exactly where this guide touches the resilience guide.

What IS part of this lesson (the concept): what failover is, its three parts, active-passive versus active-active, that the failover window counts as downtime, and how all of that looks in Enlace. With this you can reason about the reliability of a design and say "here a failover is needed, and it is going to cost such a downtime window".

What is NOT part of this lesson (the robust mechanism) → resilience-and-reliability-patterns-guide:

• Split-brain and fencing. How to really guarantee that the old primary (if it was just isolated, not dead) does not keep accepting writes once another has been promoted —"fencing it off" so it does not corrupt data—. Here we only name the danger.
• Leader election by consensus (Raft, Paxos). How several replicas agree, without ambiguity and without a central supervisor, on who is the new primary. It is a deep topic with algorithms of its own.
• Retry with backoff. When an operation fails during the failover, how to retry it safely, waiting a little longer each time (exponential backoff) so as not to drown the system that is recovering.
• Idempotency in depth. A retry is only safe if repeating the operation does no harm —creating the same short_code twice, or charging twice—. Here we mention that idempotency is the condition for retrying without fear; the resilience guide teaches how to design it (idempotency keys, operations that can be repeated with no effect).

When in a design you get to "and here we promote a replica and retry the writes that failed", that sentence opens the four doors above. This guide leaves you standing in front of the doors knowing they exist and why they matter; the resilience guide makes you cross them.

Common mistakes

Confusing "not responding" with "it is dead" (of detection). What happens: the system stops receiving heartbeats from the primary and declares it dead, promoting a replica —but the primary was alive, just isolated by a temporary network cut—. When the network comes back, there are two primaries: split-brain. Why it happens: it is impossible, from the outside, to distinguish "dead" from "isolated" from the silence alone. How to spot it: if your failover logic promotes a replacement without any mechanism that prevents the old one from continuing to write, you have a latent split-brain. How to fix it: fencing and consensus —border with the resilience guide—; the point of this lesson is to know that the danger exists and not design a naive failover that ignores it.

Believing that redundancy gives the 99.99% of the calculation (of expectation). What happens: someone computes 1 - (1-a)^2 = 99.99% for two replicas and promises that number, forgetting that each outage of the primary costs a failover window that was not in the formula. Why it happens: the formula assumes instant and perfect takeover. How to spot it: if your measured availability is worse than the calculated one, the difference is usually the accumulated failover time. How to fix it: include the failover window in the estimate (each outage = detect + decide + switch of downtime) and work on reducing it; active-active reduces it to almost zero, active-passive does not.

Setting the detection threshold too sensitive (of tuning). What happens: the system declares a component dead after a single failed health check, so a one-second network hiccup triggers a complete failover —promotion, rerouting, cold cache—, causing more disruption than the hiccup it wanted to avoid. Why it happens: it is optimized for detecting fast without considering false positives. How to spot it: if you see frequent failovers without real outages behind them, your threshold is too sensitive. How to fix it: require several consecutive failures before declaring dead (for example, 3 failed health checks), balancing detection speed against stability —the middle point depends on how expensive an unnecessary failover is in your system—.

Exercises

Exercise 1 — Active-passive or active-active. For each Enlace component, say which redundancy configuration corresponds to it and why, and describe what its failover would look like. (a) The stateless app servers. (b) The database primary. (c) The read replicas.

See solution

• (a) The app servers: active-active. They are stateless (module 6), so they are interchangeable and all serve at once. Failover: the load balancer takes the downed one out of the rotation via health check and distributes its load among the alive ones, almost instant and without promotion.
• (b) The database primary: active-passive. Only one accepts writes at a time (so that there are no conflicts). Failover: detect that it died, promote an up-to-date replica to new primary, reroute the writes. It is the failover you have to design carefully (risk of split-brain).
• (c) The read replicas: active-active among themselves. Several serve reads in parallel; if one goes down, the read load balancer distributes among the others (with N+1, without ending up short). Their failover is trivial like the app's, because a read is interchangeable. Note: a replica is also a passive candidate for primary; in that role it is active-passive.

Exercise 2 — The failover window as downtime. Enlace's primary goes down, on average, 4 times a year. Each failover takes: 10 s of detection (health checks) + 5 s of promotion + 15 s of rerouting. (a) How much write downtime a year do the failovers cause? (b) Why are reads almost unaffected? (c) If you reduced detection from 10 s to 3 s, how much downtime would you save a year?

See solution

• (a) Each failover: 10 + 5 + 15 = 30 s. Four a year: 4 × 30 = 120 s = 2 min/year of write downtime. Modest, but real, and it did not appear in the formula of lesson 2.
• (b) Because during the primary's failover, the replicas remain alive and serving reads. Only the writes stop (shorten), which are 1% of the traffic (100:1). The redirects (resolve) keep working. Enlace's asymmetry turns a primary failover into a minor degradation, not a total outage.
• (c) New window: 3 + 5 + 15 = 23 s. Four a year: 92 s. Savings: 120 - 92 = 28 s/year. Small in downtime, but every second of faster detection also reduces the risk of writes piling up; the tradeoff is not to lower the threshold so much that you trigger false failovers.

Exercise 3 — The danger of split-brain. An engineer proposes this failover for Enlace's primary: "if the supervisor does not receive heartbeats from the primary for 10 seconds, it automatically promotes the most up-to-date replica to primary, and done". Describe the scenario in which this design corrupts data, and name (without implementing them) the two ideas from the resilience guide that would prevent it.

See solution

The corruption scenario (split-brain): the primary is not dead —it is alive and healthy— but the network between the supervisor and the primary was cut for 12 seconds. The supervisor stops receiving heartbeats, believes the primary died, and promotes the replica to new primary. Now there are two primaries accepting writes at the same time: the old one (which is still alive and receiving traffic from the apps that can still reach it) and the new one (the promoted replica). The two diverge —each creates short_codes that the other does not know about, or worse, they assign the same short_code to different URLs—. When the network comes back, there is no clean way to reconcile: the data is corrupted.

The two ideas from the resilience guide that prevent it:

1. Fencing: a mechanism that physically prevents the old primary from continuing to accept writes once another has been promoted —for example, revoking its access to the storage or to the virtual address—, so that even if it is alive it cannot do harm.
2. Leader election by consensus (Raft/Paxos): instead of a supervisor deciding alone, the replicas and the system agree by majority (quorum) on who is the primary, and a majority cannot elect two at once —which makes it impossible to have two legitimate primaries simultaneously—.

The point of this lesson: recognize that the naive failover has this hole. Plugging it well is the resilience guide.

Summary and next step

In this lesson you saw the mechanism that activates redundancy. With the backup goalkeeper you understood that failover is a three-part process —detect that the starter went down, decide the replacement, switch the traffic— and that each part takes time that counts as downtime, so real redundancy never reaches the theoretical 99.99% of the calculation. You distinguished active-passive (a standby backup that is promoted, like the database) from active-active (several serving, and the alive ones absorb the downed one, like the stateless app), and you saw why the stateless component fails better. You applied it to Enlace: active-active app with trivial failover, active-passive database with failover to design, and the 100:1 asymmetry that makes a primary failover degrade only the writes while the reads keep going. And you marked the border: split-brain, fencing, consensus, retry with backoff and idempotency in depth are the resilience guide.

Before moving on you should be able to: name the three parts of failover and why each one is downtime; distinguish active-passive from active-active and say which one each Enlace component gets; explain why "not responding" is not the same as "it is dead" and how that confusion produces split-brain; and locate the border with the resilience guide.

What comes next is crossing into the module's second block. Notice the seed this lesson left: the failover trap was that the system cannot distinguish a dead component from a component isolated by the network. That network isolation has a name —network partition— and it is the protagonist of the most famous and most misunderstood topic in distributed systems. In lesson 4 you will see the CAP theorem in its correct form: when the network partitions, a system has to choose between continuing to respond (availability) and continuing to be exact (consistency), and it cannot have both. And you will dismantle once and for all the myth of "pick 2 of 3", which is the wrong way of saying something that is actually simpler and more useful.

Resources

• Designing Data-Intensive Applications, Martin Kleppmann — Chapter 5, "Handling Node Outages" and "Leader Failover" — the reference treatment of failover in the context of primary/replica replication, including why automatic failover is dangerous and the split-brain problems that here we only name.
• System Design Primer — Failover: Active-passive and Active-active — a concise summary of the two configurations of this lesson, with their tradeoffs and the term "fencing" for the split-brain problem.
• The Raft Consensus Algorithm — official site (raft.github.io) — the door to the part this lesson leaves as a border: how a group of machines elects a leader by consensus without ambiguity or split-brain. The interactive visualization makes tangible why consensus is the correct answer to the replacement choice.