Module 7: Reliability and the Consistency Tradeoff

2. Redundancy and single points of failure

Next →

Description

By the end of this lesson you will know how to identify a system's single points of failure and how to compute —by running the formula— how much availability you gain by eliminating them with redundancy. You will learn what availability is exactly as a number (uptime / (uptime + downtime)), and the two ways components compose: in series, when any one that goes down takes the request with it (and there the availabilities multiply, so each additional component subtracts reliability), and in parallel, when you put redundant copies and the group only fails if all of them fail (and there availability rises as 1 - (1-a)^N). You will run both formulas over Enlace's topology, you will see how a single weak component —the database's lone primary— drags the whole system down to 89 hours of downtime a year, and how making it redundant lowers that to 2.6. And you will meet the operational rule that falls out of all this: N+1, always keep one spare unit more than you need.

This matters because availability is the topic where intuition fails the most, and the only cure is arithmetic. People assume that "adding more machines" always helps, when adding more machines in series —a chain of dependencies where all of them have to work— actually lowers the reliability of the whole. And people underestimate how much real redundancy helps: going from one copy to two, in parallel, can turn 87 hours of downtime a year into less than one. Without the formula, those facts are surprises; with the formula, they are calculations you make before the system is in production. This lesson gives you the formula, makes you run it, and teaches you to read a topology looking for the link that, if it breaks, takes everything down with it —which is the first question any serious reliability review asks—.

Connection to the module: this is the first lesson of the reliability block, and it is the foundation of the second. Here you learn to find single points of failure and to quantify the benefit of eliminating them; lesson 3 (failover) explains the mechanism that makes that elimination possible —how the backup takes over when the primary goes down—. Without redundancy there is no possible failover (you cannot hand the work to a backup that does not exist), so this lesson comes first. The numbers you compute here —how much downtime each configuration has— are also the raw material of lesson 7, where we translate availability into the nines table and the error budget. And the topology we analyze is the one you scaled in modules 5 and 6; here we look at it with fresh eyes, searching for where it breaks.

The chain and its weakest link

Think of it this way. A chain holding up a weight is only as strong as its weakest link. It does not matter that nine of its ten links are tempered steel able to hold tons: if the tenth is plastic, the chain breaks with the first pull, and it breaks at the plastic. Adding strong links does not help; in fact, every link you add is one more chance for something to fail, because now there is one more link that can break. A chain of a single strong link is more reliable than a chain of ten links where one is weak.

A system of components in series is exactly that chain. When an Enlace request has to pass through the load balancer, then the app server, then the cache or the database —and all of them have to work for the request to be answered—, those components are in series. The availability of the whole is neither that of the best component nor the average: it is like the chain, the weakest one rules, and worse still, each additional component lowers it a bit, because it multiplies in a probability less than 1.

Now, the other side. Imagine that instead of one chain you have two chains in parallel holding the same weight, so that it is enough for one of them to hold. Now the weight only falls if both break at the same time. If each chain fails 1% of the time, the two fail together 1% of 1% —one part in ten thousand—. That is redundancy: not one more link in the same chain (which subtracts), but a whole spare chain in parallel (which multiplies the safety). The difference between series and parallel is the difference between "all of them have to work" and "it is enough for one to work", and it is all the math of reliability in two sentences.

A single point of failure (SPOF) is a component that is in series and has no partner in parallel: a lone link whose breakage takes everything down. The load balancer, if there is only one, is a SPOF. The database primary, if there is only one, is a SPOF. The task of this lesson is to find them and, with the formula, decide which are worth making redundant.

Availability as a number

Before composing anything, let us define the piece. The availability of a component is the fraction of time that it is working:

                    uptime
availability = ─────────────────────
                uptime + downtime

A component with 99.9% availability is down 0.1% of the time. In a year of 365 days that is 0.001 × 365 × 24 × 3600 s ≈ 31,536 s ≈ 8.76 hours. That is the number that lesson 7 will turn into a full table; here we only need to know that availability and downtime are two ways of saying the same thing: downtime = (1 - availability) × total_time. The closer availability is to 1, the less downtime; and as we will see, each "nine" you add divides the downtime by ten.

With that piece, the module's question becomes arithmetic: given a system made of components with certain availabilities, connected in series and in parallel, what is the availability of the whole? There are exactly two rules.

Rule 1 — In series, availabilities multiply

When N components are in series —all of them have to work for the system to work— the availability of the whole is the product of their availabilities:

A_series = a1 × a2 × ... × aN

Since each ai is less than 1, multiplying lowers the result: each component you add in series subtracts availability from the whole. Let us see it with Enlace's chain, executed:

SECONDS_PER_YEAR = 365 * 24 * 3600

def downtime_h(a):
    """Downtime in hours per year for an availability a."""
    return (1 - a) * SECONDS_PER_YEAR / 3600

# Enlace's serial chain: every request passes through the four,
# and any one that goes down takes the request with it.
lb    = 0.9999    # load balancer
app   = 0.999     # app service
db    = 0.999     # database
cache = 0.999     # cache

serial = lb * app * db * cache
print(f"series availability = {serial:.6f}  ({serial*100:.3f}%)")
print(f"downtime = {downtime_h(serial):.2f} h/year")

What to expect. When you run it:

series availability = 0.996903  (99.690%)
downtime = 27.13 h/year

Look at the number and stop on it. None of the four components is bad on its own: the worst has 99.9% (8.76 h/year). But chained together, the whole falls to 99.69%, which is 27 hours a year —more than triple the downtime of the worst individual component—. That is the trap of series: the reliability of the system is worse than that of any of its parts, because for the system to work they all have to work at the same time, and the probabilities of "all at the same time" multiply downward. This is the fact that most surprises anyone who has not done the math: a system with more components in series is, all else equal, less reliable, not more.

Rule 2 — In parallel, redundancy raises availability

Now the good news. When you put N copies of a component in parallel —it is enough for one to work for the group to work— the group only fails if all of them fail. The probability that a component is down is (1 - a); that of N independent copies being down at the same time is (1 - a)^N. So the availability of the redundant group is:

A_parallel = 1 - (1 - a)^N

Here, adding copies raises availability, and it rises fast, because raising a small number to a power makes it tiny. Let us run it with a modest component —an app with 99% availability, a component that is only down... 3.65 days a year— and see what happens when we add copies:

def parallel(a, n):
    """Availability of N redundant copies in parallel."""
    return 1 - (1 - a) ** n

a = 0.99   # a single modest app: 99%
for n in (1, 2, 3):
    grp = parallel(a, n)
    print(f"N={n}: availability = {grp*100:.4f}%   downtime = {downtime_h(grp):.2f} h/year")

What to expect. When you run it:

N=1: availability = 99.0000%   downtime = 87.60 h/year
N=2: availability = 99.9900%   downtime = 0.88 h/year
N=3: availability = 99.9999%   downtime = 0.01 h/year

This is the gift of redundancy, and it is enormous. A single app at 99% is down 87.6 hours a year —almost four days—. Put two in parallel and the group jumps to 99.99%: less than one hour a year. Put three and it is six minutes a year. Each copy you add multiplies the downtime by (1 - a) —divides it by 100 in this case, because 1 - 0.99 = 0.01—. From a mediocre component you make, with two copies, an excellent one. That is the reason "don't have single points of failure" is reliability's rule number one: going from N=1 (a SPOF) to N=2 (redundant) is the cheapest and biggest change you can make.

Putting it together: removing Enlace's SPOF

Now the full scene. Suppose that in Enlace's topology the load balancer and the app are already replicated (good, 99.99% each group), but the database has a lone primary with 99% availability —the weak link, the SPOF—. How much does that SPOF weigh, and how much do we gain by eliminating it with a second database instance in parallel?

# Enlace with a LONE database primary (weak SPOF at 99%)
sysA = 0.9999 * 0.9999 * 0.99          # lb, app (already redundant) x lone db
print("With lone primary (SPOF):")
print(f"  total = {sysA*100:.3f}%   downtime = {downtime_h(sysA):.2f} h/year")

# We make the database redundant: two instances in parallel
db_redundant = parallel(0.99, 2)       # 1 - (1-0.99)^2
sysB = 0.9999 * 0.9999 * db_redundant
print("With redundant database (two at 99% in parallel):")
print(f"  redundant db = {db_redundant*100:.4f}%")
print(f"  total = {sysB*100:.3f}%   downtime = {downtime_h(sysB):.2f} h/year")

What to expect. When you run it:

With lone primary (SPOF):
  total = 98.980%   downtime = 89.33 h/year
With redundant database (two at 99% in parallel):
  redundant db = 99.9900%
  total = 99.970%   downtime = 2.63 h/year

There is the lesson in two numbers. With the lone primary, all of Enlace —however good the load balancer and the app are— inherits the weakness of the most fragile link and falls to 98.98%, that is 89 hours of downtime a year. The database SPOF rules over the entire system, exactly as the plastic link rules over the chain. Make it redundant —a second instance in parallel— and that group rises to 99.99%, and the whole system goes to 99.97%: 2.63 hours a year. A single extra machine turned 89 hours of downtime into less than 3. That is the return of eliminating a single point of failure, and it is the reason the first question of any reliability review is "what are my SPOFs?".

An honest note that lesson 3 will develop: this formula assumes that the backup takes over instantly and without failing. In reality, going from the dead primary to the replica —the failover— takes time and can go wrong, and that time counts as downtime. Redundancy is the necessary condition (you have to have a backup), and failover is the mechanism that takes advantage of it. Here we compute the ceiling of the benefit; lesson 3 explains what separates you from it.

The N+1 rule (and N+2)

Out of all this arithmetic comes an operational rule you will see in any serious infrastructure team: N+1. If you need N units of something to serve your load —N app servers, N read replicas—, always keep N+1: one spare. That way, when one goes down (or you pull it for maintenance), the remaining N still cover the load and you are not left short. The most critical systems use N+2 (two spares), to tolerate a second one going down while you repair the first, or to be able to do maintenance on one without running out of margin.

For Enlace this is concrete. If you computed (in module 5) that you need 4 read replicas to serve the ~4,000 reads/s comfortably, N+1 says run 5: when one replica goes down, the remaining 4 still keep up, and resolve does not degrade. Redundancy is not just "two instead of one so as not to have a SPOF"; it is "always a bit more than you need, so that losing one does not leave you short". The cost is having idle capacity most of the time; the benefit is not going down the day something fails —which is when, not if—.

Common mistakes

Confusing adding components in series with making the system more robust (conceptual). What happens: someone adds one more layer —a proxy, an intermediate service, a validator— "to have more control" and unintentionally lowers the availability of the system, because that component is now in series and multiplies downward. Why it happens: "more pieces" sounds like "more solid", when in series it is the opposite. How to spot it: ask yourself of each new component "is it one more link in the chain (series, subtracts) or a parallel copy of an existing one (redundancy, adds)?". If it is series and it was not essential, you lowered availability. How to fix it: minimize the components in series on the critical path, and everything that is in series, make it redundant in parallel.

False redundancy: two copies that share a hidden SPOF (architectural). What happens: two database replicas are set up "for redundancy", but the two run on the same physical server, or in the same rack, or depend on the same shared disk, or the same network cable. The formula 1 - (1-a)^N assumes independent failures; if the two copies go down together when the shared resource goes down, the redundancy is an illusion and the shared resource is the real SPOF. Why it happens: redundancy is looked at in the logical layer (two processes) and not in the physical one (one power source). How to spot it: ask yourself "what single failure would take both copies down at once?". If there is an answer, there is your real SPOF. How to fix it: spread the copies across different failure domains —different machines, racks, availability zones— so that their failures are truly independent.

Over-redundancy where it does not weigh (of priority). What happens: a team puts three copies of a component that already had 99.99% and neglects the one with 99% —spending effort where the return is tiny and leaving the weak link intact—. Why it happens: what is redundant is what is visible or convenient, not what weighs the most. How to spot it: in the series formula, the smallest term (the least available component) dominates the product; if you make another one redundant, you barely move the total. How to fix it: find the weakest link (the smallest ai) and make it redundant first; there is the greatest return, as you saw when moving the database from 99% to 99.99% and gaining 87 hours.

Exercises

Exercise 1 — Find the SPOF. In this Enlace topology, say which components are single points of failure (they are in series and have no partner in parallel). A single load balancer → three app servers → one shared cache → two databases (primary + replica) in parallel. For each SPOF, say what would happen to the system if that component went down.

See solution
  • The single load balancer is a SPOF. It is in series (every request passes through it) and has no partner. If it goes down, no request enters the system, however healthy the app and the databases behind it are. It is the most dangerous SPOF because it is at the door.
  • The single shared cache is a SPOF if the system does not know how to work without it. If, when the cache goes down, the apps go straight to the database (cache-aside from module 4), it does not take the system down but it does degrade it (more load on the database). If the apps depend on the cache to respond, then it is a hard SPOF.
  • The three app servers are NOT a SPOF individually: they are in parallel (the load balancer distributes among them). If one goes down, the other two carry on —though N+1 is advisable so as not to end up short—.
  • The two databases are NOT a SPOF: primary + replica are in parallel for data availability. If the primary goes down, the replica is promoted (failover, lesson 3).

Fix priority: the load balancer first (SPOF at the door, takes everything down), then decide the cache policy.

Exercise 2 — Run the formula. An app service has 99.5% availability. (a) How many hours a year is a single one down? (b) And a group of two in parallel? (c) And of three? Use downtime_h(a) = (1-a) × 365 × 24 × 3600 / 3600 and parallel(a, n) = 1 - (1-a)^n.

See solution

With a = 0.995, then 1 - a = 0.005:

  • (a) N=1: 0.005 × 365 × 24 = 43.8 h/year. A single service at 99.5% is down almost two days a year.
  • (b) N=2: parallel = 1 - 0.005^2 = 1 - 0.000025 = 0.999975. Downtime = 0.000025 × 8760 h ≈ 0.219 h ≈ 13.1 min/year. From 43.8 hours to 13 minutes with a single extra copy.
  • (c) N=3: parallel = 1 - 0.005^3 = 1 - 0.000000125 = 0.999999875. Downtime ≈ 0.0011 h ≈ 3.9 s/year. Practically zero.

The repeated moral: each copy in parallel multiplies the downtime by (1-a) —here by 0.005—, so it plummets. The big jump is always from N=1 to N=2 (eliminating the SPOF).

Exercise 3 — The weak link rules. Enlace has its load balancer and its app already redundant (groups at 99.99%). The team debates between two investments with the same cost: (a) add a third copy to the app (take its group from 99.99% to ~99.9999%), or (b) make the database redundant, which today is a lone primary at 99.9%. With the series formula, compute the total downtime of each option and say which is better and why.

See solution

Common base: lb = 0.9999, app = 0.9999, db = 0.999 (lone primary).

Current state: 0.9999 × 0.9999 × 0.999 = 0.998801, downtime ≈ 0.001199 × 8760 ≈ 10.5 h/year.

  • Option (a): improve the app to 99.9999%. 0.9999 × 0.999999 × 0.999 = 0.998900, downtime ≈ 0.0011 × 8760 ≈ 9.63 h/year. Almost no change: it dropped from 10.5 to 9.6 h.
  • Option (b): redundant database at 99.9999% (two at 99.9% in parallel: 1 - 0.001^2 = 0.999999). 0.9999 × 0.9999 × 0.999999 = 0.999799, downtime ≈ 0.000201 × 8760 ≈ 1.76 h/year. It dropped from 10.5 to 1.76 h.

Option (b) is better, and by a lot. The reason is the weak-link rule: in the product, the smallest term (the db at 99.9%) dominates the result. Improving the app —which was already the strong link— barely moves the total; making the database redundant —the weak link— is what lowers the downtime from 10.5 to 1.76 hours. Always make the least available component redundant first.

Summary and next step

In this lesson you put a number on the first blackout. With the chain and its weakest link you understood the two ways to compose reliability: in series (all of them have to work) the availabilities multiply, so each additional component subtracts and the weakest link rules; in parallel (it is enough for one to work) redundancy adds as 1 - (1-a)^N, and rises fast. You ran it: four good components in series fall to 99.69% (27 h/year); an app at 99% goes from 87.6 h/year to less than 1 just by duplicating it; and the SPOF of the lone primary drags Enlace to 89 h/year, which becomes 2.6 by making it redundant. And you met the N+1 rule: always keep one spare unit more than you need, so that losing one does not leave you short.

Before moving on you should be able to: define availability as a number and convert it to downtime; distinguish series from parallel and say what happens to availability in each case; identify a topology's single points of failure; and compute with the formula how much downtime you save by making the weakest link redundant.

What comes next is the mechanism that makes redundancy real. Having a spare replica is worth nothing if the system does not know how to hand it the work when the primary goes down. In lesson 3 you will see failover: how it is detected that a component has died (health checks and heartbeats, which you already brushed against in module 6), how the backup takes over, how long that handoff takes (and why that time counts as downtime), and where the easy part ends and the hard part begins —split-brain, fencing, consensus— which is the border with the resilience guide. It is the step from "I have a backup" to "the backup goes into action when I need it".

Resources

← Previous1. Module introduction: when the machine goes down and the network splitsNext →3. Failover basics