1. Module introduction: when the machine goes down and the network splits

Description

By the end of this lesson you'll understand which two questions this module answers, and why they're the two left pending after scaling Enlace. Over six modules you worked on a single question with many faces: does it handle the load? You put numbers on it (~40 writes/s, ~4,000 reads/s, ~6 TB at five years), you modeled the Link record, you put a cache in front, you replicated it, you sharded it, and you made it stateless behind a balancer. By the end of module 6, Enlace is a system that runs on many machines and serves a lot of traffic. But "runs on many machines" opens two new questions that scale by itself doesn't answer. First: what happens when one of those machines goes down? A balancer, a primary, a replica —any piece can die on a Tuesday at three in the afternoon, and a serious system has an answer for that—. Second: when the network between two machines cuts out for a moment —and it cuts out, it's not an "if" but a "when"—, what do you prefer Enlace to do: keep responding even with slightly old data, or refuse to respond so as not to lie? This module answers both, and closes with the yardstick that measures both: how much downtime a year promising "99.9%" really means.

This matters because it's the topic that separates someone who knows how to make a fast system from someone who knows how to make a system you can trust. Scale is half the work; the other half is reliability and the consistency tradeoffs, and it's exactly the half people get most wrong —because it's full of myths—. The most famous is "the CAP theorem says you pick 2 of 3", a phrase repeated in thousands of interviews and that is wrong. This module disarms it and leaves you the correct statement, which is more useful and less mystical. It also gives you the real math of availability —why adding components in series lowers your reliability, and why redundancy raises it by an exact formula you'll run—, and it teaches you to translate an availability percentage into hours of downtime, which is the only honest way to talk about an SLA. By the end, you'll be able to look at Enlace and say, with numbers and without myths, how reliable it is, what consistency it needs, and what SLO it can promise without lying.

Connection to the module: this lesson is the map, not the territory. Here you won't design anything yet; you'll understand the order of the six lessons that follow and why they go in three blocks. The first block is reliability: lesson 2 defines availability and the single point of failure, with the composition math (series vs. parallel) run; lesson 3 introduces basic failover —detecting that something died and switching to a backup— and marks the boundary with the resilience guide. The second block is the consistency tradeoff: lesson 4 states CAP in its correct form and disarms the myth; lesson 5 completes it with PACELC (what you choose when there's no partition); and lesson 6 brings it all down to the case, choosing eventual consistency for Enlace and justifying it with the number. The third block is the yardstick: lesson 7 runs the table of nines and the error budget. Lesson 8 —the project— puts the three things together into a written decision for Enlace: its reliability plan, its CAP/PACELC classification, its consistency model, and its SLO.

The two blackouts of a house

Think of it this way. Imagine you manage the power supply of a hospital. There are two very different ways the power can fail you, and you confuse them under the word "blackout" at your own risk.

The first blackout is that something breaks. A fuse blows, a cable cuts, the generator runs out of diesel. The power goes because a physical piece failed. The answer to this blackout is redundancy: don't depend on a single piece. Have two power sources, two cables, a backup generator that starts on its own when the public grid goes down. If the hospital has a single generator and that generator fails, the generator is a single point of failure: a single piece whose death turns everything off. Removing the single points of failure —giving them a twin that takes over— is this module's reliability block (lessons 2 and 3).

The second blackout is subtler, and it's not that something breaks. Imagine the hospital has two wings, North and South, each with its own electrical panel, and the two panels coordinate over a communication line to not overload the grid. One day that communication line between the two wings cuts out —the electricity doesn't fail, the phone between the two panels fails—. Now each panel is alone and has to decide without being able to consult the other. It can take two attitudes. One: "without confirmation from the other panel, I turn on nothing new, I'd rather shut down than risk overloading" —that's preferring consistency: I don't act if I'm not sure—. The other: "I keep giving power with the information I have, and when the communication comes back we reconcile" —that's preferring availability: I keep working even though I might be uncoordinated—. Neither is "correct" in the abstract; it depends on which costs more, staying in the dark or ending up uncoordinated. That decision —what to do when the communication splits— is the heart of CAP and PACELC, the module's second block (lessons 4, 5, and 6).

The lesson I want you to take from the hospital is this: "the system failed" isn't a single problem, it's two. One is that a piece broke, and it's resolved with redundancy and failover. The other is that the communication between pieces cut out, and there's no magic repair —there's a design decision between keeping responding or keeping exact—. This whole module is learning to look at Enlace and, for each of the two blackouts, have a thought-out answer instead of a surprise.

The case: where Enlace stands at the start of this module

We don't start from scratch. Enlace is already scaled; what's left is to make it reliable and decide its consistency. Let's recall the topology we reached, and reproduce the anchor numbers —as throughout the guide, we don't quote them from memory—:

# Enlace's anchor numbers, reproduced once more (not quoted)
writes_per_month = 100_000_000
seconds_per_month = 30 * 24 * 3600            # 2,592,000 s

qps_write = writes_per_month / seconds_per_month
qps_read = qps_write * 100                     # read:write ratio = 100:1

print(f"qps_write = {qps_write:.0f} writes/s")
print(f"qps_read  = {qps_read:,.0f} reads/s")
print(f"read:write ratio = {qps_read/qps_write:.0f}:1")

What to expect. When you run it:

qps_write = 39 writes/s
qps_read  = 3,858 reads/s
read:write ratio = 100:1

Rounding: ~40 writes/s, ~4,000 reads/s, 100:1 ratio. These three numbers aren't decoration: they're the ones that will decide Enlace's consistency in lesson 6. Notice the asymmetry, because it's the key to the whole module. Enlace is a system where almost all the traffic is reading (resolve: someone clicks a short link and expects it to redirect them), and very little is writing (shorten: someone creates a new link). And Enlace's read is the most harmless in the world: returning a URL to redirect. Does it matter if that URL is one second out of date? Almost never. That combination —traffic dominated by lag-tolerant reads— is exactly the profile that makes eventual consistency the correct choice, and we'll prove it with numbers in lesson 6.

Here's the scaled topology we bring from modules 4, 5, and 6, drawn to see where this module's problems lurk:

                       ┌──────────────────┐
   clients ──────────► │  load balancer   │  ◄── what if this box goes down?
                       └────────┬─────────┘      (SPOF — lesson 2)
                    ┌───────────┼───────────┐
                    ▼           ▼           ▼
                ┌───────┐   ┌───────┐   ┌───────┐
                │ app 1 │   │ app 2 │   │ app 3 │  ◄── stateless (module 6):
                └───┬───┘   └───┬───┘   └───┬───┘      if one goes down, the others continue
                    └───────────┼───────────┘
                    ┌───────────┼───────────┐
                    ▼           ▼           ▼
                ┌───────┐   ┌───────┐   ┌───────┐
                │ cache │   │primary│──►│replica│  ◄── what if the primary dies?
                └───────┘   └───────┘   └───────┘      (failover — lesson 3)
                                │
                        replication lag ◄── is the replica behind?
                                            (consistency — lessons 4-6)

Each question mark in that diagram is a lesson of this module. The single balancer is a single point of failure (lesson 2). The primary that dies calls for a failover (lesson 3). And the replica that's behind due to replication lag —which you already met in module 5— is the door to the consistency tradeoff (lessons 4, 5, and 6). This module is walking that diagram question by question.

The map of the six lessons, in three blocks

Lesson	Block	The problem it solves, in one sentence
2	Reliability	Recognizing the single points of failure and computing how much availability rises by making them redundant
3	Reliability	What to do at the moment a piece dies: detect and switch to a backup (failover)
4	Consistency	The correct statement of CAP: under a partition, you choose between consistency and availability
5	Consistency	PACELC: what you choose when there's no partition (latency vs. consistency)
6	Consistency	Strong vs. eventual consistency, and why Enlace gets the eventual
7	The yardstick	Translating "99.9%" into hours of downtime a year, with the error budget, run

The three blocks go in this order for a reason. First reliability (lessons 2-3), because it's the most concrete and the most similar to what you already know: it's the natural continuation of "I have many machines" from module 6 —now, what happens if one goes down?—. Then the consistency tradeoff (lessons 4-6), which is more conceptual and where the myths that need disarming live; it comes after reliability because it needs the concept of a network partition, which is a type of failure, and we study failure first. And at the end the yardstick (lesson 7), because only when you understand what can bring you down (reliability) and what lags you tolerate (consistency) does it make sense to put a number on the promise —"I promise 99.9%"— and see how much downtime a year that really means.

What this module does NOT touch

It's good to mark the boundary from now, because there are neighboring topics that seem to belong here and belong to another guide in the ecosystem.

The resilience patterns in depth are the sibling guide. In lesson 3 you'll mention that a primary goes down and a replica is promoted, and you'll name failover's hard problems —split-brain (two machines that both believe they're the boss), fencing, leader election by consensus—. But how those guarantees are implemented, and the whole arsenal of patterns for a system to withstand failures without going down —circuit breaker, bulkhead, retry with backoff, and idempotency in depth— is the guide resilience-and-reliability-patterns-guide. Here idempotency is mentioned (a safe retry needs repeating an operation to do no harm), but it's taught there. When in lesson 3 I say "and here idempotency is needed", that sentence is a door to that guide; I point it out to you, I don't cross it.

Event sourcing and CQRS are from the events guide. When in the consistency lessons we talk about "reconciling replicas that diverged" or "propagating a change to several copies", you'll brush against ideas that are taken to the extreme in event-driven architecture: storing the history of changes as a log of events (event sourcing) and separating the write model from the read one (CQRS). That's the guide event-driven-architecture-guide. Here we stay in the conceptual tradeoff (strong vs. eventual); the techniques to build systems around events are from there.

The sharding and replicas detail is module 5, already seen. This module rests on the primary/replica replication and the replication lag you learned in module 5, but it doesn't re-teach them: it uses them as raw material to talk about failover and consistency. If something about "the replica is behind" sounds familiar, it's because you already saw it there; here we give it the conceptual framework.

Common mistakes

Confusing "scaling" with "making reliable" (mental-model mistake). What happens: a team puts a lot of effort into making the system fast and handle load, and assumes that for that reason it's reliable —until a single component goes down and brings everything down, or a network partition produces incoherent responses no one anticipated—. Why it happens: scale is visible (latency, QPS) and reliability is invisible until it fails. How to detect it: if you can't name your system's single points of failure or say what it does under a network partition, you scaled but didn't make it reliable. How to fix it: it's exactly what lessons 2 to 6 do —name each failure mode and a decision for each one—.

Repeating the "CAP: pick 2 of 3" myth (concept mistake). What happens: someone says "it's a CP system" or "we chose AP sacrificing C" as if the three letters were three independent knobs from which you turn one off. Why it happens: the phrase "pick two" is catchy and gets repeated without understanding it. How to detect it: if you believe you can "choose not to have P" (partition tolerance), you didn't understand CAP —partitions happen whether you like it or not; they're not an option—. How to fix it: lesson 4 gives you the correct statement —the choice between C and A happens only during a partition, and the rest of the time you have both—.

Talking about availability without translating it into numbers (rigor mistake). What happens: someone promises "high availability" or "99.9%" without ever having computed how many hours of downtime a year that is, and is then surprised when the system, meeting its SLA to the letter, was down for almost nine hours. Why it happens: the percentages close to 100 fool the intuition —99% and 99.9% sound almost the same and differ by a factor of ten—. How to detect it: if you can't say from memory (or compute in ten seconds) that 99.9% is 8.76 h/year, your availability intuition isn't calibrated. How to fix it: lesson 7 runs the complete table; after seeing it, "99.9%" stops being a slogan and becomes a budget of hours.

Exercises

Exercise 1 — Which blackout is it? For each Enlace scenario, say whether it's the "first blackout" (a piece broke → reliability, lessons 2-3) or the "second blackout" (the communication split → consistency, lessons 4-6). (a) The single load balancer restarts for an update and for 30 seconds no request enters the system. (b) The network between the primary's data center and a replica's data center cuts out for 5 seconds; both stay alive but don't talk to each other. (c) The primary server's disk is damaged and the database stops responding.

See solution

• (a) First blackout (reliability). A piece —the balancer— stopped working, and since it's the only one of its kind, it brought everything down. It's a single point of failure (lesson 2); the answer is redundancy (a second balancer) and failover (lesson 3).
• (b) Second blackout (consistency). Nothing broke: the two machines are alive, but the communication between them cut out. That's a network partition, and it forces the CAP decision: while it lasts, does the replica keep serving possibly-old reads (availability) or refuse so as not to give incoherent data (consistency)? (lessons 4-6).
• (c) First blackout (reliability). A piece —the primary's disk— broke and the database died. The answer is failover: promote a replica to primary (lesson 3), which in turn requires that a replica existed (redundancy, lesson 2).

Exercise 2 — The intruder. Of the following topics, three are taught in this module and two belong to sibling guides. Identify the two intruders and say which guide they go to. (a) Computing how much downtime a year 99.99% is. (b) Implementing a circuit breaker that stops calling a downed service. (c) Stating PACELC and classifying Enlace. (d) Storing each state change as an immutable event in a log (event sourcing). (e) Identifying the single points of failure of a topology.

See solution

The three from this module: (a) the nines table is lesson 7; (c) PACELC is lesson 5; (e) the SPOFs are lesson 2.

The two intruders:

• (b) The circuit breaker is a resilience pattern → resilience-and-reliability-patterns-guide. Here we mention that a component goes down, but the concrete pattern to stop calling it is taught there.
• (d) Event sourcing is from the event-driven-architecture-guide. Here we talk about the consistency tradeoff, but building the system around a log of events is from that guide.

Exercise 3 — Enlace's asymmetry. With the anchor numbers reproduced above (~40 writes/s, ~4,000 reads/s, 100:1 ratio), argue in two or three sentences why this asymmetry suggests —before seeing lesson 6's proof— that eventual consistency could be acceptable for Enlace. Hint: think about which operation dominates the traffic and how serious it is that its result be one second out of date.

See solution

Enlace's traffic is dominated almost entirely by reads (resolve): 4,000 per second against 40 writes, a 100 to 1. And that read is one of the most harmless possible: returning a URL to redirect someone who clicked a short link. If that URL is one second out of date —because the replica that served it is a bit behind the primary— it almost never matters: the link already existed and still points to the same thing. The only awkward case would be a just-created short_code that hasn't reached the replica yet, but that affects a tiny fraction of the traffic and only in the first seconds of a link's life. Since the dominant operation tolerates the lag, paying the cost of strong consistency (coordination, latency, lower availability under a partition) would be a bad deal. That suggests eventual consistency; lesson 6 proves it with the propagation-window numbers.

Summary and next step

In this lesson you understood that this module answers the two questions scale left pending. With the two-blackout hospital you saw that "the system failed" isn't one problem but two: one is that a piece broke —cured with redundancy and failover, the reliability block—; the other is that the communication between pieces split —there's no cure, there's a decision between keeping responding or keeping exact, the consistency-tradeoff block—. You reproduced the anchor numbers and saw Enlace's key asymmetry: 100 reads per write, and a read so harmless that it tolerates being one second behind. You met the map of the six lessons in three blocks —reliability (2-3), consistency (4-6), the nines yardstick (7)— and the project that puts them together (8). And you marked the boundary with the resilience guide (idempotency and retry in depth) and with the events one (event sourcing/CQRS).

Before moving on you should be able to: distinguish the two types of "blackout" and say which module block resolves each; explain why "CAP is picking 2 of 3" is a myth lesson 4 will disarm; and argue, from the 100:1 asymmetry, why eventual consistency looks acceptable for Enlace.

What comes next is the first block, reliability. In lesson 2 you'll put a name and a number on the first blackout: what exactly availability is, what a single point of failure is, and why the reliability math is counterintuitive —adding components in series lowers your availability, and only redundancy in parallel raises it—. And you'll run it: you'll see, with the formula run in Python, how removing Enlace's single primary as a single point of failure lowers its downtime from 89 hours a year to 2.6. It's the step from "I know a component can go down" to "I know how much it costs me for it to go down and how much I save by making it redundant".

Resources

• Designing Data-Intensive Applications, Martin Kleppmann — Chapter 5 (Replication) and Chapter 9 (Consistency and Consensus) — the canonical reference for this whole module. Chapter 5 lays the foundations of replicas and lag we use for failover; chapter 9 formalizes the linearizability and CAP we'll see in lessons 4 to 6. If you read a single distributed-systems book, let it be this one.
• System Design Primer — Availability patterns and CAP theorem — a practical and much-cited summary of the same topics (fail-over, replication, CAP), useful as a quick map before entering each lesson's detail.
• Google SRE Book — Chapter 4: Service Level Objectives — the reference treatment of SLI/SLO/SLA and the error budget we'll run in lesson 7; it's short and transformed how the industry talks about reliability.