2. Partial failure: in distributed systems, something is always broken

Overview

In lesson 1 we named partial failure in passing; here we open it up, because it is the root concept from which the remaining seven modules hang. The idea is simple to state and hard to internalize: in a distributed system, at any given instant, some parts work and others don't —and that mix is the normal state of operation, not an emergency—. It's not that "every once in a while something goes down." It's that, with enough pieces connected over the network, the probability that all of them are healthy at the same time is low, and it drops fast as you add pieces. A distributed system spends most of its life in some partially degraded state. Designing for "everything works or everything goes down" —the monolith's mental model— is designing for a state your system is almost never in.

You're going to learn three concrete things. First, the four outcomes of a remote call —success, explicit error, slowness, and the dreaded ambiguous— and why the ambiguous one (I don't know whether my charge happened) is qualitatively different from anything you knew in a single process. Second, why slowness and ambiguity don't exist inside a process and appear the moment you cross the network. And third —the measurable heart of the lesson— the state space of a system with several dependencies: you'll run in Python a count showing that with just three dependencies, each one in up/slow/down, there are 27 possible states and only one is "all healthy" —3.7%—. That number is the numerical justification for the whole guide: if the happy state is 1 of 27, you'd better design for the other 26.

Connection with the module: lesson 1 showed you one degraded state (shipping slow) and its consequence (the cascade). This lesson generalizes: it shows you that that isn't a bad day, but one of a mountain of possible states, and it gives you the precise vocabulary (the four outcomes) to talk about each one. With that, lesson 3 can explain why our intuition fails us here —the fallacies— and lesson 4 can dissect how a degraded state propagates —the cascade— already knowing precisely what "being degraded" means.

The analogy: the letter and the conversation

Think of it this way. When you talk to someone at the same table, the communication has two outcomes: they hear you and respond, or —if they faint— the whole conversation ends abruptly, and you notice instantly. There's no state of "I spoke to them, I don't know if they heard me, and I don't know if they'll answer in ten minutes or never." They're together; the response is immediate or the end is obvious.

Now imagine you send them a letter by mail. Everything changes. It can arrive and they answer (success). It can come back marked "address does not exist" (explicit error: bad, but you know what happened). It can take three weeks to arrive while you wait with no news (slowness). And —the case that had no equivalent at the table— you can be left not knowing: did the letter get lost and never arrive? Did it arrive, the person read it, answered you, and their answer got lost? From your side, those two cases look identical: silence. If your letter said "transfer me $1000," the silence is terrifying: you don't know whether to ask again (and risk being transferred $2000) or wait (and risk nothing ever happening).

The table is a function call in a process. The letter is a network call. Everything that makes a distributed system hard is in that difference: the network introduces slowness and, above all, ambiguity —the silence that doesn't distinguish "it didn't arrive" from "it arrived but the response got lost"—. Partial failure is living in the world of letters, permanently, with hundreds of letters in flight at once, some arriving, some lost, some answered whose acknowledgment got lost. This is its statement:

A network call doesn't have two outcomes (it worked / everything died), but four: success, explicit error, slowness, and ambiguity (I don't know whether it happened). In a system with many dependencies, different calls are in different outcomes at the same time, and that mix —partial failure— is the normal state. "All healthy" is just one of an explosion of possible states, and it's almost never where you are.

The four outcomes of a remote call

When orders calls payments.charge(), there aren't two possible results, there are four. It's worth seeing them one by one, because each pattern in the guide is born to attack one of them.

1. Success. payments received the request, charged, and responded "charged, id pay_123". It's the case you wrote the code for, and the only one you usually test. In a healthy system most calls land here —but "most" isn't "all," and reliability is decided in the rest—.

2. Explicit error. payments received the request and responded "I can't": card declined, insufficient funds, internal 500 error. It's an honest failure: it's bad news, but you know what happened and you know you did not charge. You can act with certainty —show "payment declined," don't retry a card decline—. Explicit errors are the easy ones; they're the ones your try/except already handles.

3. Slowness (timeout). payments received the request and is working… or not… but it doesn't respond yet. It didn't tell you no; it just leaves you waiting. This is the outcome half the guide is about, because while you wait you hold a resource (the thread, the connection), and if many calls land here at once, you exhaust the pool —the cascade from lesson 1—. Slowness is worse than the explicit error precisely because it doesn't fail: it keeps you in limbo, consuming resources, without giving you an answer to move on with.

4. Ambiguous. You received no response —the timeout expired, the connection dropped—. And here's the deep part: you don't know what state the other side ended up in. There are two possibilities that look identical from your side:

• Your request never reached payments. You didn't charge. Retrying is safe and correct.
• Your request did reach it, payments did charge, but its response got lost on the way back. You already charged. Retrying charges twice.

From orders, both cases are the same silence. There's no way —with only the call's information— to distinguish "it didn't happen" from "it happened but I didn't find out." This is the outcome that makes retrying dangerous and that forces idempotency (module 4): since you can't know whether your charge happened, you have to make retrying it harmless. The ambiguous outcome is, perhaps, the most important difference between programming in a process and programming in a distributed system.

Here are the four, with what you know and what you should do in each:

Outcome	Responded?	Do you know what happened?	Main danger	Pattern that attacks it
Success	Yes, well	Yes	None	—
Explicit error	Yes, "I can't"	Yes (it didn't happen)	Retrying something non-retryable	Selective retry (M3)
Slowness	Not yet	Not yet	Exhausting the pool waiting	Timeout (M2)
Ambiguous	Didn't respond	No (did it happen?)	Retrying and duplicating	Idempotency (M4)

Why this didn't exist in the monolith

Stop at something that seems obvious and isn't: outcomes 3 and 4 —slowness and ambiguity— don't exist inside a single process. When orders and payments live in the same process and charge() is a normal function call, there are only two results: the function returns (with a value or by raising an exception, which is the "explicit error"), or the whole process crashes (and then orders died too, so there's no one left to wonder "did I charge or not?"). There's no "the function is alive but doesn't answer me in 3 seconds," and there's no "I called the function and I don't know whether it ran." Execution is deterministic and shared: either both advance together, or both die together.

The network breaks that guarantee. By putting payments in another process, on another machine, on the other side of a cable, the two new outcomes appear: it can be alive but slow (because its CPU is saturated, or the network is congested), and it can have executed your request without you finding out (because its response got lost). Lesson 6 measures exactly how much that cable costs —an in-process call is nanoseconds; a network one, milliseconds, a thousand times more, and with the possibility of never coming back—. For now the conclusion is conceptual: partial failure is the price of distribution. It's not a defect in your code; it's a property of the terrain you now play on.

The worked example: the state space that explodes

Now let's measure it. The claim "in distributed systems, something is almost always broken" sounds like rhetorical exaggeration until you count the states. Let's model Mercado's checkout: orders depends on three services on its critical path —catalog, payments, shipping—, and let's simplify to each one being in one of three states: up (healthy), slow, or down. How many global states does the system have, and how many of them are "everything perfect"?

# l02_states.py -- the state space of Mercado's checkout.
# orders depends on 3 services; each can be up / slow / down.
# A monolith has 2 states (alive/dead). A distributed one, MANY.
from itertools import product

DEPS = ["catalog", "payments", "shipping"]
STATES = ["up", "slow", "down"]

healthy = risky = failing = 0
for combo in product(STATES, repeat=len(DEPS)):
    if all(s == "up" for s in combo):
        healthy += 1
    elif any(s == "down" for s in combo):
        failing += 1          # a dependency down -> the checkout fails
    else:
        risky += 1            # none down, some slow -> risk of exhaustion

total = len(STATES) ** len(DEPS)
print(f"dependencies on the critical path = {len(DEPS)}")
print(f"states per dependency             = {len(STATES)} {STATES}")
print(f"total system states               = {len(STATES)}^{len(DEPS)} = {total}")
print(f"  fully healthy (all up)          = {healthy}")
print(f"  degraded / at risk (some slow)  = {risky}")
print(f"  failing (some down)             = {failing}")
print(f"fraction fully healthy            = {healthy}/{total} = {healthy/total:.1%}")

# and if Mercado adds a 4th and 5th synchronous dependency...
for k in range(1, 6):
    print(f"  {k} deps -> {len(STATES)**k:>4} states, "
          f"only 1 is 'all up' ({1/len(STATES)**k:.2%})")

What to expect. Running python l02_states.py with Python 3.14.0:

dependencies on the critical path = 3
states per dependency             = 3 ['up', 'slow', 'down']
total system states               = 3^3 = 27
  fully healthy (all up)          = 1
  degraded / at risk (some slow)  = 7
  failing (some down)             = 19
fraction fully healthy            = 1/27 = 3.7%
  1 deps ->    3 states, only 1 is 'all up' (33.33%)
  2 deps ->    9 states, only 1 is 'all up' (11.11%)
  3 deps ->   27 states, only 1 is 'all up' (3.70%)
  4 deps ->   81 states, only 1 is 'all up' (1.23%)
  5 deps ->  243 states, only 1 is 'all up' (0.41%)

Read it calmly, because it's the numerical argument for the whole guide:

• With three dependencies there are 27 states, and "all healthy" is just one. The calculation is direct: 3 states per dependency, 3 dependencies, 3³ = 27 combinations. Of those 27, only one has all three services in up. The other 26 have something slow or something down. If your code only handles the "all up" state well, it handles 1 of 27 situations well.
• The 19 "failing" states are the ones with something down. As soon as a critical dependency is down, the checkout can't complete (without degradation, which is module 7). There are 19 combinations with at least one down —most of the space—.
• The 7 "risky" states are the dangerous ones: something slow, nothing down. They're the combinations where no one died but someone is slow. They look "almost healthy" and they're the ones that cause cascades, because the system keeps accepting traffic while its resources silently drain. Lesson 1's state (shipping slow) is one of these seven.
• And it grows brutally. With 4 dependencies, "all healthy" is 1 of 81 (1.23%); with 5, it's 1 of 243 (0.41%). Each dependency you add triples the number of states and makes the happy state rarer. This isn't pessimism: it's arithmetic. The more pieces you connect, the more time your system spends in some degraded state.

An honest warning about the model: this is a simplification —the real states aren't equally likely (a well-operated service is up almost always), and the count treats the 27 as if they weighed the same, which isn't true—. The point isn't that your system spends 96.3% of the time broken; a well-operated system spends most of its time in or near the healthy state. The point is qualitative and structural: there is an explosion of degraded states, your code has to do something reasonable in them, and "all healthy" is mathematically rare. Designing only for the 1-of-27 state is designing for the case that needs the least help.

The degrees of partial failure

The count above used three states per dependency so they'd fit in a table, but it's worth seeing that "partial failure" is really a spectrum, not three bins. A dependency can be:

up ──────── slow ──────── very slow ──────── timing out ──────── down
 |            |               |                    |                |
healthy   responds        at the edge        almost never      no response
          late         of exhaustion        responds in       (fails fast)
                                             time

Notice the irony of the extremes. An up (healthy) service is easy: it responds fast. A down (dead) service is also relatively easy: it fails fast —it returns "connection refused" in milliseconds, you release the resource and move on—. Hell is in the middle: the service that's almost responding on time, or responding right at the edge of the timeout. That's the one that keeps you waiting, holds your resources, and makes you doubt whether to retry. That's why throughout the guide you'll see again and again that a clean death is more manageable than degradation. A system that goes down completely is a bad day; a system that turns slow is a cascade.

Common mistakes

Modeling the world as "up or down" (the monolith's binary thinking). What happens: people design with an if service_available: ... else: ..., as if there were only two states. Why it's a mistake: it ignores the two outcomes that actually matter —slowness and ambiguity—. An "available" service that responds in 8 seconds breaks your system as much as one that's down, but your if counts it as "up." How to spot it: if in your code there's no concept of "took too long" and none of "I don't know whether it happened," you're modeling the binary world. How to fix it: design for the four outcomes, not for two. Slowness needs timeouts (M2); ambiguity needs idempotency (M4).

Treating the ambiguous outcome as an explicit error. What happens: people write except: retry() —on any failure, retry—, throwing the honest error and the ambiguous silence into the same bag. Why it's a mistake: retrying an explicit "card declined" error is useless but harmless; retrying an ambiguous charge can charge twice, because maybe the first attempt did happen and only the response got lost. How to spot it: ask yourself "if I retry this and the first time it did happen, what harm do I cause?". If the answer is "I duplicate a charge," you're treating an ambiguous outcome as if it were safe to retry. How to fix it: retries require idempotency (M4); not every error is retried the same way (M3).

Believing a timeout turns ambiguity into certainty. What happens: people add a timeout and assume that, when it expires, "the operation didn't happen." Why it's a mistake: the timeout gives you back control (you stop waiting, you release the resource), but it does not tell you what happened on the other side. The request may have arrived and executed after you stopped waiting. A timeout solves slowness (you recover your thread), not ambiguity (you still don't know whether you charged). How to spot it: if your logic says "timeout, therefore I assume it didn't happen and retry," you have a latent double-execution bug. How to fix it: the timeout (M2) and idempotency (M4) are different patterns for different outcomes, and they're almost always used together.

Exercises

Exercise 1 — Classify the outcomes. For each of these Mercado situations, say which of the four outcomes it is (success, explicit error, slowness, ambiguous) and what orders should do:

(a) payments responds "card declined". (b) orders calls shipping and after 10 seconds there's still no response. (c) orders calls payments.charge(), the connection drops before receiving a response, and there's no way to know whether the charge went through. (d) catalog returns the product record in 15 ms.

See solution

• (a) Explicit error. payments responded honestly "I can't" (card declined). orders knows it didn't charge. Action: show "payment declined" to the buyer; do not retry (retrying a card decline won't get it approved).
• (b) Slowness. shipping hasn't responded in 10 seconds. orders is holding a worker that whole time —the start of the cascade—. The correct action: to have had a timeout that would have already cut off at, say, 500 ms, to release the worker (M2). Without a timeout, this is the outcome that causes pool exhaustion.
• (c) Ambiguous. The connection dropped with no response. orders doesn't know whether it charged or not. Action: do not retry blindly —it could duplicate the charge—. It needs idempotency: retry with the same idempotency_key so that, if the charge already happened, payments doesn't repeat it (M4).
• (d) Success. catalog responded well and fast. orders continues its flow. The happy case —1 of 27—.

Exercise 2 — Count the states of a bigger checkout. Suppose Mercado adds two new synchronous dependencies to the checkout: fraud (anti-fraud verification) and inventory (stock reservation). Now orders depends on five services on the critical path, each in up/slow/down. How many global states are there? What fraction is "all healthy"? What does that tell you about adding synchronous dependencies?

See solution

With 5 dependencies and 3 states each: 3⁵ = 243 global states. "All healthy" is still just one, so the happy fraction is 1/243 ≈ 0.41% —exactly the last row of the example's output—.

What it tells you is blunt: every synchronous dependency you add to the critical path triples the state space and makes the healthy state rarer. Going from 3 to 5 dependencies took the happy fraction from 3.7% to 0.41% —almost ten times rarer—. This is a real design argument: before adding one more synchronous call to your checkout, ask yourself whether it truly has to be synchronous and on the critical path, because each one multiplies the ways the flow can degrade. Sometimes the answer is to make it asynchronous (take it off the critical path) —but that decision (sync vs async) belongs to the architectural styles guide, not this one; here we only measure the cost of having it synchronous—.

Exercise 3 — The most dangerous state. Of the 27 states of the three-dependency checkout, argue which is the most dangerous to operate: (a) all three services down, or (b) all three services slow. Justify with what you know from lesson 1.

See solution

The most dangerous is (b), all three slow, even though intuitively "all three down" sounds worse.

With all three services down, each call fails fast: orders receives "connection refused" in milliseconds, releases the worker immediately, and the checkout fails cleanly (0% success, but without exhausting resources). It's an honest bad day: the system says "I can't" and stays standing, ready to recover as soon as the services come back.

With all three services slow, each call holds a worker for seconds while it waits. The workers exhaust (lesson 1's cascade), orders stops accepting traffic, and —worse— the system appears to be trying to work, so incoming traffic is usually maintained instead of cut off. It's the state that fills the pools, spikes the p99, and makes orders drag its own callers (the storefront) down with it. A clean death recovers; degradation cascades.

This is the key intuition of the whole module, and that's why it's worth repeating: in distributed systems, "slow" is usually worse than "dead." It's counterintuitive, and it's the reason module 2 (timeouts) exists before any other: turning slowness into a fast failure —cut off and move on— is the first line of armor.

Summary and next step

In this lesson you opened the guide's root concept: partial failure. A remote call doesn't have two outcomes but four —success, explicit error, slowness, and ambiguous (I don't know whether it happened)—, and the last two don't exist inside a single process: they appear the moment you cross the network. Slowness holds resources and causes cascades (M2 attacks it with timeouts); ambiguity makes retrying dangerous (M4 attacks it with idempotency).

And you measured it: with the analogy of the letter versus the conversation at the table, and with a count run in Python that showed that Mercado's checkout, with just three dependencies in up/slow/down, has 27 possible states and only one is "all healthy" —3.7%—, dropping to 0.41% with five dependencies. The happy state is mathematically rare; designing only for it is designing for the case that needs it least. And you saw the counterintuitive lesson that runs through the whole module: "slow" is usually worse than "dead", because death fails fast and releases resources, while slowness drains them silently.

Before moving on you should be able to: name the four outcomes of a remote call and which pattern attacks each; explain why slowness and ambiguity don't exist in a monolith; read the state count and say why "all healthy" becomes rare as you add dependencies; and argue why three slow services are more dangerous than three down.

What comes next is understanding why our intuition betrays us so much here —why we design, over and over, as if the network were reliable and latency zero—. In lesson 3 we break down the eight fallacies of distributed computing: the false assumptions, so comfortable that we make them without realizing, that are behind almost every distributed bug. You'll see each fallacy bite Mercado, and you'll measure how "the network is reliable" compounds and "latency is zero" adds up.

Resources

• Release It!, 2nd ed., by Michael Nygard — Pragmatic Bookshelf — the chapter on integrations and the way remote calls fail is the direct source of this lesson's "four outcomes." Nygard insists, as we do here, that slowness and the response that never arrives are the failures that really matter.
• "Notes on Distributed Systems for Young Bloods", by Jeff Hodges — a classic essay that opens with "partial failures are the defining characteristic of distributed systems." Short, direct, and about exactly this lesson.
• "A Note on Distributed Computing", by Waldo, Wyant, Wollrath, and Kendall (Sun, 1994) — the academic paper that argues why a remote call cannot be treated like a local call: latency, partial failures, and concurrency are fundamental differences, not details. It's the theoretical root of this lesson.
• Python — itertools.product — the function with which we enumerate the state space (the Cartesian product of each dependency's states). A simple tool for reasoning about combinations, useful well beyond this example.