7. The map of Mercado's checkout failure points

Overview

This module gave you the phenomenon (partial failure), the why (the fallacies), the mechanism (the cascade), its physics (pool exhaustion), and its origin (distribution). This lesson puts it all together into a concrete, reusable artifact: the map of the failure points of Mercado's checkout. Instead of talking about failures in the abstract, we're going to walk through the real flow —orders → catalog, payments, shipping— and enumerate, systematically and exhaustively, every place where it can break and every way it can do so. It's the difference between "I know distributed systems fail" and "here is the list of the 12 exact points where this checkout can fail, what happens at each one, and which ones can take everything down."

The technique is a disciplined inventory. Each network hop (each arrow of the diagram, each call that leaves orders) is a failure point, and each point can fail in four modes —the outcomes of lesson 2 turned into an operational catalog: down, slow, wrong response (wrong) and ambiguous—. Crossing hops by modes gives the full grid of possible failures. On that grid we note two things that decide the severity: whether the hop is on the critical path (without it there's no checkout) and whether a failure of it can cascade (exhaust the orders pool). You're going to execute that inventory in Python: 3 hops × 4 modes = 12 failure points, of which 2 of 3 hops can trigger a cascade, and one —payments— has the aggravating factor of being critical and dangerous to retry. And we close with the most important table of the module: mapping each failure mode to the pattern (M2-M7) that solves it —the literal bridge to the rest of the guide—.

Connection with the module: the six previous lessons built the understanding; this one operationalizes it on the case. The map you produce here is the direct input of module 2 and beyond: when you get to "how to put in a timeout," you'll already know on which hop and against which failure mode. And it's the mold of the mini-project (lesson 8), where you'll make this same map for a new variant of the checkout, with your own hands. Here we do it together; there you do it yourself.

The analogy: the pre-flight checklist

Think of it this way. Before a plane takes off, the pilot doesn't trust their intuition or "it looks fine." They walk through a checklist point by point: flaps, rudder, fuel pressure, altimeter, landing gear… each system, one by one, checked off. The list doesn't exist because pilots are forgetful; it exists because memory and intuition skip things, and on a plane one skipped thing kills. The checklist turns "I checked the plane" (vague, incomplete, mood-dependent) into "I checked these 40 specific points, and these are the ones that need attention." It's the difference between an impression and an inventory.

Mapping the failure points of a flow is making its pre-flight checklist. "The checkout can fail" is the vague impression; the map is the inventory: these 12 points, in these ways, with this severity. And like the pilot's checklist, its value is in the mechanical exhaustiveness —walking through all the hops by all the modes, without skipping any because "that one's surely fine"—, precisely because lesson 2 taught us that "all healthy" is 1 of 27 states: you can't afford to assume a hop won't fail. This is the idea:

A map of failure points walks, systematically, through each network hop of the flow by each failure mode (down, slow, wrong, ambiguous), and notes for each whether it's on the critical path and whether it can cascade. It turns "the system can fail" (an impression) into "these are the N exact points, their severity, and the pattern that attacks each one" (an actionable inventory). It's the flow's pre-flight checklist.

The hops and the modes

Let's start by naming the two dimensions of the map.

The hops (where it can fail). In the checkout, orders makes three network calls, and each is a hop:

   orders ──► catalog     (read the product record)
   orders ──► payments    (charge)
   orders ──► shipping     (create the shipment)

Not all hops weigh the same. catalog is a read and is degradable: if it fails, orders could continue with cached data or with the minimal record —the checkout doesn't depend on it to complete—. payments and shipping are on the critical path: without charging and without creating the shipment, there's no checkout (unless you degrade the shipment, which is module 7). And payments has a unique aggravating factor: charging is non-idempotent by nature —retrying blindly can charge twice—, so its ambiguous mode is especially dangerous.

The modes (how it can fail). Each hop can fail in four ways, which are the outcomes of lesson 2 turned into a catalog:

Mode	What it is	Main danger
down	The service doesn't respond; fails fast (connection refused)	The checkout fails, but releases the resource fast
slow	The service responds super late	Holds the worker → cascade
wrong	Responds, but with bad data (format, incorrect value)	Corrupts the result silently
ambiguous	No response; you don't know if it happened	Retrying can duplicate

Notice that slow is the only one marked with "cascade": it's the mode that holds resources and exhausts the pool. down fails fast (less dangerous for the pool); wrong and ambiguous are dangerous for other reasons (corruption, duplication), but they don't exhaust the pool by themselves. This distinction is the one that will decide which pattern to apply to each point.

The worked example: executing the inventory

Now let's build the map as data and let Python walk through it, marking the critical path and the cascade risk:

# l07_inventory.py -- inventory of failure points of Mercado's checkout.
# Each HOP (network hop) is a failure point. Each can fail in
# several ways: down (no response), slow (responds late), wrong (responds
# badly), ambiguous (we don't know if it happened). We mark whether it's on
# the critical path (without it, no checkout) and whether a failure of it can CASCADE.

HOPS = [
    # caller,   callee,     sync,  critical, on_retry_unsafe
    ("orders",  "catalog",  True,  False, False),  # reads the record (degradable)
    ("orders",  "payments", True,  True,  True),   # charges (not idempotent!)
    ("orders",  "shipping", True,  True,  False),  # creates the shipment
]
MODES = ["down", "slow", "wrong", "ambiguous"]

print(f"{'hop':<22}{'critical':>9}{'sync':>6}{'cascade?':>10}  modes")
points = 0
cascade_points = 0
for caller, callee, sync, critical, unsafe in HOPS:
    # a synchronous call, on the critical path, unprotected => cascade.
    can_cascade = sync and critical
    cascade_points += 1 if can_cascade else 0
    points += len(MODES)
    hop = f"{caller} -> {callee}"
    print(f"{hop:<22}{str(critical):>9}{str(sync):>6}{str(can_cascade):>10}"
          f"  {MODES}")

print(f"\ntotal failure points (hops x modes) = {len(HOPS)} x {len(MODES)} = {points}")
print(f"hops that can trigger a cascade     = {cascade_points} of {len(HOPS)}")
print("note: payments is on the critical path AND retry-unsafe "
      "(charging twice) -> needs idempotency (M4)")

What to expect. Running python l07_inventory.py with Python 3.14.0:

hop                    critical  sync  cascade?  modes
orders -> catalog         False  True     False  ['down', 'slow', 'wrong', 'ambiguous']
orders -> payments         True  True      True  ['down', 'slow', 'wrong', 'ambiguous']
orders -> shipping         True  True      True  ['down', 'slow', 'wrong', 'ambiguous']

total failure points (hops x modes) = 3 x 4 = 12
hops that can trigger a cascade     = 2 of 3
note: payments is on the critical path AND retry-unsafe (charging twice) -> needs idempotency (M4)

Read it slowly, because it's the whole checkout seen as an attack surface:

• Twelve failure points in a three-call flow. Three hops by four modes give 12 ways this checkout —which looks so simple in the diagram— can break. And this already simplifies (it doesn't count the combinations of several hops failing at once, which lesson 2 counted as 27 states). The point is visceral: a "simple" flow has a dozen failure modes, and each one needs a design response. Ignoring 11 of them and programming only for success is what produces the incidents.
• Two of the three hops can cascade. payments and shipping are on the critical path and are synchronous, so their slow mode can exhaust the orders pool —the cascade—. catalog doesn't cascade the same way because it's degradable (although, careful, if orders calls it synchronously and with no timeout, a slow catalog also holds workers; "degradable" is a design decision we haven't made yet, not an automatic protection). The map tells you where to concentrate the armor first: the critical and synchronous hops.
• payments is the most delicate point on the map. It's the only one that combines two aggravating factors: it's on the critical path (you can't skip it) and it's non-idempotent (retrying it blindly charges twice). Its slow mode can cascade (needs timeout, M2, and bulkhead, M6) and its ambiguous mode can duplicate charges (needs idempotency, M4). A single hop, three patterns. The map highlights it so you don't treat it like the others.

From each failure to its pattern: the bridge to the guide

Here's the payoff of having made the map: each cell of the grid (hop × mode) points to a concrete pattern that attacks it, and those patterns are, one by one, the modules that follow. This is the table that turns the problem into a plan:

Hop	Failure mode	What happens to the checkout	Pattern that attacks it	Module
any	slow	Holds worker → cascade	Timeout + bulkhead	M2, M6
payments, shipping	down	The checkout fails (critical)	Circuit breaker (stop hammering)	M5
catalog	down	No record → can continue	Degradation (minimal record/cache)	M7
shipping	down	No shipment → can be deferred	Degradation (deferred shipment)	M7
any	ambiguous (transient)	Did it happen? → retry carefully	Retry + backoff + jitter	M3
payments	ambiguous	Did I charge? → don't duplicate	Idempotency (idempotency key)	M4
any	wrong	Bad data → validate/fallback	Validation + fallback	M7

Read it right to left and you'll see the guide's index: each module from 2 to 7 exists to cover a column of this map. Module 2 (timeout) attacks slow —the mode that cascades, that's why it goes first—. Module 3 (retry) and 4 (idempotency) attack ambiguous, in that order because retrying without idempotency is dangerous. Module 5 (breaker) attacks down when the service has been dead for a while. Module 6 (bulkhead) reinforces module 2 by isolating the pools. Module 7 (degradation) attacks down and wrong of the dependencies that can be dodged. The map isn't just a diagnosis: it's the executable table of contents of the rest of your learning.

A diagram of the checkout with each hop annotated with its most dangerous mode and its pattern:

graph LR
    sf[storefront] -->|checkout| orders
    orders -->|"catalog: slow/down<br/>→ timeout + degrade (M2,M7)"| catalog
    orders -->|"payments: slow/ambiguous<br/>→ timeout + idempotency (M2,M4)"| payments
    orders -->|"shipping: slow/down<br/>→ timeout + breaker + degrade (M2,M5,M7)"| shipping

Common mistakes

Mapping only the "down" mode and forgetting "slow". What happens: people do the inventory thinking "what happens if each dependency goes down?", and omit "what happens if it turns slow?". Why it's a mistake: slow is the mode that cascades —the most destructive—, and it's exactly the one intuition skips because "at least it's responding." How to spot it: if your map only has an "available yes/no?" column, you're missing the dangerous half. How to fix it: always include the four modes; this lesson's map puts slow first among the dangerous ones for a reason.

Treating all hops equally. What happens: people apply the same armor (or none) to catalog, payments, and shipping, as if they were interchangeable. Why it's a mistake: they differ in severity —catalog is degradable, payments is critical and non-idempotent, shipping is critical but deferrable—, and each calls for a different set of patterns. How to spot it: if your plan is "put a timeout on everything and done," you're ignoring that payments also needs idempotency and shipping needs degradation. How to fix it: the map notes criticality and idempotency per hop precisely to differentiate the treatment.

Confusing the map with the solution. What happens: people make a beautiful map of failure points and feel like "it's solved now." Why it's a mistake: the map is the diagnosis, not the treatment —you know where and how it can fail, but you haven't put in a single timeout yet—. How to spot it: if you have the map but the code still calls shipping without protection, you mapped the problem and didn't touch it. How to fix it: the map is the input of modules 2-7; its value is realized when each marked point receives its pattern. Diagnosing without treating is half the work.

Exercises

Exercise 1 — Extend the map. Mercado adds a fourth hop to the checkout: orders → inventory (reserve stock), which is synchronous, on the critical path (you can't sell without stock), and non-idempotent (reserving twice sets aside extra stock). Add this row to the inventory mentally: how many total failure points are there now? How many hops can cascade? What patterns does inventory need?

See solution

With four hops and four modes: 4 × 4 = 16 total failure points (before, 12).

Hops that can cascade: now 3 of 4 (payments, shipping, and inventory, all synchronous and critical; only catalog doesn't, because it's degradable). Before it was 2 of 3.

Patterns inventory needs:

• Timeout + bulkhead (M2, M6) against its slow mode —it's synchronous and critical, so its slowness cascades—.
• Idempotency (M4) against its ambiguous mode —since it's non-idempotent (reserving twice sets aside extra stock), retrying blindly is dangerous, just like payments—.
• Circuit breaker (M5) against its sustained down mode.

Note that inventory ends up looking a lot like payments: critical, non-idempotent, cascades. It's a recurring pattern —operations that mutate state and are critical are always the most delicate—, and that's why the map marks idempotency separately.

Exercise 2 — Order the armor. You have time to armor a single hop of the original checkout before a high-traffic event. With the map in hand, which one do you choose and with what pattern, and why that one before the others?

See solution

The strongest candidate is to armor shipping (or payments) against its slow mode with a timeout (M2).

The reasoning follows the map: under high traffic, the number-one risk is the cascade, and the mode that triggers it is slow on a critical, synchronous hop. A timeout attacks exactly that —it turns the indefinite wait into a fast cutoff, avoiding pool exhaustion—, and it protects not only that hop but, by not letting orders go down, everything above it (the storefront). It's the pattern with the highest "return" per unit of effort, and that's why it's module 2.

Between shipping and payments, shipping is a slightly better candidate for the first timeout because it's more prone to turning slow (it integrates external carriers) and is degradable (you can defer the shipment), while payments will also need idempotency, which is more work. But either of the two critical hops, with a timeout, is the correct answer: attacking the mode that cascades, on a critical hop, is always the first priority. (If you said catalog, reconsider: it's degradable, so its failure is the least urgent.)

Exercise 3 — Justify a cell of the mapping. In the pattern table, the cell "payments + ambiguous" points to idempotency (M4), not to "retry + backoff (M3)", even though both ambiguous modes are usually resolved by retrying. Why does payments need idempotency and not just retry like the others?

See solution

Because payments.charge() is non-idempotent by nature: charging is an operation that mutates state with a real-world effect (it takes money from someone). When the mode is ambiguous —orders received no response and doesn't know whether the charge happened—, retrying blindly (what a simple M3 retry would do) can charge twice: if the first attempt did charge and only the confirmation got lost, the retry makes a second real charge.

The solution isn't to stop retrying (sometimes you have to, because maybe the first attempt didn't arrive), but to make the retry safe: send an idempotency key —a unique identifier of that charge operation— so that payments, if it sees a key it already processed, returns the result of the original charge without charging again. That's idempotency (M4). Retry (M3) is still needed to decide when to retry; idempotency (M4) is what makes retrying not duplicate. That's why they go together and in that order: first you learn to retry well (M3), then to make retrying harmless for operations like charging (M4). The map signals that payments, being critical and mutating, needs both, while a read-only hop like catalog is content with retry.

Summary and next step

In this lesson you turned the whole module into an artifact: the map of the failure points of Mercado's checkout. You walked, systematically, through each network hop (orders → catalog/payments/shipping) by each failure mode (down, slow, wrong, ambiguous), noting criticality and cascade risk. The executed inventory gave 12 failure points (3 hops × 4 modes) in a flow of just three calls, with 2 of 3 hops able to cascade, and highlighted payments as the most delicate point —critical and non-idempotent—. It's the checkout's pre-flight checklist: the "it can fail" impression turned into an actionable inventory.

And you crossed the bridge to the rest of the guide: the table that maps each failure mode to its pattern —slow → timeout + bulkhead (M2, M6); critical down → circuit breaker (M5); degradable down → degradation (M7); ambiguous → retry (M3) or idempotency (M4); wrong → validation (M7)—. That table is, literally, the executable table of contents of modules 2 through 7: each one covers a column of your map.

Before moving on you should be able to: enumerate the checkout's hops and the four failure modes; build the hop × mode grid and count the failure points; explain why slow is the mode that cascades and why payments is the most delicate hop; and map a given failure mode to the pattern (M2-M7) that attacks it.

What comes next is doing it yourself. In lesson 8, the mini-project, you receive a variant of Mercado's checkout —with a new dependency, fraud— and produce its complete failure-mode map with Python, in the style of an FMEA analysis (Failure Mode and Effects Analysis): for each dependency, its failure mode, its effect on the checkout, its cascade risk, and the pattern that attacks it. It's your graduation from the module: you go from reading the map to drawing it, which is what an architect does when arriving at a new system and being asked "how can this fail?".

Resources

• Release It!, 2nd ed., by Michael Nygard — Pragmatic Bookshelf — the chapter on integration points is the direct source of this lesson: each integration point (each hop) is a place where the system can fail, and Nygard catalogs them by mode just like here. His line "integration points are the number-one killer of stability" is the heart of this map.
• "Failure Mode and Effects Analysis (FMEA)" — Wikipedia — the formal engineering technique (born in aeronautics and industry) of systematically enumerating failure modes and their effects. Lesson 8's mini-project is a lightweight version of this; it's worth knowing the full method.
• Google SRE Book — "Embracing Risk" — how to think about reliability as an inventory of risks that are prioritized (not everything is armored equally), which is exactly what the map lets you do: decide which hop to armor first.
• AWS Well-Architected — Reliability Pillar — the AWS framework for designing reliable systems, with its own approach to identifying failure points and applying the right pattern to each. It complements this lesson's "failure → pattern" table with a cloud provider's perspective.