Module 8: Project — Make Mercado's Checkout Resilient
8. Project: make Mercado's checkout resilient
Overview
This is the guide's final deliverable. Over seven modules you learned six patterns, each with its measurement; over seven lessons of this module you combined them on Mercado's checkout, step by step, measuring each contribution. Now you package everything into a project you'd produce —and defend— on a real team: you take Mercado's checkout without protection, armor it by combining the six patterns in the correct order, and deliver the evidence that it worked.
The shape of the deliverable is the one that governs the whole guide, now over the complete flow: you measure the fragility → apply the patterns → measure again. You don't deliver "resilient code"; you deliver the code plus the before/after table that proves the success_rate went from 45.6% to 100% and the p99 from 10133 ms to 1043 ms, with zero overcharge, under the same storm and the same seed. And you deliver the justification: the failure → pattern table that explains which pattern solves which failure, and why each one is where it is. This lesson gives you the statement, the rubric by which you know you did it right, and —in a <details>— the complete reference solution: the executable resilient checkout and the executed metrics table. And with it the guide closes: a review of the eight modules and the map of where to go next.
Connection with the module: lessons 2 to 6 built the armoring piece by piece; lesson 7 consolidated the evidence. This lesson turns it into a deliverable project: statement, rubric, reference solution. It's the application of the guide's complete method to a single flow, from start to finish, and the point where you prove —with your own hands and with numbers— that you know how to take a fragile system and make it reliable. After the project, the guide closes and shows you the three natural paths to continue: event-driven, SRE and system design.
The deliverable, at a glance
By the end of the project you'll have produced three things:
- The resilient checkout (code): the version of
ordersthat combines the six patterns in the correct order —timeout + retry + idempotency onpayments; timeout + circuit breaker + bulkhead + degradation onshipping; load shedding over the flow—. - The before/after metrics table (evidence):
success_rate, p99, rejected requests and overcharge, measured WITHOUT and WITH protection, under the same storm and the same seed. - The justification (the failure → pattern table): for each failure mode of the checkout, the pattern that attacks it, on which axis it acts, and what it leaves pending for the next one.
The statement
You receive Mercado's checkout without protection —the fragile orders that calls catalog, payments and shipping without a single defense—. Your task:
A. Measure the baseline. Run the bare checkout under the canonical storm (shipping down between second 15 and 45; payments failing 10% transiently; 40 checkouts/s over a pool of 40 workers, seed 42). Record: success_rate, p50, p99, rejected requests, fail_pay, fail_ship, and the ledger's overcharge.
B. Armor payments (the critical dependency). Apply, in this order: timeout (derived from the healthy p99 × ~2), retry with exponential backoff + jitter and a budget, and idempotency (an idempotency_key per operation, the same on all the retries; the server deduplicates). Justify why payments receives retry (insist) and not a breaker.
C. Armor shipping (the deferrable dependency). Apply: timeout, circuit breaker (with failure_threshold and cooldown), bulkhead (concurrency cap), and degradation (when shipping fails, complete the order and defer the shipment to a recovery queue). Justify why shipping receives a breaker (give up) and not retry.
D. Protect the flow under overload. Apply load shedding (reject when exceeding a pool-occupancy threshold) and demonstrate it with a traffic peak (7.5× the capacity, with a client deadline), measuring the congestion collapse with and without shedding.
E. Measure the after and deliver. Run the armored checkout under the same storm and the same seed. Deliver the before/after table, the ladder rung by rung, and the failure → pattern table. For each pattern, say what failure it solves and what it leaves pending.
Step-by-step guide
Step 1 — Instrument and measure the fragility. Start from the lesson 2 discrete-event simulation. Run it without any pattern and note the five "before" numbers. Don't invent the fragility: measure it. Save these numbers; they're the left half of your table.
Step 2 — Derive the timeouts. Don't invent them. From each dependency, take the healthy p99 and multiply it by ~2 for the read timeout: catalog ~100 ms, payments ~650 ms, shipping ~400 ms. The connect timeout short and uniform (~1 s). Note the justification —a timeout without a derivation is a magic number—.
Step 3 — Armor payments, and verify the idempotency. Apply timeout + retry + idempotency. And make the measurement that separates an armoring from a disaster: run the same retry with and without the server's deduplication, and compare the overcharge. If without idempotency the overcharge is $0, your storm isn't exercising the lost responses —adjust it until the risk is visible—, because the value of idempotency is only seen when there are retries over charges whose response was lost.
Step 4 — Armor shipping, and don't expect the success to rise. Apply timeout + breaker + bulkhead. Measure, and observe that the success_rate doesn't rise —it even drops—. This is correct: they're resource patterns. If it surprises you, go back to lesson 5. Note that the resources improved (calls saved, p50) even though the outcome hasn't yet.
Step 5 — Add the degradation, and harvest. Catch shipping's failure (open breaker, full bulkhead, timeout) and complete it as a deferred shipment, enqueuing it for recovery. Measure: the success_rate should jump close to 100%, and the fail_ship of step 4 should turn almost exactly into deferred. That correspondence is your proof that the degradation harvested what the breaker prepared.
Step 6 — Protect under overload. Add load shedding. Demonstrate it with a peak (7.5× the capacity) and a client deadline: measure the congestion collapse without shedding (super low success, huge p99, phantom charges) against the bounded service with shedding.
Step 7 — Build the tables and the justification. Gather the ladder, the before/after and the failure → pattern table. For each pattern, one sentence: what failure it solves, on which axis, what it leaves pending. The justification is as much a deliverable as the code.
Rubric: how you know you did it right
Your project is complete when you can answer affirmatively to all of this:
- Did you measure the "before" with concrete numbers (success_rate, p99, rejections, overcharge), not with a qualitative description?
- Is each timeout derived from its dependency's
p99, and not an invented round number? - Does
paymentshave retry + idempotency, and did you demonstrate with the ledger that without idempotency there would be an overcharge (not just claim it)? - Does
shippinghave breaker + bulkhead, and did you understand (and note) why that alone does not raise thesuccess_rate? - Does the degradation turn
shipping's failures into successful deferred checkouts, with a recovery queue (not a lost shipment)? - Did you demonstrate the load shedding with a real peak, measuring the congestion collapse with and without it?
- Does your before/after table use the same seed and the same storm on both sides, and show
success_rateand p99 and overcharge? - Does your failure → pattern table map each failure mode to its pattern, its axis, and what it leaves pending?
- Can you explain, for each pattern, what failure it solves and which it leaves for another?
If you answer "no" to any, there's your next iteration. The measurement is the proof; without it, you didn't do the project, you just wrote code.
Common mistakes in the project
Applying the six patterns to the two dependencies equally. What happens: retry and breaker are put on both, "so they're well protected." Why it's a mistake: payments (critical, transient failure) calls for retry; shipping (deferrable, persistent failure) calls for a breaker. Putting retry on shipping hammers a dead one; putting a breaker on payments contributes nothing (it never accumulates consecutive failures). How to spot it: your two dependencies have exactly the same patterns. How to fix it: apply according to the failure mode, not "across the board." The asymmetry —insist on the critical, give up on the deferrable— is the sign that you armored with judgment.
Measuring only the "after". What happens: the armoring is applied and only the final table is delivered. Why it's a mistake: without the measured "before," you can't quantify the improvement or defend the decision. How to spot it: you don't have a comparison table. How to fix it: measure both worlds with the same seed. The project's value is the contrast.
Delivering the 100% without looking at the ledger. What happens: the 100% success_rate is celebrated and the overcharge isn't verified. Why it's a mistake: the success_rate is identical with and without idempotency; the double charge only appears in the ledger. How to spot it: your table has no overcharge row. How to fix it: correctness (the exact charge) is as much a part of the deliverable as performance. A 100% success with $31,657 of overcharge isn't a resilient checkout.
Exercises
Exercise 1 — Change the storm. The reference solution uses shipping down 30 s and payments at 10%. Modify the storm: make payments (not shipping) the dependency that goes down 30 seconds, and shipping the one that fails 10% transiently. Without running, predict: which patterns would change dependency? Would the armored success_rate still be near 100%?
See solution
If payments goes down 30 seconds (persistent failure) and shipping fails 10% transiently, the patterns would have to change dependency according to the failure mode, not according to the name:
paymentsdown (persistent): the retry alone is no longer enough —insisting against apaymentsdead 30 seconds is hammering a corpse—. It would need a circuit breaker (stop hitting it). But here's the serious problem:paymentsis critical and not deferrable. You can't "degrade" a charge (confirming the order without charging would be giving away product). So whenpayments' breaker opens, the checkout has to fail honestly —there's no deferred shipment that helps—. The armoredsuccess_ratewould not reach 100%: during the 30 seconds of a downedpayments, the checkouts can't complete, because there's no way to degrade the charge.shippingtransient (10%): nowshippingcalls for retry (its failures are short, a retry resolves them), not a breaker. And since it's deferrable, if the retry fails, the degradation is still available.
The deep lesson: the choice of patterns depends on the failure mode (persistent → breaker; transient → retry) and on whether the operation is deferrable (yes → degradation available; no → you have to fail honestly). When the persistently-downed dependency is critical and not deferrable (like payments), no pattern saves it —you can only contain the damage and wait for it to recover—. That's why the original design put shipping (deferrable) as the one that goes down persistently: it's the failure the degradation can rescue. That a downed payments is unrescuable isn't a defect of the armoring; it's the reality that some dependencies are irreplaceable, and the work then is to minimize the downtime (an operations/SRE problem, not a pattern one).
Exercise 2 — The chain's budget. The checkout calls catalog → payments → shipping in sequence. With the timeouts (100 + 650 + 400 ms) plus payments' retries, what's the worst-case latency of a checkout that doesn't degrade? If the client has a deadline of 2000 ms, does it fit? What would you do if it didn't fit?
See solution
The worst case without degradation, summing the sequential chain:
catalog: up to 100 ms (its timeout).payments: up to 650 ms per attempt, and with a budget of 3 attempts plus 2 backoffs of up to 800 ms each → in the worst case, ~650 + 800 + 650 + 800 + 650 ≈ 3550 ms forpaymentsalone (three complete attempts with two maximum backoffs between them).shipping: up to 400 ms (its timeout).
The theoretical worst case is around ~4050 ms, which does not fit in the 2000 ms deadline. In practice it's rare (it requires payments to fail its first two attempts with maximum backoffs), that's why the measured p99 is 1043 ms and not 4050 —the theoretical worst case is much more pessimistic than the real p99—. But the risk exists, and the options if it doesn't fit are the module 2 ones:
- Bound the retry to the chain's budget: each
paymentsretry is limited to the deadline's remaining time (min(read_timeout, remaining)), and if there's no budget left, it stops retrying. That way the worst case is bounded to the deadline, at the cost of fewer retries when the chain runs tight. - Lower
payments' retry budget (from 3 to 2) or the maximum backoff (from 800 to 400 ms): reduces the worst case at the cost of recovering fewer transient failures. - Parallelize the independent:
catalogdoesn't depend onpayments; launching them in parallel saves ~100 ms of the budget. - Defer
shippingalways (not just when it fails): taking it off the checkout's critical path saves its 400 ms of the budget —a decision that brushes the boundary with event-driven architecture (the shipment as an asynchronous event)—.
The important thing: the chain's budget reveals that the worst case doesn't fit and forces you to an explicit design decision, instead of hiding the problem until a client suffers a 4-second timeout.
Exercise 3 — Defend an omission. A demanding reviewer says: "you put a bulkhead on shipping, but not on catalog. catalog could also go down. Why the omission?". Answer with cost/benefit judgment, not with "I forgot."
See solution
The omission is deliberate and defensible by cost/benefit. A bulkhead (and any pattern) has a cost: complexity, reserved resources, one more place where something can be misconfigured. It's justified only where the benefit exceeds it, and a bulkhead's benefit is proportional to the contagion risk of that dependency.
shipping is a clear candidate for a bulkhead because in the storm it hangs (huge latency), and a dependency that hangs is exactly the one that hogs the pool and contaminates the others —the bulkhead confines it—. catalog, on the other hand, is a fast, healthy read (~20 ms): its typical failure mode is "down" (it fails fast, with a connection error in milliseconds), not "hung." A dependency that fails fast doesn't hog the pool —it releases the thread right away—, so the contagion the bulkhead prevents almost never happens with catalog. Putting a bulkhead on it would be paying the cost without harvesting the benefit.
That said, the honest answer includes a nuance: if catalog could hang (for example, if its database degrades and the reads start taking seconds), then it would indeed deserve a bulkhead —the criterion isn't "it's catalog" but "can it hang and hog the pool?"—. The design decision is documented like this: "I didn't put a bulkhead on catalog because its observed failure mode is a fast outage, not a hang; if its latency profile changed, I'd reevaluate it." That's armoring with judgment: not putting all the patterns everywhere "just in case," but putting each pattern where its corresponding failure is a real risk, and knowing how to say why you omitted it where you omitted it.
Reference solution
See the complete solution (executable code + executed tables)
The solution is a discrete-event simulation, deterministic, that armors the checkout pattern by pattern and measures each rung. The heart is the simulate function, which activates each pattern with a flag, so the same storm is run with different levels of armoring.
# mercado_capstone.py -- Mercado's checkout, from baseline to complete armoring.
# Discrete-event simulation, deterministic (fixed seed). Python 3.14.
# Thread-per-request model: each checkout takes ONE `orders` worker end to
# end; the worker is held even during the backoff waits.
import heapq
import random
# --- scenario (identical in all runs: the same storm) ---
DURATION, ARRIVAL_RATE, SEED = 60.0, 40.0, 42
ORDERS_POOL, ORDERS_QUEUE = 40, 40
CATALOG_MS, PAYMENTS_MS, SHIPPING_MS = 20.0, 120.0, 200.0
SHIP_DOWN = (15.0, 45.0) # shipping DOWN in this window
SHIP_HANG_MS = 5000.0 # no timeout, a downed shipping holds this
P_CHARGE_LOST, P_TRANSIENT = 0.05, 0.05 # 10% of transient failures in payments
PAY_TRANS_MS = 50.0
# --- pattern parameters ---
SHIP_TIMEOUT_MS, PAY_TIMEOUT_MS = 400.0, 650.0
RETRY_BUDGET, BACKOFF_BASE_MS, BACKOFF_CAP_MS = 3, 100.0, 800.0
CB_THRESHOLD, CB_COOLDOWN_S = 5, 5.0
SHIP_BULKHEAD, SHED_WATERMARK = 8, 0.90
def ship_is_down(t):
return SHIP_DOWN[0] <= t < SHIP_DOWN[1]
def simulate(timeout_on=False, retry_on=False, idempotent=True,
breaker_on=False, bulkhead_on=False, degrade_on=False,
shed_on=False, deadline_ms=None, queue_size=ORDERS_QUEUE,
arrival_rate=ARRIVAL_RATE, duration=DURATION, seed=SEED):
rng = random.Random(seed)
free, queue, ship_inflight = ORDERS_POOL, [], 0
cb = {"state": "CLOSED", "fails": 0, "opened_at": 0.0, "probe": False}
ledger = {"charged": 0.0}
idem_seen, charge_count, all_reqs = set(), {}, []
st = {"offered": 0, "success": 0, "deferred": 0, "fail_payment": 0,
"fail_shipping": 0, "rejected": 0, "shed": 0, "deadline_fail": 0,
"lat": [], "ship_calls": 0, "saved_by_cb": 0}
events, seq = [], 0
def push(t, kind, req):
nonlocal seq
heapq.heappush(events, (t, seq, kind, req)); seq += 1
t, oid = 0.0, 0
while True:
t += rng.expovariate(arrival_rate)
if t >= duration:
break
req = {"oid": oid, "arrival": t, "amount": round(rng.uniform(10, 500), 2),
"attempt": 0, "held": False, "idem": f"chg-{oid}"}
charge_count[oid] = 0; all_reqs.append(req)
push(t, "arrive", req); st["offered"] += 1; oid += 1
def release(req, now):
nonlocal free
if req["held"]:
free += 1; req["held"] = False
if queue:
nxt = queue.pop(0); free -= 1; nxt["held"] = True
push(now + CATALOG_MS / 1000.0, "catalog_done", nxt)
def finish(req, now, outcome):
lat = (now - req["arrival"]) * 1000.0
if deadline_ms is not None and outcome in ("success", "deferred") \
and lat > deadline_ms:
outcome = "deadline_fail" # the user gave up; worker wasted
st[outcome] += 1; st["lat"].append(lat); release(req, now)
def on_arrive(req, now):
nonlocal free
if shed_on and (ORDERS_POOL - free) >= SHED_WATERMARK * ORDERS_POOL:
st["shed"] += 1; st["lat"].append(0.0); return # load shedding
if free > 0:
free -= 1; req["held"] = True
push(now + CATALOG_MS / 1000.0, "catalog_done", req)
elif len(queue) < queue_size:
queue.append(req)
else:
st["rejected"] += 1; st["lat"].append((now - req["arrival"]) * 1000.0)
def payment_outcome():
r = rng.random()
if r < P_CHARGE_LOST:
return "lost" # server charges, response lost
if r < P_CHARGE_LOST + P_TRANSIENT:
return "transient" # rejects, does NOT charge
return "ok"
def server_charge(req):
if idempotent and req["idem"] in idem_seen: # dedup by idempotency_key
return
ledger["charged"] += req["amount"]
idem_seen.add(req["idem"]); charge_count[req["oid"]] += 1
def run_payment(req, now):
oc = payment_outcome(); req["_pay"] = oc
if oc in ("ok", "lost"):
server_charge(req) # real side effect on the server
delay = (PAYMENTS_MS if oc == "ok" else PAY_TRANS_MS if oc == "transient"
else (PAY_TIMEOUT_MS if timeout_on else PAYMENTS_MS))
push(now + delay / 1000.0, "payment_done", req)
def on_catalog_done(req, now):
req["attempt"] = 1; run_payment(req, now)
def on_payment_done(req, now):
oc = req["_pay"]
if oc == "ok":
start_shipping(req, now); return
if retry_on and req["attempt"] < RETRY_BUDGET: # retry with backoff+jitter
req["attempt"] += 1
back = min(BACKOFF_CAP_MS, BACKOFF_BASE_MS * (2 ** (req["attempt"] - 2)))
push(now + rng.uniform(0, back) / 1000.0, "payment_retry", req)
else:
finish(req, now, "fail_payment")
def shipping_skipped(req, now): # shipping wasn't called (breaker/bulkhead)
finish(req, now, "deferred" if degrade_on else "fail_shipping")
def start_shipping(req, now):
nonlocal ship_inflight
if breaker_on:
if cb["state"] == "OPEN":
if now - cb["opened_at"] >= CB_COOLDOWN_S:
cb["state"] = "HALF_OPEN"; cb["probe"] = False
else:
st["saved_by_cb"] += 1; shipping_skipped(req, now); return
if cb["state"] == "HALF_OPEN":
if cb["probe"]:
st["saved_by_cb"] += 1; shipping_skipped(req, now); return
cb["probe"] = True
if bulkhead_on and ship_inflight >= SHIP_BULKHEAD:
shipping_skipped(req, now); return
ship_inflight += 1; st["ship_calls"] += 1
if ship_is_down(now):
req["_ship"] = "fail"
hold = SHIP_TIMEOUT_MS if timeout_on else SHIP_HANG_MS
push(now + hold / 1000.0, "shipping_done", req)
else:
req["_ship"] = "ok"
push(now + SHIPPING_MS / 1000.0, "shipping_done", req)
def on_shipping_done(req, now):
nonlocal ship_inflight
ship_inflight -= 1
if req["_ship"] == "ok":
if breaker_on:
cb["fails"] = 0
if cb["state"] == "HALF_OPEN":
cb["state"] = "CLOSED"; cb["probe"] = False
finish(req, now, "success")
else:
if breaker_on:
cb["fails"] += 1
if cb["state"] == "HALF_OPEN":
cb["state"] = "OPEN"; cb["opened_at"] = now; cb["probe"] = False
elif cb["fails"] >= CB_THRESHOLD:
cb["state"] = "OPEN"; cb["opened_at"] = now
finish(req, now, "deferred" if degrade_on else "fail_shipping")
H = {"arrive": on_arrive, "catalog_done": on_catalog_done,
"payment_done": on_payment_done, "payment_retry": run_payment,
"shipping_done": on_shipping_done}
while events:
now, _, kind, req = heapq.heappop(events)
H[kind](req, now)
completed = st["success"] + st["deferred"]
lat = sorted(st["lat"])
p = lambda q: (lat[min(len(lat) - 1, int(round(q / 100 * (len(lat) - 1))))]
if lat else 0)
correct = sum(r["amount"] for r in all_reqs if charge_count[r["oid"]] >= 1)
return {"sr": 100.0 * completed / st["offered"], "p50": p(50), "p99": p(99),
"rejected": st["rejected"], "fail_pay": st["fail_payment"],
"fail_ship": st["fail_shipping"], "deferred": st["deferred"],
"ship_calls": st["ship_calls"], "saved_by_cb": st["saved_by_cb"],
"shed": st["shed"], "deadline_fail": st["deadline_fail"],
"overcharge": round(ledger["charged"] - correct, 2) + 0.0}
ALL_ON = dict(timeout_on=True, retry_on=True, breaker_on=True,
bulkhead_on=True, degrade_on=True) # shedding: only under overload
def row(name, r):
print(f"{name:<24} sr={r['sr']:5.1f}% p50={r['p50']:4.0f}ms p99={r['p99']:6.0f}ms "
f"rej={r['rejected']:4d} fpay={r['fail_pay']:4d} fship={r['fail_ship']:4d} "
f"defer={r['deferred']:4d} ship_calls={r['ship_calls']:4d} over=${r['overcharge']:>10,.2f}")
if __name__ == "__main__":
print("=== the armoring ladder (same storm, seed 42) ===")
row("0. baseline", simulate())
row("1. +timeout", simulate(timeout_on=True))
row("2. +retry+idem", simulate(timeout_on=True, retry_on=True))
row("3. +breaker+bulkh", simulate(timeout_on=True, retry_on=True,
breaker_on=True, bulkhead_on=True))
row("4. +degradation", simulate(**ALL_ON))
print("\n=== idempotency (same armoring, dedup on/off) ===")
print(" WITH idem:", simulate(**ALL_ON, idempotent=True)["overcharge"])
print(" WITHOUT idem:", simulate(**ALL_ON, idempotent=False)["overcharge"])
print("\n=== load shedding under overload (300/s, deadline 2000 ms) ===")
no = simulate(**ALL_ON, deadline_ms=2000.0, queue_size=100000, arrival_rate=300.0)
yes = simulate(**ALL_ON, shed_on=True, deadline_ms=2000.0, arrival_rate=300.0)
print(f" WITHOUT shedding: sr={no['sr']:.1f}% p99={no['p99']:.0f}ms deadline_fail={no['deadline_fail']}")
print(f" WITH shedding: sr={yes['sr']:.1f}% p99={yes['p99']:.0f}ms shed={yes['shed']}")The executed output (Python 3.14.0, fixed seed, deterministic):
=== the armoring ladder (same storm, seed 42) ===
0. baseline sr= 45.6% p50= 340ms p99= 10133ms rej= 916 fpay= 147 fship= 240 defer= 0 ship_calls=1334 over=$ 0.00
1. +timeout sr= 45.7% p50= 340ms p99= 670ms rej= 0 fpay= 233 fship=1068 defer= 0 ship_calls=2164 over=$ 0.00
2. +retry+idem sr= 50.5% p50= 540ms p99= 1263ms rej= 0 fpay= 1 fship=1185 defer= 0 ship_calls=2396 over=$ 0.00
3. +breaker+bulkh sr= 34.0% p50= 140ms p99= 1043ms rej= 0 fpay= 1 fship=1581 defer= 0 ship_calls= 831 over=$ 0.00
4. +degradation sr=100.0% p50= 140ms p99= 1043ms rej= 0 fpay= 1 fship= 0 defer=1581 ship_calls= 831 over=$ 0.00
=== idempotency (same armoring, dedup on/off) ===
WITH idem: 0.0
WITHOUT idem: 31657.47
=== load shedding under overload (300/s, deadline 2000 ms) ===
WITHOUT shedding: sr=5.1% p99=35812ms deadline_fail=17126
WITH shedding: sr=56.0% p99=878ms shed=7934
The before/after table (the central deliverable):
| Metric (same storm, seed 42) | BEFORE | AFTER |
|---|---|---|
success_rate | 45.6% | 100.0% |
| Latency p99 | 10133 ms | 1043 ms |
| Requests rejected (pool exhaustion) | 916 | 0 |
Charge failures (fail_pay) | 147 | 1 |
Shipment failures (fail_ship) | 240 | 0 |
| Checkouts with a deferred shipment | 0 | 1581 |
| Overcharge | $0 | $0 (would be $31,657 without idempotency) |
The failure → pattern table (the justification):
| Failure mode | Pattern | Axis | Leaves pending |
|---|---|---|---|
| Slow dependency exhausts the pool | Timeout | Resource | The dependency keeps failing |
| Dependency hogs the shared pool | Bulkhead | Resource | Doesn't complete the checkout |
Transient failure of payments | Retry (backoff+jitter) | Outcome | The retry's double charge |
| Charge retry duplicates | Idempotency | Outcome | (closes the retry's risk) |
Persistently dead shipping | Circuit breaker | Resource | Doesn't create the shipment |
Checkout fails hard when shipping goes down | Degradation | Outcome | (deferred shipment) |
| Overload jams even the charge | Load shedding | Resource | The rejected aren't served |
The code combines the six patterns in the correct order: on payments, the timeout enables the retry, and the idempotency makes it safe; on shipping, the timeout feeds the breaker, the bulkhead covers the window before, and the degradation harvests what both prepare; the load shedding protects the flow under overload. The same storm, the same seed, and the success_rate from 45.6% to 100% with the p99 from 10133 to 1043 ms and $0 of overcharge.
Close of the guide: what you built, module by module
You started with a checkout that went down when a dependency hiccuped, and you finish with one that survives the partial failure of its parts —and you know how to prove it with numbers—. Let's recap the complete arc of the eight modules:
- Why distributed systems fail. Partial failure as the normal state: in distributed, something is always slow or down. The cascading failure measured: a slow dependency exhausts the pool and knocks everything down. There you installed the problem.
- Timeouts. The silent killer: without a timeout, an endless wait exhausts the threads. The pattern that enables all the others, because it turns an infinite wait into a bounded failure.
- Retries, backoff and jitter. Retrying well: the retry storm that buries the one that was recovering, and its cure —space out (backoff), desynchronize (jitter), limit (budget)—.
- Idempotency. Why a retry charges twice, and the
idempotency_keythat makes it safe. The condition that makes the retry an armoring and not a weapon. - Circuit breakers. Stop hitting a dead one: the three-state machine (
CLOSED/OPEN/HALF_OPEN) that gives up in time and recovers on its own. - Bulkheads. Isolate so a failure doesn't spill over: compartments per dependency, so the one that hangs doesn't hog the healthy ones' pool.
- Graceful degradation and load shedding. Fail soft, not hard: complete the checkout even if a part fails (defer), and reject early under overload to protect the core.
- The capstone. The six patterns combined in the correct order over the complete checkout, measured end to end: 45.6% → 100%, p99 10133 → 1043 ms, overcharge $0.
The idea that holds everything: local failure isn't eliminated, it's contained. You didn't make shipping stop going down or payments stop hiccuping —that doesn't depend on you—. You made their failures stop turning into the outage of the whole checkout. That distinction —contain instead of cure— is the heart of resilience, and now you carry it to any distributed system you touch.
Where to go next
This guide gave you the mechanics of making a synchronous call reliable, given that it exists. Three natural paths extend it, each a sibling guide of the ecosystem:
- Event-driven architecture. The capstone deferred the shipment by enqueuing it. When that idea —communicating by asynchronous events instead of synchronous calls— becomes the system's main style, you enter event-driven architecture: queues, brokers, the outbox pattern, the bus's exactly-once, at-least-once delivery. It's the natural evolution of the degradation that defers work: instead of deferring as an exception, communicating that way by design. The event-driven architecture guide covers that mechanics that here stayed out of boundary.
- SRE (Site Reliability Engineering). You measured resilience in a simulation; in production it's measured with real observability —metrics, traces, alerts—, objectives are set with SLOs and error budgets, and it's tested with chaos engineering (injecting failures on purpose to verify the defenses work). The health checks and minimal metrics this guide brushed against are the entry point to the SRE discipline, which operates reliable systems at scale.
- System design. This guide's patterns are pieces of a bigger design. When to use each one, how they compose with scaling (replicas, sharding), consistency (CAP, PACELC) and structure (styles and boundaries) is the work of systems design. The system-design-fundamentals guide taught you to scale; the architecture and boundaries one, to structure; this one, to make it reliable. Together they're the craft of designing distributed systems.
The axis that unites them is the same one you closed the capstone with: a serious system isn't judged because it works when everything is healthy, but because it survives when its parts fail —and because you can measure that it does—. With this guide you have that capability. The three paths deepen it.
Summary and next step
In this project you integrated the guide's six patterns over Mercado's complete checkout, in the correct order, and produced the deliverable you'd defend on a team: the resilient code, the executed before/after table (45.6% → 100%, p99 10133 → 1043 ms, overcharge $0 versus $31,657 without idempotency), and the failure → pattern table that justifies each decision. You practiced the cycle that governs the whole guide —measure → apply → measure— from start to finish, and learned to armor with judgment: each pattern where its corresponding failure is a real risk, none to spare, in the order that respects their dependencies.
The project's deepest lesson is the one that closes the guide: resilience is proven by measuring, not by citing. The before/after table is what turns "I armored the checkout" into a reproducible engineering decision, and the habit of distinguishing what each pattern solves from what it leaves pending is what lets you combine them with a head in any system.
With this you finish the resilience and reliability patterns guide. You know how to take a service that calls unreliable dependencies and make it survive their failures —with timeouts, retries, idempotency, circuit breakers, bulkheads, degradation and load shedding—, applied with judgment and proven with numbers. What follows —event-driven, SRE, system design— builds on this base. But the central capability is already yours: to look at a fragile distributed flow and turn it into a reliable one, and to prove that you did.
Resources
- Release It!, 2nd ed., by Michael Nygard — Pragmatic Bookshelf — the canonical source of the six patterns and the book that presents them as an integrated system. The reference to reread now that you combined them all. In English.
- Google SRE Book and The Site Reliability Workbook — the entry point to the SRE path: SLOs, error budgets, observability, overload handling and incident response. How reliability is operated in production at scale. Free and in English.
- Building Microservices, 2nd ed., by Sam Newman — O'Reilly — the bridge toward system design and event-driven: how these resilience patterns fit into the complete design of a microservices system. In English.
- AWS Builders' Library — Amazon's collection of articles on resilience in production (timeouts, retries, idempotency, load shedding, dependencies); each one deepens one of the capstone's patterns from real operation. Free and in English.