8. Project: correct retries for Mercado's calls

Overview

This is the module's practical closing. The seven previous lessons gave you the pieces one by one —why retry, the retry storm measured, exponential backoff, jitter, the budget, and when not to retry—. Now it's your turn to weave them into a single deliverable that demonstrates the module's central thesis with your own hands: take the raw call from orders to payments, put the complete retry on it —transient + backoff + jitter + budget—, and prove with the measured simulation the before and after: the storm against the control. It's not a new tool; it's the whole module condensed into a retry helper built and measured.

What's really evaluated here isn't that you achieve 100% success —that's the result— but that you can demonstrate the before and after with numbers. Anyone can write a retry with backoff. The deliverable that matters is the evidence: the call without control burying payments (289.6x load, 3% success), the same call with the complete retry recovering (1.5x, 100%), and the justification of which piece solves which part of the problem. A student who delivers the helper without the "before" measurement didn't demonstrate that they understood what problem they solved; one who delivers the storm, the control, and the explanation of each decision demonstrated that they'll no longer let a network call become an incident.

Connection with the module: this lesson is the practical exam of module 3 and its closing. It collects the retry you were building piece by piece (the TransientError from lesson 2, the backoff from 4, the jitter from 5, the max_attempts and retry_on from lessons 6 and 7) and the payments simulation (lessons 3 to 6), and presents them as a project with deliverables and a reference solution. After the statement, it closes the module and points you to module 4: the retry you deliver here has a hidden danger —retrying the charge can duplicate the charge—, and plugging it with an idempotency_key is exactly what module 4 builds on top of what you leave here.

Analogy: the report of the engineer who put out the fire

Think of a fire engineer called to explain why a building caught fire and how they prevented it in the building next door. Their job doesn't end when the second building is safe; it ends when they deliver the report that demonstrates what failed and what fixed it: the photo of the first building in flames (the problem), the photo of the second intact (the solution), and the list of which measure —the sprinklers, the fire doors, the exits— stopped which part of the fire. An engineer who only says "the second one didn't burn" proves nothing; one who delivers the before, the after, and the causal explanation demonstrates that they understood the fire and know how to prevent it.

Your project is that report. The first building in flames is the call without control unleashing the retry storm; the second intact is the same call with the complete retry; and the list of measures is the justification of which piece —backoff, jitter, budget, retry_on— solves which part of the problem. Delivering the complete report, not just "it ended up at 100%," is what separates "I changed the code" from "I understood the storm and know how to prevent it." And like any good fire report, yours ends by pointing out the risk that still remains —the duplicate charge— and whose job it is to solve it (module 4).

The project: formal statement

Your task is to take the call from orders to payments without protection and leave it with correct retries, measured and justified. You start from this starting point —the raw call, with nothing—:

# orders_naive.py — orders charges without any protection.
def checkout(order_id, amount_cents):
    # If payments fails, this raises. If the caller retries immediately,
    # and many callers do it at once, the retry storm is born.
    return payments.charge(order_id, amount_cents)

You must:

1. Build the complete retry helper with the module's four pieces: retry only transient errors (retry_on), with exponential backoff (base * 2^attempt), full jitter (uniform(0, ceiling)), and local budget (max_attempts). Suggested signature: retry(fn, *, max_attempts, base, cap, retry_on).
2. Wrap orders' call with that retry, choosing justified parameters for the case of a charge (think: high or low max_attempts? which errors go in retry_on? is retrying a charge safe?).
3. Measure the before and after with the simulation of payments recovering from an outage: the no-control policy ("none") against the backoff + jitter policy, showing peak req/s, load amplification, and success_rate side by side.
4. Write the justification: a table that associates each piece of the retry with the part of the problem it solves, and an honest note about the risk that remains (the duplicate charge) and why its solution is module 4.

Deliverables

• The retry helper (retry.py) with the four pieces and a clean signature.
• orders wrapped (orders_resilient.py) with the chosen and commented parameters.
• The before/after table with the actual simulation output (fixed seed).
• The justification: which piece solves what, and the note about the duplicate-charge danger.

Analyze your result before seeing it solved

Before looking at the reference solution, attempt the project and ask yourself these questions about your result. Does your retry retry only the transient errors, or does it also swallow the 400s and the 401s? If a charge gets a read timeout, can your design duplicate the charge —and did you acknowledge it in the justification—? Did you choose a low max_attempts (2-3) and can you argue why, with lesson 2's diminishing returns and lesson 6's amplification? Is your jitter full (uniform(0, ceiling)) or did you leave a deterministic backoff that synchronizes? And the most important: does your measurement show both rows —the storm and the control— with the same seed, so the comparison is legitimate? If any answer makes you uncomfortable, that's exactly the part the reference solution will clarify.

Reference solution

1. The complete retry helper

# retry.py — the module 3 retry: transient + backoff + jitter + budget.
import random
import time


class TransientError(Exception):
    """Momentary error worth retrying: timeout, 503, 429, network."""


def retry(fn, *, max_attempts=3, base=0.5, cap=10.0, retry_on=(TransientError,)):
    """Retries fn() with exponential backoff + full jitter, only on retry_on.
    - max_attempts: total attempts (includes the first). LOW on purpose.
    - base, cap:    wait ceiling = min(cap, base * 2^(attempt-1)).
    - jitter:       the real wait is uniform(0, ceiling)  (full jitter).
    - retry_on:     ONLY these errors are retried; the rest propagates at once."""
    attempt = 0
    while True:
        attempt += 1
        try:
            return fn()                              # success: return and finish
        except retry_on:
            if attempt >= max_attempts:              # budget exhausted: give up
                raise
            ceiling = min(cap, base * (2 ** (attempt - 1)))
            time.sleep(random.uniform(0, ceiling))   # full jitter, not the exact value
        # note: any exception OUTSIDE retry_on (400, 401, 422) is not
        # caught here and propagates immediately -> we don't retry the permanent.

The four pieces are in four precise places: retry_on in the except (only the transient); base * 2 ** (attempt - 1) (exponential backoff); random.uniform(0, ceiling) (full jitter); and if attempt >= max_attempts (local budget). Everything that's not a TransientError —a 400, a ValidationError— doesn't even enter the except: it propagates on the spot, fulfilling lesson 7's first rule.

2. orders wrapped, with justified parameters

# orders_resilient.py — orders charges with correct retries for a CHARGE.
from retry import retry, TransientError

def checkout(order_id, amount_cents):
    # max_attempts LOW (2): charging is dangerous to retry (lesson 7) and the
    # returns decline fast (lesson 2). One retry, and we give up.
    # retry_on only transient: a validation 400 is NOT retried.
    # small base (0.3s) so as not to punish the user's latency in the good case.
    return retry(
        lambda: payments.charge(order_id, amount_cents),
        max_attempts=2,
        base=0.3,
        cap=5.0,
        retry_on=(TransientError,),   # timeout / 503 / 429 / network error
    )
    # KNOWN DANGER: if the charge got a READ timeout (payments did charge,
    # but the response was lost), this retry DUPLICATES the charge. The solution
    # is an idempotency_key that payments deduplicates -> Module 4.

The decisions, justified: max_attempts=2 (low) because charging is the most dangerous operation to retry and retries yield little after the first; retry_on only transient so as not to retry a 400; small base=0.3 so as not to add perceptible latency for the user in the normal case. And the KNOWN DANGER comment is part of the deliverable, not decoration: it names the risk the retry can't solve on its own.

3. The measurement: before and after

We run the simulation of payments recovering from a 5 s outage (capacity 50/s, 40 checkouts/s arriving), with the no-control policy and with backoff + jitter, same seed:

# measure.py — the before (storm) and the after (control), side by side.
for label, policy in [("WITHOUT control (storm)", "none"),
                      ("WITH full retry", "jitter")]:
    m = run_storm(policy=policy)     # same simulation, same seed 42
    print(label, m["peak_req_s"], m["amplification"], m["success_rate"])

What to expect. On my machine (Python 3.14, seed 42), the before and after:

  scenario               | peak req/s | req/s @30s | total reqs | amplif. | success
  ---------------------------------------------------------------------------------
  no backoff (storm)     |     23,120 |     11,708 |    695,043 |  289.6x |     3%
  backoff + jitter       |        179 |         41 |      3,695 |    1.5x |   100%

The fire report, in two rows. Before (the raw call of the starting point, retried without control): payments receives 289.6 times its load, with a peak of 23,120 req/s, and only 3% of the checkouts finish well —the 5 s outage became a one-minute collapse—. After (the same call with the complete retry): the load drops to 1.5x, the peak to 179 req/s, and the success rises to 100%. The only difference between the two rows is how the client retries. That contrast —289.6x against 1.5x, 3% against 100%— is your evidence that you understood the module.

4. The justification: which piece solves what

And the honest note that closes the report: the complete retry solved the storm, but left a danger alive. Every retry of the charge is a potential duplicate charge —if the original attempt actually charged (read timeout) and the retry charges again, the buyer pays double—. The retry can't solve this on its own, because it has no way to know that its retry is the "same" charge as the original. The solution is an idempotency_key that payments recognizes and deduplicates, and building it is module 4. Retrying well (this module) and retrying safely (module 4) are the two halves of armoring a charge.

Common mistakes in the project

Delivering the "after" without the "before." What happens: the 100%-success row is shown and it's declared done. Why it happens: the good result feels like the proof. How to spot it: if your report doesn't have the storm row (289.6x, 3%), you didn't demonstrate what problem you solved —only that your code works—. How to fix it: measure both policies with the same seed. The pedagogical value is in the contrast, not in the final number. Without the before, the after means nothing.

High max_attempts on the charge. What happens: max_attempts=5 is set "to maximize the checkout's success." Why it happens: more attempts = more success, in lesson 2's intuition. How to spot it: for a charge, each extra retry is a potential extra duplicate charge, and the returns already declined. How to fix it: on the system's most dangerous operation, max_attempts goes as low as possible (1 or 2). Prudence here is worth more than the last point of success.

Retrying the charge without naming the duplicate-charge danger. What happens: the retry on charge is delivered as if it were safe, with no mention of the risk. Why it happens: in the simulation, charge doesn't visibly duplicate (we don't model the monetary effect in the storm). How to spot it: ask yourself what happens if the original attempt charged and the retry does too. How to fix it: the justification must include the duplicate-charge note and refer to module 4. Retrying a charge without acknowledging that risk is exactly the mistake lesson 7 measures in money.

Extension exercises

Exercise 1 — The complete retry_on. Extend retry_on to the realistic list of transient errors you'd retry against payments, and name two errors you'd deliberately leave out. Justify each exclusion.

retry_on = (
    TimeoutError,          # connection or read timeout
    ConnectionError,       # the network went down, the connection closed
    ServiceUnavailable,    # 503: payments protected itself under load
    TooManyRequests,       # 429: payments asks to slow down (respect Retry-After)
)

Exercise 2 — Parameter sensitivity. Run the simulation with base=0.1 (very short backoff) and with base=2.0 (long backoff), both with jitter, against the 5 s outage. Predict and then verify: how do the amplification and success_rate change? Is there a point where a too-short backoff approaches the storm?

Exercise 3 — Add the aggregate budget. The project uses a local budget (max_attempts). Sketch how you'd add an aggregate budget (token bucket, lesson 6) to the simulation, and predict how the storm row would change if the token bucket limited the retries to 20% of the traffic.

Summary and module closing

With this project you closed module 3. You built the complete retry helper —transient + exponential backoff + full jitter + local budget— with each piece in its precise place, and applied it to the call from orders to payments with justified parameters for a charge (low max_attempts, retry_on only transient). And you demonstrated the before and after with the measured simulation: the call without control unleashing the storm (289.6x load, 3% success), and the same call with the complete retry recovering (1.5x, 100%). You delivered the fire report: the problem, the solution, the list of which piece solves what, and the honest note about the danger that remains.

Looking at the whole module: you started with a decision that module 2's timeout left in your hand —retry or not—. You learned that retrying works because failures are usually transient (lesson 2), but that retrying without control unleashes a retry storm that buries the dependency (lesson 3). You disarmed the storm with three pieces: backoff to space out (lesson 4), jitter to de-synchronize (lesson 5), budget so as not to hammer a dead one (lesson 6). And you sharpened the discipline of when not to retry: the permanent (pure waste) and the non-idempotent (duplicate charge) (lesson 7). All measured, nothing from memory.

The module ends pointing out its own debt. The retry you built is powerful and, in equal measure, dangerous: every retry of a charge can charge twice. The only way to retry a charge with peace of mind is to make it idempotent —an idempotency_key that guarantees that two attempts of the same charge charge only once—, and that's module 4. Retrying well (what you just learned) and retrying safely (what comes next) are the two halves of armoring Mercado's most delicate operation. See you there.

Resources

• "Timeouts, retries, and backoff with jitter" — Marc Brooker, Amazon Builders' Library — the reference text that gathers all the pieces of this project in one place, from the experience of operating services at scale.
• "Exponential Backoff And Jitter" — Marc Brooker, AWS Architecture Blog — the original measurement that justifies the full jitter you used in the helper.
• Release It! (2nd edition) — Michael Nygard — the stability patterns (retries, circuit breaker, bulkhead) that this module starts and the following ones continue; the canonical source of the ecosystem.
• Tenacity documentation and resilience4j — Retry — the production libraries (Python and JVM) that implement the retry you built here by hand; use them in a real system instead of reinventing them, now that you understand what they do inside.
• Stripe — Idempotent requests — the preview of module 4: how a real gateway makes it safe to retry a charge with an idempotency_key.