8. Project: put timeouts on Mercado's checkout and measure

Overview

This is the module's capstone. Up to now you learned the timeout piece by piece —what it is, why it's the first defense, where to put it, what value to give it, what it promises, how to chain it—. Now you apply it whole, with your own hands, to Mercado's checkout. The task has a shape that will repeat in every module of the guide and that I want you to internalize as a method: you take an unprotected system, measure its fragility with numbers, apply the pattern, and measure again to prove it helped. Not "I put in timeouts because the book says so"; "I put in timeouts, and here's the table showing the healthy traffic went from 62/167 to 167/167." The before/after measurement is the deliverable, because it's what turns a belief into a defensible engineering decision.

You're going to work with orders' checkout that calls payments, shipping, and catalog, over a shared pool —the same scenario you measured in lesson 3, now as your project—. First you instrument it and provoke the incident (hung shipping) to see the exhausted pool and the healthy traffic go down. Then you choose the timeouts with the p99 method, apply them separating connection and read, add the budget in the chain, and measure again. The deliverable is the resilient code, the before/after metrics table, and —most important— the justification of why each number is what it is.

Connection with the module: this lesson introduces no new concept; it integrates the seven previous ones into a workflow. Lesson 3 gave you the pool simulator; lesson 4, the connect/read separation; lesson 5, the p99 method; lesson 6, the time contract; lesson 7, the budget. Here you put them together. By the end, you'll have done end to end the "measure the fragility → apply the pattern → measure the improvement" cycle that is the backbone of the whole guide, and you'll be ready for module 3, where the pattern that stands on these timeouts is the retry.

The deliverable, at a glance

By the end of the project you'll have produced three things:

1. The instrumented and armored checkout (code): the version with connection and read timeouts, values derived from the p99, and a time budget in the chain.
2. The before/after metrics table: success_rate of the healthy traffic, rejected requests, orders' latency p99, and pool occupancy, measured WITHOUT and WITH timeouts under the same incident and the same seed.
3. The justification: why you chose each timeout (what p99 you measured, what margin you applied), why you separated connection and read, and what failure the timeout solves and what failure it does not solve (making clear what's left for the following modules).

Step 1 — Take the fragile checkout and measure its fragility

We start from lesson 3's orders pool simulator, which already models the complete scenario: pool of 8 threads, queue of 8, browse traffic (to catalog, healthy) and checkout (to shipping, today hung). Your first job is to run it without a timeout and record the fragility. This is the line of code that represents "the fragile checkout": the call without a limit.

# The FRAGILE checkout: no timeout. hold = shipping's full latency.
if timeout_ms is not None and latency > timeout_ms:
    hold = timeout_ms
    req["outcome"] = "timeout"
else:
    hold = latency            # no timeout, hold = 3000 ms hung
    req["outcome"] = "served"

Run the simulation with timeout_ms=None and note the "before" numbers. These are the ones you already saw in lesson 3, now as your baseline measurement:

=== WITHOUT timeout (the fragile checkout) ===
  browse  (catalog, healthy): served= 62/167  rejected=105  latency p50=123ms p99=2373ms
  checkout(shipping, hung)  : served= 17  timeout=  0  rejected= 33
  pool: avg busy=7.3/8   time at 100% full=86%

Translate it to the deliverable's metrics. The success_rate of the healthy traffic (browse) is 62/167 = 37% —63% of the catalog reads, which don't touch shipping, fail—. The latency p99 of orders for that traffic is 2373 ms. The pool is at 100% 86% of the time. This is the picture of the fragility: a service that goes down with its dependency, dragging along traffic that had nothing broken. Save these numbers; they're the left half of your table.

Step 2 — Choose the timeouts with the p99 method

Before applying timeouts, decide their values with lesson 5's method, not by eye. You need each dependency's healthy latency p99. Measure it (or, in the project, take it from lesson 5's measurement for shipping):

• shipping: p99 ≈ 143 ms (measured over 50,000 healthy calls). Read timeout = p99 × ~2 ≈ 300 ms.
• payments: suppose you measure p99 ≈ 320 ms. Read timeout ≈ 650 ms.
• catalog: p99 ≈ 40 ms. Read timeout ≈ 100 ms.

And the connect timeout of each comes separately, from your network's healthy handshake (lesson 4): in the same datacenter, ~1 s is slack and safe for all three. So the configuration is:

TIMEOUTS = {
    #             (connect, read) in seconds
    "catalog":  (1.0, 0.100),   # read = p99(40ms) x ~2
    "payments": (1.0, 0.650),   # read = p99(320ms) x ~2
    "shipping": (1.0, 0.300),   # read = p99(143ms) x ~2
}

Note the justification now, while you have it fresh: each read timeout is the healthy dependency's p99 times a margin of ~2× —enough not to cut healthy traffic (false timeouts ~0%, lesson 5) and enough to release the thread fast on a hang—. The connect is short and uniform because a healthy handshake in the datacenter is milliseconds; 1 s detects a down host without punishing the normal variation. This justification is part of the deliverable: a timeout without justification is a magic number.

Step 3 — Apply the timeouts and measure again

Now run the same simulation with shipping's read timeout (300 ms) active. In the simulator, that's simply timeout_ms=300. In orders' real code, it's passing the tuple to the call:

import requests

def create_shipment(order_id, deadline_ms):
    # phase timeout (connect, read) + bounded by the budget (step 4)
    connect_to, read_to = TIMEOUTS["shipping"]
    effective_read = min(read_to, deadline_ms / 1000)   # budget, lesson 7
    resp = requests.post(
        "http://shipping/shipments",
        json={"order_id": order_id},
        timeout=(connect_to, effective_read),
    )
    return resp.json()

Run the simulation with timeout_ms=300 and note the "after" numbers:

=== WITH timeout = 300 ms (the armored checkout) ===
  browse  (catalog, healthy): served=167/167  rejected=  0  latency p50=21ms p99=31ms
  checkout(shipping, hung)  : served=  0  timeout= 50  rejected=  0
  pool: avg busy=3.2/8   time at 100% full=0%

The success_rate of the healthy traffic rose to 167/167 = 100%, the rejected requests dropped to 0, orders' p99 fell from 2373 ms to 31 ms, and the pool never reaches 100%. This is the right half of your table.

Step 4 — Add the budget in the chain

orders' real checkout doesn't make a single call; it chains catalog → payments → shipping. Apply lesson 7's budget: the app gives orders a deadline (say 1200 ms), and each call is bounded by the remaining time. The skeleton of the budgeted checkout:

import time

CLIENT_DEADLINE_MS = 1200

class DeadlineExceeded(Exception):
    pass

def checkout(order):
    start = time.monotonic()

    def remaining_ms():
        return CLIENT_DEADLINE_MS - (time.monotonic() - start) * 1000

    def call(dep, fn, *args):
        rem = remaining_ms()
        if rem <= 0:
            raise DeadlineExceeded(f"no budget before calling {dep}")
        connect_to, read_to = TIMEOUTS[dep]
        effective_read = min(read_to, rem / 1000)   # min(local, remaining)
        return fn(*args, timeout=(connect_to, effective_read))

    product  = call("catalog",  read_catalog,   order.product_id)
    payment  = call("payments", charge,         order, product)
    shipment = call("shipping", create_shipment, order)   # bounded by what remains
    return confirm(order, payment, shipment)

Verify with the budget simulation (lesson 7) that, in the worst case, orders responds within the 1200 ms instead of overshooting to 1340 ms. Document the case where the budget clips shipping's timeout below its p99: it's the signal that the chain is at the deadline's limit, and an honest note for your deliverable ("if this happens often, we have to parallelize or defer shipping").

Step 5 — Build the table and the justification

Put it all together in the before/after table, which is the heart of the deliverable:

Metric (under hung shipping)	WITHOUT timeout	WITH timeout
success_rate healthy traffic (browse)	37% (62/167)	100% (167/167)
Healthy requests rejected	105	0
p99 latency of orders (healthy traffic)	2373 ms	31 ms
Pool: % of time at 100% full	86%	0%
Pool: busy threads on average	7.3 / 8	3.2 / 8
checkout (to shipping)	17 served super slow, 33 rejected	50 fast timeout

And the justification in prose, which accompanies the table:

• What the timeout solves: the contagion. The healthy catalog traffic went from 37% to 100% success because the timeout keeps the hung checkout requests from monopolizing the pool. The timeout saved the one that had nothing broken.
• What the timeout does NOT solve: the checkout itself still fails (50 timeouts) —shipping is broken and the timeout doesn't cure it—. That's left for the following modules: retry with backoff (M3) in case it's transient, stop hammering shipping with a breaker (M5), isolate its pool with a bulkhead (M6), and complete the order with a deferred shipment instead of failing (M7, degradation).
• Why each number: the read timeouts come from each healthy dependency's p99 × ~2; the connect timeout is short and uniform (healthy handshake); the budget bounds the chain to the client's 1200 ms deadline.

Rubric: how you know you did it well

Your project is complete when you can answer yes to all of this:

• Did you measure the "before" with concrete numbers (success_rate, rejections, p99, pool occupancy), not with a qualitative description?
• Does each timeout have a justification derived from the dependency's p99, and isn't it an invented round number?
• Did you separate connect and read, with the connect short and the read from the p99?
• Did you apply the time budget to the chain and verify that the worst case fits in the client's deadline?
• Does your before/after table show the improvement in the healthy traffic (not just in the checkout), which is where the contagion lives?
• Did you explicitly distinguish what failure the timeout solves (the contagion) from which it doesn't (the checkout that fails because shipping is broken), pointing out what later pattern attacks what's left?

If you answer "no" to any, there's your next iteration. The measurement is the proof; without it, you didn't do the project, you just wrote code.

Common mistakes in the project

Measuring only the checkout and declaring "it didn't help." What happens: the student sees that the checkout still fails (50 timeouts) with a timeout and concludes the timeout didn't help. Why it happens: the wrong metric is measured —the timeout doesn't fix the checkout, it fixes the contagion—. How to spot it: your conclusion ignores the 167/167 browse saved. How to fix it: the improvement lives in the healthy traffic (browse from 37% to 100%); that's the cascading failure avoided. The broken checkout is other modules' problem.

Inventing the timeouts instead of deriving them. What happens: "300 ms for everything" is set without measuring each dependency's p99. Why it happens: it's faster and "it worked in lesson 3." How to spot it: catalog (p99 ~40 ms) with a 300 ms timeout is unnecessarily slack, and payments (p99 ~320 ms) with 300 ms would cut healthy traffic. How to fix it: each dependency has its own p99 and therefore its own read timeout; derive one per dependency, not a global number.

Skipping the "before" measurement. What happens: the timeout is applied directly and only the "after" is delivered. Why it happens: it seems redundant to measure something "we already know is bad." How to spot it: you have no comparative table, only a final snapshot. How to fix it: the project's value is the contrast; without the measured "before," you can't quantify the improvement or defend the decision. Measure both worlds with the same seed.

Exercises

Exercise 1 — Derive payments' row. Add payments as a third traffic type to the simulator, with a healthy p99 of 320 ms, and suppose that during the incident payments isn't hung but slow (responds in 1500 ms, not infinite). What read timeout do you give it, and would you expect its behavior under a timeout to be the same, better, or worse than that of a hung shipping? Justify.

See solution

payments' read timeout: p99(320 ms) × ~2 ≈ 650 ms. It's larger than shipping's (300 ms) because payments is legitimately slower —its healthy tail reaches further—, and cutting it at 300 ms would generate false timeouts on healthy charges.

Behavior under a timeout: slow payments (1500 ms) would be less harmful without a timeout than hung shipping (infinite/3000 ms), because at least it responds in 1500 ms and releases the thread, while the hung one holds it indefinitely. But with a timeout the result is the expected and healthy one: since 1500 ms > 650 ms, each call to slow payments times out at 650 ms and releases the thread —just as fast as shipping at 300 ms, only with a higher ceiling in line with its p99—. The key difference: 650 ms of holding per call is more than 300 ms, so under the same load, slow payments presses the pool more than shipping (more thread time held per request). This reinforces lesson 5: the timeout is your detection latency, and a dependency with a higher p99 necessarily holds the thread longer before cutting —one more reason to isolate the pools by dependency (bulkhead, M6), so that slow payments doesn't share a pool with catalog—.

Important note on payments: retrying a charge that timed out is dangerous (it might have charged before you cut —lesson 2's "unknown state"—), so here the timeout cuts but you should not retry without idempotency (M3 and M4).

Exercise 2 — The budget that doesn't fit. With a 1200 ms deadline and timeouts of catalog (100 ms), payments (650 ms), and shipping (300 ms), the sum is 1050 ms + overhead. Suppose that one day payments degrades and its healthy p99 rises to 900 ms, so its read timeout should rise to ~1800 ms so as not to cut healthy traffic. What happens to the checkout's budget, and what design options do you have?

See solution

What happens to the budget: if payments needs 1800 ms so as not to cut healthy traffic, but the client's total deadline is 1200 ms, then payments alone doesn't fit in the deadline —its healthy read timeout (1800 ms) already exceeds the entire budget (1200 ms)—. The budget would clip payments to at most ~1100 ms (what remains after catalog), well below its p99 of 900 ms + margin, so you'd start cutting healthy charges. It's the "budget < p99" conflict of lesson 7, in its acute form: the chain no longer honestly fits in the deadline.

Design options (none is "tighten the timeout blindly"):

• Parallelize: if catalog is independent of payments, launch them in parallel so catalog doesn't consume sequential budget before payments. You gain ~100 ms, but it's not enough if payments alone already asks for 1800 ms.
• Defer shipping: take the shipment creation off the checkout's critical path —confirm the order after payments and create the shipment asynchronously (degradation, M7 and arch-styles)—. That frees shipping's 300 ms from the budget, but still doesn't solve that payments alone already exceeds 1200 ms.
• Renegotiate the deadline: if payments legitimately needs 1800 ms when degraded, maybe the checkout can't promise 1200 ms that day; the business decides whether to raise the deadline to 2 s or accept cutting payments.
• Attack the cause: payments' p99 rising to 900 ms is itself an incident to investigate (overload? slow deploy?); the timeout is a defense, not a cure.

The important thing: the budget revealed that the chain doesn't fit, and forces you into an explicit design decision instead of hiding the problem. That transparency is the budget's value.

Exercise 3 — Transfer the pattern. Outside Mercado, describe another system that calls an unreliable external dependency (for example, a service that queries a third-party API for exchange rates, or a backend that calls an email provider). Apply the project's cycle: what would you measure "before," what p99 would you need, what timeouts would you put, and what failure would the timeout solve and which wouldn't?

See solution

Let's take a backend that sends transactional emails by calling an external provider (SendGrid, SES, etc.) from a user registration flow.

• What to measure "before": under a slow/down email provider, how many user registrations fail or hang even though the registration itself (creating the account in your database) is healthy? If the call to the email provider has no timeout and shares a pool with the rest of the backend, a down provider can exhaust the pool and take down the whole backend —just as shipping took down catalog—. You'd measure: success_rate of the traffic that doesn't send email, rejected requests, pool occupancy.
• What p99 you need: the healthy latency of the email provider's API (typically you measure it responds in, say, p99 ≈ 400 ms). Read timeout ≈ 400 × 2 = ~800 ms. Short connect timeout (~1 s), though being an external API over the internet, the connect might need more margin than in your own datacenter.
• What timeouts: timeout=(1.0, 0.8) on the call to the provider, with a budget if the registration has a UX deadline.
• What the timeout solves: the contagion —a down email provider stops exhausting the pool and taking down the rest of the backend; the registrations that don't depend on email keep working—.
• What it does NOT solve: the email itself isn't sent. But here the natural solution is degradation (M7): don't block the registration on the email —confirm the account and enqueue the email to retry it asynchronously when the provider comes back—. The welcome email shouldn't be on the registration's critical path; the timeout avoids the contagion, and deferring the email (enqueuing it) keeps the user from even noticing the provider's failure.

The cycle is identical to Mercado's: measure the fragility, derive timeouts from the p99, apply, measure the improvement, and distinguish the contagion (which the timeout solves) from the symptom (which degradation/retry solve). That cycle is transferable to any call to an unreliable dependency.

Module closing and where it goes next

You finished the timeout module, and you finished it by measuring. You took Mercado's fragile checkout, quantified its fragility (37% success in the healthy traffic, pool at 100% 86% of the time), applied the pattern with judgment —connection and read timeouts, values from the p99, budget in the chain— and proved the improvement with a table (100% success, a pool that breathes). Above all, you practiced the cycle that governs the whole guide: measure → apply → measure, and you learned to distinguish what a pattern solves from what it leaves pending.

What the timeout leaves pending is the agenda of the following modules, and now it makes sense. The checkout that times out still fails; module 3 asks whether it's worth retrying it —and discovers the retry storm, backoff, and jitter, all resting on the fact that now the calls fail fast and in a bounded way thanks to the timeout—. Retrying a charge that timed out requires the charge to be idempotent (module 4), because of the "unknown state" you saw in lesson 2. Stopping hammering a dead shipping altogether, saving even the cost of the timeout, is the circuit breaker (module 5). Keeping slow shipping from sharing a pool with payments is the bulkhead (module 6). And completing the checkout with a deferred shipment instead of failing is graceful degradation (module 7). Each of those patterns stands on the timeout you just mastered: without it, none gets a chance to act before the pool exhausts.

Summary and next step

In this project you integrated the module's seven concepts into the "measure → apply → measure" cycle on Mercado's checkout. You measured the "before" (success_rate of the healthy traffic 37%, 105 rejections, orders' p99 2373 ms, pool at 100% 86% of the time), chose the timeouts with the p99 method (read = p99 × ~2 per dependency, connect short and uniform), applied them separating connection and read, added the budget in the chain to fit in the client's deadline, and measured the "after" (success_rate 100%, 0 rejections, p99 31 ms, a pool that never fills). The deliverable is the code, the before/after table, and the justification of each number.

The project's deepest lesson is methodological: a pattern is justified by measuring, not by citing. The before/after table is what turns "I put in timeouts" into a defensible engineering decision, and the habit of distinguishing "what the pattern solves" (the contagion) from "what it leaves pending" (the broken checkout) is what lets you combine patterns with judgment instead of piling them up.

With this you master the timeout end to end: what it is, why it's the first defense, where to put it, what value to give it, what it promises, and how to chain it. What comes next is module 3: retries, backoff, and jitter —what to do after the timeout fires—. You'll see that retrying naively creates a retry storm that kills the service that was recovering, and that the cure is exponential backoff with jitter. And that whole module rests on what you just built: the calls now fail fast and in a bounded way, which is the precondition for being able to retry them wisely.

Resources

• Michael T. Nygard, Release It!, 2nd ed. (Pragmatic Bookshelf, 2018) — the stability patterns chapter shows how timeout, circuit breaker, and bulkhead combine in a real system; the mental frame for what this project starts and the following modules complete. In English.
• Marc Brooker, "Timeouts, retries, and backoff with jitter", Amazon Builders' Library — aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter. The article that integrates timeouts with what comes in module 3 (retries and backoff); read it as a bridge. Free and in English.
• Google SRE Book, "Addressing Cascading Failures" — sre.google/sre-book/addressing-cascading-failures. The end-to-end treatment of how a partial failure becomes total and how timeouts, deadlines, and the rest of the defenses avoid it. The big-picture view of the whole guide. Free and in English.
• requests documentation — requests.readthedocs.io/en/latest/user/advanced/#timeouts — and httpx — www.python-httpx.org/advanced/timeouts. The references you'll use when implementing this project's timeouts in real code. In English.