8. Project: isolate Mercado's checkout dependencies and measure

Overview

This is the module's capstone. You learned the bulkhead piece by piece —the contagion that motivates it, what it is, how it's implemented, how it's sized, the Titanic warning, the second axis of traffic classes, and its cost—. Now you apply it whole, with your own hands, to Mercado's checkout. The task has the shape that repeats in every module of the guide, and that by now is your method: you take an unisolated system, measure its fragility with numbers, apply the pattern, and measure again to prove it worked. Not "I put bulkheads because the book says so"; "I put bulkheads, and here's the table showing that catalog and payments went from 19% and 17% to 100% while the broken shipping was confined." The before/after measurement is the deliverable, because it's what turns a belief into a defensible engineering decision.

You're going to work with orders' checkout, which calls payments, shipping and catalog over a shared pool —the same scenario you measured in lesson 2, now as your project—. First you instrument it and provoke the incident (shipping hung) to see the contagion: the three dependencies collapsing even though only one is broken. Then you partition the pool into a compartment per dependency, size it with judgment, and measure again. The deliverable is the isolated code, the before/after metrics table, and —most importantly— the justification of which failure the bulkhead solves and which it leaves to the other patterns.

Connection with the module: this lesson doesn't introduce a new concept; it integrates the previous seven into a workflow. Lesson 2 gave you the shared-pool simulator and the contagion; lesson 3, the definition and the two forms; lesson 4, the measured solution and the sizing; lesson 5, the warning to isolate all the resources; lesson 6, the axis of traffic classes; lesson 7, the cost. Here you put them together. When you finish, you'll have done from start to finish the cycle "measure the contagion → isolate → measure the isolation," and you'll be ready for module 7 (degradation —what to respond to the buyer whose shipping compartment is full—) and for the module 8 capstone, where the bulkhead combines with the circuit breaker.

The deliverable, at a glance

By the end of the project you'll have produced three things:

1. The isolated checkout (code): the version with a compartment (thread pool) per dependency, each sized by its dependency's healthy traffic, with the connections also isolated (the bulkhead to the top).
2. The before/after metrics table: each dependency's success_rate, with the shared pool and with the bulkheads, measured under the same incident (shipping hung) and the same seed.
3. The justification: why each compartment has the size it has, which failure the bulkhead solves (the contagion to the healthy dependencies) and which failure it does not solve (the broken shipping), making clear what remains for the circuit breaker (cutting) and degradation (responding to the user).

Step 1 — Take the shared checkout and measure the contagion

We start from the lesson 2 simulator, which already models the complete scenario: a shared pool of 12 threads, queue of 12, and traffic to the three dependencies with shipping hung. Your first job is to run it with the shared pool and record the contagion. This is the line of code that represents "the unisolated checkout": the three dependencies mapped to the same pool.

# The UNISOLATED checkout: a shared pool for the three dependencies.
pools   = {"all": {"free": 12, "cap": 12, "queue": [], "qcap": 12}}
pool_of = {"catalog": "all", "payments": "all", "shipping": "all"}

Run the simulation with shipping hung and note the "before" numbers. These are the ones you already saw in lesson 2, now as your baseline measurement:

=== SHARED POOL (the unisolated checkout) ===
  catalog  : served= 45/233  rejected=188   success_rate= 19.3%
  payments : served= 17/101  rejected= 84   success_rate= 16.8%
  shipping : served= 24/117  rejected= 93   success_rate= 20.5%

Translate it to the deliverable's metric. The healthy traffic's success_rate collapsed: catalog to 19.3% and payments to 16.8%, even though neither of them has anything broken. The only really broken dependency is shipping (20.5%), but it dragged the other two down to its same basement. This is the picture of the contagion: one dependency's failure turned into the outage of all three, from sharing a pool. Save these numbers; they're the left half of your table.

Step 2 — Size the compartments with Little's law

Before partitioning, decide the size of each compartment with the lesson 4 method, not by eye. You need the expected concurrency of each dependency under healthy traffic: concurrency = arrival rate × latency (Little's law), plus a margin of ~2× to absorb spikes.

• catalog: arrives ~40 req/s (every 25 ms), latency ~0.02 s → concurrency ≈ 40 × 0.02 = 0.8 → with margin → sub-pool of 2–3 threads.
• payments: arrives ~17 req/s (every 60 ms), latency ~0.12 s → concurrency ≈ 17 × 0.12 = 2 → with margin → sub-pool of 4 threads.
• shipping: arrives ~20 req/s (every 50 ms), healthy latency ~0.15 s → concurrency ≈ 20 × 0.15 = 3 → with margin → sub-pool of 4–5 threads.

Crucial note about shipping: you size it by its healthy traffic (when it works, ~150 ms), not by the hang. When shipping hangs, its compartment will fill and reject —and that's exactly what we want: to confine—. You don't size shipping's sub-pool to "withstand the hang"; you size it for its normal operation and let the bulkhead do its job when it fails. For the project, we'll use a split of 4/4/4 (12 total threads, the same budget as the shared pool), which covers the three concurrencies with margin.

Note the justification now, while you have it fresh: each sub-pool covers its dependency's healthy concurrency by a margin of ~2×, enough not to reject healthy traffic in natural spikes and contained enough that a hang doesn't take more than its quota. The total (12) respects the thread budget —we don't inflate resources, we just partition them—. This justification is part of the deliverable: a compartment without a size justification is a magic number.

Step 3 — Isolate and measure again

Now run the same simulation changing only the pool assignment: from a shared one to a compartment per dependency. In the simulator it's this two-line change:

# The ISOLATED checkout: a compartment (sub-pool) per dependency.
pools = {
    "catalog":  {"free": 4, "cap": 4, "queue": [], "qcap": 4},
    "payments": {"free": 4, "cap": 4, "queue": [], "qcap": 4},
    "shipping": {"free": 4, "cap": 4, "queue": [], "qcap": 4},
}
pool_of = {"catalog": "catalog", "payments": "payments", "shipping": "shipping"}

In orders' real code, it's a ThreadPoolExecutor per dependency —with its connection pool also separate, so the wall reaches the top (lesson 5)—:

from concurrent.futures import ThreadPoolExecutor

# One compartment per dependency: threads AND connections isolated.
pools = {
    "catalog":  ThreadPoolExecutor(max_workers=4),
    "payments": ThreadPoolExecutor(max_workers=4),
    "shipping": ThreadPoolExecutor(max_workers=4),
}
conn_pools = {                    # isolated connections: the bulkhead to the top (lesson 5)
    "catalog":  make_db_pool(max_conn=4),
    "payments": make_http_pool(max_conn=4),
    "shipping": make_http_pool(max_conn=4),
}

class BulkheadFull(Exception):
    pass

def call(dep, fn, *args):
    try:
        future = pools[dep].submit(fn, *args)         # runs in dep's compartment
    except RuntimeError:
        raise BulkheadFull(dep)                        # sub-pool + queue full: fast reject
    return future.result(timeout=TIMEOUTS[dep])        # module 2 timeout, still applies

Run the isolated simulation with shipping hung and note the "after" numbers:

=== BULKHEAD 4/4/4 (the isolated checkout) ===
  catalog  : served=241/241  rejected=  0   success_rate=100.0%
  payments : served=101/101  rejected=  0   success_rate=100.0%
  shipping : served=  8/117  rejected=109   success_rate=  6.8%

catalog's success_rate rose to 100% and payments' to 100%: the two healthy dependencies, which the contagion had sunk, are intact. The broken shipping was confined to its compartment (6.8%): it serves less than shared because it no longer steals others' resources, and that's correct. This is the right half of your table.

Step 4 — Verify that the bulkhead reaches the top

Before declaring victory, apply the Titanic lesson: verify that no shared resource is left through which the failure overflows. The most common trap is isolating the threads but sharing the connection pool. If in your real code the three dependencies draw connections from a single pool, a hung shipping can drain it and drown catalog despite the separate thread pools —you measured it in lesson 5: catalog drops to 15.3% with isolated threads but shared connections, against 100% with both isolated—.

Go through the list of resources and confirm that each one is partitioned by dependency (or that it can't propagate the failure):

• Threads: a ThreadPoolExecutor per dependency. ✓ (step 3)
• Connections: a connection pool per dependency, not a shared one. ✓ (step 3, conn_pools)
• Queues: each sub-pool with a bounded queue (that rejects, not that grows in memory without limit).
• CPU/process: if any dependency does heavy CPU work, consider whether the thread pool suffices or you need a separate process.

Document in your deliverable which resources you isolated and which you decided not to isolate (with the reason: "can't hang," "doesn't share a finite resource"). The bulkhead you didn't check is the one that overflows.

Step 5 — Build the table and the justification

Put everything together in the before/after table, the heart of the deliverable:

success_rate under shipping hung	SHARED pool	With BULKHEADS
catalog (healthy)	19.3% (45/233)	100% (241/241)
payments (healthy)	16.8% (17/101)	100% (101/101)
shipping (HUNG)	20.5% (24/117)	6.8% (8/117)
Blast radius (services down)	3 of 3	1 of 3

And the justification in prose, which accompanies the table:

• What the bulkhead solves: the contagion. The healthy traffic of catalog and payments went from ~18% to 100% because the isolation prevents shipping's hung checkouts from monopolizing the pool. The bulkhead reduced the blast radius from "3 of 3 services down" to "1 of 3": Mercado went from "everything down" to "everything works except creating shipments."
• What the bulkhead does NOT solve: shipping itself is still broken (6.8%) —it's hung, and no isolation cures it—. That's left for other patterns: the circuit breaker (M5) stops calling shipping entirely when it confirms it's dead, saving even the timeout cost; the degradation (M7) completes the order with a deferred shipment instead of failing. The bulkhead isolates; the breaker cuts; the degradation responds. They combine in the module 8 capstone.
• Why each size: each sub-pool covers its dependency's healthy concurrency (Little's law) by a margin of ~2×; the total (12) respects the thread budget —we partition, we don't inflate—.

Rubric: how you know you did it right

Your project is complete when you can answer affirmatively to all of this:

• Did you measure the "before" with the success_rate per dependency (not an aggregate), showing that healthy catalog and payments dropped to the level of the broken shipping?
• Does each compartment have a size derived from Little's law (rate × latency) plus margin, and not an invented number?
• Did you respect the total thread budget (partition, don't inflate)?
• Did you verify that the bulkhead reaches the top —connections and queues isolated, not just threads— so the failure doesn't overflow through a hidden shared resource?
• Does your before/after table show the improvement in the healthy traffic (catalog, payments), which is where the contagion lives, and not just in shipping?
• Did you explicitly distinguish which failure the bulkhead solves (the contagion) from which it doesn't (the broken shipping), naming which later pattern attacks what remains (breaker, degradation)?
• Did you express the result as a reduction of the blast radius (from 3 of 3 to 1 of 3 services)?

If you answer "no" to any, there's your next iteration. The measurement is the proof; without it, you didn't isolate, you just reorganized the code.

Common mistakes in the project

Measuring only shipping and declaring "the bulkhead didn't work". What happens: the student sees that shipping still fails (6.8%, worse than the 20.5% shared) and concludes that the isolation made things worse. Why it happens: the broken dependency is measured instead of the healthy ones. How to spot it: your conclusion ignores that catalog and payments went from ~18% to 100%. How to fix it: the bulkhead is judged by the healthy traffic saved, not by the broken dependency. That shipping serves less isolated is correct: its shared throughput was stolen from the healthy ones.

Isolating the threads and forgetting the connections. What happens: separate thread pools are created but the three dependencies share the connection pool, and in production catalog goes down just the same when shipping hangs. Why it happens: the connection pool is an easy-to-forget shared resource (the Titanic trap). How to spot it: catalog failing with "no connection" with free threads. How to fix it: isolate the connections too (step 4). The bulkhead must reach the top on all the resources that can propagate the failure.

Over-partitioning "just in case". What happens: a compartment is created for each endpoint, and under healthy rotating spikes the isolated system rejects more than the shared one. Why it happens: after the module, isolation always feels good. How to spot it: rejections of healthy traffic in dependencies while others are idle —lesson 7's elasticity loss—. How to fix it: isolate along real failure boundaries (dependencies that hang independently), not any distinction; for each compartment, name the failure it contains.

Exercises

Exercise 1 — Redesign the split under a tight budget. Your machine only tolerates 9 threads total (not 12). catalog needs healthy concurrency ~1, payments ~2, shipping ~3. Propose a split of the 9 threads, justify it, and say what compromise you accept versus the comfortable split of 12.

See solution

With 9 threads and the healthy concurrencies (catalog ~1, payments ~2, shipping ~3, summing to 6), a reasonable split with some margin is catalog=2, payments=3, shipping=4 (sum 9), prioritizing giving margin to the higher-concurrency ones. Another option, if you want to protect the charge more: catalog=2, payments=4, shipping=3.

Justification: each sub-pool covers its healthy concurrency with a small margin; catalog, being super fast (~1 concurrency), makes do with 2; payments and shipping, slower, receive more. The compromise versus the comfortable 12: less margin for spikes. With 4 threads, catalog absorbed spikes with room to spare; with 2, a spike of catalog could brush rejections. You accept a bit more risk of rejection in healthy spikes in exchange for fitting in the budget of 9. What doesn't change is the protection against contagion: as long as each dependency has its compartment (even one of 2), a hung shipping can't touch catalog's or payments' threads, so the healthy ones stay protected. The tight budget trims the margin (internal elasticity), not the containment (isolation). That's the correct hierarchy of sacrifices: first yield margin, never the wall.

Exercise 2 — The compartment that overflows. You implemented per-dependency bulkheads and in the simulator test catalog gives 100%. But in production, when shipping really hangs, catalog drops to ~20%. catalog's thread pool has free threads. Diagnose which shared resource was left unisolated and how you'd confirm it.

See solution

The symptom —catalog dropping with free threads— is the exact signature of the Titanic overflow (lesson 5): the threads are isolated, but another shared resource isn't. Candidate number one is the shared database connection pool: a hung shipping holds connections (or, if catalog and shipping share a backend, shipping drains the common pool), and catalog can't execute its reads even with threads, because it can't get a connection. It's exactly the scenario you measured (15.3% with shared connections vs 100% with separate ones).

How to confirm it:

1. Observe why catalog fails: if the errors are "no connection available" / "connection pool timeout" (not "thread pool full"), the bottleneck is the connections, not the threads.
2. Look at the connection pool's occupancy during the incident: if it's at 100% while catalog's thread pool has free threads, there's the overflow.
3. Check the configuration: is there a single shared connection pool / DataSource, or one per dependency?

The fix: give each dependency (or at least shipping, the one that hangs) its own bounded connection pool, so its hang doesn't drain catalog's connections. The simulator gave 100% because it didn't model the connections; production does have them. Lesson: test the isolation against all the resources, not just the ones your simulator models. The bulkhead you didn't check is the one that overflows.

Exercise 3 — Transfer the pattern. Outside Mercado, take a system that calls several unreliable dependencies (for example, an API gateway that routes to five microservices, or another store's checkout service that calls inventory, payments and fraud). Apply the project's cycle: what would you measure "before," how would you isolate, which resources would you verify, and which failure would the bulkhead solve and which not?

See solution

Let's take an API gateway that routes requests to five microservices (users, products, orders, search, recommendations) from a single client app.

• What to measure "before": over a shared worker pool in the gateway, hang one of the microservices (say recommendations, usually the slowest and least critical) and measure the success_rate of the other four. If they share a pool, a hung recommendations exhausts the gateway's workers and knocks down users, products, orders, search too —the whole gateway goes down because of the least important microservice—. You measure: success_rate per microservice, and the blast radius (how many of the five go down when one hangs?).

• How to isolate: a thread pool (or concurrency semaphore) per microservice in the gateway. Size each one by its healthy traffic (Little's law). The critical ones (orders, payments) get compartments with margin; the secondary ones (recommendations) get a small compartment —in fact, you want recommendations to be the first to reject under pressure, not the one that knocks the others down—.

• Which resources to verify (bulkhead to the top): threads per microservice (✓), outgoing HTTP connections (do they share an HTTP client with a global connection limit? isolate it), bounded queues per compartment. If the gateway queries a shared cache or database, isolate it too.

• What the bulkhead solves: the contagion —a hung recommendations stops knocking down the gateway; users, products, orders and search keep working—. It reduces the blast radius from "5 of 5" to "1 of 5", and on top of that confines the damage to the least critical service.

• What it does NOT solve: recommendations itself is still down. Natural solution: degradation (M7) —the gateway returns a response without recommendations (an empty list or a fallback) instead of failing the whole page— and circuit breaker (M5) —it stops calling recommendations while it's dead—. The bulkhead avoids the contagion; degradation and breaker handle the symptom of the downed service.

The cycle is identical to Mercado's: measure the contagion per dependency, isolate with sized compartments, verify that the bulkhead reaches the top, measure the blast-radius reduction, and distinguish the contagion (which the bulkhead solves) from the symptom (which breaker and degradation solve). Transferable to any system that calls several unreliable dependencies.

Module close and where it goes next

You finished the bulkhead module, and you finished it by measuring. You took Mercado's unisolated checkout, quantified the contagion (catalog 19.3% and payments 16.8% —healthy, sunk by shipping—), partitioned the pool into a compartment per dependency sized with judgment, verified that the bulkhead reached the top, and proved the improvement with a table (the healthy ones at 100%, the blast radius from 3 of 3 to 1 of 3). Above all, you practiced the cycle that governs the whole guide —measure → isolate → measure— and learned to distinguish what a pattern solves from what it leaves pending.

What the bulkhead leaves pending is the agenda of the modules that follow, and now it makes sense. shipping is still broken (6.8%); its compartment fills and rejects. Module 7 asks what to respond to the buyer whose checkout fell into that full compartment: not to fail hard, but to degrade —complete the order with a deferred shipment, return a fallback, an "order confirmed, shipment in preparation"—, so the contained failure isn't even noticed on the user's side. And module 8, the guide's capstone, combines everything: timeout (M2) to bound the time per thread, retry/backoff/idempotency (M3-M4) to retry safely, circuit breaker (M5) to cut the traffic to a dead shipping, bulkhead (M6, this module) to contain the blast radius from second zero, and degradation (M7) to respond well to the user. The bulkhead is the wall that makes the other patterns act on a contained fire instead of on a house in flames.

Summary and next step

In this project you integrated the module's seven concepts into the "measure the contagion → isolate → measure the isolation" cycle over Mercado's checkout. You measured the "before" (shared pool, shipping hung: catalog 19.3%, payments 16.8%, blast radius 3 of 3), sized the compartments with Little's law (rate × latency × margin), partitioned the pool respecting the thread budget, verified that the bulkhead reached the top (connections and queues isolated, not just threads), and measured the "after" (catalog and payments at 100%, shipping confined to 6.8%, blast radius 1 of 3). The deliverable is the code, the before/after table and the justification of sizes and boundaries.

The project's deepest lesson is the same one that runs through the guide: a pattern is justified by measuring, not by citing, and each pattern solves one thing and leaves another. The bulkhead solves the contagion —one dependency's failure turning into the outage of all— and you measure it as a reduction of the blast radius; it doesn't solve the broken shipping, which is left for the breaker (cutting) and the degradation (responding). Isolating well is sizing each compartment by its real failure boundary, with the bulkhead reaching the top, paying the fair cost of elasticity and complexity, neither too much nor too little.

With this you master the bulkhead from start to finish: the contagion that motivates it, what it is and its two forms, how it's sized, the Titanic warning, the axis of traffic classes and its cost. What follows is module 7: graceful degradation and load shedding —what to do with the user whose compartment filled—: instead of failing hard when shipping doesn't respond, complete the checkout with a deferred shipment, so the failure the bulkhead contained is, for the buyer, almost invisible. And then the capstone (module 8), where all the guide's patterns combine over the same Mercado checkout.

Resources

• Michael T. Nygard, Release It!, 2nd ed. (Pragmatic Bookshelf, 2018) — the stability patterns chapter shows how bulkhead, timeout and circuit breaker combine in a real system; the mental framework for what this project integrates and module 8 completes. In English.
• Netflix Technology Blog, "Making the Netflix API More Resilient" — netflixtechblog.com/making-the-netflix-api-more-resilient-a8ec62159c2d. The production case of isolating each dependency in its own thread pool and combining it with circuit breakers —exactly the architecture this project simulates and module 8 combines—. Free and in English.
• Google SRE Book, "Addressing Cascading Failures" — sre.google/sre-book/addressing-cascading-failures. The end-to-end treatment of how a partial failure turns total and how resource isolation, timeouts and the rest of the defenses avoid it. The guide's big-picture view. Free and in English.
• resilience4j documentation, "Bulkhead" and "Getting Started" — resilience4j.readme.io/docs/bulkhead. The reference you'll use when implementing this project's bulkheads in real code (ThreadPoolBulkhead, SemaphoreBulkhead), and how they compose with the circuit breaker and the timeout in a single decorator. In English.