7. The cost of isolation

Overview

The previous five lessons sold the bulkhead, and rightly so: it isolates the failures, contains the blast radius, returns the healthy traffic to 100%. This lesson is the engineer's honesty: isolating costs. There's no free defense, and the bulkhead pays its price in three currencies —more reserved resources, less elasticity, and more complexity to size and tune—. Understanding that cost isn't to discourage you from using bulkheads; it's so you use them with judgment and don't fall into the opposite mistake of lesson 2: isolating nothing is fragile, but over-partitioning —cutting the system into too many tiny compartments— is just as fragile, for different reasons.

The central cost, the one we'll measure, is the loss of elasticity. A shared pool has a virtue we sacrifice on isolating: when one part of the system has a spike and another is idle, the idle one lends its capacity to the one that needs it. A common pool absorbs spikes because it shares the capacity. On isolating into compartments, that idle capacity is locked up: shipping's sub-pool, even if it's empty, can't lend a single thread to catalog's sub-pool that's under a spike. You isolated the risk, but you also isolated the capacity. And that, under certain load patterns, makes the isolated system reject traffic the shared one would have served.

Connection with the module: this is the counterweight lesson. Lessons 2 to 6 gave you the benefit of isolation (by dependency and by class); here we measure its cost, so the decision to isolate is a balance and not a reflex. The lesson 8 project will ask you to justify not only that you isolated, but how much —how many compartments, of what size—, and that justification is impossible without understanding the cost this lesson quantifies. It's also the lesson that connects with module 8 (the capstone), where combining a bulkhead with a breaker and degradation requires knowing how much isolation is enough and how much is too much.

The virtue you lose: the parking lot analogy

Imagine an office building with a parking lot of 120 spots shared among three companies that rent floors. On a normal day, company A uses 40 spots, B uses 30, C uses 50 —they fit comfortably—. But on the days when company A has a big meeting and arrives with 70 cars, it finds a spot, because B that day only brought 20 and C 30: B's and C's idle spots absorb A's spike. The shared parking lot is elastic: the unused capacity of some cushions the spikes of others.

Now divide the parking lot into three sections closed with a barrier: 40 exclusive spots for A, 40 for B, 40 for C —no one uses anyone's—. This has an advantage: if company A brings trucks that leak oil and dirty its 40 spots, the problem stays in its section; B's and C's are clean (isolation). But it has a cruel cost: on the day of A's big meeting, its 40 spots fill and cars 41 onward are left outside, in the street, while B's and C's sections have empty spots A can't use —there's a barrier—. The isolated parking lot lost the elasticity: B's and C's idle capacity no longer cushions A's spike. You isolated the dirt (the oil), but you also isolated the space.

That's the bulkhead's deal in one image. The shared pool shares the risk (A's oil dirties everyone) and the capacity (A's spots absorb B's spike). The bulkhead separates both: it protects you from the shared risk, but you give up the shared capacity. The design question isn't "isolate or not?" in the abstract, but "does the risk I isolate justify the elasticity I lose?".

Worked example: the healthy spike the bulkhead rejects

We measure the elasticity cost with a scenario where all the dependencies are healthy —no hangs, no failures—, but one has a traffic spike. catalog receives an avalanche (one read every 5 ms, ~40 ms each: it needs ~8 threads of concurrency), while payments and shipping are quiet. We compare the shared pool of 12 against the bulkheads of 4/4/4. Same seed. This is the real output:

### COST OF ISOLATION: a HEALTHY spike of catalog, all deps OK ###
(catalog spiking; payments and shipping quiet and healthy)

  catalog under the spike:
    SHARED POOL (lends the idle threads)      : 100.0%  (served 1201, rejected   0)
    BULKHEAD (capped to its 4 threads)        :  50.2%  (served  601, rejected 596)

What to expect. Here the result inverts relative to the whole module: the shared pool wins. Under catalog's healthy spike, the shared pool serves 100% (1201 requests, zero rejections), because catalog borrows the idle threads of payments and shipping —which don't need them that day— and thus absorbs its spike with the 12 threads. The bulkhead, on the other hand, has catalog capped to its 4 threads: no matter how empty payments' and shipping's 8 threads are, catalog can't touch them, so it rejects 596 of 1197 requests —half of a perfectly healthy traffic—, with eight threads idle looking on. The isolation that saved catalog when shipping hung (lesson 4) is the same one that now condemns it when catalog has a spike and the others are idle.

That's the elasticity cost, measured: the bulkhead turns idle capacity into unreachable capacity. In the shared pool, no thread is wasted: if catalog needs them and the others don't, catalog uses them. In the bulkhead, shipping's threads are shipping's even if shipping doesn't use them and catalog is drowning. Isolating the risk had as its price losing the spike absorption. The same wall that contains one dependency's flooding prevents one's idle capacity from cushioning another's spike.

The three currencies of the cost

The elasticity cost is the subtlest, but not the only one. The bulkhead pays in three currencies:

1. More reserved resources. To give each dependency a compartment with margin, you reserve threads (and connections, and memory) per dependency. If each one needs 4 threads with margin and you have 3 dependencies, that's 12 threads reserved, many of them idle most of the time —because the three rarely spike at once—. The shared pool would serve the same traffic with fewer total threads, precisely because it shares them. Isolating is, in part, paying for capacity that sits idle to have it guaranteed when needed.

2. Less elasticity. What we just measured: one compartment's idle capacity doesn't cushion another's spike. The isolated system is more predictable (each dependency knows exactly what it has) but less adaptable (it can't reallocate on the fly). Under irregular loads —spikes that rotate among dependencies— the shared pool makes better use of the same resources.

3. More sizing and operational complexity. Each compartment is a size to choose, measure and tune. With a shared pool you have one number to size; with per-dependency bulkheads you have one per dependency, and with per-class bulkheads, one per class per dependency —the combinatorics grow—. Each one badly sized is a problem: too small rejects healthy traffic, too big wastes. And each one has to be readjusted when the traffic changes. Isolation exchanges "one pool to tune" for "many compartments to tune," and that operational burden is real.

The opposite mistake: over-partitioning

Lesson 2 showed the danger of isolating nothing. This lesson shows the opposite danger: isolating too much. A system cut into twenty tiny compartments —one for each endpoint, one for each traffic variant— suffers the worst of the three currencies: each compartment is so small that it rejects healthy traffic at the slightest spike (no margin to absorb variance), the elasticity is lost entirely (nothing is lent to anything), and the complexity of sizing twenty numbers is unmanageable. The over-partitioned system can reject more healthy traffic, in aggregate, than one with a well-sized shared pool —recreating the fragility that isolation came to avoid, from the other extreme—.

The practical rule to not over-partition: isolate along the real failure boundaries, not any imaginable distinction. Two dependencies that fail independently and can hang deserve separate compartments (the risk justifies the cost). But two endpoints that call the same healthy dependency, with the same latency profile and no reason to fail separately, probably don't —isolating them only pays the cost without buying useful isolation—. The question before creating a compartment is always: "what independent failure does this wall contain, and is that isolation worth the elasticity and complexity it costs?". If you can't name the failure it contains, the wall isn't worth its price.

How to recover some elasticity without losing the isolation

The deal between isolation and elasticity isn't always all-or-nothing; there are intermediate designs that recover some elasticity while keeping the containment:

• Compartments with a shared cushion. Instead of splitting the 12 threads into rigid 4/4/4, you give each dependency a guaranteed reserve (3 each = 9) and leave 3 threads in a common pool anyone can take when its reserve is exhausted. That way each dependency has its guaranteed floor (isolation) but can stretch to the shared cushion under a spike (some elasticity). The cushion, being small, limits how much a broken dependency can invade —it contains the damage without locking up all the capacity—.
• Elastic limits with a ceiling. A compartment can have a guaranteed base size and a larger ceiling it grows to if there's free capacity, yielding it when another dependency reclaims it. It's more complex to implement, but it brings the best of both worlds closer.
• Isolating only what hangs. Not all dependencies need a compartment. The ones that do network I/O and can hang, yes. A purely local and fast operation, that can't hang waiting for anyone, rarely needs its own bulkhead —leaving it in the common pool preserves elasticity without risking contagion, because there's nothing to contain—.

These designs don't eliminate the deal —there's always tension between isolating and sharing—, but they soften it. The decision is still yours: how much elasticity you give up for how much isolation you gain, measured against your system's load patterns and real failures.

Common mistakes

Isolating everything "just in case" without naming the failure it contains. What happens: a compartment is created for each endpoint or each traffic distinction, out of prudence. Why it happens: isolation always feels good after the previous lessons. How to spot it: compartments that never contained a real failure, many of them idle, and rejections of healthy traffic in spikes a shared pool would have absorbed. How to fix it: for each compartment, name the independent failure it contains; if you can't, merge it with another. Isolation is justified by a real failure boundary, not by symmetry or diffuse prudence.

Sizing the compartments without margin to save resources. What happens: so as not to "waste" threads, each sub-pool is sized exactly to the average concurrency (no margin), and then it rejects healthy traffic at every natural spike. Why it happens: idle capacity is seen as waste. How to spot it: frequent rejections of healthy traffic with the compartment's average occupancy below 100% (the rejections happen in the spikes, not on average). How to fix it: the margin over the average concurrency (the ~2× of lesson 4) isn't waste; it's what absorbs the variance. A compartment with no margin recreates the fragility on the rejection side. The cost of isolation includes paying that margin.

Forgetting that the shared pool wins under healthy rotating spikes. What happens: a system whose dependencies are all healthy and only have spikes that rotate (never spike at once) is isolated, and the isolated system rejects more than the shared one. Why it happens: isolation was applied where the problem wasn't the failure but the spike. How to spot it: rejections in healthy dependencies while others are idle —the exact signature of the elasticity loss—. How to fix it: if the problem is absorbing spikes of healthy traffic (not containing failures), the shared pool or a shared-cushion design may be better. The bulkhead is for containing failures, not for absorbing spikes; using it for the latter pays its cost without collecting its benefit.

Exercises

Exercise 1 — Isolate or not? For each pair of operations, decide whether they deserve separate compartments or it's better to leave them in a common pool, naming the failure the isolation would contain (or its absence): (a) an HTTP call to payments and an HTTP call to shipping; (b) two endpoints that both read from catalog with the same latency profile; (c) a fast read from a local in-memory cache and an HTTP call to an external service.

See solution

• (a) payments and shipping: separate compartments. Failure it contains: each can hang independently (network, overload, deploy), and if they share a pool, the one that hangs knocks down the other (lesson 2). The risk is real and justifies the cost. Isolate.

• (b) two endpoints that read from catalog with the same profile: probably a common pool. What independent failure would separating them contain? Almost none: they call the same dependency, with the same latency, and fail together (if catalog goes down, both go down). Isolating them pays the cost (less elasticity, two numbers to size) without buying useful isolation. Leave them together —unless one is interactive and the other batch, in which case the partition axis is the traffic class, not the endpoint (lesson 6)—.

• (c) local cache vs external HTTP: separate, but for an asymmetric reason. The HTTP call can hang and needs its compartment (with a timeout and a bulkhead). The local cache read can't hang waiting for anyone (it's process memory), so it doesn't need its own bulkhead —and in fact putting it in a separate thread pool would only add overhead—. Here "isolate" means: the HTTP in its compartment, the local cache in the direct flow. Not everything needs the same treatment; you isolate what can hang.

The rule in action: you isolate where there's an independent failure to contain (a, the HTTP of c); you don't isolate where there isn't (b, the cache of c). Naming the failure is the test.

Exercise 2 — The elasticity cost, with numbers. In the measurement, under catalog's healthy spike, the shared pool served 100% and the bulkhead 50.2%. Explain exactly why the bulkhead rejected 596 requests while having 8 idle threads, and what intermediate design could have served more without losing all the isolation.

See solution

Why it rejected with idle threads: in the 4/4/4 bulkhead, catalog is capped to its 4 threads. catalog's spike needs ~8 threads of concurrency (one read every 5 ms, ~40 ms each → 40/5 = 8). With only 4 threads available for catalog and a bounded queue, half the requests don't fit and are rejected —even though payments' and shipping's 8 threads are completely idle that day—. The bulkhead's wall prevents catalog from touching those 8 threads: they're payments' and shipping', period. The capacity exists but is unreachable. That's why 596 rejections with 8 empty threads.

Intermediate design: a shared cushion. Instead of rigid 4/4/4, you give each dependency a guaranteed reserve of 3 threads (9 total) and leave 3 threads in a common pool anyone can take when it exhausts its reserve. Under catalog's healthy spike, catalog would use its 3 guaranteed plus the 3 of the shared cushion = 6 threads, serving much more than with 4 (though maybe not 100%, because the cushion is small). And it keeps isolation: if shipping hangs, it can only invade the cushion of 3 (not the 9 guaranteed), so catalog is left with its 3 no matter what. The cushion exchanges a bit of isolation (the 3 shared threads are a bounded contagion vector) for a good deal of elasticity (it absorbs spikes). It's the midpoint between the shared pool (all elasticity, zero isolation) and the rigid bulkhead (all isolation, zero elasticity).

Exercise 3 — The capstone's tension. In module 8 you'll combine a bulkhead with a circuit breaker. A colleague proposes: "if I have a circuit breaker, I don't need a bulkhead: the breaker cuts shipping when it hangs, so it never fills the pool." Evaluate the argument considering the cost and benefit of each one.

See solution

The argument underestimates the time window before the breaker acts. The circuit breaker doesn't open instantly: it needs to accumulate a number of failures (or timeouts) to decide that shipping is dead. In those first seconds of the incident —while shipping starts hanging but the breaker is still counting failures— shipping can fill the shared pool and drown catalog and payments, before the breaker cuts. The bulkhead protects during that window: from the first instant, shipping can't take more than its quota, no matter how long the breaker takes to open.

On the cost: the bulkhead pays elasticity and complexity (this lesson); the breaker pays little (a failure counter and a timer). But they do different things: the breaker cuts the source (stops calling shipping, saving even the timeout), the bulkhead contains the blast radius (limits how much shipping can occupy while it's still being called). Together: the bulkhead holds from second zero and contains the damage; the breaker, once the problem is confirmed, turns off the useless traffic and gives shipping room to recover. Removing the bulkhead leaves the initial window exposed; removing the breaker keeps paying the cost of calling a dead service. The combination isn't redundant: it covers risks at different moments. That's what you'll build in the capstone —and knowing the cost of each is what lets you size them without waste—.

From the balance to the capstone

You already have both sides of the bulkhead. The benefit (lessons 2 to 6): it contains failures, shrinks the blast radius, protects the healthy and critical traffic. The cost (this lesson): more reserved resources, less elasticity —measured: a healthy spike of catalog goes from 100% served in the shared pool to 50.2% in the bulkhead, with idle threads it can't touch— and more sizing and operational complexity. And the opposite mistake to lesson 2's: over-partitioning recreates the fragility on the rejection side. Isolating is a balance, not a reflex: it's justified by a real failure boundary, with the margin and number of compartments that boundary warrants.

What follows is putting all the module's content together with your own hands. Lesson 8 is the project: you take Mercado's checkout over a shared pool, measure the contagion, apply per-dependency bulkheads, measure the isolation, size the compartments with judgment (paying the fair cost, neither too much nor too little), and deliver the before/after table distinguishing what the bulkhead solves —the contagion— from what it leaves for the breaker and degradation —the broken shipping—. It's the module's "measure → isolate → measure" cycle, complete.

Summary and next step

In this lesson you measured and named the cost of isolation, the other side of everything before. The central cost is the loss of elasticity: one compartment doesn't lend its idle capacity to another. You measured it with a healthy spike of catalog (all dependencies OK): the shared pool serves 100% because catalog borrows the idle threads of payments and shipping; the bulkhead serves only 50.2% —it rejects 596 requests with 8 unreachable idle threads—, because the wall that isolates the risk also locks up the capacity. Isolation turns idle capacity into unreachable capacity.

You learned the three currencies of the cost (more reserved resources, less elasticity, more sizing and operational complexity); the over-partitioning mistake —too many tiny compartments recreate the fragility on the rejection side—; the rule to avoid it (isolate along real failure boundaries, naming the failure each wall contains); and the intermediate designs (shared cushion, elastic limits with a ceiling, isolating only what hangs) that recover some elasticity without losing all the containment.

Before moving on you should be able to: explain why the shared pool wins under a healthy rotating spike; name the three currencies of the cost; and apply the "what failure does this wall contain?" test to decide whether a compartment is worth its price.

What follows is the integrating project. Lesson 8 has you walk through the module's complete cycle over Mercado's checkout —measure the contagion, isolate by dependency, measure the isolation, size with judgment, and deliver the before/after table—, closing the module with the bulkhead applied end to end and its clear boundary with the breaker (which cuts) and degradation (which responds to the user of the flooded compartment).

Resources

• Michael T. Nygard, Release It!, 2nd ed. (Pragmatic Bookshelf, 2018) — the Bulkhead chapter discusses the deal between isolation and resource utilization, and warns against partitioning beyond what the failure boundaries justify. The source of this lesson's balance. In English.
• Marc Brooker, "Some risks of coordinating only sometimes" and other Amazon Builders' Library posts on isolation — aws.amazon.com/builders-library. They deal with the trade between isolating loads (containment) and sharing capacity (elasticity/utilization), this lesson's central cost, at AWS scale. Free and in English.
• Google SRE Book, "Handling Overload" — sre.google/sre-book/handling-overload. How capacity reservation by class (isolation) interacts with global utilization, and why over-reserving wastes while under-reserving rejects healthy traffic; this lesson's sizing tension. Free and in English.
• resilience4j documentation, "Bulkhead" (maxConcurrentCalls and maxWaitDuration parameters) — resilience4j.readme.io/docs/bulkhead. Shows that each bulkhead is a set of parameters to size; the operational complexity (third currency of the cost) made concrete configuration. In English.