5. The Titanic analogy

Overview

The previous lessons gave you the measured pattern: one compartment per dependency returns the healthy traffic to 100%. This lesson settles the mental image that governs the bulkhead —a ship's watertight compartments— and, through it, teaches you the pattern's most important and least-told lesson: a bulkhead only isolates if it's complete. And for that there's no better teacher than the most famous ship that sank despite having watertight compartments: the Titanic.

The term "bulkhead" comes literally from naval engineering: it's the watertight wall, the transverse wall that divides a ship's hull into watertight compartments, so that a leak in one doesn't flood the others. The idea is old and good. The Titanic had it: sixteen compartments separated by bulkheads, designed to float even with several flooded. And even so it sank. Understanding why —what failed in its bulkheads, which were there— is understanding the difference between a bulkhead that really contains and one that only seems to. That difference, in your code, is the difference between really isolating and believing you isolated.

Connection with the module: this is the mental-image lesson and its warning. Lessons 2 to 4 gave you the problem and the solution measured; here you extract the model that lets you reason about any bulkhead —the concept of blast radius— and discover the failure mode that ruins apparently correct bulkheads: overflow through a shared resource that was left below (or above) the wall. You'll measure it with an "incomplete" bulkhead that isolates the threads but shares the connections, and you'll see the water overflow just the same. Lesson 6 will use this idea —isolate all the relevant resources— to isolate by traffic class; lesson 7 will put the cost of so much isolation on the table.

The Titanic's compartments, and why they weren't enough

The Titanic was divided into sixteen watertight compartments by fifteen transverse bulkheads. The design was genuinely good for its time: the ship could stay afloat with the first four compartments completely flooded, or with any pair of adjacent compartments flooded. For almost any imaginable accident —a collision that broke one or two compartments—, the bulkheads would have contained the water and the ship would have floated. The watertight compartments were, on paper, a solid defense.

The iceberg did something the design didn't contemplate: instead of a blow concentrated on one or two compartments, it tore the hull along its length, opening five or six compartments at once. That alone exceeded the design margin (four). But the fatal detail, the one that turns this story into an engineering lesson, is another: the bulkheads didn't reach the top. They rose only a few meters above the waterline —up to deck E, in some cases D—, not up to the top deck. They were tall walls, but with the top edge open.

While the ship was level, that didn't matter: each compartment contained its water. But as the bow flooded, the ship tilted forward, and then the water of the front compartment, already full, overflowed over the top of the bulkhead into the next compartment, which until that moment was dry. As that one filled, it overflowed into the next. Like an ice tray you tilt: the water passes from one cell to the next over the walls, one after another, because the walls don't reach the edge. The bulkheads were there, they were real, and even so the water went around them over the top. The Titanic didn't sink from not having compartments; it sank because its compartments weren't watertight all the way up: they left a path over which the water jumped from one to another.

The lesson for your code: isolate all the resources, not just one

Here's the exact translation to software, and it's the central warning of this lesson. You can give each dependency its own thread pool —build the wall of threads— and feel that you isolated. But if those dependencies share another resource underneath —a database connection pool, the same event loop, the same saturated CPU, the same unbounded queue that grows in memory—, then one's failure will find that shared resource and overflow to the others over your wall of threads, exactly like the water overflowed over the Titanic's bulkheads. Your thread bulkhead was real, but it didn't reach the top: there was another level where the compartments were still connected.

The most common case in practice: you isolate the thread pools per dependency, but the three dependencies draw their connections from a single database connection pool. When shipping hangs holding a connection, it drains the shared connection pool; and then catalog, even if it has free threads in its own sub-pool, can't execute anything because it can't get a connection. Your threads were isolated; your connections, no. The water jumped over the edge of the bulkhead that didn't reach the top.

Worked example: the bulkhead that didn't reach the top

We measure exactly that scenario. Two dependencies, catalog and shipping, each with its own sub-pool of 6 threads —the threads are isolated—. But in one case they share a single pool of 6 connections (the incomplete bulkhead, like the Titanic's bulkheads) and in another each has its own connection pool (the complete bulkhead, bulkheads to the top). A hung shipping holds a connection while it waits. Same seed. This is the real output:

INCOMPLETE (threads isolated, connection pool SHARED)
  catalog  : served= 36/235  success_rate= 15.3%
  shipping : served= 12/115  success_rate= 10.4%

COMPLETE   (threads AND connection pool isolated)
  catalog  : served=241/241  success_rate=100.0%
  shipping : served= 15/115  success_rate= 13.0%

What to expect. Read it as the Titanic's story. In the incomplete bulkhead, catalog drops to 15.3% —it collapses just the same!—, despite having its own six threads completely available. Why, if its threads are isolated? Because a hung shipping holds the connections of the shared pool, and catalog can't execute without a connection: its threads are free but useless, waiting for a connection shipping doesn't release. The water overflowed over the wall of threads, through the resource they did share. In the complete bulkhead, where each dependency also has its own connection pool, catalog goes back to 100%: now the wall reaches the top, and shipping's flooding finds no path toward catalog's compartment.

The moral, measured: isolating one resource isn't enough if another stays shared. The bulkhead must cover all the resources through which a slow dependency can propagate its failure —threads, connections, memory, CPU, locks—. A compartment with a tall wall but a hole at the bottom floods just the same. In your service, the question isn't "did I isolate the threads?", but "is there any finite resource the dependencies still share that a slow one can hog?". If the answer is yes, there's your bulkhead that doesn't reach the top.

The blast radius: the metric of isolation

The Titanic analogy also gives the concept that best summarizes what a bulkhead is for: the blast radius. The blast radius of a failure is how much of the system goes down when one part fails. Without a bulkhead, the blast radius of a hung shipping is the whole system: all of orders goes down, dragging catalog and payments. With a bulkhead, the blast radius of that same hung shipping is only the shipment: orders stays up, catalog and payments work, and only shipment creation degrades. The bulkhead doesn't reduce the probability that shipping fails —that's impossible—; it reduces the radius of that failure.

You can put numbers on it, using the module's measurements. The blast radius, measured as "which dependencies go down when shipping hangs":

                        blast radius of "shipping hung"
  SHARED pool     :  catalog DOWN (19%) + payments DOWN (17%) + shipping DOWN  -> 3 of 3 services
  With BULKHEADS  :  catalog OK (100%) + payments OK (100%) + shipping down     -> 1 of 3 services

From three affected services to one. That's the bulkhead's job expressed in its natural metric: contain the blast radius. When you design or review a system, the useful question isn't "can this fail?" (everything can fail), but "when this fails, what else goes down with it?". If the answer is "half the system," you're missing a bulkhead. If it's "only this function," the bulkhead is there and reaches the top.

A bulkhead that reaches the top: the list of resources

For your bulkhead to be "all the way up," it's worth explicitly going through the resources the dependencies can share, because each one is a possible overflow path:

• Threads / workers: the base case of lesson 4. One thread pool per dependency.
• Connections (to database, to services): the case we just measured. One connection pool per dependency (or per failure boundary), not a shared one.
• Memory / queues: an unbounded queue is a bulkhead with the top edge open —the water (the queued requests) isn't contained, it overflows as memory that grows until it knocks down the process—. Each compartment's queues should be bounded.
• CPU / the same process: if all the dependencies run in the same process and one does heavy CPU work, it can drown the others even if they have "their own" threads (the threads compete for the same cores). Sometimes real isolation requires separate processes or machines.
• Shared locks: if two dependencies take the same lock and one holds it while it hangs, the other blocks —a bulkhead with a secret passage—.

You don't always need to isolate all of them; you need to isolate those through which a failure can propagate. The criterion is the same as the blast radius: for each shared resource, ask yourself "if a dependency hogs this resource when it fails, does it knock down the others?". If yes, that resource needs its own compartment.

Common mistakes

Declaring victory after isolating a single resource. What happens: the team separates the thread pools per dependency, tests the hung-shipping scenario, sees it improve, and considers the bulkhead done —without noticing that the connections are still shared—. Why it happens: thread isolation is the most visible and the one tutorials teach; the shared resources "underneath" are easy to forget. How to spot it: an incident where a slow dependency knocks down another despite having separate thread pools —the signature of overflow—. How to fix it: go through the list of resources (threads, connections, memory, CPU, locks) and verify that no finite one stays shared in a way that a failure hogs it. The bulkhead must reach the top at all levels.

Unbounded queues inside the compartments. What happens: each dependency has its thread pool, but with an unlimited queue; when the dependency hangs, the queue grows without brakes and consumes the process's memory, knocking everyone down. Why it happens: an unbounded queue seems "kinder" (it never rejects), but it only moves the overflow from "rejection" to "memory." How to spot it: the process's memory growing during a single-dependency incident, with OOM at the end. How to fix it: bound each compartment's queue; let it reject fast when it fills. A bulkhead with an infinite queue is a bulkhead with the top edge open: the water overflows as memory.

Confusing "I have compartments" with "I'm safe". What happens: it's assumed that, by having bulkheads, no failure can propagate —as it was assumed that the Titanic was "unsinkable"—. Why it happens: the existence of the defense gives a false sense of completeness. How to spot it: there's no real test that the failure stays contained; only the presence of the pattern. How to fix it: test the isolation as you tested everything in this guide —provoke one dependency's failure and measure the blast radius—. A compartment you never flooded on purpose is a compartment you don't know is watertight. (Testing resilience with fault injection is a topic of the testing ecosystem; here the idea suffices: don't trust the bulkhead you didn't test.)

Exercises

Exercise 1 — Find the bulkhead that doesn't reach the top. A team isolated its dependencies with a thread pool each. Even so, when shipping hangs, catalog goes down. List at least three shared resources "underneath" that could be causing the overflow, and for each say how you'd verify it.

See solution

Three candidate shared resources that overflow despite the separate thread pools:

1. Shared database connection pool. If the three dependencies draw connections from a single pool, a shipping that holds connections when it hangs drains them, and catalog can't get any even if it has free threads (the case measured in this lesson). How to verify it: look at whether there's a single connection pool/DataSource for everything, and observe whether during the incident catalog fails with "no connection available" with its threads idle.

2. Unbounded queue/memory. If shipping's thread pool has an unlimited queue, when it hangs it accumulates requests in memory until the whole process runs out of RAM, knocking catalog down too (same process). How to verify it: check the queue size of each pool; observe whether the process's memory grows during the incident.

3. CPU / same process. If the dependencies share the same process and shipping (or its hang handling) consumes CPU intensively, catalog competes for the same cores and slows down. How to verify it: process CPU at 100% during the incident; catalog's latency rising even though catalog itself is healthy.

Others possible: a global lock both take, a shared HTTP client with its own connection limit, a shared rate limiter. The general method: for each finite resource, ask yourself "do the dependencies share it, and can one hog it when it fails?". That's the bulkhead that doesn't reach the top.

Exercise 2 — The blast radius before and after. Describe the blast radius of "catalog's database gets slow" in two designs: (a) orders with a shared pool for the three dependencies; (b) orders with per-dependency bulkheads. What goes down in each case?

See solution

• (a) Shared pool: the blast radius is all of orders. A slow catalog (even a read) holds threads from the shared pool; with enough catalog traffic, those slow threads fill the pool, and then payments and shipping —which don't touch the catalog database— also run out of threads. The customer can't browse nor charge nor ship. The whole checkout goes down because of a slow read.

• (b) Per-dependency bulkheads: the blast radius is only catalog. The slow catalog fills its own sub-pool and starts rejecting (or degrading) catalog reads, but payments and shipping have their compartments intact: charging and shipping keep working. The customer who already has the product in the cart can complete the purchase even if the catalog is slow. The failure was confined to the browsing function.

The difference is exactly the Titanic's: in (a), one compartment's flooding sinks the ship; in (b), one compartment floods and the ship floats. Note that here the dependency "that fails" is catalog, not shipping —the bulkhead protects in any direction: it isolates each dependency from the failures of all the others, not just from a predefined culprit—.

Exercise 3 — When does the bulkhead need to be a separate process? Thread-pool isolation works for waiting failures (I/O that hangs). Give an example of a failure that a separate thread pool within the same process doesn't contain, and explain what level of isolation would be needed.

See solution

A separate thread pool doesn't contain a failure that affects the whole process, because the thread pools share the process. Examples:

• Intensive CPU consumption: if a dependency (or the handling of its requests) does heavy CPU work —for example, deserializing giant responses or a loop that goes off—, it saturates the cores that all the threads share. catalog's thread pool has its threads, but those threads don't get CPU time. Isolation needed: separate processes or containers with CPU limits (cgroups), or even different machines.

• Memory leak / OOM: if a dependency accumulates objects and exhausts the process's memory, the Out Of Memory kills the whole process, with all its thread pools inside. Isolation needed: separate processes, so an OOM in one doesn't kill the others.

• A crash / segfault / uncaught exception that knocks down the process: same problem; the thread pool doesn't help if the process dies.

The isolation ladder goes from lower to higher strength (and cost): semaphore (concurrency) → thread pool (threads) → process → container with limits → machine. Each level isolates more types of failure, at more cost. For I/O-waiting failures (this module's case), the thread pool suffices; for failures that affect the process (CPU, memory, crash), you have to go up to separate processes or containers. Choosing the level is choosing which failures you want to contain against how much cost you accept —lesson 7's balance—.

From the image to the traffic classes

The Titanic left you two ideas that govern every bulkhead. First: the pattern is measured by the blast radius —how much of the system goes down when one part fails—, and the bulkhead's job is to shrink it. Second, and crucial: a bulkhead only isolates if it reaches the top —if it covers all the resources through which a failure can overflow—; you isolated the threads but shared the connections, and the water jumped just the same (15.3% vs 100%, measured). A compartment with a hole floods.

Up to here we isolated by dependency —one compartment for payments, one for shipping, one for catalog—. But isolation has another axis, just as useful: the traffic class. Sometimes the failure doesn't come from which dependency you call, but from what type of request arrives —a batch job that drowns the interactive traffic, a premium customer who deserves guaranteed capacity against the spikes of free traffic—. Lesson 6 takes the bulkhead to that axis: isolating by traffic class, so one type of work doesn't drown another, using the same tools (thread pools, semaphores) over a different partition.

Summary and next step

In this lesson you settled the mental image of the bulkhead —a ship's watertight compartments— and extracted its most important lesson through the Titanic: a bulkhead only isolates if it's complete. The Titanic had compartments, but its bulkheads didn't reach the top deck, so as the bow tilted the water overflowed from one compartment to the next over the walls. In software, the equivalent is isolating one resource (the threads) while sharing another (the connections, the memory, the CPU): the failure finds the shared resource and overflows just the same. You measured it: with isolated threads but a shared connection pool, catalog drops to 15.3% despite having free threads; with threads and connections isolated, it goes back to 100%.

You learned the concept of blast radius —how much of the system goes down when one part fails— as the bulkhead's natural metric: without isolation, a hung shipping knocks down 3 of 3 services; with bulkheads, 1 of 3. And you learned the isolation ladder —semaphore, thread pool, process, container, machine—: each level contains more types of failure (waiting, CPU, memory, crash) at more cost, and the correct level depends on which failure you want to contain.

Before moving on you should be able to: tell the Titanic's lesson and its translation to software; list the resources a bulkhead must cover to "reach the top"; and express a bulkhead's effect as a reduction of the blast radius.

What follows is a second axis of isolation. Lesson 6 takes the bulkhead beyond the dependency, to the traffic class: isolating batch work from interactive, premium traffic from free, the critical from the secondary, so one type of request doesn't drown another. Same tools (thread pools, semaphores), different partition.

Resources

• Michael T. Nygard, Release It!, 2nd ed. (Pragmatic Bookshelf, 2018) — the Bulkhead chapter uses exactly the naval analogy and warns that the partition must cover the correct resource so the failure doesn't propagate through another path. The source of the image and the warning. In English.
• Encyclopedia Titanica, "Watertight Subdivision / bulkheads" — www.encyclopedia-titanica.org. The historical reference on the Titanic's sixteen compartments, the bulkheads that didn't reach the top deck, and how the water overflowed from one to another. The engineering fact behind the analogy. In English.
• Adrian Cockcroft (ex-Netflix), talks on blast radius and isolation — for example "Microservices and the Art of Taming the Dependency Hell Monster". The concept of blast radius as a design metric for resilient systems, from the one who popularized it in practice. Look it up in English.
• Google SRE Book, "Addressing Cascading Failures" — sre.google/sre-book/addressing-cascading-failures. Covers how hidden shared resources (connections, memory, queues) propagate a failure despite the apparent partitions; this lesson's overflow, at production scale. Free and in English.