What is the bulkhead pattern, and how does it stop one failing dependency from taking down your whole service?
A slow payment provider shouldn't be able to freeze your whole checkout flow — bulkheads wall off each dependency's resources so one failure stays contained.
LEARN BY DOINGPut your intuition to work3 steps
What is the bulkhead pattern, and how does it stop one failing dependency from taking down your whole service?
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.#The ship that doesn't sink from one hole
Bulkheads are named after the watertight compartments in a ship's hull. If a torpedo breaches one compartment, that compartment floods, the crew seals the doors around it, and the rest of the ship stays afloat and keeps sailing. Nobody builds a ship as one giant open room below deck, because a single puncture anywhere would sink the whole vessel. Most backend systems, left to their default architecture, are built exactly like that giant open room.
#The failure mode you're protecting against
Picture an order service that calls three downstream dependencies — payments, inventory, and a recommendation engine — and all three calls go through the same shared HTTP client backed by one thread pool of, say, 50 threads. Under normal conditions this is fine: each call is fast, threads return to the pool quickly, and 50 threads comfortably serve all three dependencies.
Now the recommendation engine starts timing out — maybe it's under a load spike, maybe a downstream database query regressed. Every request to it now takes 30 seconds instead of 30 milliseconds. Threads that call the recommendation engine stop returning to the pool because they're blocked waiting on a response. Within a few seconds, all 50 threads are stuck waiting on the slow dependency. At that point, requests to payments and inventory — both completely healthy — have no threads left to run on. They queue up and start timing out too. A single slow dependency has just taken down calls to two unrelated, perfectly healthy services. This is resource-exhaustion cascading failure, and it's one of the most common ways a "minor" incident in one dependency turns into a full outage.
#What a bulkhead actually does
The bulkhead pattern partitions the shared resource — usually a thread pool, a connection pool, or a semaphore limiting concurrency — so each downstream dependency gets its own bounded slice instead of drawing from one shared pot. If the payments client gets 15 threads, inventory gets 15, and recommendations gets 15 (plus a small shared reserve), then recommendations timing out can exhaust its own 15 threads, but the other 30 keep serving payments and inventory without interruption. The failure is contained to the compartment it started in.
#Two ways to build the wall
| Approach | How it works | Trade-off |
|---|---|---|
| Thread-pool isolation | Each downstream dependency gets its own dedicated executor/thread pool and queue | Strong isolation, even from a call that blocks the thread itself, but costs more memory and context-switching overhead across many pools |
| Semaphore isolation | A counting semaphore caps how many concurrent calls to a dependency are in flight, reusing the caller's own thread | Cheap — no extra threads — but a call that truly hangs the calling thread still ties it up, giving weaker isolation |
Semaphore isolation is usually enough when the client library reliably respects timeouts and never blocks the calling thread indefinitely; thread-pool isolation is the safer default when you don't fully trust that.
#Sizing the compartments
A bulkhead sized too small becomes an artificial bottleneck even when the downstream dependency is perfectly healthy — you throttle yourself. A reasonable starting point comes from Little's Law: the number of concurrent slots you need is roughly the expected throughput (requests per second) multiplied by the expected latency (in seconds), plus headroom for variance and retries. A dependency called 100 times per second with a 50ms p99 latency needs roughly 5 concurrent slots at steady state — but you size for the tail, not the average, and add margin for the traffic spikes and retry storms that are exactly when bulkheads matter most.
#Bulkheads pair with circuit breakers — they don't replace them
A bulkhead limits how much of your resources a failing dependency can consume; it does not stop you from calling that dependency at all. If recommendations is down hard, you'll still burn through your 15 allotted threads on every single request, timing each one out individually. A circuit breaker sitting in front of the same call detects the elevated failure rate and starts short-circuiting requests immediately — failing fast or falling back to a cached/default response — instead of letting every one of those 15 threads wait out a full timeout. Used together, the bulkhead caps the blast radius and the circuit breaker cuts the bleeding once the pattern is clear.
#The same idea, several places in your stack
The bulkhead principle shows up well beyond thread pools: a dedicated database connection pool per downstream service so a lock-heavy reporting query can't starve the checkout path's connections, per-tenant rate limits or queues in a multi-tenant system so one noisy customer can't degrade everyone else, and dedicating separate node pools or pods to latency-sensitive workloads versus batch jobs in Kubernetes. The mechanism changes, but the goal is always the same: a failure in one compartment should be observable, contained, and recoverable — not something that silently drains a resource every other part of the system also depends on.
🧠 Check your understanding
0/3 · 0/3 answered1. In the shared-thread-pool scenario, why do healthy dependencies start failing when only one dependency is slow?
2. What is the main trade-off of semaphore isolation compared to thread-pool isolation?
3. Why are bulkheads and circuit breakers used together rather than one replacing the other?