How do cascading failures happen in a distributed system, and how do you actually stop one from taking down everything?

#What a cascading failure actually is

A cascading failure is what happens when a failure — often a small, localized one — in a single component triggers failures in the components that depend on it, which then trigger failures in their dependents, until degradation that started in one deep, unremarkable service takes down the entire system. The insidious part is that the trigger is frequently not an outage at all: a service that is merely slow is usually far more dangerous than a service that is cleanly down, because a hard failure fails fast, while a slow dependency invites every caller to sit around waiting, tying up resources the whole time.

#How the propagation actually mechanically happens

Picture four services in a call chain: an API Gateway calls Service B, which calls Service C, which calls Service D. Service D develops a latency spike — a slow downstream database query, a garbage-collection pause, disk contention, doesn't matter what. Service C calls D synchronously with no timeout configured, so every request into C that touches D now sits there waiting. Those requests hold open threads, or connections in a fixed-size connection pool, and within seconds C's own pool is fully exhausted by requests that are all waiting on a dependency that isn't responding. Now C itself looks slow or unresponsive to Service B — for exactly the same reason D looked slow to C — and B's resources start exhausting the same way. This keeps climbing until the API Gateway itself is exhausted, and now every single request into the system fails or hangs, even the ones that have nothing to do with whatever originally slowed down deep in the chain.

Two things make this worse in practice. First, naive retry logic: when a caller sees a timeout or an error, an unconditioned retry immediately resends the request, which adds more load onto a dependency that is already struggling — a "retry storm" that can turn a recoverable blip into a full outage. Second, shared infrastructure: if B, C, and D's messages all flow through the same queue cluster or the same connection pool to a shared datastore, a backlog caused by one slow consumer can exhaust disk or memory on infrastructure that unrelated services also depend on — a "noisy neighbor" effect that spreads the blast radius sideways, not just up the call chain.

#The core mitigations

• Timeouts on every network call. No call should be allowed to wait indefinitely. Timeouts should be set aggressively relative to the caller's own SLA, and ideally the timeout budget is propagated down the call chain as a deadline (this is what gRPC's deadline propagation gives you for free) so that a request that has already spent 150ms of its 200ms budget doesn't let a downstream call try for another 500ms.
• Circuit breakers. After a configured rate of failures or slow responses to a specific dependency, the breaker opens: further calls fail immediately (or fall back to cached/default data) without even attempting the network call, which stops piling more load onto a struggling dependency and gives it room to recover. The breaker periodically lets a small number of calls through in a "half-open" state to check whether the dependency has recovered before fully closing again.
• Load shedding. When a service is approaching saturation, it deliberately rejects a portion of incoming requests — ideally the lowest-priority ones first — rather than accepting all of them and having every single request degrade or the process crash outright. Serving 80% of requests well beats attempting 100% and delivering close to none.
• Bulkheads. Partition resources — thread pools, connection pools, even separate service instances or node pools — per dependency or per tenant, so that one slow or failing dependency can only exhaust the pool allocated to it, not the shared pool every other code path also relies on. The name comes from the watertight compartments in a ship's hull that keep flooding in one section from sinking the whole vessel.
• Backpressure, retry budgets, and jitter. Retries should be bounded (a small number of attempts), backed off exponentially, and jittered so that many clients retrying at once don't synchronize into another load spike, and a system-wide retry budget should cap the total fraction of extra traffic retries are allowed to add.

#Putting it together

None of these mechanisms are alternatives to each other — a mature service typically has all of them: an aggressive timeout on every outbound call, a circuit breaker wrapping each of those calls, a bulkhead separating the resource pool for each downstream dependency, and load shedding at the edge for when everything upstream is still not enough. The reason interviewers push hard on this topic is that it is exactly the kind of thing that is easy to state in the abstract and easy to skip in a real system under deadline pressure — and it's usually the missing piece in the retro after a real outage.