What happens when many worker processes all start up and hit the same resource (database, cache) at once, and how do you prevent that thundering herd?
Synchronized startups turn a routine scale-up or deploy into a self-inflicted outage unless the demand hitting the shared resource is spread out in time.
LEARN BY DOINGPut your intuition to work3 steps
What happens when many worker processes all start up and hit the same resource (database, cache) at once, and how do you prevent that thundering herd?
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.#What a Startup Stampede Looks Like
A thundering herd is what happens when a large number of independent workers all decide to act on the same shared resource at (or near) the same instant, and that resource's capacity is nowhere near enough to absorb all of them at once. The "startup" flavor shows up constantly in real deployments: an autoscaler reacts to a traffic spike and brings 2 replicas up to 20 in one step; a rolling restart (or a bad deploy that crash-loops and recovers) brings every existing replica down and back up within the same few seconds; a scheduled job fires on every replica at the top of the minute because it's driven by wall-clock time rather than a single leader.
Whatever the trigger, the shape of the problem is the same: each new process runs its own startup path, and that path usually does real work against a shared resource — opening a pool of database connections, priming a local in-memory cache by querying the database or a config service, registering with service discovery, or fetching secrets. When that startup path runs on 20 processes within the same second, the shared resource sees 20x its steady-state load, all in a single burst.
#The Cache Stampede Variant
A closely related failure hits a single hot cache key rather than a fleet of processes: a widely-requested cache entry expires, and the next request that misses recomputes it from the database — a slow, expensive query. Under real traffic, it isn't one request that misses; it's the hundreds or thousands of concurrent requests that were all being served from that key in the same instant. All of them miss together, and all of them independently hammer the database with the same expensive recomputation at once. This is the same pattern as the process-startup case — synchronized demand hitting a resource with real capacity limits — just triggered by a TTL boundary instead of a deployment event.
#Why It Cascades Instead of Just Being Slow
The dangerous part is the feedback loop at the bottom of the unmitigated path: the initial burst causes real latency and timeouts, which triggers retries from upstream callers or the app's own retry logic, and those retries land on the same already-overloaded resource, adding to the load instead of backing off from it. This is a retry storm, and it's what turns a brief, survivable spike into a sustained outage — the database or cache doesn't get a chance to recover because the retries keep arriving as fast as the original requests did.
#Mitigations
| Technique | What it does | Best for |
|---|---|---|
| Jittered startup delay | Each instance waits a small random amount before doing its expensive init work | Spreading N simultaneous process starts over a window |
| Staggered / rolling rollout | Deployment brings up new instances a few at a time instead of all at once, gated by readiness | Deploys and scale-up events |
| Connection pool caps + lazy connect | Cap max connections per instance and open them on demand rather than all eagerly at startup | Reducing peak connection count during startup |
| Request coalescing (single-flight) | Only one caller actually recomputes a missing value; concurrent callers wait on that one result | Cache stampede on a hot key |
| Probabilistic early expiration | Recompute a cache entry slightly before it actually expires, with randomized jitter per key | Avoiding many keys expiring in lockstep |
| Stale-while-revalidate | Serve the stale value immediately while one background request refreshes it | Latency-sensitive reads where slightly stale data is acceptable |
| Leader election for warmup | Exactly one instance performs a one-time expensive operation; others wait or skip it | Cache priming, schema migrations, one-time registration |
| Circuit breakers / bulkheads | Stop sending requests to an overloaded dependency instead of queuing behind it | Preventing cascading failure once the herd has already hit |
| Backpressure at the resource | A connection pooler (e.g., PgBouncer) or rate limiter in front of the shared resource itself | Capping load regardless of how many callers exist |
#The "Full Jitter" Shape
A useful general formula for spreading out both retries and startup delays is "full jitter": wait a random duration between zero and an exponentially growing cap — delay = random(0, min(maxDelay, baseDelay * 2^attempt)) — rather than a fixed or purely exponential delay. Fixed delays reproduce the exact same stampede on every retry round, just shifted in time; jitter breaks the synchronization instead of just moving it.
#Readiness Gates Matter Too
A subtler version of this problem comes from health checks rather than application code: if every new pod's readiness probe passes at nearly the same moment (because they all took roughly the same time to boot), the load balancer starts sending full production traffic to all of them simultaneously — the stampede hits the fleet's own downstream dependencies, not just the database connection step. Spreading initialDelaySeconds, adding jitter to a pod's own warmup before it reports ready, or leaning on a conservative maxSurge/maxUnavailable in a rolling update all address this by controlling when traffic starts arriving, not just when the process starts running.
🧠 Check your understanding
0/3 · 0/3 answered1. What is the defining characteristic of a thundering herd / startup stampede?
2. Why can a naive retry-with-fixed-backoff policy make a thundering herd outage worse rather than better?
3. What does 'request coalescing' (single-flight) solve in the cache stampede variant of this problem?