How do you keep a cache consistent when you have multiple application instances writing to it?

#Why "just update the cache" doesn't scale past one instance

When a single process owns both the cache and the writes, keeping them in sync is trivial: update the value, update the cache, done. The moment you scale out to multiple application instances — which is the normal case for anything behind a load balancer — that simplicity disappears. Now several processes can write the same key at nearly the same time, in an order that has nothing to do with the order their writes actually reach the database, and each of them may also be holding a private, in-process (L1) copy of the data that nobody else can see or invalidate directly.

The failure mode is almost never "the cache is wrong forever." It's transient staleness that shows up as confusing, hard-to-reproduce bugs: a user changes their plan, refreshes, and still sees the old plan for a few seconds — but only sometimes, and only on some pods.

#Prefer invalidation over replication

The single most important design decision is: on a write, delete the cache entry rather than set it to the new value. Setting the cache on every write looks appealing because it avoids an extra miss, but it introduces a race that's easy to lose:

1. Instance A starts writing v2 to the DB.
2. Instance B starts writing v3 to the DB (a moment later, legitimately newer).
3. B's DB write commits first, B sets cache = v3.
4. A's DB write commits second (say it was delayed by a slow replica or a retry), A sets cache = v2.
5. The cache now holds v2 — stale — even though the DB correctly holds v3.

Deleting the key instead of setting it removes this race almost entirely: whichever write finishes last, the cache ends up empty, and the next reader repopulates it from the database, which is the only place that has a consistent view of "the latest value." An empty cache is never wrong; a cache holding the wrong value is.

#The one race that survives: delete-then-repopulate

Even "delete on write" has a narrow window: a reader can race between your write and your delete, read the old value from the DB (on a lagging replica, for instance), and re-populate the cache with it right after you deleted the key — leaving stale data behind indefinitely, or until TTL. The standard mitigation is the delayed double delete: delete the key, do the write, delete the key again after a short delay (long enough to cover typical replication lag or in-flight reads, e.g. 500ms to a few seconds). It doesn't make the system linearizable, but it closes the realistic window without adding a distributed lock to every write path.

#Propagating invalidation to every instance

Deleting the shared L2 (Redis, Memcached) handles the shared cache. It does nothing for the private L1 caches sitting inside each application process — those still hold the old value until their own TTL expires, because nothing told them otherwise. That's what a pub/sub? invalidation channel is for: the instance that performs the write publishes a small "this key changed" message, and every instance (including the writer itself, for uniformity) subscribes and evicts that key from its local L1 the moment the message arrives.

Redis gives you this mechanism natively: a plain PUBLISH/SUBSCRIBE channel that every instance listens on, or — since Redis 6 — client-side caching via CLIENT TRACKING, where Redis itself remembers which client cached which key and pushes an invalidation message automatically when that key changes, without you having to publish it by hand. Either way, the shape is the same: one shared broadcast channel that turns N independent, unaware caches into N caches that agree with each other within one network round-trip.

#Watch out for thundering herds after a broadcast invalidation

If you invalidate a hot key across every instance at once — a popular product's price, a config flag everyone reads — you can create the opposite problem: all instances miss at the same moment and hammer the database simultaneously. Guard against this with request coalescing (a single in-flight fetch per key, with concurrent callers waiting on it rather than each issuing their own query) or a short-lived lock around the repopulation step.

#A quick decision guide

🧠 Check your understanding
0/2 · 0/2 answered
1. On a write, which is generally safer for keeping a shared cache correct?
Writing to the cache before writing to the databaseSetting the cache to the new value immediatelyDoubling the TTL so it expires less oftenDeleting the cache key and letting the next read repopulate it
2. Why is a pub/sub invalidation channel not sufficient on its own to guarantee cache correctness?
Message delivery is fire-and-forget; a disconnected or restarting instance can miss the message entirelyRedis pub/sub cannot scale past a handful of subscribersIt only works with Memcached, not RedisPub/sub messages are limited to 8 bytes