How does distributed locking work — Redis Redlock vs. ZooKeeper — and what can go wrong?

#Why you need a lock outside any single process

A mutex or a language-level lock only coordinates threads inside one process's memory. The moment you run more than one instance of a service -- which is the normal case, not the exception, for anything that needs to scale or survive a restart -- coordinating 'only one of you should do this right now' requires a lock that lives somewhere all instances can see: typically Redis or ZooKeeper (or etcd, which behaves similarly to ZooKeeper for this purpose). The canonical example is a scheduled job?: five instances of a service all wake up at 2am to run a nightly reconciliation, and exactly one of them should actually do the work.

#A single-node Redis lock

The building block is SET key value NX PX ttl: set the key only if it does not already exist (NX), with an expiry (PX) so a crashed holder cannot lock everyone else out forever. The value should be a random token unique to this acquisition, not a constant -- that token is what lets you release your own lock safely, using a compare-and-delete instead of a blind delete.

import redis
import uuid

r = redis.Redis(host='localhost', port=6379)

RELEASE_SCRIPT = '''
if redis.call('get', KEYS[1]) == ARGV[1] then
    return redis.call('del', KEYS[1])
else
    return 0
end
'''

def acquire_lock(lock_name, ttl_ms=10000):
    token = str(uuid.uuid4())
    acquired = r.set(lock_name, token, nx=True, px=ttl_ms)
    return token if acquired else None

def release_lock(lock_name, token):
    release = r.register_script(RELEASE_SCRIPT)
    # Only deletes the key if it still holds OUR token -- a blind DEL
    # could otherwise release a lock some other client has since acquired.
    return release(keys=[lock_name], args=[token])

This is correct against the single most common bug in hand-rolled Redis locks, which is releasing with a plain DEL and accidentally deleting a lock some other client acquired after yours expired.

#Redlock: spreading the lock across a quorum

A single Redis node is a single point of failure -- if it goes down before the lock expires, nobody can acquire or release it. Redlock addresses this by asking for the same lock on N independent Redis nodes (typically 5), counting it as acquired only if a majority respond successfully within a time budget well under the TTL, and accounting for clock drift between nodes. Release is attempted against all N nodes regardless of which ones granted it.

#The classic caveat: pauses break the timing assumption

Redlock's safety depends on an assumption that is easy to state and surprisingly easy to violate: no client can pause for longer than the lock's TTL between acquiring the lock and finishing its work. A long garbage-collection pause, a hypervisor migrating a VM, or the process simply being descheduled under load can all violate it.

The fix is not a better algorithm for the lock itself -- it's making the protected resource reject stale writers. A fencing token is a number that increases every time the lock is granted; the resource records the last token it accepted and rejects any write carrying a lower one. Client A's write in the diagram above would carry an old token and get rejected, even though A still (wrongly) believes it holds the lock.

#ZooKeeper: consensus instead of timing

ZooKeeper locks work differently: a client creates a sequential ephemeral znode under a lock path (/locks/job-0000001), and holds the lock when its znode has the lowest sequence number among siblings -- otherwise it watches the next-lowest one and waits. 'Ephemeral' means the znode is tied to the client's session: if the client's process dies or its connection drops, ZooKeeper deletes the znode automatically and the next client in line acquires the lock, no TTL guesswork required. The sequence number doubles as a natural fencing token.

CuratorFramework client = CuratorFrameworkFactory.newClient(
    "zk1:2181,zk2:2181,zk3:2181",
    new ExponentialBackoffRetry(1000, 3));
client.start();

InterProcessMutex lock = new InterProcessMutex(client, "/locks/nightly-reconciliation");

if (lock.acquire(10, TimeUnit.SECONDS)) {
    try {
        runReconciliationJob();
    } finally {
        lock.release();
    }
} else {
    log.warn("could not acquire lock, another instance is running the job");
}