How does database replication actually work — leader-follower vs multi-leader, and what can go wrong?

#Why replicate a database at all

A single database instance is a single point of failure and a single ceiling on read throughput. Replication keeps copies of the same data on multiple nodes so that the system survives losing a machine, can serve reads from wherever the client is geographically closest, and can keep serving traffic while one node is down for maintenance. The complexity it buys you is: now there are multiple copies of the truth, and you have to decide what happens when they temporarily disagree.

#Leader-follower (single-leader) replication

In the most common topology, one node — the leader (or primary/master) — accepts all writes. It appends every write to a replication log (Postgres? calls it the WAL, MySQL calls it the binlog), and one or more followers (replicas/standbys) continuously stream and apply that log to reproduce the leader's state. Reads can be served from the leader or, for workloads that can tolerate slightly stale data, from a follower — which is the most common way teams scale reads without scaling writes.

Replication to the followers can be synchronous (the leader waits for at least one follower to confirm it has the write before acknowledging the client — no data loss on leader failure, but higher write latency) or asynchronous (the leader acknowledges immediately and ships the log in the background — low latency, but a leader crash can lose the last few writes that never made it to any follower). Most production systems run a hybrid: one synchronous follower for durability, and the rest? asynchronous for scale.

#Multi-leader replication

Multi-leader replication lets more than one node accept writes, typically one leader per data center, with each leader replicating its writes to the others. The appeal is obvious for multi-region systems: a write in Europe doesn't have to cross an ocean to reach a single leader in the US before it's acknowledged. The cost is equally obvious: two leaders can accept conflicting writes to the same row at nearly the same time, and now the system has to decide how to reconcile them, because there's no single log ordering everything.

Common conflict-resolution strategies include last-write-wins (simple, but silently discards one of the writes based on a timestamp — dangerous if both writes mattered), storing both versions and resolving in the application (what Dynamo-style "siblings" do), or using data structures that merge deterministically regardless of order (CRDTs — conflict-free replicated data types — are the modern answer for counters, sets, and similar structures).

#What can go wrong: replication lag

Because followers apply the log asynchronously, they are always slightly behind the leader — this is replication lag, and it creates real, user-visible bugs even when nothing has technically failed:

• Read-your-writes violations — a user submits a comment (written to the leader), the page reloads and reads from a lagging follower, and their own comment appears to have vanished.
• Monotonic read violations — a user refreshes twice, hitting two different followers with different lag, and sees a newer comment thread on the first refresh and an older one on the second.
• Causality violations — a user sees an answer to a question before the follower has replicated the question itself.

Teams work around this with techniques like routing a user's own reads to the leader for a short window after they write, or reading from "the replica that's at least caught up to timestamp T."

#What can go wrong: failover

When a leader dies, something has to promote a follower to be the new leader — this is failover, and it's where replication gets genuinely dangerous if done naively.

The failure modes to name in an interview:

1. Split brain — if the old leader is only network-partitioned (not actually dead), it may keep accepting writes from clients that still see it, at the same time a new leader has been promoted and is also accepting writes. Now you have two leaders and diverging data, with no log ordering between them. Fencing mechanisms (STONITH — "shoot the other node in the head", or a generation/epoch number that clients and followers reject stale leaders by) exist specifically to prevent this.
2. Lost writes on promotion — if the promoted follower wasn't fully caught up, the writes that only existed on the old leader are gone the moment a client starts writing to the new one.
3. Choosing the wrong follower — picking the follower with the most advanced replication position (not just "the first one that responds") is itself a small consensus problem, which is why serious systems use a proven algorithm (Raft, Paxos, or a coordination service like ZooKeeper/etcd) rather than an ad hoc script.