What are the common distributed-system failure modes — network partitions, split-brain, clock skew — and how do you defend against them?

#Distributed systems fail in ways single machines don't

A single process either runs or it doesn't; you rarely have to reason about it being "half-alive." A distributed system introduces an entirely new failure surface because the network itself is unreliable, and because there is no shared clock — two machines can each behave perfectly correctly in isolation and still produce a badly wrong result together. The failure modes worth knowing cold for a senior interview are network partitions, split-brain (a specific and dangerous consequence of a partition), and clock skew, along with the defenses that make each one survivable rather than catastrophic.

#Network partitions

A partition is any situation where a subset of nodes can't communicate with another subset, even though every individual node is otherwise healthy. This is not exotic — it happens from a flaky top-of-rack switch, a misconfigured security group, a cross-region link saturating, or a routine deploy that briefly drops connections. The dangerous part isn't the partition itself; it's that each side of the partition typically cannot tell the difference between "the other node is down" and "the other node is fine but unreachable from here" — and the correct response to those two situations is completely different (failing over is right in the first case, and actively harmful in the second).

#Split-brain: two leaders, one truth

Split-brain is what happens when a partition causes two disjoint subsets of a cluster to each independently conclude they should have a leader, resulting in two nodes simultaneously believing they are the leader and accepting writes. Both sides continue operating "correctly" from their own point of view — that's what makes it so dangerous, because there's no error to detect locally. The damage is discovered later, when the partition heals and you find two divergent histories of writes that both claim to be authoritative and cannot simply be concatenated.

In the diagram, Node 1 is cut off but keeps accepting writes from Client X because nothing locally tells it to stop — it can't reach the other nodes to find out it's been outvoted. Meanwhile Nodes 2 and 3 still form a majority, correctly elect a new leader, and start accepting writes from Client Y. Two authoritative-looking histories now exist for the same data, and reconciling them after the network heals may require discarding one side's writes entirely.

#Defending against split-brain: quorums and fencing

The standard defense is majority quorum: a node may only act as leader if it can confirm it's talking to a strict majority of the cluster (⌊n/2⌋ + 1). With an odd number of nodes, a partition can produce at most one side with a majority — so in the 3-node example, the minority side (1 node) mathematically cannot also claim a majority, and a correctly implemented node should step down rather than keep serving writes once it detects it has lost quorum. This is exactly what consensus protocols like Raft and Paxos formalize: leadership is a lease backed by majority agreement, not a permanent title.

A second, complementary defense is fencing tokens: every time leadership changes, the new leader gets a monotonically increasing token, and any downstream resource (a shared storage system, a database) is required to reject writes carrying an older token than the highest one it has already seen — even if the old leader is still alive and still convinced it's in charge, its writes get physically rejected at the point of actual effect. This closes the gap that quorum alone doesn't: quorum stops a minority node from thinking it's a valid leader, but a node can lag on noticing that (a GC pause right after losing an election, for instance), and fencing tokens catch that residual case at the storage layer.

#Clock skew: when "later" lies to you

Every machine's clock drifts from true time, NTP synchronization is only approximately accurate (and can jump backward when it corrects), and virtualized/containerized environments can add their own timing anomalies. This matters enormously any time correctness depends on comparing timestamps from different machines — for example, a last-write-wins conflict resolution strategy that picks "whichever write has the later timestamp" can silently pick the earlier write in wall-clock terms if the machine that produced the "later" timestamp actually has a fast clock.

The standard defenses don't try to make clocks perfect; they route around needing them to be. Logical clocks (Lamport timestamps) assign an ordering based on causality — "this event happened after that event because a message carrying that event's timestamp arrived first" — rather than wall-clock time, giving a correct partial order without any clock synchronization at all. Vector clocks extend this to detect concurrent (not just ordered) events, which is what lets a system like Dynamo say "these two writes genuinely happened concurrently, hand them to the application to merge" instead of guessing. At the far end, Google's Spanner uses TrueTime, an API that returns not a single timestamp but a confidence interval [earliest, latest] backed by atomic clocks and GPS receivers in every datacenter, and simply waits out the uncertainty window before committing a transaction — turning "clocks are imprecise" into a bounded, known quantity instead of an unknown one.