The Trouble with Distributed Systems

Why distributed systems are fundamentally unreliable — clocks drift, networks lie, and processes pause for longer than you think.

Everything else in distributed data systems — replication, partitioning, consensus — is really a set of strategies for coping with three uncomfortable facts about the physical world a distributed system runs on. None of them are edge cases; they’re the normal operating conditions of any real network.

The network is unreliable

A request can be lost, delayed arbitrarily, duplicated, or reordered — and the sender generally has no way to distinguish “the request was lost” from “the response was lost” from “the remote node is just slow.” This ambiguity is why timeouts are the only practical detection mechanism, and why they’re such a blunt instrument: too short, and you falsely declare healthy nodes dead; too long, and failure detection is unacceptably slow.

Clocks can’t be trusted

Machines have two kinds of clocks, and conflating them is a classic source of bugs:

Even with NTP, clocks on different machines drift and can be off by tens or hundreds of milliseconds — enough that “which write happened last?” by comparing timestamps across machines is not a safe operation. This is the actual root cause of many “last write wins” bugs in distributed databases: it’s not really about time, it’s about the clock being an unreliable proxy for ordering.

# Dangerous: comparing wall-clock timestamps from different nodes
# to decide which write "happened first" — clock skew makes this
# silently wrong, sometimes by hundreds of milliseconds.
if event_a.timestamp > event_b.timestamp:
    winner = event_a  # not actually guaranteed to be the later event

Processes pause for longer than you’d expect

A process (and the thread you’re relying on to hold a lock, or to keep leadership) can be paused for an unbounded amount of time — a garbage collection pause, the hypervisor live-migrating the VM, the OS swapping to disk, a laptop’s lid being closed mid-request. From every other node’s point of view, a long GC pause is indistinguishable from a crash.

Fig. 8 — A paused leader, with and without fencing tokens

The old leader was paused long enough for the cluster to correctly elect a new one — but when it resumes, it still believes it's the leader and writes anyway. Storage has no way to tell the two writes apart, and accepts both.

Why this all matters

Every algorithm in the Consistency & Consensus topic exists specifically to let a system make correct decisions despite these three facts, without ever assuming synchronized clocks, reliable delivery, or bounded pause times. Systems that quietly assume otherwise work fine in testing and then fail unpredictably in production under real network and GC conditions.