Part II — Distributed Data
Replication
Keeping copies of the same data on multiple machines — and living with the consequences when they briefly disagree.
Replication means keeping a copy of the same data on multiple machines connected via a network. You do it for three reasons: to keep data geographically close to users (lower latency), to keep the system running even if some machines fail (availability), and to scale out read throughput (more machines can serve more reads).
The dominant approach is leader-based replication: writes go to one designated leader, which propagates them to followers.
The leader acknowledges the client as soon as it's written locally — writes are fast, but a crashed leader can lose the most recent writes, and reads from Follower B can return stale data during the lag window.
The central tradeoff: synchronous vs. asynchronous
Synchronous replication guarantees a follower has an up-to-date copy before the client’s write is acknowledged — strong durability, at the cost of write latency (and total unavailability if that follower is unreachable). Asynchronous replication never blocks on followers — writes are fast, but the leader can acknowledge a write that a crash then loses forever, and followers can serve stale reads while catching up.
Most real systems use semi-synchronous replication in practice: one follower is synchronous (guaranteeing at least one up-to-date copy survives a leader crash) and the rest are asynchronous (so a slow follower doesn’t stall every write).
Beyond single-leader
- Multi-leader replication lets writes happen on more than one node (e.g. one leader per data center), which removes the single point of write contention but introduces write conflicts that have to be resolved when two leaders accept conflicting writes to the same record.
- Leaderless replication (Dynamo-style — Cassandra, Riak) has clients write directly to multiple replicas and read from multiple replicas, using quorums to reason about consistency instead of a single leader. Covered in depth under Consistency & Consensus.
# A common mitigation for stale reads on a lagging follower:
# force reads that immediately follow a write back to the leader.
def read_after_write(user_id, key):
if just_wrote(user_id, key):
return read_from_leader(key)
return read_from_any_replica(key)