Partitioning

Replication copies the same data to multiple nodes. Partitioning (sharding) does the opposite: it splits a large dataset into smaller pieces (partitions) spread across many nodes, so both the data volume and the query load can scale horizontally. The two are almost always combined — each partition is itself replicated.

The whole design problem is: given a key, which partition does it live on? Get this wrong and some partitions become overloaded “hot” partitions while others sit idle.

Key range partitioning

Each partition owns a contiguous range of keys (like the volumes of a paper encyclopedia). This is great for range queries — “give me all events between 9am and 10am” touches one or two partitions — but it’s exactly what creates the hotspot in the diagram above: if keys are assigned in some naturally increasing order (timestamps, auto-incrementing IDs), every new write goes to the same “newest” partition.

Hash partitioning

Run the key through a hash function first, then assign by hash value. This scatters keys evenly and eliminates hotspots from sequential keys — the tradeoff is that range queries no longer map to a contiguous set of partitions, since consecutive keys are now scattered essentially at random.

def partition_for_key(key: str, partition_count: int) -> int:
    return hash(key) % partition_count

Request routing

Once data is scattered across nodes, a request for a given key has to reach the right one. Common approaches: a routing tier that tracks partition ownership (like a ZooKeeper-backed metadata service), gossip protocols where any node can forward a request to the right owner, or a smart client that caches the partition map itself.