Distributed Systems for Backend Engineers · Data Distribution and Agreement
Partitioning and Sharding
Split data across nodes by key range or hash.
Partitioning (sharding) splits a dataset so each node holds a subset. Key-range partitioning keeps ranges together (good for range scans, risk of hot spots on sequential keys). Hash partitioning spreads load evenly (range queries hit all shards). Consistent hashing / fixed virtual partitions minimize data movement when nodes change. Secondary indexes are either local (per partition; scatter-gather queries) or global (partitioned by the indexed term; async updates).
Partitioning schemes
| Criterion | Key-range | Hash | Directory / lookup service |
|---|---|---|---|
| Range scans | Efficient | Scatter-gather | Depends |
| Load distribution | Hot spots on sequential keys | Even | Flexible |
| Rebalancing | Split/merge ranges | Consistent hashing / vnodes | Update directory |
| Examples | HBase, Spanner, CockroachDB ranges | Cassandra, DynamoDB | Vitess, custom shard maps |
Hot-spot example: key = timestamp (range partitioning) → all writes hit the latest range
Fix: prefix with a hash/bucket: key = hash(device_id) % 16 + ":" + timestamp
Tenant sharding: tenant_id → shard via lookup table (move big tenants to their own shard)
Common pitfalls
- Sequential keys on range-partitioned stores (write hot spots).
- Cross-shard joins and transactions everywhere.
- Choosing shard keys that don't match access patterns.
Key points
- Range vs hash trade-offs
- Rebalance with minimal movement
- Shard key = access pattern
Further reading: DynamoDB: Partition key design
Practice questions
Why do monotonically increasing keys create hot spots in range-partitioned stores?
Answer
All new writes fall into the last key range, so one partition handles all inserts.
What's the downside of hash partitioning?
Answer
Range queries can't target one partition and must scatter to all, and related keys are spread apart.
Open in the app to mark lessons complete, take notes and track your progress.