Distributed Systems for Backend Engineers · Coordination
Distributed Locks and Fencing Tokens
Mutual exclusion that survives pauses and partitions.
Distributed locks (Redis SET NX PX, ZooKeeper/etcd leases, database advisory locks) coordinate exclusive work. The danger: a lock holder pauses (GC, VM migration) past its lease, another process acquires the lock, and both act. Fencing tokens fix this: each lock grant carries a monotonically increasing token, and the protected resource rejects requests with an older token. Use consensus-backed locks (etcd/ZooKeeper) for correctness; Redis locks are fine for efficiency (avoiding duplicate work), not correctness.
Lock options
| Lock type | Correctness under pauses/partitions | Fencing token | Use for |
|---|---|---|---|
| Redis SET NX PX (single node) | Weak | Not built in | Efficiency (dedupe work) |
| Redlock | Debated | No | Efficiency |
| etcd / ZooKeeper lease | Strong (with fencing) | Revision / zxid | Correctness-critical |
| PostgreSQL advisory / row lock | Strong within the DB | Use DB transactions | Work tied to that DB |
Fencing in practice
client A acquires lock → token 33 → long GC pause → lease expires
client B acquires lock → token 34 → writes to storage with token 34
client A wakes → writes with token 33 → storage rejects (33 < 34) ✓
Storage check: UPDATE files SET data=$1, fence=$2 WHERE id=$3 AND fence < $2
Common pitfalls
- Assuming lock expiry means the old holder stopped working.
- Redis locks guarding money movements.
- Locks without TTLs (deadlock on crash).
Key points
- Leases expire while holders still run
- Fencing tokens enforce safety at the resource
- Consensus-backed locks for correctness
Further reading: Kleppmann: How to do distributed locking
Practice questions
Why can two processes believe they hold the same distributed lock?
Answer
A holder can pause (GC, VM stall) past its lease expiry while another process acquires the lock; when the first resumes, it still thinks it holds it.
What does a fencing token do?
Answer
It's a monotonically increasing number issued with each lock grant; the protected resource rejects operations carrying a token older than one it has already seen.
Open in the app to mark lessons complete, take notes and track your progress.