Distributed Systems for Backend Engineers · Capstone
Capstone: Diagnose a Consistency Bug
Find the cause of duplicate processing and lost updates.
Scenario
A nightly billing job runs on two pods using a Redis lock (SET billing NX PX 60000). Occasionally customers are billed twice. Separately, customers report profile edits 'reverting' — the profile service uses multi-region active-active writes with last-write-wins on updated_at from each region's app servers.
Requirements
- Explain both bugs
- Propose correct designs
Constraints
- Keep multi-region profile writes
Deliverables
- Explain the double billing and fix it.
- Explain the reverting profiles and fix them.
Draft each deliverable below before revealing the model answer and rubric.
Further reading: Kleppmann: Distributed locking
Deliverables
Explain the double billing and fix it.
Model answer
The job runs longer than the 60 s lease (or pauses in GC); the lock expires, the second pod acquires it and both bill. Fix: make billing idempotent per customer+period (unique constraint on invoices), use a consensus-backed lease with fencing tokens (or Kubernetes leader election / DB advisory lock), renew the lease while working, and check leadership before each batch.
Rubric
Strong answers fix it at the resource (idempotency/fencing), not just by lengthening the TTL.
Explain the reverting profiles and fix them.
Model answer
LWW with wall-clock timestamps from different regions: clock skew lets an older write carry a newer timestamp and win, discarding the real latest edit. Fix: hybrid logical clocks or per-field version vectors; field-level merges (CRDT registers per field); or home each profile to one region for writes (single-leader per user) with fast local reads elsewhere.
Rubric
Look for recognition of clock skew + LWW data loss and a principled conflict strategy.
Open in the app to mark lessons complete, take notes and track your progress.