Resilience Patterns · Capstone
Capstone: Recover From a Metastable Failure
Diagnose and fix a system that stays down after the trigger is gone.
Scenario
A cache cluster was restarted during a deploy. Cache hit rate dropped from 95% to 0%, database load jumped 20×, latency rose, clients timed out and retried, and the cache never warmed because database queries timed out before populating it. The system stayed degraded for 2 hours until traffic was manually blocked.
Requirements
- Explain the feedback loop
- Short-term recovery runbook
- Long-term prevention
Constraints
- Cannot add database capacity quickly
Deliverables
- Explain why the system didn't recover on its own.
- Write the recovery runbook and long-term fixes.
Draft each deliverable below before revealing the model answer and rubric.
Further reading: USENIX: Metastable failures in the wild
Deliverables
Explain why the system didn't recover on its own.
Model answer
Feedback loop: cold cache → DB overload → slow queries time out → cache not populated → still cold; client retries multiply load; timeouts kept connections busy. Load stayed above DB capacity, so the system remained in the overloaded state (metastable) even though nothing was 'broken'.
Rubric
Look for the self-sustaining loop explanation (cold cache + retries + timeouts).
Write the recovery runbook and long-term fixes.
Model answer
Runbook: shed load at the edge (rate limit or block non-critical traffic), disable client retries via flag, let caches warm (prioritize hot keys, warm from snapshot), then ramp traffic gradually. Long-term: rolling cache restarts (never all nodes), cache warming before cutover, request coalescing/single-flight, stale-if-error with persistent cache, retry budgets, DB concurrency limits/load shedding, alerts on hit-rate drops, game day for cache loss.
Rubric
Strong answers include gradual ramp-up and structural fixes that remove the feedback loop.
Open in the app to mark lessons complete, take notes and track your progress.