Resilience Patterns · How Distributed Systems Fail
Cascading and Metastable Failures
How overload spreads and why systems stay down after the trigger ends.
A cascading failure spreads as one overloaded component makes its callers slow, which makes their callers slow. A metastable failure is a state where the system stays overloaded even after the trigger is removed, sustained by feedback loops: retries, cache misses after a cold start, queue backlogs, or clients reconnecting en masse. Recovery often requires shedding load or blocking retries. Prevention: bounded retries, load shedding, warming caches, and capacity headroom.
Metastability feedback loops
| Feedback loop | How it sustains overload | Break it with |
|---|---|---|
| Retries | Failed requests multiply | Retry budgets, backoff, breakers |
| Cold caches | Misses hit the database | Cache warming, request coalescing, stale-if-error |
| Queue backlog | Old work processed after clients gave up | TTLs/deadlines on messages, LIFO under overload |
| Reconnect storms | All clients reconnect at once | Jittered reconnects, connection limits |
Common pitfalls
- Restarting everything at once (cold caches, reconnect storms).
- Processing stale queued work nobody wants.
- Assuming removing the trigger ends the incident.
Key points
- Overload spreads through callers
- Feedback loops cause metastability
- Shed load to recover
Further reading: USENIX: Metastable failures in the wild
Practice questions
What makes a failure 'metastable'?
Answer
The system remains in a bad, overloaded state after the initial trigger is gone because feedback loops (retries, cache misses, backlogs) keep load above capacity.
Why can processing queued requests in FIFO order worsen overload?
Answer
The oldest requests' clients have probably timed out; serving them wastes capacity while new requests also time out.
Open in the app to mark lessons complete, take notes and track your progress.