What is a cascading failure, and how do you prevent one?

Summary

A failure that grows by positive feedback: one component fails or slows, its load shifts to others or callers retry, those become overloaded and fail too, until the whole system is down. Typical triggers are overload, retry storms, resource exhaustion and slow dependencies tying up threads.

Longer Answer

Prevention:

• Limit load: load shedding, rate limits, bounded queues.
• Retry discipline: capped exponential backoff with jitter, retry budgets, no retries at every layer.
• Timeouts and deadlines propagated through the call chain so work for abandoned requests stops.
• Bulkheads: separate pools per dependency so one slow dependency cannot consume all threads.
• Circuit breakers to stop calling a failing dependency.
• Capacity headroom so losing one instance or zone does not overload the rest.

Recovery may require stopping traffic entirely, then ramping back up gradually, because a cold system cannot absorb the full load at once.

Key Points

• Positive-feedback failure spread.
• Overload, retries, resource exhaustion.
• Shedding, backoff, deadlines, bulkheads, breakers.
• Recover with gradual ramp-up.

Deep dive: Resilience Patterns