Resilience Patterns · How Distributed Systems Fail
Failure Modes of Distributed Systems
Crashes, slowness, partitions, gray failures and dependency failures.
Components fail in more ways than "down". Fail-stop crashes are the easy case. Slowness (a dependency at p99 = 5 s) is worse: it ties up threads, connections and memory upstream. Gray failures are partial (some requests fail, health checks pass). Network partitions split clusters. Dependency failures propagate through call graphs. Resilience engineering assumes all of these will happen and limits their blast radius.
Failure modes and defenses
| Failure | Symptom | Main defenses |
|---|---|---|
| Crash | Connection refused, 5xx | Redundancy, health checks, retries |
| Slowness | Timeouts, pool exhaustion | Timeouts, bulkheads, circuit breakers |
| Gray failure | Some requests fail; health OK | Outlier detection, deep health signals |
| Partition | Nodes can't talk | Quorum design, fencing, retries |
| Overload | Rising latency then collapse | Load shedding, backpressure, autoscaling |
| Bad deploy/config | Errors after change | Canaries, fast rollback, flags |
Common pitfalls
- Designing only for crashes.
- Health checks that test liveness but not actual functionality.
- Ignoring config changes as a failure source.
Key points
- Slowness is worse than crashes
- Gray failures fool health checks
- Limit blast radius
Further reading: Google SRE: Addressing cascading failures
Practice questions
Why is a slow dependency often more dangerous than a crashed one?
Answer
Crashes fail fast; slow responses hold threads, connections and memory in callers, spreading resource exhaustion upstream.
What is a gray failure?
Answer
A partial failure where some operations fail or degrade while health checks still pass, so automated failover doesn't trigger.
Open in the app to mark lessons complete, take notes and track your progress.