Resilience Patterns · How Distributed Systems Fail
SLO-Driven Resilience
Decide how much resilience you need from user-facing objectives.
Resilience has costs — redundancy, complexity, latency. Use SLOs to decide where to invest: critical user journeys (checkout, login) get redundancy, fallbacks and strict budgets; internal reports may tolerate hours of downtime. Map dependencies as critical (failure breaks the journey) or optional (degrade gracefully), and remember the composed availability of serial dependencies multiplies: 5 dependencies at 99.9% give at most ~99.5%.
Matching resilience to objectives
| Journey | SLO | Resilience investment |
|---|---|---|
| Checkout | 99.95% success, p99 < 1 s | Multi-AZ, fallbacks, breakers, idempotency, canaries |
| Product page | 99.9%, p99 < 500 ms | Caching, CDN, degrade recommendations |
| Admin reports | 99%, minutes OK | Simple retries, async generation |
| Analytics ingest | No data loss, delays OK | Durable queues, replay |
Composed availability of serial dependencies:
0.999 ^ 5 ≈ 0.995 (≈ 3.6 hours of downtime per month)
Optional dependencies with fallbacks don't count against the critical path.
Common pitfalls
- Same resilience effort for everything.
- Ignoring the availability math of call chains.
- Treating every dependency as critical.
Key points
- SLOs prioritize investment
- Critical vs optional dependencies
- Serial availability multiplies
Further reading: Google SRE Workbook: Implementing SLOs
Practice questions
Why does a service calling five 99.9%-available dependencies serially struggle to reach 99.9% itself?
Answer
Availabilities multiply for serial dependencies: 0.999^5 ≈ 99.5%, so it needs fallbacks, caching or redundancy to do better than its dependencies.
How does classifying dependencies as optional improve resilience?
Answer
Optional dependencies can fail without failing the request (fallback or omission), removing them from the critical path's availability math.
Open in the app to mark lessons complete, take notes and track your progress.