Resilience Patterns · Verifying Resilience
Resilience Reviews and Production Readiness
A checklist to apply to every service before launch.
Bake resilience into the delivery process with a production readiness review: timeouts and retries on every dependency, idempotency for unsafe operations, breakers and bulkheads where needed, graceful shutdown, health checks, autoscaling and load-shedding behavior, dashboards and SLO alerts, runbooks, capacity tests, backup/restore drills, and a rollback plan. Revisit after incidents and major changes.
Production readiness checklist (excerpt)
[ ] SLOs defined; burn-rate alerts; dashboards
[ ] Every outbound call: timeout, retry policy (budget), breaker/bulkhead if critical
[ ] Unsafe operations idempotent (keys / natural idempotency)
[ ] Load shedding / concurrency limits; tested in stress test
[ ] Graceful shutdown; readiness/liveness probes
[ ] Multi-AZ; failover tested; backups restored in last 90 days
[ ] Feature flags / kill switches for risky features
[ ] Runbooks for top alerts; on-call trained
[ ] Dependency register with owners and fallbacks
Common pitfalls
- Checklists filled out without evidence.
- Reviews only at launch, never after.
- Backups never test-restored.
Key points
- Readiness review before launch
- Evidence, not checkboxes
- Revisit after incidents
Further reading: Google SRE: Production readiness reviews
Practice questions
Why require evidence (e.g., a stress test report) in readiness reviews?
Answer
Self-reported checkboxes are often optimistic; evidence shows the mechanisms actually work under realistic conditions.
What's the point of regularly restoring backups?
Answer
Backups that have never been restored may be incomplete, corrupt or too slow to restore; drills verify RPO/RTO.
Open in the app to mark lessons complete, take notes and track your progress.