Resilience Patterns · Verifying Resilience
Game Days and Incident Drills
Practice failures as a team before they happen for real.
A game day simulates an incident (region failover, database primary loss, dependency outage, key compromise) with the real team, tools and runbooks. Goals: verify technical resilience and human response — detection, paging, communication, decision-making, runbook accuracy. Schedule them regularly, rotate scenarios and participants, and track action items like postmortems.
Game day scenarios
| Scenario | Validates |
|---|---|
| Database primary failover | Replication, connection retries, app reconnect logic |
| Region evacuation | Multi-region readiness, DNS, capacity |
| Third-party outage (payments) | Fallbacks, status communication |
| Expired TLS certificate | Monitoring, renewal automation |
| Leaked credential | Rotation runbook speed |
Game day agenda (2 hours)
0:00 brief: scenario, roles (commander, scribe, comms), abort criteria
0:10 inject failure; responders detect via alerts only
0:40 mitigation; measure time to detect / mitigate
1:10 restore; verify steady state
1:20 debrief: what worked, runbook gaps, action items with owners
Common pitfalls
- Pre-announcing exactly what will break (tests nothing).
- Only senior engineers participating.
- No follow-through on findings.
Key points
- Practice technical and human response
- Measure detection and mitigation time
- Track action items
Further reading: AWS Well-Architected: Game days
Practice questions
What do game days test that automated chaos experiments don't?
Answer
Human and organizational response: alerting, paging, coordination, communication and the accuracy of runbooks.
Which metrics should you record during a game day?
Answer
Time to detect, time to engage, time to mitigate/restore, customer impact, and runbook steps that failed or were missing.
Open in the app to mark lessons complete, take notes and track your progress.