Observability and Operations · Operating Production Systems
Incident Response and Blameless Postmortems
Run incidents calmly and learn from them.
Effective incident response has clear roles (incident commander, communications lead, operations/subject experts), a shared channel and timeline, and a bias toward mitigation first (roll back, fail over, disable a feature flag) before root-cause analysis. Communicate status regularly to stakeholders and customers. Afterwards, write a blameless postmortem: timeline, impact, contributing factors, what went well, and action items with owners and deadlines.
Postmortem template
- Summary & customer impact (duration, % users, revenue/SLO impact)
- Timeline (detection → mitigation → resolution, with timestamps)
- Contributing factors (technical + process; no individual blame)
- What went well / what was lucky
- Action items: [owner] [due] prevent / detect faster / mitigate faster
Common pitfalls
- Debugging for an hour before trying the rollback.
- Blame-focused reviews that discourage transparency.
- Action items that never get done (track them).
Key points
- Roles: commander, comms, operations
- Mitigate first
- Blameless postmortems with tracked actions
Further reading: Google SRE: Managing incidents
Practice questions
Why is the incident commander usually not the person debugging?
Answer
Coordinating people, decisions and communication is a full-time job; separating it lets responders focus while someone keeps the big picture.
What makes a postmortem 'blameless'?
Answer
It assumes people acted reasonably with the information they had and focuses on systemic contributing factors and improvements rather than individual fault.
Open in the app to mark lessons complete, take notes and track your progress.