The Tech Lead Handbook · Leading Across the Team and Beyond
Leading Incidents and Blameless Postmortems
Run incidents with clear roles instead of everybody debugging at once, and turn them into learning with blameless postmortems.
Scenario
Example team: at 18:40 payments start failing for 20% of customers. Eight engineers join a call; three restart services, two read logs, the tech lead answers questions from support, sales and the CTO in parallel. A restart briefly makes it worse. It takes 90 minutes to find a misconfigured certificate.
What you see on the team
- Everyone debugging, nobody coordinating.
- Conflicting changes made at the same time.
- Stakeholders asking for updates in five channels.
- A postmortem that names the engineer who changed the certificate; nobody volunteers information next time.
The common mistake
Treating an incident as a debugging session rather than an operation with roles, and the postmortem as a search for who caused it.
Why it happens
Engineers are trained to fix things, so everyone jumps in. Without explicit roles, coordination and communication fall on the most senior person, who also tries to debug. After the incident, blame feels like accountability. Google's SRE book chapter on managing incidents describes separating roles — incident command, operations, communication, planning — so each person has one job; the postmortem culture chapter explains why postmortems must be blameless to surface the real causes.
The playbook
Roles, declared in the first minutes:
| Role | Does | Doesn't |
|---|---|---|
| Incident commander (IC) | Coordinates, decides, keeps a timeline, assigns work | Debug or make changes |
| Operations lead | Investigates and changes systems, one change at a time | Talk to stakeholders |
| Communications lead | Status updates on a fixed cadence, one channel | Debug |
| Scribe (optional) | Records actions and times | — |
Rules during the incident:
- Mitigate first (roll back, fail over, disable a feature), root-cause later.
- One change at a time, announced before it's made.
- Updates every 30 minutes even if nothing changed.
Blameless postmortem within a few days:
- Timeline, impact, contributing factors (plural), what went well, action items with owners and dates.
- Ask "how did the system make this easy to get wrong?", not "who did it?".
PagerDuty publishes its incident response process openly; it's a good template to adapt.
What to say
Opening an incident call:
"I'm incident commander. Ana is operations lead — all changes go through her. Lee handles communications; updates in #incident-payments every 30 minutes. Everyone else: tell me what you're investigating before you start. First goal: stop the bleeding. Ana, can we roll back the 18:15 deploy?"
In the postmortem:
"We're not looking for who changed the certificate. We're looking for why a certificate change could reach production without a check, and why it took 60 minutes to suspect it."
Trade-offs
- Formal roles feel heavy for small incidents; define severity levels and use the full structure only above a threshold.
- Mitigating before root-causing sometimes destroys evidence; snapshot logs and state before rolling back when it's cheap.
- Blameless doesn't mean no accountability: people are accountable for learning and for action items, not punished for honest mistakes.
Signals it's working
- Roles are named within five minutes of declaring an incident.
- Time to mitigate falls; fewer "fixes" make things worse.
- Postmortems list several contributing factors and action items that get done.
- Engineers volunteer their own mistakes early.
Key points
- Run incidents with roles: commander, operations, communications; the commander doesn't debug.
- Mitigate first, one change at a time, regular updates in one channel.
- Postmortems are blameless: look for system causes and contributing factors, not culprits.
- Track postmortem actions to completion.
Practice questions
Eight engineers join an incident call and all start debugging. As tech lead, what do you do first?
Answer
Declare roles: name an incident commander (who doesn't debug), an operations lead through whom all changes go, and a communications lead with a fixed update cadence; then focus on mitigation such as rolling back the last deploy.
Why should postmortems be blameless?
Answer
Blame makes people hide information, so real causes stay unknown. Blameless postmortems ask how the system allowed the mistake, which surfaces contributing factors and produces fixes that prevent recurrence.
What does a good postmortem contain?
Answer
Impact and timeline, multiple contributing factors, what went well, and action items with owners and due dates — written without blame and reviewed until the actions are done.
Open in the app to mark lessons complete, take notes and track your progress.