Site Reliability Engineering interview question · Most asked
How should incident response be organised? Describe the key roles.
Summary
Use a lightweight incident command structure: an incident commander who coordinates and makes decisions, an operations lead who works on mitigation, a communications lead who updates stakeholders and status pages, and a scribe who keeps the timeline. In small incidents one person may hold several roles, but the commander role is always explicit.
Longer Answer
Principles from incident command systems:
- Declare early: a declared incident that turns out minor is cheap; an undeclared one that grows is expensive.
- Mitigate first, diagnose later: roll back, fail over, shed load or disable a feature to stop user impact; root cause comes afterwards.
- One coordination channel, a written live state document, regular updates at a stated cadence, and explicit hand-offs when people change.
- The commander does not debug; they keep the big picture, assign work and decide.
Afterwards: a blameless postmortem with tracked action items.
Key Points
- Commander, ops, comms, scribe.
- Declare early.
- Mitigate before root-causing.
- Single channel, regular updates, explicit hand-offs.
Deep dive: The Tech Lead Handbook
Explanation
What interviewers look for
Emphasis on mitigation first and on clear roles and communication.
Follow-up questions
- When would you roll back without knowing the cause?
More most-asked Site Reliability Engineering questions
- What is Site Reliability Engineering, and how does it relate to DevOps?
- What makes a good alert? Explain symptom-based alerting and alert fatigue.
- What is an error budget, and how should it be used?
- Define SLI, SLO and SLA, and explain how they relate.
- What is burn-rate alerting, and why is it better than alerting on raw error rates?
- How do you choose good SLIs for a service?
- What are the four golden signals?
- What is the difference between monitoring and observability? Explain metrics, logs and traces.