Observability and Operations · Operating Production Systems
Dashboards and Runbooks
Build dashboards people actually use and runbooks that shorten incidents.
Good dashboards tell a story top-down: SLOs and user-facing symptoms first, then service RED metrics, then dependencies and resources. Annotate deployments. Every page-worthy alert should link to a runbook: what the alert means, how to verify impact, common causes, mitigation steps (with commands), and escalation contacts. Review runbooks after incidents and automate repeated steps.
# Runbook: CheckoutHighErrorRate
Meaning: >2% of checkout requests return 5xx for 5 minutes (SLO burn).
Verify: Dashboard "Checkout / Overview" → error ratio by route & region.
Common causes:
1. Payment provider outage → check status page; enable fallback: `flags set payments.fallback=on`
2. Bad deploy (deploy marker within 30 min) → `deploy rollback checkout`
3. DB connection exhaustion → check pool saturation panel; scale read replicas
Escalate: #payments-oncall, then Payments EM.
Common pitfalls
- Dashboards with no clear starting point.
- Alerts without runbooks.
- Runbooks that are out of date after architecture changes.
Key points
- Symptoms first, causes second
- Every alert links to a runbook
- Automate repetitive runbook steps
Further reading: Google SRE Workbook: On-call
Practice questions
What belongs at the top of a service dashboard?
Answer
User-facing health: SLO status and error budget, request rate, error ratio and latency percentiles, plus deploy annotations.
How do runbooks reduce incident duration?
Answer
They give responders verified steps and context immediately, avoiding rediscovery under pressure, and enable less experienced on-call engineers to mitigate safely.
Open in the app to mark lessons complete, take notes and track your progress.