SLOs, Error Budgets and Alerting — Observability and Operations

An SLI measures user experience (e.g., proportion of checkout requests that succeed in under 500 ms). An SLO sets a target over a window (99.9% over 30 days). The error budget (0.1%) is the allowed unreliability — spend it on releases and experiments; when it's exhausted, prioritize reliability. Alert on burn rate (how fast the budget is being consumed) with multi-window rules, so pages are urgent and actionable while slow burns become tickets.

SLO: 99.9% of checkout requests succeed (non-5xx) over 30 days → budget = 0.1%

Page if:   burn rate > 14.4 over 1h  AND  > 14.4 over 5m     (2% of budget in 1 hour)
Page if:   burn rate > 6   over 6h   AND  > 6   over 30m     (5% of budget in 6 hours)
Ticket if: burn rate > 1   over 3d                            (budget would run out)

burn rate = (observed error ratio) / (1 - SLO)

Common pitfalls

SLOs on internal metrics users don't feel (CPU).
100% targets (no room to change anything).
Alerting on every error spike instead of budget burn.

Key points

SLI → SLO → error budget
Burn-rate alerts, multi-window
Budget policy guides release decisions

Further reading: Google SRE Workbook: Alerting on SLOs

Practice questions

Why alert on burn rate instead of a raw error-rate threshold?

Answer

Burn rate ties alerts to user impact relative to the SLO, catching both fast severe incidents and slow chronic degradation while ignoring harmless blips.

What should happen when the error budget is exhausted?

Answer

Per the agreed policy: pause risky launches, focus engineering on reliability work, and resume feature velocity once the budget recovers.