Observability and Operations · Operating Production Systems
Capacity Planning and Performance Monitoring
Spot saturation before it becomes an outage.
Track saturation signals (CPU throttling, memory, connection pool waits, queue depth and age, disk, thread pools), forecast growth from trends and planned events, and load test to know each component's limits. Alert on leading indicators (pool wait time, queue age) rather than only on failures, and keep headroom for failover (N+1).
Leading indicators to alert on (warning, not paging):
- DB connection pool: wait time p95 > 50 ms for 10 min
- Queue: oldest message age > 5 min
- Disk: projected full within 7 days (predict_linear)
- Kubernetes: CPU throttling > 25% for a deployment
predict_linear(node_filesystem_avail_bytes[6h], 7 * 24 * 3600) < 0
Common pitfalls
- Capacity alerts only when things already fail.
- Ignoring seasonal peaks and launches.
- Load tests that don't resemble production traffic.
Key points
- Monitor saturation and leading indicators
- Forecast and load test
- Keep failover headroom
Further reading: Google SRE: Handling overload
Practice questions
Why is queue age often a better alert than queue length?
Answer
Age directly reflects user-visible delay regardless of message rate; a long queue may be fine if it drains fast, while old messages mean work is stuck.
What does N+1 capacity mean?
Answer
Having enough capacity to handle peak load even if one instance/zone fails.
Open in the app to mark lessons complete, take notes and track your progress.