Why use percentiles instead of averages for latency, and how should percentiles be aggregated?

Summary

Averages hide the tail: a mean of 120 ms can coexist with 5% of users waiting over 2 seconds. Percentiles (p50, p95, p99) describe what users actually experience, and the tail matters even more when one page load fans out to many backend calls. Percentiles cannot be averaged across hosts; aggregate histograms and compute percentiles from them.

Longer Answer

If a page makes 20 backend calls, the chance that at least one hits that backend's p99 is about 18%, so backend p99 becomes typical user experience. Prometheus histograms (or native/exponential histograms) store bucket counts that can be summed across instances, after which histogram_quantile estimates percentiles. Averaging per-host p99 values gives a meaningless number.

For SLOs, threshold-based SLIs ("proportion of requests under 300 ms") are often clearer than percentile targets and are easy to aggregate.

Key Points

• Averages hide the tail.
• Fan-out makes tails common.
• Aggregate histograms, never percentiles.
• Threshold SLIs aggregate cleanly.