Measuring Performance: Percentiles, Profiling and Load Tests

Define performance goals as percentiles under load ("p99 < 300 ms at 2,000 RPS"), not averages. Measure in production with RED metrics and real-user monitoring, find hotspots with profilers (CPU, allocation, async) and traces, and verify capacity with load tests that model real traffic (k6, Gatling, Locust). Change one thing at a time and re-measure.

// k6 load test: ramp to 2,000 RPS, fail if p99 >= 300 ms or errors >= 1%
import http from "k6/http";
export const options = {
  scenarios: { ramp: { executor: "ramping-arrival-rate", startRate: 100, timeUnit: "1s",
    preAllocatedVUs: 500, stages: [{ target: 2000, duration: "10m" }, { target: 2000, duration: "20m" }] } },
  thresholds: { http_req_duration: ["p(99)<300"], http_req_failed: ["rate<0.01"] },
};
export default () => http.get("https://staging.example.com/api/products?limit=20");

Common pitfalls

• Optimizing by intuition.
• Load tests with unrealistic data or closed-model generators (coordinated omission).
• Reporting averages that hide tail latency.

Key points

• Goals as percentiles at a load
• Profile and trace to find bottlenecks
• Load test with realistic arrival rates

Further reading: Grafana k6 documentation