latency

Measuring Real LLM Latency: p50, p95, and p99

An average latency number hides the requests that ruin your UX. Here is how to measure LLM latency with percentiles, why p95/p99 matter more than the mean, and how to read tail latency on a gateway.