Data & Decisions
Average Response Time Hides Slow Requests: Choose Percentiles Carefully
Two executed synthetic comparisons show equal means with different tails and reject averaged worker percentiles. A completed review separates missing timeout durations, request counts and histogram estimates from observed quantiles.
An average response time can stay unchanged while some requests become much slower. Read the distribution alongside the mean: define the request population and measurement window, show counts, and choose percentiles that answer a specific tail-latency question. These are request-level statistics; without a user-to-request mapping, they do not state what fraction of users had a slow interaction. Calculate an overall percentile from the combined observations or a suitable aggregated distribution. Averaging the percentiles of servers or time windows does not recover the percentile of all requests.
This LandOffer.ai guide contains two completed synthetic comparisons and a measurement-review decision. The arithmetic was actually executed locally; the service, durations and review are invented. No production telemetry, monitoring account, load test or real user experience was measured. The example shows what its recorded numbers support, with unknown timeouts and aggregation limits kept visible.

Define the response whose duration you are reporting
Start with the event boundary. Server handling time, gateway time and a user's complete interaction can have different starts and finishes. A server measurement might omit connection setup, retries, network travel or client rendering. Name the endpoint, region, response class, measurement location and time interval before comparing two values.
Google's SRE monitoring chapter recommends examining the latency distribution and distinguishes successful-request latency from error latency. A quickly returned error can make an overall duration look better without making the service useful. Keep errors visible rather than simply discarding them and labeling the remaining distribution the experience of every attempt.
For the fictional Laurel preview service, analyst Ren defines a five-minute slice, 10:00–10:05 UTC, for one endpoint in one region. The duration starts when the server accepts work and ends when it produces its successful response. All durations below are milliseconds. These are authored observations, not a claim that an actual service behaves at these particular speeds.
Ren keeps exactly 1,000 completed successful requests in each comparison dataset. Timeouts stay outside these percentile arrays because their exact completion durations are unavailable. A separate attempt-level calculation below includes the known timeout count; the two denominators are never interchangeable.
Completed comparison: the same mean, different tails
The first dataset contains 1,000 durations of 120 milliseconds. The second contains 960 durations of 50 milliseconds, 30 of 500 milliseconds and 10 of 5,700 milliseconds. Repeated values make the boundary cases easy to audit; this deliberately simplified distribution is not a realistic performance benchmark.
The executed Python 3.12.14 fixture checks both sums at 120,000 milliseconds. Dividing by 1,000 gives a mean of 120 milliseconds in both cases. Its 22 checks also cover the ranks, rejected aggregation calculations, cumulative counts and timeout denominators; they verify authored arithmetic, not telemetry collection. Yet the second dataset has 40 responses above 300 milliseconds, including ten very long responses. The equality of the means cannot make those waits disappear.
For this exercise, a percentile uses the nearest-rank convention: sort the observations, multiply the requested fraction by the count, round that position upward and select the observation at that one-based position. Other tools may interpolate or estimate quantiles. Record the actual method before expecting a dashboard to reproduce these discrete values.
| Statistic on completed successes | Uniform dataset | Mixed dataset | Reading supported by these inputs |
|---|---|---|---|
| Count | 1,000 | 1,000 | The comparison has equal numbers of recorded completed successes. |
| Mean | 120 ms | 120 ms | Equal means do not establish equal distributions. |
| p50 | 120 ms | 50 ms | The middle position improves in the mixed dataset. |
| p95 | 120 ms | 50 ms | Position 950 remains inside the 960 fast observations. |
| p99 | 120 ms | 500 ms | Position 990 is the last of the 30 intermediate observations. |
| p99.9 | 120 ms | 5,700 ms | Position 999 reaches the slowest group. |
| Count at or below 300 ms | 1,000 of 1,000 | 960 of 1,000 | The mixed dataset meets this threshold for 96% of completed successes. |
The p99 result deserves attention: it is 500 milliseconds even though the slowest ten observations take 5,700 milliseconds. A percentile describes a position in the ordered sample, not the maximum and not the shape beyond that position. Repeated values also mean the proportion at or below a reported value can exceed the nominal percentile.
This is not an argument to replace every average with p99.9. The mean remains useful for the total recorded duration divided across requests; p50 describes the middle; higher percentiles expose different tail positions. Choose the combination that addresses the decision, and retain the count and distribution so that a single attractive statistic cannot stand in for the whole result.
Correct an overall percentile assembled from server summaries
The second comparison is a separate synthetic dataset from the same five-minute interval. Worker East records 990 requests, all at 50 milliseconds. Worker West records ten requests, all at 2,000 milliseconds. Their measurement boundaries and units are identical; pooling them is meaningful within this authored case.
Each worker's p95 equals its only recorded duration. An unweighted average of those p95 values is 1,025 milliseconds. Weighting them by request count gives 69.5 milliseconds. Neither equals the pooled p95, which is 50 milliseconds: position 950 falls within the 990 East observations.
| Aggregation attempted | Actual calculation | Result and interpretation |
|---|---|---|
| Average the two p95 values | (50 + 2,000) / 2 | 1,025 ms; not the combined request p95. |
| Request-weight the two p95 values | (990 × 50 + 10 × 2,000) / 1,000 | 69.5 ms; still not the combined request p95. |
| Pool all request observations, then select position 950 | 990 values at 50, then ten at 2,000 | Pooled p95 = 50 ms. |
| Combine duration sums and counts | 69,500 / 1,000 | Pooled mean = 69.5 ms. Its equality to the weighted-p95 calculation here is coincidental. |
That coincidence occurs because every value within each worker is identical, so each local percentile equals its local mean. It does not create a general rule for averaging percentiles. Even with correct request weights, retaining only those two local quantiles discards the distributions needed to locate a pooled rank. Pooling the retained raw arrays is what supports the 50-millisecond result here; no monitoring-backend query was executed.
The Prometheus histogram and summary guidance distinguishes precomputed summary quantiles from aggregatable distribution representations. Its current page recommends native histograms where supported and explains estimation and bucket-resolution limits. Check your instrumentation and backend rather than assuming that changing a chart's aggregation setting can reconstruct a distribution that was never retained.
The same mistake occurs across time. An average of twelve five-minute p95 values is a summary of twelve selected values, not automatically the hour's request p95. Different traffic volumes compound the problem, but weighting alone does not repair it. Retain compatible distribution information for the desired population and interval, or label the available summary accurately.

Keep histogram approximation and sample size visible
For the mixed dataset, the retained cumulative counts are 960 at or below 100 milliseconds, 960 at or below 300, 990 at or below 1,000 and 1,000 at or below 6,000. These are overlapping cumulative buckets. Adding all four counts would count many requests repeatedly; 1,000 remains the total.
The exact stored observations show p99 at 500 milliseconds. The coarse buckets alone locate its rank in the interval above 300 and at or below 1,000 milliseconds. They do not reveal that every observation in that interval equals 500. A monitoring query may interpolate within buckets, producing an estimate that differs from this nearest-rank calculation on raw values.
For a fixed 300-millisecond threshold, an inclusive bucket at exactly 300 gives the completed-success fraction directly: 960 divided by 1,000. With no matching boundary, a tool may estimate the fraction or be unable to answer the query as intended. Ren therefore labels 960/1,000 as the exact count-based result for this fixture and leaves any backend-specific interpolated result unreported. Examine the actual tool semantics and resolution. Do not publish more precision than the retained distribution supports.
CloudWatch's statistics definitions explain that statistics apply over specified periods and distinguish percentiles from averages. The page also documents requirements for computing percentiles from raw data or particular statistic sets. Those are CloudWatch's documented rules, not evidence that this local calculation ran in AWS or that another backend uses the same algorithm.
High percentiles are especially fragile in tiny windows. In a separate 20-observation nearest-rank sample, p99 selects position 20, the maximum. One observation represents 5% of that sample. Reporting that value with several decimal places does not create information about a stable future one-percent tail. The exercise supplies no confidence interval or independent sampling assumptions that would support one.
Low traffic is therefore a reason to show the count and reconsider the reporting window, not to fabricate a smooth tail. A longer window may provide more observations while concealing a short-lived incident. Keep an appropriate shorter-window view alongside it when diagnosing a spike, and state which question each view answers.
Completed readback: revise the claim before choosing a fix
Ren's fictional October 28 review originally proposed the headline “Average unchanged, so response performance is unchanged.” The completed record withdraws that claim. The mixed sample has the same mean and a lower p50, but worse p99 and p99.9, with 40 recorded successes exceeding 300 milliseconds. The data supports that descriptive comparison, not a cause or a customer-impact estimate.
Service owner Mae adds twelve known timeout attempts to the mixed interval. Their precise completion durations are missing, so Ren does not enter zeros, substitute an arbitrary duration or place them in the exact percentile array. The record shows 1,012 attempts, 1,000 completed successes and twelve timeouts. It cannot report an exact all-attempt latency percentile from these records.
For a separate, explicitly authored decision rule, a good attempt must succeed within 300 milliseconds; every timeout counts as bad. The resulting fraction is 960 good attempts divided by 1,012 attempts, approximately 94.86%. It differs from the 96% completed-success fraction because it answers a different question. This is a calculation under a local example definition, not an approved service objective or a universal timeout policy.
Mae accepts responsibility for investigating the slow-response group using authorized diagnostic evidence and checking the missing timeout instrumentation. Her first deliverable is the measurement-gap explanation and a request-path review, not an already verified performance fix. Ren accepts replacing averaged worker p95 values with a correctly described pooled distribution and retaining the old calculation as a labeled rejected result. Neither action has actually been applied to a monitoring system; the completed asset is the decision record and arithmetic.
The review makes no claim that West caused a real incident, that a particular database needs optimization or that any customer experienced these authored waits. It also makes no promise that a future deployment will improve the tail. Those conclusions require actual measurements, a suitable comparison and evidence about the relevant request path.
Related workplace guides are available from LandOffer.ai. Before approving the next latency report, have its owner read back the event boundary, window, count, quantile method, aggregation rule and treatment of missing or failed requests. If only local p95 summaries remain, record that the combined p95 is unavailable instead of approving an averaged substitute. Keep the observed tail and its uncertainty beside the mean, then assign the next investigation without reporting a planned change as a verified improvement.
Sources and Further Reading
- Google SRE: Monitoring Distributed Systems — success and error latency, distributions and tail monitoring; checked October 28, 2026.
- Prometheus: Histograms and Summaries — quantile aggregation, histogram representations and resolution; current documentation checked October 28, 2026.
- Amazon CloudWatch: Statistics Definitions — periods, average and percentile definitions, and data requirements; documentation only, with no AWS account test.


