Latency, throughput and percentiles

Capsule description

Performance isn't a number. It's a distribution. When a manager asks you "how fast is the API?", answering "200ms on average" is like answering "how's the weather" with the annual average temperature. Technically true, practically useless.

In this capsule you'll internalize three concepts that are the mandatory language of any serious performance conversation:

1. Latency vs throughput: why optimizing one can kill the other.
2. Percentiles (p50, p95, p99): why the average lies and percentiles tell the truth.
3. Your workload (read-heavy / write-heavy / mixed): why it matters before you pick any technique.

You won't write production code here. You'll run simple simulations with Python and numpy to see with your own eyes why the average deceives. By the end, you'll be able to defend in any technical discussion why reporting p95 matters more than reporting the average — and you'll remember this capsule every time you see a dashboard that only shows the mean.

Latency vs throughput

They're two different metrics that people confuse all the time.

Mental model: the bank teller

Imagine a bank with a single teller.

• Latency is: "how long one customer takes from arriving to leaving". If the teller handles each transaction in 60 seconds, latency is 60 seconds.
• Throughput is: "how many customers the bank serves per minute". If there's only one teller, maximum throughput is 1 customer/minute.

Now open 10 tellers in parallel:

• Latency is still 60 seconds per customer — each teller takes the same time.
• Throughput went up to 10 customers/minute — you multiplied capacity without touching individual speed.

And the other way around, optimize the single teller (better training, shorter forms):

• Latency drops to 30 seconds — each customer is faster.
• Throughput rises to 2 customers/minute — without opening more tellers, you serve twice as many.

Latency is the speed of one. Throughput is the volume of many. They're not the same thing.

Applied to your API

LATENCY    = time from request to response, for each request
             (measured in ms or seconds)
             an individual user's metric

THROUGHPUT = requests processed per unit of time
             (measured in RPS — requests per second)
             a system capacity metric

The hidden trade-off: optimizing one can kill the other

A real case: you have an endpoint that runs a 100ms query. You decide to "optimize" it by caching with an in-memory lock. Now the first request takes 100ms (same as before) but the rest wait on that lock. Result:

• If requests arrive sequentially: latency improves (post-cache: 5ms), throughput goes up too.
• If requests arrive in parallel (the realistic scenario): the lock serializes them. Individual latency drops at p50, but p99 explodes because the unlucky ones wait in line. Throughput falls because concurrency disappears.

Without measuring under concurrent load, you don't detect this. By measuring only sequential latency (the classic "I tested it on my machine"), the bug goes unnoticed.

Rule: any optimization that changes concurrency (locks, DB connections, external calls) requires measuring latency and throughput simultaneously, under concurrent load.

Why the average lies: the case for percentiles

The problem with the mean

You have 10 requests with these latencies in ms:

[50, 55, 60, 58, 62, 55, 60, 58, 56, 4500]

• Mean: 511 ms
• Median (p50): 58 ms
• p95: 4,500 ms (the worst of the worst)

Which one reflects the real user experience?

• If you report the mean (511ms): you're lying — 9 out of 10 users had <65ms.
• If you report the median (58ms): you're hiding the problem — 1 in every 10 users suffers 4.5 seconds.
• If you report p50 + p95: you tell the full story: "the typical case is 58ms, but the worst 5% is 4,500ms".

The 4,500ms outlier could be: a slow query on a cache miss, a call to an external service that failed, a GC pause, lock contention. If you only look at the mean, you don't see it. If you only look at the median, you're actively hiding it.

Definition of percentiles

A percentile pN is the value below which N% of the data falls.

• p50 (median): 50% of requests were faster than this number.
• p95: 95% of requests were faster; the worst 5% is above it.
• p99: 99% were faster; the worst 1% is above it.
• p99.9: for high-scale systems, the worst 0.1%.

Typical HTTP latency distribution:
                                                    p99
                                                     │
        ████████████                                 │
      ██████████████████                             │
    ████████████████████████                         │
   ████████████████████████████                      │
   ████████████████████████████████  ▒▒  ░░  ░░  ░░  │
   |─────|─────|─────|─────|─────|─────|─────|─────|
  0ms   50    100   150   200   500   1s   2s   5s
                  ↑           ↑              ↑
                 p50         p95           p99

Most requests are fast. But the "tail" is where the user's real pain lives.

Worked example: the average can lie a lot

Let's simulate two APIs with the same mean but radically different distributions.

# simulate_distributions.py
import numpy as np

# API A: consistent latencies around 200ms
api_a = np.concatenate([
    np.random.normal(loc=200, scale=20, size=10000)  # 10k "normal" requests
])

# API B: most are fast (50ms), but 5% go to 3 seconds
api_b = np.concatenate([
    np.random.normal(loc=50, scale=10, size=9500),   # 9500 fast requests
    np.random.normal(loc=3000, scale=200, size=500)  # 500 slow requests
])

def report(name, data):
    print(f"\n--- {name} ---")
    print(f"  Mean:   {np.mean(data):.0f} ms")
    print(f"  p50:    {np.percentile(data, 50):.0f} ms")
    print(f"  p95:    {np.percentile(data, 95):.0f} ms")
    print(f"  p99:    {np.percentile(data, 99):.0f} ms")
    print(f"  Max:    {np.max(data):.0f} ms")

report("API A (consistent)", api_a)
report("API B (bimodal)",    api_b)

Expected output (numbers vary slightly because of np.random):

--- API A (consistent) ---
  Mean:   200 ms
  p50:    200 ms
  p95:    233 ms
  p99:    247 ms
  Max:    275 ms

--- API B (bimodal) ---
  Mean:   197 ms
  p50:    50 ms
  p95:    204 ms
  p99:    3162 ms
  Max:    3573 ms

Reading it: If you only look at the mean (~200ms in both cases) you'd think the two APIs are equivalent. They are not.

• API A: consistent experience. Every user waits between 150 and 280ms. Predictable.
• API B: bimodal experience. 95% of users feel ~50ms (excellent), but 5% suffer 3+ seconds. That 5% is your worst nightmare — the 1-star reviews, the support tickets, the silent churn.

If you had to pick which one to optimize first, it would be API B without hesitation. But if you only look at the mean, you don't even detect that it has a problem.

The detail almost everyone misses: the percentile you pick has to be deeper than the tail

Look at API B's output again and notice something uncomfortable: p95 is 204ms, not 3 seconds. Didn't we just say the worst 5% sits at 3,000ms?

We did, and that's the trap. The slow group is exactly 5% of the requests. p95 is, by definition, the boundary of that 5%: it lands right at the edge of the cliff, still on the fast side. Anyone reporting only p95 for this API will conclude "200ms, all good" — and will be just as blind as the person reporting the mean.

The one that uncovers the problem is p99 (3,162ms), because the worst 1% is fully inside the slow tail.

The rule that follows: a percentile only reveals a tail if the tail is bigger than 100 - N. If 5% of your requests are slow, p95 can't see it; you need p99. If 1% is slow, even p99 barely sees it; you need p99.9. That's why you never report one percentile: you report p50, p95 and p99 together. The shape of the distribution lives in the differences between them, not in any isolated number.

Why do high percentiles matter so much?

Marc Brooker (AWS Principal Engineer) explains it well: on a typical web page that makes 30 requests to the backend, an "average" user experiences your API's p99, not its p50.

Mathematically: if a request has a 1% chance of being "slow" (>1s), the probability that a user makes 30 requests without hitting a single slow one is:

P(no slow request) = 0.99 ^ 30 = 0.74

Only 74% of users escape the "worst 1%". 26% experience it. If your product requires 100 requests per session (a typical dashboard), that number drops to 36%.

Conclusion: p99 isn't "the rare case". It's the typical experience of an active user. That's why obsessing over it isn't perfectionism — it's respect for 26% of your user base.

Moving averages, maximums, and other problematic numbers

Other numbers you'll see that can deceive you:

"Maximum latency"

Max: 12,400 ms

What does it tell you? Almost nothing. A single outlier (a GC, a query that got stuck, a networking glitch) can be the max. It isn't actionable.

Better report: p99 (worst 1%) or p99.9 (worst 0.1%) depending on scale.

"Average latency over the last 5 minutes"

What you see in many New Relic or Datadog dashboards. Useful for spotting trends, useless for diagnosis. A 5-minute moving average hides short but real spikes.

Better: percentiles per interval (p99 over 1 minute buckets).

"Average throughput"

Throughput: 1,200 RPS

Averaged over how long? Under what load? With what error rate? Without that context, the number isn't comparable.

Better: "sustained throughput of X RPS with error rate <0.1% for 5 minutes on machine Y".

Your workload: read-heavy, write-heavy or mixed

Before choosing what to optimize, identify what dominates your app: reads, writes, or both in similar proportions. It changes the tuning decisions.

Read-heavy (typical of product / content APIs)

• 95%+ of queries are SELECT.
• Example: public blog, product catalog, analytics dashboard.
• Optimizations that pay off: indexes, eager loading, aggressive caching, read replicas.
• Optimizations that do NOT pay off: tuning autovacuum, changing isolation levels.

Write-heavy (typical of logs, telemetry, ingestion)

• Significant volume of INSERT/UPDATE/DELETE.
• Example: event ingestion pipeline, audit logs, IoT.
• Optimizations that pay off: batch inserts, partitioning, autovacuum tuning, fewer indexes (more indexes = more overhead on writes).
• Optimizations that do NOT pay off: read replicas, aggressive caching (the data changes constantly).

Mixed (typical of B2B SaaS)

• Reads and writes in comparable proportion.
• Example: productivity app, e-commerce with lots of orders.
• Optimizations that pay off: depends on the specific endpoint — you need to measure per endpoint, not per app.

How to identify it in PostgreSQL (a preview of module 5)

-- Ratio of tuples read vs modified, per table
SELECT
  schemaname || '.' || relname AS table_name,
  seq_tup_read + idx_tup_fetch AS tuples_read,
  n_tup_ins + n_tup_upd + n_tup_del AS tuples_modified
FROM pg_stat_user_tables
ORDER BY tuples_read DESC
LIMIT 10;

(In module 5 you'll go deep on pg_stat_*. Here we show it only so you know it exists.)

Why does this matter in real work?

Three situations where what you learned saves you hours or protects your reputation:

1. Conversations with stakeholders. Your PM asks you "is the app fast?". Possible answers:

• ❌ "Yes, it's fast." (means nothing)
• ❌ "The average is 250ms." (can hide a p99 of 8s)
• ✅ "p50 is 180ms and p95 is 320ms on the main endpoint. We're fine for typical users. The /reports endpoint has a p95 of 4.2s — that one is a problem."

2. Reviewing a colleague's code. A colleague sends a PR saying "I optimized the endpoint, it's twice as fast now". Mandatory questions:

• Did you measure with which tool and what concurrent load?
• Did p50 or p99 improve? Because if only p50, p99 could have gotten worse.
• Did throughput hold or drop? Because "faster" without throughput can be a badly-made trade-off.

3. Senior technical interviews. "Tell me about a time you optimized an API." The junior answer starts with the solution. The senior answer starts with: "first I measured a baseline with X, I identified that the problem was in the p95 of endpoint Y, I hypothesized Z, applied the change, re-measured, and reported the improvement with concrete numbers". The process is the answer, not the technique.

Traps and common mistakes

Mistake 1 (conceptual): confusing p99 with "the worst case"

Symptom: "p99 is 800ms, that means no user waits more than 800ms".

Why it's wrong: p99 says that 99% of requests are faster than 800ms. The remaining 1% can range from 801ms to 30 seconds. If you have 100k requests per hour, that 1% is 1,000 requests/hour suffering worse than 800ms. That's not "almost nobody".

How to detect it: look at the max and p99.9. If the gap between p99 and max is huge, you have a dangerously long tail.

Mistake 2 (practical): measuring only sequential latency

Symptom: you run the benchmark with -c1 -t1 (a single client, a single thread). You report that the API runs at 50ms. Production falls over at 100 RPS.

Why it happens: without concurrency there's no contention (locks, connection pool, cache hit rate, GC). Your measurement reflects a scenario that doesn't exist in production.

How to fix it: always measure under representative load. If production sees 200 RPS, your benchmark has to generate ≥200 sustained RPS.

Mistake 3 (conceptual): assuming the mean is close to the median

Symptom: "the mean and the median are usually similar in real data, I'm not worried".

Why it's wrong: HTTP latencies are almost never normally distributed. They have long tails to the right (long-tail). In long-tail distributions, the mean is always greater than the median and outliers distort it. The intuition you learned with normally distributed data (mean ≈ median) does not apply here.

How to tell: run the simulate_distributions.py snippet and look at how the mean and median differ in API B.

Mistake 4 (practical): comparing percentiles across runs without a critical eye

Symptom: your p95 was 200ms on Monday and 220ms on Tuesday. You report that "it got 10% worse".

Why it's problematic: percentiles have natural noise. A 5-10% variation between runs is normal in real systems. To detect real regressions you need: (a) multiple runs, (b) significance tests or confidence intervals, (c) a reproducible baseline.

How to fix it: run N times (at least 3-5), report the median of the percentiles, and only shout "regression" if it exceeds the natural variation range you measured.

Exercises

Exercise 1: Calculate percentiles by hand

You have 20 latencies (ms) from an endpoint:

[120, 135, 142, 138, 145, 150, 148, 132, 140, 138,
 145, 155, 142, 138, 140, 148, 152, 145, 138, 4200]

Compute the mean, median (p50), p95 and p99. Which metric is the most misleading? Which is the most revealing?

import numpy as np

data = [120, 135, 142, 138, 145, 150, 148, 132, 140, 138,
        145, 155, 142, 138, 140, 148, 152, 145, 138, 4200]

print(f"Mean:   {np.mean(data):.1f} ms")
print(f"p50:    {np.percentile(data, 50):.1f} ms")
print(f"p95:    {np.percentile(data, 95):.1f} ms")
print(f"p99:    {np.percentile(data, 99):.1f} ms")
print(f"Max:    {np.max(data)} ms")

Mean:   344.6 ms
p50:    142.0 ms
p95:    357.3 ms
p99:    3431.4 ms
Max:    4200 ms

Exercise 2: Identify bimodal vs unimodal

Look at these two latency descriptions and decide which one is bimodal:

API X:

Mean: 250ms, p50: 245ms, p95: 320ms, p99: 380ms, Max: 420ms

API Y:

Mean: 250ms, p50: 80ms, p95: 1800ms, p99: 2400ms, Max: 3100ms

Which one has a bimodal distribution and why?

Exercise 3: Latency vs throughput trade-off

You have an endpoint that currently has:

• p50: 100ms
• p95: 200ms
• Sustained throughput: 500 RPS

You implement an optimization that adds a global lock to avoid race conditions. Afterward:

• p50: 60ms
• p95: 800ms
• Sustained throughput: 200 RPS

Was the optimization good or bad? Justify it.

Exercise 4: Report like a senior

Your colleague says: "I added cache to the /products endpoint, it's faster now". You're their tech lead. List the minimum 5 questions you have to ask before approving the change.

Exercise 5: Real bimodality with simulation

Modify the simulate_distributions.py snippet to create an API "C" that has a mean of 100ms but a p99 of 5 seconds. What proportion of outliers do you need? Confirm with code.

import numpy as np

# Naive attempt: 99% at ~50ms and 1% at ~5000ms
api_c = np.concatenate([
    np.random.normal(loc=50, scale=10, size=9900),
    np.random.normal(loc=5000, scale=100, size=100)
])

print(f"Mean:   {np.mean(api_c):.0f} ms")
print(f"p50:    {np.percentile(api_c, 50):.0f} ms")
print(f"p95:    {np.percentile(api_c, 95):.0f} ms")
print(f"p99:    {np.percentile(api_c, 99):.0f} ms")

Mean:   99 ms      ← meets "mean ~100ms"
p50:    50 ms
p95:    67 ms
p99:    135 ms     ← ✗ does NOT meet it: we expected ~5000ms

import numpy as np

# Fix: 98.8% at ~50ms and 1.2% at ~5000ms
api_c = np.concatenate([
    np.random.normal(loc=50, scale=10, size=9880),
    np.random.normal(loc=5000, scale=100, size=120)
])

print(f"Mean:   {np.mean(api_c):.0f} ms")
print(f"p50:    {np.percentile(api_c, 50):.0f} ms")
print(f"p95:    {np.percentile(api_c, 95):.0f} ms")
print(f"p99:    {np.percentile(api_c, 99):.0f} ms")

Mean:   109 ms     ← ~100ms (the toll for raising the tail)
p50:    50 ms
p95:    67 ms
p99:    4923 ms    ← ✓ now it does land inside the slow tail

Exercise 6: Read-heavy or write-heavy?

For each app, decide whether it's read-heavy, write-heavy, or mixed, and what type of optimization you'd prioritize:

a) A public blog with 1M visits/month and 5 new posts per week. b) An IoT backend that receives 50,000 events per second from sensors and is queried 100 times a day for reports. c) A B2B SaaS for project management where each user creates ~10 tasks and queries ~200 times a day.

Summary and next step

In this capsule you internalized:

• Latency ≠ throughput. Optimizing one can worsen the other. Measure both whenever you change something that touches concurrency.
• The average lies when there are outliers (which is always, in HTTP latencies). Report percentiles: p50, p95, p99 at minimum.
• p99 isn't "the rare case" — it's the typical experience of an active user making many requests per session.
• Your workload defines what to optimize. Read-heavy, write-heavy and mixed require different tactics. Identify it before picking a technique.

Before moving on you should be able to:

• Explain to a non-technical colleague why "the average is 200ms" can be a useful lie
• Recognize a bimodal distribution by looking at p50 vs p95
• List at least 5 critical questions to ask someone who says "I optimized it"

Next capsule — Reproducible baselines. You now know what to measure; in capsule 03 you'll learn how to measure well: warmup, context, multiple runs, what to document so someone else can reproduce your numbers. Without reproducibility, everything you measure is anecdotal.

Resources

1. Gil Tene — "How NOT to Measure Latency" (full talk) — 40 min on why almost every latency benchmark is done wrong. Mandatory.
2. Marc Brooker — "Tail latency might matter more than you think" — explains with math why p99 matters more than it seems.
3. Heinrich Hartmann — "Statistics for Engineers" — technical reference on percentiles, aggregations, and how NOT to average percentiles.
4. Brendan Gregg — Chapter 2 of Systems Performance (2nd ed.) — "Methodologies": how to reason about performance.
5. HdrHistogram — the canonical data structure for reporting latencies. If you measure latencies in serious work, you'll use this or an equivalent.
6. Coda Hale — "Metrics, Metrics Everywhere" — a classic talk on how to report honest metrics in production.
7. PostgreSQL Documentation — Monitoring with pg_stat_statements — a preview of module 5, useful for understanding where your PostgreSQL metrics will come from.

Module 1 — Database Performance & Query Tuning Guide