Percentiles: p50 and p95

Description

Everything you did in this module up to now measured one run at a time. A real business never asks "how long did that specific run take?" — it asks "how fast is the system, in general, for the people using it?" That question has no single correct answer with one number. An average is one possible answer, but — as Module 1 already warned — it hides the users having the worst time. This lesson introduces a better answer: percentiles, calculated over a real batch of twelve Reservo runs.

Connection to the module

This lesson scales total_run_latency_ms (lesson 05) from one run to a batch — the first time this module works with more than one run at a time. The batch of twelve runs you build here is the same one lesson 07 (to identify which tool dominates) and lesson 08 (the mini-project's final report) reuse — build it carefully, because it never gets rebuilt.

Analogy: customer number 95 out of every 100

Imagine a coffee shop that serves, on a busy day, a hundred customers. If you ask the owner "how long does a customer wait in line?", and they answer with the average of the hundred wait times, that figure might sound reasonable — "two minutes, on average" — and still hide something important: ninety-five customers waited less than two minutes, but five waited fifteen minutes each, because they arrived right when the coffee machine jammed. The average of those hundred numbers still comes out to "two minutes," because those five customers' fifteen minutes get diluted among the ninety-five who waited little. If you're one of those five, "two minutes on average" doesn't describe your experience at all.

The 95th percentile (p95) answers a different, more honest question: "if I sort the hundred customers from shortest to longest wait time, how long did the customer in position 95 wait?" That figure — not the average — is what a serious business uses to promise something real: "95% of our customers wait less than X minutes" is a promise you can verify, and one that doesn't break just because a few extreme cases exist. The 50th percentile (p50, the median) answers the "typical" customer's question: the one right in the middle of the sorted line. A healthy system has a low p50 (most people wait little) and a p95 that doesn't spike too far above the p50 (the extreme cases aren't too extreme). When p95 spikes far above p50, that's, almost always, the first real symptom that something's wrong — long before the average moves enough for anyone to notice.

The batch: twelve Reservo runs, none repeated on purpose

Before calculating anything, put together the batch the rest of this lesson works over — twelve different tasks, each exercising a different combination of the four tools:

import reservo_agent as ra

TOOL_LATENCY_MS = {
    "list_rooms": 40,
    "get_quote": 25,
    "book_room": 120,
    "cancel_booking": 90,
}


def total_run_latency_ms(history):
    latency_ms = 0
    tool_use_name = {}
    for turn in history:
        if turn["role"] != "assistant" or not isinstance(turn["content"], list):
            continue
        for block in turn["content"]:
            if block["type"] == "tool_use":
                tool_use_name[block["id"]] = block["name"]
    for turn in history:
        if turn["role"] != "user" or not isinstance(turn["content"], list):
            continue
        for block in turn["content"]:
            if block["type"] == "tool_result" and not block.get("is_error"):
                name = tool_use_name.get(block["tool_use_id"])
                latency_ms += TOOL_LATENCY_MS.get(name, 0)
    return latency_ms


def tu(id_, name, input_):
    return {"type": "tool_use", "id": id_, "name": name, "input": input_}


def step(*blocks):
    return {"stop_reason": "tool_use", "content": list(blocks)}


def end(text):
    return {"stop_reason": "end_turn", "content": [{"type": "text", "text": text}]}


batch = [
    ("Cuánto cuesta Focus basic 2h", [
        step(tu("t01", "get_quote", {"room": "Focus", "tier": "basic", "hours": 2})),
        end("Focus basic 2h cuesta $50.00.")]),
    ("Qué salas hay disponibles", [
        step(tu("t01", "list_rooms", {})),
        end("Tenemos Focus, Studio y Boardroom.")]),
    ("Reserva Studio basic 1h para Luis", [
        step(tu("t01", "get_quote", {"room": "Studio", "tier": "basic", "hours": 1})),
        step(tu("t02", "book_room", {"room": "Studio", "tier": "basic", "hours": 1, "member": "Luis"})),
        end("Reservé Studio basic 1h para Luis. Confirmación #1.")]),
    ("Cancela la reserva 1", [
        step(tu("t01", "cancel_booking", {"id": 1})),
        end("Cancelé la reserva #1.")]),
    ("Reserva Boardroom pro 1h para Sofía, con la lista primero", [
        step(tu("t01", "list_rooms", {})),
        step(tu("t02", "get_quote", {"room": "Boardroom", "tier": "pro", "hours": 1})),
        step(tu("t03", "book_room", {"room": "Boardroom", "tier": "pro", "hours": 1, "member": "Sofía"})),
        end("Reservé Boardroom pro 1h para Sofía. Confirmación #2.")]),
    ("Cotiza Focus premium y luego pro 3h", [
        step(tu("t01", "get_quote", {"room": "Focus", "tier": "premium", "hours": 3})),
        step(tu("t02", "get_quote", {"room": "Focus", "tier": "pro", "hours": 3})),
        end("Focus pro 3h cuesta $60.00.")]),
    ("Reserva Focus pro 3h para Ana, con corrección", [
        step(tu("t01", "list_rooms", {})),
        step(tu("t02", "get_quote", {"room": "Focus", "tier": "premium", "hours": 3})),
        step(tu("t03", "get_quote", {"room": "Focus", "tier": "pro", "hours": 3})),
        step(tu("t04", "book_room", {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"})),
        end("Reservé Focus pro 3h para Ana. Confirmación #3.")]),
    ("Reserva y cancela Studio pro 2h para Diego", [
        step(tu("t01", "book_room", {"room": "Studio", "tier": "pro", "hours": 2, "member": "Diego"})),
        step(tu("t02", "cancel_booking", {"id": 4})),
        end("Reservé y cancelé Studio pro 2h para Diego.")]),
    ("Compara Focus pro y Boardroom pro 2h", [
        step(tu("t01", "get_quote", {"room": "Focus", "tier": "pro", "hours": 2})),
        step(tu("t02", "get_quote", {"room": "Boardroom", "tier": "pro", "hours": 2})),
        end("Focus pro 2h cuesta $40.00 y Boardroom pro 2h cuesta $128.00.")]),
    ("Reserva Boardroom basic 1h para Carla, con horas inválidas primero", [
        step(tu("t01", "book_room", {"room": "Boardroom", "tier": "basic", "hours": 0, "member": "Carla"})),
        step(tu("t02", "book_room", {"room": "Boardroom", "tier": "basic", "hours": 1, "member": "Carla"})),
        end("Reservé Boardroom basic 1h para Carla. Confirmación #5.")]),
    ("Qué salas hay y cuánto cuesta Studio pro 4h", [
        step(tu("t01", "list_rooms", {})),
        step(tu("t02", "get_quote", {"room": "Studio", "tier": "pro", "hours": 4})),
        end("Studio pro 4h cuesta $128.00.")]),
    ("Reserva Focus pro 3h para Marta y luego cancela", [
        step(tu("t01", "list_rooms", {})),
        step(tu("t02", "get_quote", {"room": "Focus", "tier": "pro", "hours": 3})),
        step(tu("t03", "book_room", {"room": "Focus", "tier": "pro", "hours": 3, "member": "Marta"})),
        step(tu("t04", "cancel_booking", {"id": 6})),
        end("Reservé y cancelé Focus pro 3h para Marta.")]),
]

latencies = []
for i, (question, script) in enumerate(batch, start=1):
    final, history = ra.run_reservo_agent(question, script)
    lat = total_run_latency_ms(history)
    latencies.append(lat)
    print(f"run {i:2}: {lat:4} ms  -- {question}")

What to expect:

run  1:   25 ms  -- Cuánto cuesta Focus basic 2h
run  2:   40 ms  -- Qué salas hay disponibles
run  3:  145 ms  -- Reserva Studio basic 1h para Luis
run  4:   90 ms  -- Cancela la reserva 1
run  5:  185 ms  -- Reserva Boardroom pro 1h para Sofía, con la lista primero
run  6:   25 ms  -- Cotiza Focus premium y luego pro 3h
run  7:  185 ms  -- Reserva Focus pro 3h para Ana, con corrección
run  8:  210 ms  -- Reserva y cancela Studio pro 2h para Diego
run  9:   50 ms  -- Compara Focus pro y Boardroom pro 2h
run 10:  120 ms  -- Reserva Boardroom basic 1h para Carla, con horas inválidas primero
run 11:   65 ms  -- Qué salas hay y cuánto cuesta Studio pro 4h
run 12:  275 ms  -- Reserva Focus pro 3h para Marta y luego cancela

Twelve runs, twelve different latencies — well, almost: runs 5 and 7 tie at 185, and runs 1 and 6 tie at 25. Look at run 12: 275 ms, the same ceiling you already calculated in lesson 04 — it's, exactly, the script that uses all four tools, each once.

p50: the middle value, with the nearest-rank method

The most direct way to calculate a percentile — the nearest-rank method — is simple: sort the values from smallest to largest, and take the value at position ceil(p/100 * n) (counting from 1).

import math


def percentile(sorted_values, p):
    """Percentil por el metodo nearest-rank: ordena y toma el valor en la
    posicion ceil(p/100 * n) (1-indexado). Siempre devuelve un valor que
    ocurrio de verdad -- nunca interpola entre dos runs."""
    n = len(sorted_values)
    rank = math.ceil(p / 100 * n)
    rank = max(1, min(rank, n))
    return sorted_values[rank - 1]


sorted_latencies = sorted(latencies)
print("latencias ordenadas:", sorted_latencies)
print("n runs             :", len(sorted_latencies))

p50 = percentile(sorted_latencies, 50)
print("p50 (nearest-rank) :", p50, "ms")

What to expect:

latencias ordenadas: [25, 25, 40, 50, 65, 90, 120, 145, 185, 185, 210, 275]
n runs             : 12
p50 (nearest-rank) : 90 ms

With n=12, ceil(0.5 * 12) = 6 — position 6 (1-indexed) in the sorted list. Counting from the start (25, 25, 40, 50, 65, 90, ...), the sixth value is 90 — run 4, "Cancela la reserva 1," which only calls cancel_booking. This batch's p50 is 90 ms: half the runs take less than that, the other half take more (or the same).

p95: the same method, the higher cutoff point

p95 = percentile(sorted_latencies, 95)
print("p95 (nearest-rank) :", p95, "ms")

What to expect:

p95 (nearest-rank) : 275 ms

ceil(0.95 * 12) = ceil(11.4) = 12 — position 12, that is, the last value in the sorted list: 275, run 12, the one using all four tools. Stop here, because it's one of this entire module's most honest lessons: with only twelve runs, p95 ends up being, literally, the batch's slowest run. That's not a calculation error — it's a direct consequence of having little data: 0.95 * 12 = 11.4 is so close to the end of the list that any rounding up almost always takes you all the way to the last position. With twelve samples, "the 95th percentile" and "the worst case observed" nearly coincide — and that coincidence disappears as the batch grows to hundreds or thousands of runs, where p95 starts to really represent "95% of cases," not "the one case worse than all the others."

Comparing p50, p95, and the average

import statistics

mean = statistics.mean(latencies)
print(f"p50 (mediana, nearest-rank) : {p50} ms")
print(f"promedio (mean)             : {mean:.1f} ms")
print(f"p95 (nearest-rank)          : {p95} ms")

What to expect:

p50 (mediana, nearest-rank) : 90 ms
promedio (mean)             : 117.9 ms
p95 (nearest-rank)          : 275 ms

Three figures, three different readings of the same batch. The p50 (90 ms) says: "half of Reservo's customers had an experience of 90 ms or less." The average (117.9 ms) sits closer to p50 than to p95 — but already pulled upward by the few long runs, exactly as in the coffee shop analogy. The p95 (275 ms) says: "the worst case we observed in this batch took almost triple the typical customer's." If Reservo had to promise something to its product team, "on average we respond in 118 ms" sounds fine but hides the customer who had to wait 275; "95% of our customers wait less than 275 ms" is a more honest promise, even though it sounds less favorable.

The trap of statistics.quantiles with little data

Python has a ready-made function for calculating percentiles, statistics.quantiles, and it's worth knowing — but it's also worth seeing, run for real, how it can surprise you with a batch this small.

q_inclusive = statistics.quantiles(latencies, n=100, method="inclusive")
q_exclusive = statistics.quantiles(latencies, n=100, method="exclusive")
print("p95 con method='inclusive':", q_inclusive[94], "ms")
print("p95 con method='exclusive':", q_exclusive[94], "ms")
print("máximo real del lote      :", max(latencies), "ms")

What to expect:

p95 con method='inclusive': 239.25 ms
p95 con method='exclusive': 297.75 ms
máximo real del lote      : 275 ms

Look carefully at the method='exclusive' last line: 297.75 ms — a p95 higher than the slowest run that actually happened (275). This isn't a Python bug — statistics.quantiles with method='exclusive' (the default value if you don't specify method) uses an interpolation formula meant for large samples, and with only twelve data points it can extrapolate beyond the observed maximum. method='inclusive' gives 239.25 — a more reasonable number (between 210 and 275), but one that doesn't correspond to any real run either: it's a point interpolated between two observations. Neither of statistics.quantiles's two numbers is "the" correct p95 — both are valid, under different percentile definitions, and both differ from the 275 this lesson's nearest-rank method gave. This is why observability/latency_model.py (lesson 08) uses this lesson's manual percentile function, not statistics.quantiles: with a simple method of its own, every reported percentile is always a value some real run produced — never an interpolated figure nobody actually experienced.

Common mistakes

1. Reporting only the average, and never a percentile. This batch's average (117.9 ms) sounds reasonable — and completely hides that the slowest run took more than double. A latency report without at least p50 and p95 doesn't give anyone enough information to know whether the system has a long tail of bad cases.

2. Trusting a p95 calculated over a small sample as if it were a solid figure. This lesson showed it plainly: with twelve runs, p95 almost always coincides with the worst case observed — useful for learning the concept, insufficient for a real business decision. Lesson 08 is going to insist on this: a production p95 needs, at minimum, hundreds of samples to start being reliable.

3. Mixing the percentile method from two different sources without noticing. If a dashboard reports p95 with one method (say, method='inclusive') and your regression code calculates it with another (nearest-rank), you're going to be comparing two numbers that don't mean exactly the same thing — this lesson's 239.25 versus 275 difference is direct proof that the method matters, not just the data.

4. Thinking statistics.quantiles(n=100) always returns an observed value. It doesn't — with interpolation (this function's default behavior), it can return a number no real run produced, and even, as you saw with method='exclusive', one that exceeds the batch's real maximum.

5. Calculating percentiles over a batch that mixes completely different types of runs without thinking about it. This lesson's batch mixes simple queries (25 ms) with complex bookings (275 ms) on purpose, to make the example rich — but in a real system, if "check the price" and "book with four tools" are operations a business wants to measure separately, calculating a single p95 over both mixed together can hide more than it reveals. This guide doesn't split the batch by task type — that's outside its scope — but it's worth keeping in mind.

Exercises

Exercise 1: Calculate p50 and p95 without the slowest run (Easy)

Remove run 12 (275 ms, the slowest) from the batch, and recalculate p50 and p95 over the remaining eleven runs with this lesson's percentile function.

sin_el_mas_lento = sorted(lat for lat in latencies if lat != 275)
print("latencias (11 runs):", sin_el_mas_lento)
print("p50:", percentile(sin_el_mas_lento, 50), "ms")
print("p95:", percentile(sin_el_mas_lento, 95), "ms")

latencias (11 runs): [25, 25, 40, 50, 65, 90, 120, 145, 185, 185, 210]
p50: 90 ms
p95: 210 ms

Exercise 2: Double the batch and confirm p50 and p95 don't change (Medium)

Duplicate the latencies list (each value appears twice, in the same order), and recalculate p50 and p95 over the batch of 24 values. Do the figures change compared to the original batch of 12?

doble = sorted(latencies + latencies)
print("n runs:", len(doble))
print("p50:", percentile(doble, 50), "ms")
print("p95:", percentile(doble, 95), "ms")

n runs: 24
p50: 90 ms
p95: 275 ms

Exercise 3: Design a batch of four runs where p50 and p95 are identical (Hard)

Is it possible to build a batch (of any size, using this module's latencies) where p50 and p95 give exactly the same number? Reason first about what condition the batch would have to be in for that to happen, then build it and confirm with code.

lote_uniforme = [185, 185, 185, 185]
print("p50:", percentile(sorted(lote_uniforme), 50), "ms")
print("p95:", percentile(sorted(lote_uniforme), 95), "ms")

p50: 185 ms
p95: 185 ms

Summary and next step

• We put together a batch of twelve real Reservo runs, with latencies ranging from 25 to 275 ms, none repeated on purpose.
• We calculated p50 (90 ms) and p95 (275 ms) with the nearest-rank method — simple, always returns a value some real run produced — and compared them against the average (117.9 ms): the average hides the runs having the worst time, exactly as in the coffee shop analogy.
• We confirmed, with real execution, that with only twelve samples p95 almost always coincides with the worst case observed — an honest limitation of working with small batches, not a flaw in the method.
• We saw, with real numbers, that statistics.quantiles can give different results depending on its interpolation method — including a p95 exceeding the batch's real maximum — and why this guide prefers the nearest-rank method, simpler and always anchored to a real data point.

Next lesson: 07 — Latency as an Operational Signal. With p50 and p95 now calculated, we identify which tool, specifically, dominates the batch's total latency — and trace the boundary between what this guide measures (an agent's steps) and what sre-and-incident-response-guide measures (the infrastructure behind it).

Additional resources

1. Python — statistics — statistics.mean, statistics.median, and statistics.quantiles, with their different interpolation methods (inclusive/exclusive) confirmed in this lesson.
2. Python — math.ceil — The central function of this lesson's nearest-rank percentile method.
3. Anthropic — Building effective agents — On why a real user's perceived latency depends on the distribution's tail, not just the average case.
4. Python 3.14 — What's New — The version every percentile in this lesson, correct and surprising alike, ran on.