Latency benchmark

Latency is the dimension easiest to measure wrong. A sample of 1 request tells you nothing — the variance is huge. An average without percentiles hides the worst case, which is what matters for the user experience.

In this capsule you're going to build a Python harness that measures P50, P95 and P99 latency for each provider we covered in the path. At the end you'll have a defensible table with numbers you can show in a meeting.

By the end you'll be able to:

• Design a latency benchmark that avoids the typical mistakes (cold start contaminating data, small sample, irrelevant prompts)
• Implement the harness for OpenAI, OpenRouter, Ollama, Modal with the same structure
• Interpret P50/P95/P99 correctly and explain the difference
• Export results to JSON/CSV for analysis and cross-comparisons

Why it matters

Your product doesn't deliver "average latency". If your P95 is 25s, one user in 20 waits 25s. For an interactive chatbot, that's a broken product even if the average looks fine.

Knowing how to measure latency percentiles lets you:

• Decide whether a provider meets your real SLA (not their marketing SLA)
• Detect when a provider degrades (P99 goes from 8s to 30s in a day) before users report it
• Negotiate with vendors by showing your own data, not anecdotes

Mental model: percentiles

Don't use the average (mean). The average is the naive person's favorite metric: a single 30s request "drowns out" 100 requests of 1s and gives you a 1.3s average — hiding that one user suffered 30s.

Requests: [1.0, 1.1, 0.9, 1.2, 1.0, 30.0, 1.1, 0.8, 1.3, 1.0]
Mean:  3.94s   ← misleading (a user waited 30s!)
P50:   1.05s   ← honest: the median
P95:   30.0s   ← honest: the worst 5%

Harness design

Five explicit design decisions, all important:

1. Warm-up before measuring. The first request to a provider with a cold start (Modal, Ollama just started) has anomalous latency. Send 3-5 dummy requests first, ignore those.

2. Sample size of 50-100. Fewer and the percentiles aren't stable. More is overkill (takes a very long time, especially with local Ollama).

3. Prompts representative of your case, not generic ones. Benchmarking "Hi, how are you?" doesn't predict the behavior with a 2000-token RAG prompt. Use 3-5 typical prompts from your product.

4. Same amount of output tokens. If OpenAI generates 50 tokens and Modal 500, the second "takes longer" but not because it's slower, but because it generates more. Fix max_tokens equal.

5. Measure wall-clock time, not server time. What matters to the user is from "I send a request" to "I receive a response", including the network. Use time.perf_counter() around the full HTTP call.

Implementation: modular harness

Create benchmark_latency.py:

# benchmark_latency.py
import os
import time
import json
import statistics
from dataclasses import dataclass, field
from typing import Callable

# ============================================================
# Data models
# ============================================================
@dataclass
class BenchmarkResult:
    provider: str
    model: str
    samples: int
    p50: float
    p95: float
    p99: float
    max_: float
    errors: int
    raw_latencies: list[float] = field(default_factory=list)


# ============================================================
# Representative prompts (adjust to your real case!)
# ============================================================
PROMPTS = [
    "Explain REST in 3 sentences.",
    "Summarize the advantages of PostgreSQL over MongoDB for transactional data.",
    "What is prompt injection and how is it mitigated?",
    "Write a Python function that validates emails with regex.",
    "Compare Docker and Kubernetes in terms of when to use each one.",
]


# ============================================================
# Measurement helpers
# ============================================================
def percentile(values: list[float], p: int) -> float:
    if not values:
        return 0.0
    ordered = sorted(values)
    idx = int(len(ordered) * p / 100)
    return ordered[min(idx, len(ordered) - 1)]


def measure_latency(
    name: str,
    model: str,
    invoke: Callable[[str], None],
    samples: int = 50,
    warmup: int = 3,
) -> BenchmarkResult:
    """
    Measure the latency of a provider.

    `invoke(prompt)` is a callable that calls the provider.
    Its latency is measured; its response is discarded.
    """
    print(f"\n→ Benchmarking {name} ({model})")
    print(f"  Warmup: {warmup} requests (ignored)")

    # Warm-up
    for i in range(warmup):
        try:
            invoke(PROMPTS[i % len(PROMPTS)])
        except Exception as e:
            print(f"  Warmup error #{i}: {e}")

    print(f"  Measuring {samples} samples...")
    latencies = []
    errors = 0
    for i in range(samples):
        prompt = PROMPTS[i % len(PROMPTS)]
        start = time.perf_counter()
        try:
            invoke(prompt)
            latency = time.perf_counter() - start
            latencies.append(latency)
        except Exception as e:
            errors += 1
            print(f"  Sample #{i} error: {e}")
        if (i + 1) % 10 == 0:
            print(f"    {i + 1}/{samples} done")

    return BenchmarkResult(
        provider=name,
        model=model,
        samples=len(latencies),
        p50=percentile(latencies, 50),
        p95=percentile(latencies, 95),
        p99=percentile(latencies, 99),
        max_=max(latencies) if latencies else 0,
        errors=errors,
        raw_latencies=latencies,
    )


# ============================================================
# Adapters per provider
# ============================================================
MAX_TOKENS = 150  # Fixed for a fair comparison

def openai_adapter(prompt: str):
    from openai import OpenAI
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=MAX_TOKENS,
    )


def openrouter_adapter(prompt: str):
    from openai import OpenAI
    client = OpenAI(
        base_url="https://openrouter.ai/api/v1",
        api_key=os.environ["OPENROUTER_API_KEY"],
    )
    client.chat.completions.create(
        model="mistralai/mistral-7b-instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=MAX_TOKENS,
    )


def ollama_adapter(prompt: str):
    import httpx
    httpx.post(
        "http://localhost:11434/api/chat",
        json={
            "model": "mistral",
            "messages": [{"role": "user", "content": prompt}],
            "stream": False,
            "options": {"num_predict": MAX_TOKENS},
        },
        timeout=120,
    ).raise_for_status()


def modal_adapter(prompt: str):
    import httpx
    httpx.post(
        os.environ["MODAL_BASE_URL"] + "/chat",
        headers={"Authorization": f"Bearer {os.environ['MODAL_API_TOKEN']}"},
        json={"prompt": prompt, "max_tokens": MAX_TOKENS},
        timeout=120,
    ).raise_for_status()


# ============================================================
# Runner
# ============================================================
def main():
    results: list[BenchmarkResult] = []

    if os.environ.get("OPENAI_API_KEY"):
        results.append(
            measure_latency("OpenAI", "gpt-4o-mini", openai_adapter)
        )
    if os.environ.get("OPENROUTER_API_KEY"):
        results.append(
            measure_latency(
                "OpenRouter", "mistral-7b-instruct", openrouter_adapter
            )
        )
    if os.environ.get("OLLAMA_BASE_URL") or _is_ollama_running():
        results.append(
            measure_latency("Ollama (local)", "mistral", ollama_adapter)
        )
    if os.environ.get("MODAL_BASE_URL"):
        results.append(
            measure_latency("Modal", "mistral-7b-instruct-v0.3", modal_adapter)
        )

    # Table
    print("\n\n=== RESULTS ===\n")
    print(f"{'Provider':<20} {'Model':<28} {'P50':>7} {'P95':>7} {'P99':>7} {'Max':>7} {'Err':>5}")
    print("-" * 84)
    for r in results:
        print(
            f"{r.provider:<20} {r.model:<28} "
            f"{r.p50:>6.2f}s {r.p95:>6.2f}s {r.p99:>6.2f}s "
            f"{r.max_:>6.2f}s {r.errors:>5}"
        )

    # Export to JSON
    with open("latency_results.json", "w") as f:
        json.dump(
            [
                {
                    "provider": r.provider,
                    "model": r.model,
                    "samples": r.samples,
                    "p50": r.p50,
                    "p95": r.p95,
                    "p99": r.p99,
                    "max": r.max_,
                    "errors": r.errors,
                }
                for r in results
            ],
            f,
            indent=2,
        )
    print("\n→ Results saved to latency_results.json")


def _is_ollama_running() -> bool:
    import httpx
    try:
        httpx.get("http://localhost:11434/api/tags", timeout=2)
        return True
    except Exception:
        return False


if __name__ == "__main__":
    main()

Run

Configure the API keys / URLs you want to benchmark:

export OPENAI_API_KEY=sk-...
export OPENROUTER_API_KEY=sk-or-...
# Ollama runs on localhost; it needs no config if it's active
export MODAL_BASE_URL=https://your-user--llm-api-final-web.modal.run
export MODAL_API_TOKEN=...

pip install openai httpx
python benchmark_latency.py

Expected output (illustrative numbers, you'll get your own):

→ Benchmarking OpenAI (gpt-4o-mini)
  Warmup: 3 requests (ignored)
  Measuring 50 samples...
    10/50 done
    ...

=== RESULTS ===

Provider             Model                           P50     P95     P99     Max   Err
------------------------------------------------------------------------------------
OpenAI               gpt-4o-mini                   1.42s   2.31s   3.18s   3.20s     0
OpenRouter           mistral-7b-instruct           2.14s   3.05s   4.42s   4.51s     1
Ollama (local)       mistral                       4.21s   5.93s   6.85s   7.12s     0
Modal                mistral-7b-instruct-v0.3      1.95s   2.74s  18.34s  19.20s     0

How to read the results

Look at patterns, not absolute numbers:

OpenAI: low P50, P99 close to P50 → consistent. Good product for predictable latency.

OpenRouter: Similar to OpenAI but a step slower. Latency added by the proxy layer.

Ollama (local): higher P50 because your hardware (local CPU/GPU) is less powerful than the providers', but consistent (P50→P99 close) because you don't share infrastructure with anyone.

Modal: competitive P50, very high P99 — the worst 1% is off the charts. Why? Cold starts. If your Modal doesn't have min_containers=1, some requests hit a cold container and pay 18-30s. It's exactly the pattern you diagnose with percentiles.

This is worth more than the table: understanding why Modal's P99 is high tells you what to configure (warm pool, snapshot) to fix it.

Benchmark variations

Variation 1 — Different prompt length. Your real product handles prompts of various sizes. Modify PROMPTS to include short prompts (50 chars) and long ones (2000 chars). You'll see that some providers degrade more than others with long prompts.

Variation 2 — Concurrency. The previous benchmark is sequential: 1 request at a time. Add concurrency (with asyncio or concurrent.futures) to measure P95 under 10 parallel requests. This reveals throughput limitations that the sequential benchmark hides.

Variation 3 — Different times of day. Latency varies by hour (providers have uneven traffic). Run the benchmark at 10am and at 11pm your time; compare. Sometimes there are differences of 30%+.

Common traps

Trap 1 — "I measured over 10 minutes. Result: OpenAI won." 10 minutes is a very small window. OpenAI's degradation from congestion is erratic. For reliable data, run the benchmark 3 times at different moments and combine.

Trap 2 — "I didn't warm up and Modal came out terrible." Without warmup, the first request to Modal can be 60s (full cold start). That drags the average. The harness above already has warmup=3 — leave it active.

Trap 3 — "I used ridiculously short prompts." "Hi" as a prompt doesn't predict your real case. If your product sends prompts of 1500 characters, use prompts of 1500 characters in the benchmark.

Trap 4 — "I compare latency but the models are different." GPT-4o vs Mistral 7B isn't a comparison of infrastructure — it's a comparison of model + infrastructure. Be explicit in your report: "OpenAI serves gpt-4o-mini at 1.4s P50; OpenRouter serves Mistral 7B at 2.1s P50". Don't say "OpenAI is faster than OpenRouter" in the abstract.

Trap 5 — "My laptop benchmarks Ollama on an integrated GPU." If your Ollama runs on a MacBook M2 with 16GB unified memory, the numbers will be very different from Ollama on a machine with an A100. Report your hardware.

Exercise

Modify the harness to:

1. Accept the --samples N argument via CLI (instead of hardcoding 50)
2. Accept --prompts file.txt to load prompts from a file (one per line)
3. Print, in addition to percentiles, the minimum latency (P0) and the coefficient of variation (std/mean) to show consistency

import argparse

def parse_args():
    p = argparse.ArgumentParser()
    p.add_argument("--samples", type=int, default=50)
    p.add_argument("--prompts", type=str, help="File with one prompt per line")
    return p.parse_args()


def load_prompts(path: str | None) -> list[str]:
    if not path:
        return PROMPTS
    with open(path) as f:
        return [line.strip() for line in f if line.strip()]


# In main():
args = parse_args()
prompts = load_prompts(args.prompts)
# ... pass `samples=args.samples` and `prompts=prompts` to the harness

# In measure_latency, add:
min_ = min(latencies) if latencies else 0
cv = statistics.stdev(latencies) / statistics.mean(latencies) if len(latencies) > 1 else 0

# In the print, add Min and CV columns

Summary

You learned:

• ✅ Percentiles (P50/P95/P99) tell the real story, not the average
• ✅ Modular harness with adapters per provider (same structure, different client)
• ✅ Five design decisions: warmup, sample size, representative prompts, fixed tokens, wall-clock
• ✅ Interpreting results: cold start (Modal high P99), throughput (degradation with concurrency)

Checkpoint: if you have a JSON with percentiles of at least 2 providers and you understand what your P99 says about each one, you're ready.

Next capsule

03 — Cost benchmark. Latency is half the trade-off; cost is the other half. We're going to calculate the real cost per request for each provider at three different traffic scales, and see when the economic winner changes.

Resources

1. Brendan Gregg — Latency Heat Maps — visualizing latency beyond percentiles.
2. The Tail at Scale (Dean, Barroso) — classic paper on why P99 matters more than P50.
3. OpenAI API status — check if the provider was degraded during your benchmark.
4. HDR Histogram — tool for high-resolution latency distributions.