Reproducible baselines

Capsule description

A baseline is the snapshot of "how your system behaves before touching anything". It's the line against which you'll compare every change. If that snapshot is badly taken — no context, no warmup, a single run with a warm cache, on localhost while Slack is eating CPU — then all your future comparisons are noise.

In this capsule you'll learn the discipline of building baselines that someone else (or you in three months) can reproduce and understand. It's the difference between a benchmark that's useful and a number in a screenshot that nobody knows how to interpret.

You'll build your first BENCHMARKS.md with the structure you'll reuse for the rest of the guide: environment context, explicit methodology, honest numbers. By the end, you'll have a Python script that measures latencies properly — with warmup, multiple runs, and percentiles aggregated correctly.

What a baseline is (and what it isn't)

A reproducible baseline has three mandatory properties:

1. Documented: someone who wasn't there can repeat the experiment.
2. Stable: if you run it twice in a row, the numbers vary little (controlled noise).
3. Representative: it reflects a realistic scenario, not a degenerate case.

Not a baseline:

• "I tested it on my machine and it's fast."
• A Datadog screenshot of the last 24h with no load context.
• The first number that comes out of wrk -d10s http://localhost:8000/.
• A single time curl.

That IS a baseline:

• "With PostgreSQL 16.2 running on a MacBook Pro M2 Max 32GB, a dataset of 100k rows in books and 500k in reviews, pool size 20, FastAPI app with uvicorn --workers 4, after 30s of warmup, across 5 runs of 60s with 50 concurrent connections, the median of the percentiles was: p50=45ms, p95=320ms, p99=850ms."

The second version is longer, yes. That's what separates "anecdote" from "evidence".

The 5 most common sources of noise (and how to control them)

1. Cold cache vs warm cache

PostgreSQL has shared_buffers. The operating system has a page cache. Your disk has a cache. The first time you run a query, everything is cold and it takes longer. The fifth time, everything is warm and it flies.

Run 1: 850ms  ← cold cache
Run 2: 240ms  ← warm page cache
Run 3: 180ms  ← warm shared_buffers
Run 4: 165ms  ← stable
Run 5: 162ms  ← stable

Solution: warmup. Run the benchmark without measuring for N seconds before you start measuring. Standardize: either everything is measured cold (representing your users' first hit), or everything is measured warm (representing sustained load). Mixing them biases the result.

2. Other things running on your machine

Slack, the browser with 60 tabs, Docker Desktop, Spotify, macOS indexing, a TypeScript build in another project. All of that steals CPU/IO and your numbers dance around.

Solution:

• Close everything non-essential.
• If you can, measure on a dedicated machine (server, VM, CI runner).
• Document what was running (Spotify open, Docker running PostgreSQL).

3. Warm connection pool vs cold pool

The first request of the benchmark opens N new connections to PostgreSQL. That takes time. The following ones reuse them.

Solution: after the warmup, the connections are already open. If your benchmark is very short (<10s), you may be mostly measuring the pool setup.

4. Non-representative data

You measured with 100 rows in a table that has 10M in production. PostgreSQL chooses different plans depending on table size — a sequential scan on 100 rows is optimal, on 10M it's a disaster.

Solution:

• Use datasets that are representative in size.
• If you can't (they don't fit locally), at least generate synthetic data that reproduces production's cardinality distribution (how many authors, how many reviews per book on average, etc.).
• Document the exact size: books: 100k rows, reviews: 500k rows, authors: 5k rows.

5. Local network vs real network

localhost adds ~0.05ms of latency. A real network between your app and the DB adds 1-5ms (same datacenter) or 50-200ms (different regions). If you measure on localhost and deploy with the DB in another region, your p50 will explode.

Solution:

• If the app and the DB run on the same host in production, measure on the same host. If they're separate, measure them separate (Docker Compose with the DB in another container helps).
• Document the topology: app and DB in the same Docker container, local bridge network.

The formula for an honest baseline

For each endpoint you measure:

1. WARMUP: run the scenario for 30-60s without measuring.
2. RUNS:   run the scenario N times (minimum 3, ideally 5).
3. PER RUN: measure and store p50, p95, p99 of THAT individual run.
4. AGGREGATION: report the MEDIAN of the percentiles across the N runs.
5. CONTEXT: document hardware, software, data, tool, parameters.

Critical point: NEVER average percentiles. The average of three p99s (200, 800, 250) is not 416ms — that "metric" means nothing mathematically. Report the median of the percentiles, or report all three values. Anything else is mathematically suspect. (This is one of the things Heinrich Hartmann insists on heavily in his "Statistics for Engineers" talks.)

Worked example: measuring an endpoint with discipline

You're going to measure a simple HTTP endpoint using only Python + httpx (the wrk tool comes in the next capsule). The goal: see how a baseline is built from first principles before learning specialized tools.

Setup

mkdir honest-baseline
cd honest-baseline
python -m venv venv
source venv/bin/activate
pip install httpx numpy

To have something to measure, spin up a minimal FastAPI app (in another terminal):

pip install fastapi uvicorn

# api_demo.py
from fastapi import FastAPI
import time
import random

app = FastAPI()

@app.get("/fast")
def fast():
    return {"ok": True}

@app.get("/slow")
def slow():
    # Simulates a variable operation: 50ms typical, 5% of the time 1-2s
    if random.random() < 0.05:
        time.sleep(random.uniform(1.0, 2.0))
    else:
        time.sleep(0.05)
    return {"ok": True}

Run it:

uvicorn api_demo:app --workers 1

A disciplined measurement script

# measure.py
import asyncio
import time
import statistics

import httpx
import numpy as np


URL = "http://localhost:8000/slow"
WARMUP_SECONDS = 10
RUN_SECONDS = 30
N_RUNS = 5
CONCURRENCY = 20


async def hit_endpoint(client: httpx.AsyncClient) -> float:
    """Makes one request and returns its latency in ms."""
    start = time.perf_counter()
    await client.get(URL)
    return (time.perf_counter() - start) * 1000


async def workload(duration_s: float) -> list[float]:
    """Generates concurrent load for `duration_s` seconds."""
    latencies: list[float] = []
    deadline = time.perf_counter() + duration_s

    async with httpx.AsyncClient(timeout=10.0) as client:

        async def worker():
            while time.perf_counter() < deadline:
                latencies.append(await hit_endpoint(client))

        await asyncio.gather(*(worker() for _ in range(CONCURRENCY)))

    return latencies


def report_run(label: str, latencies: list[float]) -> dict:
    arr = np.array(latencies)
    return {
        "label": label,
        "n": len(latencies),
        "p50": float(np.percentile(arr, 50)),
        "p95": float(np.percentile(arr, 95)),
        "p99": float(np.percentile(arr, 99)),
        "max": float(np.max(arr)),
    }


async def main():
    print(f"[warmup] {WARMUP_SECONDS}s discarded")
    await workload(WARMUP_SECONDS)

    runs = []
    for i in range(1, N_RUNS + 1):
        print(f"[run {i}/{N_RUNS}] measuring {RUN_SECONDS}s...")
        latencies = await workload(RUN_SECONDS)
        run = report_run(f"run {i}", latencies)
        runs.append(run)
        print(
            f"  n={run['n']}  p50={run['p50']:.0f}ms  "
            f"p95={run['p95']:.0f}ms  p99={run['p99']:.0f}ms  "
            f"max={run['max']:.0f}ms"
        )

    print("\n=== AGGREGATED (median across runs) ===")
    print(f"median p50: {statistics.median(r['p50'] for r in runs):.0f}ms")
    print(f"median p95: {statistics.median(r['p95'] for r in runs):.0f}ms")
    print(f"median p99: {statistics.median(r['p99'] for r in runs):.0f}ms")


if __name__ == "__main__":
    asyncio.run(main())

Run it:

python measure.py

Expected output (numbers vary by hardware):

[warmup] 10s discarded
[run 1/5] measuring 30s...
  n=11420  p50=51ms  p95=107ms  p99=1640ms  max=1980ms
[run 2/5] measuring 30s...
  n=11380  p50=50ms  p95=115ms  p99=1720ms  max=1992ms
[run 3/5] measuring 30s...
  n=11440  p50=51ms  p95=110ms  p99=1680ms  max=1875ms
[run 4/5] measuring 30s...
  n=11402  p50=51ms  p95=108ms  p99=1700ms  max=1944ms
[run 5/5] measuring 30s...
  n=11430  p50=51ms  p95=112ms  p99=1690ms  max=1968ms

=== AGGREGATED (median across runs) ===
median p50: 51ms
median p95: 110ms
median p99: 1690ms

Note: the exact numbers above are illustrative. Your hardware, Python version and system load will produce different numbers. What matters is the shape of the output: the runs are consistent with each other (low variance), and the percentiles (p50 and p99) tell very different stories, just as we'd expect from the simulated endpoint.

Why this script is well built

• ✅ Explicit warmup (10s discarded) — avoids measuring a cold cache and a cold pool.
• ✅ Multiple runs (5) — controls for the noise of an isolated run.
• ✅ Concurrent load (20 connections) — reflects a realistic scenario, not a sequential one.
• ✅ Percentiles per run, not by aggregating raw latencies — aggregating all latencies and computing p99 over that also works, but reporting the median of the per-run percentiles is more robust to corrupted runs.
• ✅ Reports the max — because sometimes it's informative (a big GC pause, a near-timeout).

What it's still missing

• ⚠️ It only measures latency. It doesn't measure sustained throughput, error rate, or behavior under increasing load. That's what wrk (capsule 05) and locust (capsule 06) are for.
• ⚠️ Concurrency is fixed at 20. To find the "knee" (the point where throughput stops scaling and latency explodes) you need to vary the concurrency.

That's why from capsule 04 onward you'll use dedicated tools. But understanding the principles from Python guarantees that when you use wrk, pgbench or locust, you know what they're doing underneath.

The BENCHMARKS.md template

Every time you measure seriously, you write it in BENCHMARKS.md. It's the document that lives in the repo, that your future self and your colleagues can read, and that gets updated every time you apply an optimization.

Minimum template:

# BENCHMARKS - Bookstore API

## Environment context

- **Hardware:** MacBook Pro M2 Max, 32GB RAM, internal SSD
- **OS:** macOS 14.5
- **PostgreSQL:** 16.2 (local install via `brew`)
- **Python:** 3.12.3
- **FastAPI:** 0.110.2
- **SQLAlchemy:** 2.0.30 (async, asyncpg driver)
- **Pool config:** `pool_size=20`, `max_overflow=10`
- **App server:** `uvicorn --workers 4`
- **Topology:** app and DB on localhost, no real network

## Data

- `books`: 100,000 rows
- `authors`: 5,000 rows
- `reviews`: 500,000 rows
- Generated with `seed.py` (commit hash: `abc123`)

## Methodology

- Tool: `wrk` 4.2.0
- Warmup: 30s discarded
- Runs: 5 runs of 60s each
- Concurrency: 50 connections, 4 threads
- Reporting: median of the percentiles across the 5 runs

## Baseline (Module 1) — date: 2026-05-15

| Endpoint | p50 | p95 | p99 | sustained RPS | Errors |
|----------|-----|-----|-----|---------------|--------|
| `GET /books` | 45ms | 220ms | 850ms | 980 | 0% |
| `GET /books?author=tolkien` | 180ms | 2,100ms | 8,400ms | 320 | 0% |
| `GET /orders?page=5000` | 4,200ms | 8,500ms | 12,400ms | 45 | 0.2% (timeouts) |
| `GET /stats/total-sales` | 11,800ms | 16,200ms | 19,500ms | 8 | 0.5% (timeouts) |

**Observations:**
- `/books?author=tolkien` shows a p99 ~46x its p50 → suspicious bimodal distribution, possible N+1.
- `/orders?page=5000` collapses with a large OFFSET (expected).
- `/stats/total-sales` is dying under COUNT(*) on a large table (expected).

## After Module X — date: ...

(To be filled in after each optimization module.)

What makes this template good?

• Exhaustive context. Versions, hardware, configuration. Reproducible.
• Metadata about the data. Table sizes, how they were generated (with a commit hash if it's a script).
• Explicit methodology. Tool, warmup, runs, concurrency. Someone else can repeat it.
• A table per endpoint. Each endpoint has its own line with percentiles, RPS and errors.
• Qualitative observations. Not just numbers — what stands out to you.
• Space for "After". The file grows with each module. You compare baseline vs each iteration.

What does NOT go in BENCHMARKS.md

• ❌ "It's faster" without numbers.
• ❌ A single run without warmup.
• ❌ "Average" latency without percentiles.
• ❌ Conclusions like "Postgres is slow" — that's an opinion, not a measurement.

How to deal with noisy runs

Sometimes, even with warmup and multiple runs, one run comes out clearly off-pattern:

Run 1: p99=1700ms
Run 2: p99=1680ms
Run 3: p99=4800ms   ← obvious outlier
Run 4: p99=1690ms
Run 5: p99=1710ms

What do you do with Run 3?

Option 1: discard it and document it. "Run 3 discarded, p99=4800ms — coincided with an automatic OS backup."

Option 2: investigate before discarding. Sometimes the "outlier" is real (a GC pause, connection contention, your code has an edge case that only appears sporadically). If it happens again on re-runs, it isn't noise — it's a symptom.

What you must NOT do:

• ❌ Discard runs silently.
• ❌ Report only "the good runs" without mentioning that you discarded any.
• ❌ Assume any outlier is noise without investigating.

Heuristic: if more than 1 out of every 5 runs is off-pattern, you don't have a stable baseline — you have a system with real problems you need to investigate before continuing.

Why does this matter in real work?

Three situations where a badly built baseline costs you:

1. Migration to another version / framework. You're going to migrate from PostgreSQL 14 to 16. Without a reproducible PG14 baseline, you can't prove PG16 improved things (or made them worse). If you just "tested it and it seems fine", whoever comes after you can't confirm anything — and if a regression shows up in production, you have no evidence.

2. A big refactor. Your team decides to rewrite the search module. Without a pre-refactor baseline, you can't demonstrate that the refactor didn't introduce a performance regression. "Subjectively it feels the same" isn't a valid defense in a postmortem.

3. Justifying investment. You ask the CTO for budget for a new DB / more infra / a month to optimize. Without honest numbers about the current state, everything is opinion. With a reproducible BENCHMARKS.md, you have evidence that justifies the investment and a metric to measure ROI afterward.

Traps and common mistakes

Mistake 1 (conceptual): "averaging percentiles"

Symptom: you run 5 runs, take the p99 of each, and compute the mean of those 5 values.

Why it's wrong: percentiles are not additive. Averaging the p99 of 5 runs doesn't produce a number with a clear statistical interpretation. The mean of p99s can be dragged by a single run with extreme outliers and does not represent typical behavior.

How to fix it: report the median of the runs' percentiles. Or aggregate ALL the latencies from ALL the runs into a single array and compute p99 over the combined array (this is mathematically valid, but it requires access to each run's raw latencies).

Mistake 2 (practical): not documenting the context

Symptom: "p95 was 200ms three months ago; now it's 350ms, why?".

Why it happens: the original baseline documented neither the PostgreSQL version, nor the data size, nor how many workers Uvicorn had. You have no way to reproduce the original experiment, so you can't tell whether the regression comes from your code or from changes in another layer (more data, a different config, a different library version).

How to fix it: every BENCHMARKS.md carries an exhaustive "Environment context" section. If you're unsure whether something matters, document it — the cost of documenting is low, the cost of not documenting and needing to reproduce is high.

Mistake 3 (practical): insufficient warmup

Symptom: the benchmark's first request takes 5,000ms, the rest take 50ms. Your p99 rises because of the cold start, not because of a real problem.

Why it happens: the connection pool wasn't open, PostgreSQL's shared_buffers was empty, Python's JIT compilation hadn't run. The first request pays for all of it.

How to fix it: run 30-60s of discarded warmup before you start measuring. The informal rule: warm up until two consecutive runs give similar numbers.

Mistake 4 (conceptual): "I tested it in production at 3am, it's fast"

Symptom: you measure during low-traffic hours and report those numbers as "the app's performance".

Why it's wrong: without real traffic (warm cache, busy pool, contention), the numbers don't reflect behavior under the load that actually matters. An app can be fine at 3am and collapse at 9am when the users show up.

How to fix it: production baselines are done with shadow traffic, mirroring, or against replicas with synthetic load that replicates the real pattern. In the context of this guide, we do them with wrk/locust generating representative load.

Exercises

Exercise 1: Spot a badly built baseline

A colleague shows you this "baseline":

"I tested the API with curl http://localhost:8000/products, it took 80ms. It's fine."

List the minimum 5 problems with this baseline and how you'd fix them.

Exercise 2: Apply the disciplined script

Take the measure.py script from the capsule and modify it to:

1. Measure the /fast endpoint (the one that does NOT have variable latency).
2. Also report min in addition to p50/p95/p99/max.
3. Validate that the difference between p50 and p99 is under 50ms (a sign of a consistent distribution).

# measure_fast.py
import asyncio
import time
import statistics

import httpx
import numpy as np


URL = "http://localhost:8000/fast"   # <-- we change the endpoint
WARMUP_SECONDS = 10
RUN_SECONDS = 30
N_RUNS = 5
CONCURRENCY = 20


async def hit_endpoint(client: httpx.AsyncClient) -> float:
    start = time.perf_counter()
    await client.get(URL)
    return (time.perf_counter() - start) * 1000


async def workload(duration_s: float) -> list[float]:
    latencies: list[float] = []
    deadline = time.perf_counter() + duration_s

    async with httpx.AsyncClient(timeout=10.0) as client:

        async def worker():
            while time.perf_counter() < deadline:
                latencies.append(await hit_endpoint(client))

        await asyncio.gather(*(worker() for _ in range(CONCURRENCY)))

    return latencies


def report_run(label: str, latencies: list[float]) -> dict:
    arr = np.array(latencies)
    return {
        "label": label,
        "n": len(latencies),
        "min": float(np.min(arr)),
        "p50": float(np.percentile(arr, 50)),
        "p95": float(np.percentile(arr, 95)),
        "p99": float(np.percentile(arr, 99)),
        "max": float(np.max(arr)),
    }


async def main():
    print(f"[warmup] {WARMUP_SECONDS}s discarded")
    await workload(WARMUP_SECONDS)

    runs = []
    for i in range(1, N_RUNS + 1):
        print(f"[run {i}/{N_RUNS}] measuring {RUN_SECONDS}s...")
        latencies = await workload(RUN_SECONDS)
        run = report_run(f"run {i}", latencies)
        runs.append(run)
        print(
            f"  n={run['n']} min={run['min']:.1f} p50={run['p50']:.1f} "
            f"p95={run['p95']:.1f} p99={run['p99']:.1f} max={run['max']:.1f}"
        )

    p50_med = statistics.median(r["p50"] for r in runs)
    p99_med = statistics.median(r["p99"] for r in runs)

    print("\n=== AGGREGATED (median across runs) ===")
    print(f"median p50: {p50_med:.1f}ms")
    print(f"median p99: {p99_med:.1f}ms")

    delta = p99_med - p50_med
    print(f"\nDelta p99 - p50: {delta:.1f}ms")
    if delta < 50:
        print("✅ Consistent distribution (delta < 50ms)")
    else:
        print("⚠️  Distribution with a tail — investigate")


if __name__ == "__main__":
    asyncio.run(main())

Exercise 3: Build your first BENCHMARKS.md

Take the output of Exercise 2 (with your real numbers), and write a minimal BENCHMARKS.md following the template. Make sure to include: environment context, data, methodology, and a table with the percentiles for the /fast endpoint.

# BENCHMARKS - Demo API

## Environment context

- **Hardware:** MacBook Air M1, 16GB RAM
- **OS:** macOS 14.5
- **Python:** 3.12.3
- **FastAPI:** 0.110.2
- **App server:** `uvicorn --workers 1`
- **Topology:** everything on localhost

## Data

N/A — the `/fast` endpoint doesn't touch the DB, it just returns static JSON.

## Methodology

- Tool: custom Python script (`measure_fast.py`)
- Warmup: 10s discarded
- Runs: 5 runs of 30s each
- Concurrency: 20 async connections
- Reporting: median of the percentiles across the 5 runs

## Baseline (Module 1, exercise) — date: 2026-05-15

| Endpoint | min | p50 | p95 | p99 | max | total n |
|----------|-----|-----|-----|-----|-----|---------|
| `GET /fast` | 0.4ms | 1.2ms | 2.8ms | 4.1ms | 18.5ms | 270,400 |

**Observations:**
- Consistent distribution (p99-p50 = 2.9ms < 50ms ✅).
- Max=18.5ms is likely a Python GC pause — investigate if it recurs.

Exercise 4: Detect noisy runs

You have the following p99s from 6 consecutive runs: [180, 175, 178, 182, 5400, 179]. What do you do with Run 5?

Exercise 5: Identify the missing context

Read this "baseline" and list what's missing:

Endpoint /products
p50: 120ms
p95: 450ms
p99: 1200ms
Taken last Monday.

Summary and next step

In this capsule you learned:

• A reproducible baseline documents context, methodology and numbers — not just numbers.
• The 5 main sources of noise: warm/cold cache, other things running, a cold pool, non-representative data, local vs real network.
• The honest formula: warmup → N runs → percentiles per run → median of the percentiles → exhaustive context.
• Never average percentiles — use the median of the percentiles, or aggregate raw latencies and compute p99 over the combined array.
• Outliers get investigated before they get discarded. If more than 1 in 5 runs is an outlier, your system has a real problem, not noise.
• BENCHMARKS.md is the document that lives in the repo and grows with each optimization module.

Before moving on you should be able to:

• List everything that goes in the "Environment context" section of a BENCHMARKS.md
• Defend why you discarded a specific run, with a documented root cause
• Distinguish natural noise (~5-10% variance between runs) from a real outlier

Next capsule — pgbench. You now know how to build well-made baselines in general. Now we go to the first specialized tool: pgbench, which ships with PostgreSQL and measures the database isolated from your app. It helps you answer "is the bottleneck the DB or is it my Python code?".

Resources

1. Heinrich Hartmann — "Statistics for Engineers" (article) — an essential technical reference on why you should NOT average percentiles.
2. Heinrich Hartmann — QCon talk "Statistics for Engineers" — the same material in talk format, 40 min.
3. Brendan Gregg — "USE Method" — a systematic methodology for identifying bottlenecks in any system.
4. PostgreSQL Documentation — Server Configuration: Resource Consumption — parameters that affect benchmark reproducibility (shared_buffers, work_mem).
5. Aphyr — "The trouble with timestamps" — why measuring time in distributed systems is subtler than it looks.
6. Andrei Alexandrescu — "Writing Quick Code in C++, Quickly" — a talk on microbenchmarking; although it's C++, the principles about warmup, noise and measurement are universal.
7. Marc Brooker — "Why are benchmarks so hard?" — a piece on how easy it is to fool yourself with "obvious" benchmarks.

Module 1 — Database Performance & Query Tuning Guide