Reporting improvements with `BENCHMARKS.md`

Capsule description

Measuring well is half the job. The other half is communicating what you measured in an honest, reproducible, and actionable way. This capsule is the shortest one in code and the longest in professional judgment — you'll learn the discipline of writing performance reports that an auditor (a senior colleague, a CTO, your future self) accepts without objection.

Poorly reported benchmarks are like the misleading charts in the press: they technically "don't lie", but they lead the reader to the wrong conclusion. Your job is to write reports that lead the reader to the right conclusion without tricks: the before/after table anyone can understand and reproduce, the explicit caveats, the metrics chosen with judgment.

By the end of this capsule, you'll have the canonical BENCHMARKS.md template you'll use for the rest of the guide, and you'll know how to spot (in your own work and in others') the most common anti-patterns of misleading reports.

What is a good benchmark report?

Three properties:

1. Reproducible. Someone else, on another machine, can repeat the experiment and compare.
2. Honest. It reports what was measured as-is, including what didn't improve or got worse. It doesn't cherry-pick "the good runs".
3. Actionable. It leads the reader to a clear decision: approve the change, revert, or investigate further.

A poorly reported benchmark fails at least one. Let's look at the three most common failure patterns.

Anti-pattern 1: the "press release"

🚀 NEW OPTIMIZATION!! 🚀

After a week of work, we optimized the /products endpoint
and it's now **3x faster**!

We look forward to your applause!

What's missing?

• It doesn't say which tool was used.
• It doesn't say which metric improved 3x. Average? p99? Throughput?
• It doesn't say under what load (concurrency).
• It doesn't say versus which baseline (commit, date, context).
• It doesn't mention whether anything got worse (tail latency, throughput, error rate).
• It has a marketing tone, not an engineering one.

Diagnosis: "technically true" but communicationally empty. Impossible to refute, impossible to reproduce. Useful for demos to non-technical stakeholders, useless for deciding whether to merge the PR.

Anti-pattern 2: the context-free report

## Optimization of the /products endpoint

| Metric | Before | After |
|--------|--------|-------|
| Latency | 850ms | 200ms |

✅ 76% improvement

Better than the press release, but still bad:

• Is "latency" the average, the median, p95?
• Was the measurement done with wrk? locust? time curl?
• How many runs? With warmup?
• Were the "Before" and the "After" measured on the same machine with the same version of PostgreSQL?
• Did throughput hold up? An optimization that improves individual latency but kills throughput can be a regression in disguise.

Diagnosis: it looks like a serious report but it isn't reproducible. In three months, when someone wants to know "where did you get those numbers?", the answer is "I don't remember".

Anti-pattern 3: the cherry-picked report

## Optimization of the /products endpoint

After the change, p99 improved from 2,400ms to 1,850ms (-23%).

Suspicious. Here's why:

• Why only p99? What about p50, p95?
• Why only -23% when the capsule's promise was to "improve dramatically"?
• Reporting ONE percentile instead of all three can hide that p50 got worse.
• How many runs? A single run can show -23% from random noise.

Diagnosis: possibly honest, possibly cherry-picked. The way to avoid suspicion is to always report all three percentiles + throughput + error rate, even if some didn't improve or got worse.

The canonical BENCHMARKS.md template

This is the template you'll use for the rest of the guide. It's exhaustive on purpose — the cost of over-documenting is low; the cost of not documenting enough and needing to reproduce is high.

# BENCHMARKS — [project/repo name]

## Environment context

- **Hardware (client — measurement tool):** [CPU, RAM, model]
- **Hardware (server — where the app runs):** [CPU, RAM, model]
- **OS:** [exact version]
- **PostgreSQL:** [exact version, e.g.: 16.2]
- **Python:** [exact version]
- **Main framework:** [FastAPI X.Y, SQLAlchemy X.Y, etc.]
- **DB driver:** [asyncpg X.Y / psycopg X.Y]
- **App server:** [`uvicorn --workers N`]
- **Pool config:** `pool_size=N, max_overflow=M`
- **Topology:** [app and DB on the same host / DB on another VM / estimated RTT]
- **Other loads on the machine:** [Slack open / nothing]

## Data

- **Table X:** N rows
- **Table Y:** N rows
- **Data generation:** [script `seed.py`, commit hash `abc123`]
- **Distribution:** [if applicable — e.g.: "real skew: top 10 authors have 50% of the books"]

## Methodology

- **Tool:** [`wrk` 4.2.0 / `locust` 2.27.0 / `pgbench` 16.2]
- **Exact command:** `<full command with all flags>`
- **Warmup:** [discarded duration]
- **Runs:** [N runs]
- **Concurrency:** [N connections / virtual users]
- **Duration per run:** [seconds]
- **Aggregation across runs:** [median of the percentiles / etc.]

## Baseline (Module 1) — date: [YYYY-MM-DD], commit: [hash]

| Endpoint | p50 | p95 | p99 | Sustained RPS | Errors |
|----------|-----|-----|-----|---------------|--------|
| `GET /endpoint1` | 45ms | 220ms | 850ms | 980 | 0% |
| `GET /endpoint2` | ... | ... | ... | ... | ... |

**Observations:**
- [Observation 1: what caught your attention, what hypothesis you have]
- [Observation 2]

---

## After Module X — [name of the applied change] — date: [YYYY-MM-DD], commit: [hash]

### Applied change
[Concrete description of the change. Example: "Added composite index (author_id, published_year) on books"]

### Results

| Endpoint | p50 baseline | p50 after | Δ p50 | p95 baseline | p95 after | Δ p95 | p99 baseline | p99 after | Δ p99 | RPS baseline | RPS after | Δ RPS |
|----------|--------------|-----------|-------|--------------|-----------|-------|--------------|-----------|-------|--------------|-----------|-------|
| `GET /endpoint1` | 45ms | 32ms | -29% | 220ms | 95ms | -57% | 850ms | 110ms | -87% | 980 | 1,420 | +45% |

### Observations
- [What improved, what didn't, what got worse]

### Caveats
- [If the methodology changed relative to the baseline]
- [If the dataset changed in size]
- [Anything that affects comparability]

Why each section matters

• Environment context: without this, it isn't reproducible. Period.
• Data: same data size = valid comparison. If the dataset grew between baseline and after, you declare it.
• Methodology: the tool and the exact command. Without this, you can't repeat it.
• Table with baseline + after side by side: the difference is seen, not inferred. The Δ column keeps the reader from doing mental math.
• Observations: the qualitative part. What caught your attention. Hypotheses to investigate.
• Caveats: if something changed that affects the comparison, you say so explicitly. This is what separates an honest report from a cherry-picked one.

The golden rule: measure the SAME thing before and after

For a before/after comparison to be valid, everything must be equal except the change you applied:

• ✅ Same machine, same PostgreSQL config.
• ✅ Same dataset (same tables, same rows, same distribution).
• ✅ Same measurement tool with the same parameters (-c 50 -t 4 -d 60s --latency).
• ✅ Same warmup, same number of runs.
• ✅ Same OS load (close Slack, browser, etc.).

If anything changes beyond the change you're measuring, the numbers aren't comparable — it's comparing apples to oranges.

Example of an invalid comparison

Baseline (on MacBook Pro M1, PostgreSQL 14, 100k rows):
  p99 = 850ms

After (on MacBook Pro M2 Max, PostgreSQL 16, 1M rows):
  p99 = 200ms

Did it improve because of the change? Or because of the faster machine? Or did it get worse because there's now 10x more data but the numbers improved thanks to the hardware? Impossible to tell. The report is useless even though the numbers are real.

How to fix it: measure baseline and after under identical conditions (same machine, same PG version, same dataset). If any condition changed, declare it in "Caveats" and treat the comparison as merely indicative.

How to write the observations

Observations are where your professional judgment shows. Some useful patterns:

Pattern 1: what improved + magnitude + why (hypothesis)

- p95 of `/books?author=X` dropped 92% (2,100ms → 168ms). Confirms the module 4 hypothesis: the N+1 was responsible for the latency.

Pattern 2: what did NOT improve (important to report)

- `/orders?page=5000` didn't improve with the applied optimization (4,200ms → 4,180ms). The composite index we added doesn't apply to this endpoint, which suffers from a large OFFSET (module 8). Expected: this endpoint was NOT a target of this module.

Pattern 3: what got worse (the most important to report)

- p99 of `/books?author=X` improved 87%, but sustained throughput dropped 12% (1,420 → 1,250 RPS). Hypothesis: the change added overhead on queries that don't use the index. Investigate whether the trade-off is worth it or whether it can be mitigated.

Pattern 4: caveats that affect the conclusion

- **Caveat:** the baseline dataset was 100k books. For this after we regenerated to 100k books but the author distribution changed slightly. A re-measurement with a bit-exact dataset is pending. Confidence in the delta: medium-high.

The critical part: observations aren't a formality. They're where the reader understands what the change means. A report without observations is just a table — useful but not complete.

Caveats: the art of declaring limitations honestly

Caveats are the section that best demonstrates technical honesty. Typical categories:

Measurement caveats

• "Only 3 runs per level; variance not statistically characterized."
• "Coordinated omission not corrected (we used wrk, not wrk2)."
• "p99 may be underestimated if locust's sampling doesn't capture every request under high load."

Environment caveats

• "Measured with app and DB on the same host. Production has a ~3ms RTT between app and DB; the real numbers in production will be proportionally higher for chatty endpoints."
• "Client hardware is the same machine as the server — CPU contention."

Data caveats

• "Synthetic dataset generated with seed.py. The real production distribution has a top 10 authors with 60% of the volume, not replicated here."
• "100k books is 1/100 of production scale."

Scope caveats

• "We only measured read endpoints. The write endpoints (POST/PUT/DELETE) were not part of this baseline."
• "PgBouncer is NOT in the measurement path; production does have it."

Why does this matter? Because it protects you. When the result in production isn't exactly the same as in your baseline, you'll have documented why. And when a senior colleague reviews your report, they'll see that you thought about these things — that builds credibility.

Worked example: a complete report

Imagine you applied an optimization in module 4 (eliminating N+1 with selectinload). Here's how you report it:

# BENCHMARKS — Bookstore API

## Environment context

- **Hardware (client):** MacBook Pro M2 Max, 32GB RAM
- **Hardware (server):** same machine (localhost)
- **OS:** macOS 14.5
- **PostgreSQL:** 16.2 (local install via brew)
- **Python:** 3.12.3
- **FastAPI:** 0.110.2
- **SQLAlchemy:** 2.0.30 (async, asyncpg driver)
- **Driver:** asyncpg 0.29.0
- **App server:** `uvicorn --workers 4`
- **Pool config:** `pool_size=20, max_overflow=10`
- **Topology:** everything on localhost
- **Other loads:** Slack and browser closed during measurements

## Data

- `books`: 100,000 rows
- `authors`: 5,000 rows
- `reviews`: 500,000 rows
- Generated with `bench/seed.py`, commit `seed-v2-abc123`

## Methodology

- **Tool:** `wrk` 4.2.0
- **Command:** `wrk -t4 -c50 -d60s --latency <url>`
- **Warmup:** 30s discarded before each run
- **Runs:** 5 runs of 60s each
- **Aggregation:** median of the percentiles across the 5 runs

## Baseline (Module 1) — 2026-05-15, commit `bookstore-baseline`

| Endpoint | p50 | p95 | p99 | Sustained RPS | Errors |
|----------|-----|-----|-----|---------------|--------|
| `GET /books` | 45ms | 220ms | 850ms | 980 | 0% |
| `GET /books?author=tolkien` | 180ms | 2,100ms | 8,400ms | 320 | 0% |
| `GET /books/<id>` | 12ms | 35ms | 95ms | 4,200 | 0% |

**Observations:**
- `/books?author=tolkien` shows a suspicious bimodal distribution (p99 / p50 ≈ 47x). Hypothesis: a hidden N+1 — confirm with `pg_stat_statements` (module 5).
- `/books/<id>` is healthy; not an optimization target.
- We didn't measure write endpoints because the broken app is read-heavy.

---

## After Module 4 — eager loading on `/books?author=X` — 2026-05-22, commit `n+1-fix-def456`

### Applied change
On the `/books?author=X` endpoint, we replaced the default `lazy="select"` with
`selectinload(Book.reviews)` + `joinedload(Book.author)`. It reduces from N+1 queries
(1 for books + 1 per author + N per reviews) to 2 queries total.

### Results

| Endpoint | p50 base | p50 after | Δ p50 | p95 base | p95 after | Δ p95 | p99 base | p99 after | Δ p99 | RPS base | RPS after | Δ RPS |
|----------|---------|-----------|-------|----------|-----------|-------|----------|-----------|-------|----------|-----------|-------|
| `GET /books` | 45ms | 47ms | +4% | 220ms | 235ms | +7% | 850ms | 870ms | +2% | 980 | 950 | -3% |
| `GET /books?author=X` | 180ms | 35ms | -81% | 2,100ms | 168ms | -92% | 8,400ms | 280ms | -97% | 320 | 1,850 | +478% |
| `GET /books/<id>` | 12ms | 12ms | 0% | 35ms | 34ms | -3% | 95ms | 92ms | -3% | 4,200 | 4,250 | +1% |

### Observations
- ✅ **`/books?author=X` improved dramatically.** p99 dropped 97% (8,400ms → 280ms) and RPS rose ~5.8x. Confirms the baseline hypothesis.
- ⚠️ **`/books` (full listing) got slightly worse** (~5-7% in latency, -3% in throughput). Eager loading added overhead to queries that don't need it as much. Action: apply it selectively, not to every relationship — pending for a refactor.
- ✅ **`/books/<id>` was not affected** (expected, that endpoint doesn't use the loaded relationship).

### Caveats
- Same dataset, same config, same tool. Valid comparison.
- 5 runs per endpoint. Variance across runs <5% for all reported percentiles.
- The `/books` delta (+4-7%) is close to the measurement noise floor — confirm with a re-run.
- Production has PgBouncer in transaction mode, not present in this baseline. The exact delta may vary +/-10% in production.

Why this report is well made:

• Exhaustive, replicable context.
• Table with baseline + after side by side, explicit Δ columns.
• It reports what improved AND what got worse (it doesn't hide the +5% on /books).
• Real, useful caveats, not formalities.
• Observations that lead to a decision: approve the change (a dramatic improvement justifies the minor cost on another endpoint) and plan a refactor to mitigate the regression.

Visualizations: tables vs charts

For BENCHMARKS.md the rule is Markdown tables. Reasons:

• ✅ Version-controllable in git, diff-able.
• ✅ They render in GitHub/GitLab without assets.
• ✅ Copyable to Slack/Notion/email without friction.
• ✅ Searchable (Ctrl+F finds "p99").

Charts have their place:

• 📊 For presentations to non-technical stakeholders.
• 📊 For live production dashboards (Grafana, Datadog).
• 📊 For latency distributions (histograms) that are hard to visualize in a table.

But in BENCHMARKS.md, tables win. If you need to show a distribution, you can generate an image and reference it, but the source of truth remains the table.

Reporting when there was NO improvement (also important)

A less-taught but equally common situation: you applied a change and it didn't move the needle. How do you report that?

## After Module X — [change that did NOT work] — date, commit

### Applied change
[Description]

### Results

| Endpoint | p50 base | p50 after | Δ p50 | p95 base | p95 after | Δ p95 | ... |
|----------|---------|-----------|-------|----------|-----------|-------|-----|
| `GET /target` | 220ms | 218ms | -1% | 580ms | 590ms | +2% | ... |

### Observations
- ❌ The change did NOT improve latency significantly. The hypothesis was that [X caused the problem], but the numbers don't confirm it.
- Investigate another hypothesis before reverting or continuing.

### Decision
[Revert / Keep but look for the next optimization / Other]

Why reporting failures matters:

• You document which hypotheses were already ruled out — you save time for your future self and colleagues.
• It builds credibility: someone who only reports successes raises suspicion, someone who reports failures honestly is trustworthy.
• It's the scientific mindset applied to engineering: hypothesis → experiment → result, regardless of the sign of the result.

Why does this matter in real work?

1. PRs approved without argument. When your optimization PR comes with an updated, rigorous BENCHMARKS.md with an honest before/after, the reviewer doesn't need to fight you over whether it improved. The data is right there. The PR gets approved in one pass.

2. Serious postmortems. When something fails in production and you need to show you did your job, your version-controlled BENCHMARKS.md is evidence: "here's the pre-deploy baseline, here are the canary numbers, here's the delta — the regression is measured and tracked".

3. Justifying the time investment. "I spent two weeks optimizing" is vague. "I spent two weeks optimizing, p99 of the main endpoint dropped from 8s to 280ms (-97%), throughput rose 5.8x" is concrete. The second justifies raises, promotions, budgets.

4. Knowledge transfer. When someone new joins the team and wonders "why is /books?author implemented with selectinload?", the commit with its updated BENCHMARKS.md answers: "because the N+1 had put p99 at 8s and this change brought it down to 280ms". The "why" of the code lives in the benchmarks.

Traps and common mistakes

Mistake 1 (conceptual): comparing against a baseline that's no longer valid

Symptom: "The baseline had p99=850ms six months ago. Now we measure p99=600ms with the change. -29%, big improvement."

Why it's problematic: in 6 months the dataset changed (more rows), the PostgreSQL version, the Python version, code in other parts you don't control. The baseline is no longer comparable.

How to fix it: re-measure the baseline right before applying the change (on the same pre-change commit, in the same environment). Only then does the delta reflect the pure change.

Mistake 2 (practical): showing only the percentile that improved the most

Symptom: you report only p99 because it improved 50%, you hide that p50 improved only 5% and p95 got 10% worse.

Why it's problematic: it's cherry-picking. Another reader might think "if you only report p99, what happened to the others?". It generates well-founded distrust.

How to fix it: always report all three percentiles + throughput + error rate. If any got worse, you say so explicitly and explain why you think it happened. Honesty builds credibility.

Mistake 3 (conceptual): mixing a change + refactor + tweaks in a single benchmark

Symptom: "I refactored the endpoint, added an index, changed the ORM, and all together: p99 improved 70%."

Why it's problematic: you don't know which change was responsible for what. If something gets worse later, you can't revert a specific change — you have to revert everything.

How to fix it: ONE change per benchmark. You apply the index → measure → report. You apply the refactor → measure → report. You apply the ORM change → measure → report. More laborious, but it gives actionable data.

Mistake 4 (practical): not version-controlling BENCHMARKS.md

Symptom: BENCHMARKS.md is in .gitignore or you only keep it locally.

Why it's problematic: the history is lost. In 6 months you can't know how the endpoint was before the module 4 change. Each release is a black box.

How to fix it: BENCHMARKS.md lives in the repo, gets committed with every significant change. Ideally, part of CI: if the PR changes performance-relevant code, require the file to be updated.

Mistake 5 (conceptual): assuming better numbers always = better product

Symptom: "p99 improved 50%, let's merge." Without thinking about whether operational complexity increased.

Why it's problematic: a performance improvement that significantly increases complexity (a new piece, new config, a new failure mode) may not be worth it. The benchmark doesn't capture that.

How to fix it: the report's observations should include the operational cost. "This optimization adds a Redis cache that your team has to maintain. Is the p99 improvement from 850ms to 200ms worth the operational cost? — yes/no, justify it."

Exercises

Exercise 1: Identify problems in a real report

Read this report and list all the problems you find:

## Search module optimization

We improved search speed by 60%!

| Metric | Before | After |
|--------|--------|-------|
| Latency | 800ms | 320ms |

This was achieved by adding an index. Tests were done on my local machine with test data.

Exercise 2: Rewrite the report from Exercise 1

Take the report from the previous exercise and rewrite it applying the canonical template. Make up the missing but realistic data — the goal is to practice the format, not the numbers.

# BENCHMARKS — search-service

## Environment context

- **Hardware (client):** MacBook Pro M2, 16GB RAM
- **Hardware (server):** same machine (localhost)
- **OS:** macOS 14.5
- **PostgreSQL:** 16.2
- **Python:** 3.12.3, FastAPI 0.110.2
- **App server:** `uvicorn --workers 4`
- **Pool config:** `pool_size=10, max_overflow=5`
- **Topology:** everything on localhost

## Data

- `documents`: 250,000 rows
- `tags`: 8,500 rows
- Generated with `seed.py`, commit `seed-v1-xyz789`

## Methodology

- **Tool:** `wrk` 4.2.0
- **Command:** `wrk -t4 -c30 -d60s --latency http://localhost:8000/search?q=postgres`
- **Warmup:** 30s discarded
- **Runs:** 5 runs of 60s
- **Aggregation:** median of the percentiles

## Baseline — 2026-05-10, commit `pre-search-fix-aaa111`

| Endpoint | p50 | p95 | p99 | RPS | Errors |
|----------|-----|-----|-----|-----|--------|
| `GET /search?q=<x>` | 800ms | 1,400ms | 2,800ms | 35 | 0% |

**Observations:**
- High p99 and very low RPS (35) suggest a sequential scan on the search field.
- Hypothesis: a missing GIN index on `documents.content` (full-text search).

---

## After — add GIN index — 2026-05-12, commit `search-fix-bbb222`

### Applied change
Added `CREATE INDEX idx_docs_fts ON documents USING GIN (to_tsvector('english', content));`
Modified the query to use `@@ to_tsquery(...)` instead of `LIKE`.

### Results

| Endpoint | p50 base | p50 after | Δ p50 | p95 base | p95 after | Δ p95 | p99 base | p99 after | Δ p99 | RPS base | RPS after | Δ RPS |
|----------|---------|-----------|-------|----------|-----------|-------|----------|-----------|-------|----------|-----------|-------|
| `GET /search?q=<x>` | 800ms | 320ms | -60% | 1,400ms | 480ms | -66% | 2,800ms | 720ms | -74% | 35 | 145 | +314% |

### Observations
- p99 dropped 74%. Confirms the seq scan hypothesis.
- RPS rose 4x. The improvement is real and significant.
- p50 is still high (320ms) — investigate whether there's additional overhead in the app beyond the query.

### Caveats
- Same dataset, same environment, same tool. Valid comparison.
- The new index adds ~80MB on disk; acceptable.
- Write queries to `documents` are now ~10% slower (formal measurement pending). Acceptable given the app is read-heavy.

Exercise 3: Invent reasonable caveats

For each situation, write the caveat you would put in BENCHMARKS.md:

a) You measured with a 10k-row dataset; production has 5M. b) You measured with wrk (not wrk2), your app has a lot of variance. c) You measured without PgBouncer; production has it. d) You measured only 1 run because the deadline was pressing. e) The Mac model changed between the baseline and the after.

Exercise 4: Decide what to report when something gets worse

You applied an optimization and the results are:

                p50    p95    p99    RPS
Baseline:       80ms   200ms  450ms  1200
After change:   75ms   215ms  890ms  950

Do you approve the change? How do you report it?

## After Module X — [change Y] — 2026-05-15, commit `xyz`

### Applied change
[Description]

### Results

| Metric | Baseline | After | Δ |
|--------|----------|-------|---|
| p50 | 80ms | 75ms | -6% |
| p95 | 200ms | 215ms | +7% |
| p99 | 450ms | 890ms | **+98%** ⚠️ |
| RPS | 1,200 | 950 | **-21%** ⚠️ |

### Observations
- ✅ p50 improved 6% (marginal improvement).
- ❌ **p99 got 98% worse** and **RPS dropped 21%**. The improvement in the typical case doesn't make up for the deterioration in the tail and throughput.
- Hypothesis: the change added some kind of contention/serialization that benefits light loads but hurts concurrency.

### Decision
**REVERT the change.** The trade-off isn't favorable: a small gain in p50 vs significant regressions in p99 and RPS.

### Next steps
- Investigate which part of the change adds contention.
- Consider a variant that preserves the p50 improvement without affecting the tail.

Exercise 5: Create your personalized template

Create an empty BENCHMARKS.md file in a project of yours (real or for practice). Include only the "Environment context" and "Methodology" sections filled in with your real data. It's the first half of the baseline you'll produce in capsule 08.

# BENCHMARKS — my-fastapi-project

## Environment context

- **Hardware (client — measurement tool):** MacBook Air M1, 16GB RAM
- **Hardware (server — where the app runs):** same machine (localhost)
- **OS:** macOS 14.5
- **PostgreSQL:** 16.2 (via Homebrew)
- **Python:** 3.12.3
- **Main framework:** FastAPI 0.110.2
- **ORM:** SQLAlchemy 2.0.30 (async)
- **DB driver:** asyncpg 0.29.0
- **App server:** `uvicorn --workers 4`
- **Pool config:** `pool_size=20, max_overflow=10`
- **Topology:** app and DB on the same host (localhost)
- **Other loads on the machine:** none during measurements

## Data

(To fill in before the real baseline — depends on the dataset you use.)

## Methodology

- **Tools:** `wrk` 4.2.0 (HTTP) and `pgbench` 16.2 (pure DB)
- **Warmup:** 30s discarded before each run
- **Runs:** 5 runs of 60s each
- **Aggregation:** median of the percentiles across runs
- **Concurrency for the HTTP baseline:** -t4 -c50

(Continue with the Baseline and After sections once you have them.)

Summary and next step

In this capsule you learned:

• A good benchmark report is reproducible, honest, and actionable.
• Three classic anti-patterns: the press release (empty), the context-free report (not reproducible), the cherry-picking (hides what got worse).
• The canonical template has: environment context, data, methodology, baseline, after with a side-by-side table, observations, caveats.
• Golden rule: measure the SAME thing before and after. If something changes (dataset, hardware, tool), declare it explicitly or re-measure.
• Report what got worse as honestly as what improved. That builds credibility and keeps regressions disguised as improvements from making it into main.
• Report failures too — which hypotheses you ruled out is valuable information for your future self.

Before moving on you should be able to:

• Apply the canonical BENCHMARKS.md template to any change
• Detect at least the three anti-patterns (press release, no context, cherry-picking) in others' reports
• Write reasonable caveats instead of pretending the benchmark is perfect
• Decide to approve/revert a change based on a before/after table

Next capsule — Module project: the Bookstore API baseline. You have everything: the mindset, the percentiles, the reproducible baselines, the three tools (pgbench, wrk, locust), and the report template. Capsule 08 is where you apply it all to a concrete case: a simplified API with known problems. You'll produce the real BENCHMARKS.md that will be the baseline against which all the optimizations from modules 2 through 7 are compared.

Resources

1. Brendan Gregg — "Performance Engineering Communication" — how to communicate performance in team contexts.
2. Heinrich Hartmann — "Statistics for Engineers" — a reference on percentiles and honest aggregations.
3. Edward Tufte — The Visual Display of Quantitative Information — the classic on how to show data honestly. Applies directly to benchmark tables.
4. Datadog — "Best practices for monitoring and incident review" — modern patterns for how to report performance in production.
5. Marc Brooker — "Why benchmarks?" — on the discipline of honest benchmarking.
6. Honeycomb Engineering Blog — "Observability culture" — performance reports integrated into team culture.
7. Google SRE Workbook — Chapter on SLIs — how to choose metrics that reflect user experience, not ease of measurement.

Module 1 — Database Performance & Query Tuning Guide