Module 1: Performance Mindset & Benchmarking

`pgbench`: benchmarking raw PostgreSQL

Capsule description

When your API is slow there are two possible questions: is your Python app slow (FastAPI, SQLAlchemy, serialization, business logic) or is your database slow (heavy queries, missing indexes, lock contention, saturated IO)?

If you only measure the full API with wrk or locust, you can't answer that question. You're measuring the sum. To isolate the database you need to hit it directly, without going through your app — and that's what pgbench exists for, the official benchmarking tool that has shipped with PostgreSQL for decades.

In this capsule you'll install and master pgbench. You'll run the standard TPC-B-like benchmark, understand what it reports (TPS, average latency, latency stddev), and write custom SQL scripts to measure your app's specific queries. By the end you'll have pgbench integrated into your diagnostic kit, and you'll know exactly when to use it and when not to.


What pgbench is and what it's for

pgbench is the benchmarking tool that comes in the postgresql-contrib package (in many distros it comes in the main package). It is not a production load tool — it's a microbenchmark that measures how fast PostgreSQL executes specific queries under concurrency.

Mental model: where pgbench fits in your kit

                    ┌──────────────────────┐
   HTTP client →   │   FastAPI (Python)   │   ← wrk / locust measure this
                    │   ─ JSON serializ.   │     (the full API)
                    │   ─ ORM (SQLAlchemy) │
                    │   ─ conn. pool       │
                    └─────┬────────────────┘
                          │
                          ▼
                    ┌──────────────────────┐
                    │     PostgreSQL       │   ← pgbench measures only this
                    │   ─ planner          │     (the isolated DB)
                    │   ─ executor         │
                    │   ─ buffers / IO     │
                    └──────────────────────┘
  • wrk / locust: measure the sum — the whole stack from HTTP down to PostgreSQL and back.
  • pgbench: measures only PostgreSQL — from the connection to the query result.

If wrk reports p95=800ms and pgbench running the same query reports p95=20ms, you know the problem lives in your app (Python, ORM, serialization). If pgbench also reports 800ms, the problem lives in the DB.

That distinction is worth gold during diagnosis.

When to use pgbench

Do use it:

  • To get a baseline of "how fast PostgreSQL runs on this hardware with this config" — useful when changing machine, cloud provider, or PostgreSQL version.
  • To validate that a configuration change (e.g.: raising shared_buffers, adjusting work_mem) actually made an impact.
  • To measure your app's specific queries in isolation, comparing against the latency you see over HTTP.
  • To reproduce locks, tuple contention, or connection pool effects under synthetic load.

Don't use it:

  • To measure your full app (that's wrk/locust).
  • To replace EXPLAIN ANALYZE (module 2) when you want to understand why a query is slow — pgbench tells you how slow, not why.
  • For "stress testing" of real production (it generates very uniform synthetic traffic; production is usually bursty).

Installation

pgbench ships with PostgreSQL. If you have PostgreSQL installed, you probably already have it.

macOS (Homebrew)

# If you installed PostgreSQL with brew
brew install postgresql@16  # if you don't have it

# Verify
pgbench --version
# pgbench (PostgreSQL) 16.2

Linux (Debian/Ubuntu)

# Ships with postgresql-client or the server package
sudo apt-get install postgresql-contrib

pgbench --version
# pgbench (PostgreSQL) 16.x

Docker

If you have PostgreSQL running in Docker:

# Launch a container with PostgreSQL 16
docker run --name pg-bench-demo \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_USER=postgres \
  -p 5432:5432 \
  -d postgres:16

# Run pgbench from inside the container
docker exec -it pg-bench-demo pgbench --version

Verify the connection

Before moving on, make sure you can connect to a DB:

# Create a DB to experiment with
createdb pgbench_demo

# Verify
psql -d pgbench_demo -c "SELECT version();"

The standard benchmark: TPC-B-like

pgbench ships with a default benchmark based on TPC-B (an 80s standard for OLTP). It isn't perfect, but it's the "Hello World" of PostgreSQL benchmarks — everyone knows it, everyone understands it.

Step 1: initialize the dataset

# -i: initialize (creates the tables and loads data)
# -s 10: scale factor. Multiplies the table sizes (see below).
# Roughly 150MB on disk with -s 10
pgbench -i -s 10 pgbench_demo

Expected output:

dropping old tables...
creating tables...
generating data (client-side)...
1000000 of 1000000 tuples (100%) done (elapsed 1.23 s, remaining 0.00 s)
vacuuming...
creating primary keys...
done in 4.21 s (drop tables 0.01 s, create tables 0.02 s, client-side generate 1.50 s, vacuum 0.30 s, primary keys 2.38 s).

This created four tables. The proportions are based on TPC-B and all of them depend on the scale factor s:

TableRowsWith -s 10
pgbench_accounts100000 × s1,000,000
pgbench_tellers10 × s100
pgbench_branchess10
pgbench_history0 at start; grows with each transaction0

Notice the asymmetry: accounts is huge and branches is tiny (10 rows!). That's not an oversight — it's the point of the benchmark. Every TPC-B transaction updates a row in branches, so with -s 10 you have 10 rows fighting over all the concurrent writes. That lock contention is exactly what TPC-B wants to stress. If you raise the scale factor, contention drops and TPS rises — which is why comparing TPS across different scale factors is meaningless.

You can check it yourself:

psql -d pgbench_demo -c "
SELECT 'pgbench_accounts' AS table_name, count(*) FROM pgbench_accounts
UNION ALL SELECT 'pgbench_tellers',  count(*) FROM pgbench_tellers
UNION ALL SELECT 'pgbench_branches', count(*) FROM pgbench_branches;"
    table_name    |  count
------------------+---------
 pgbench_accounts | 1000000
 pgbench_tellers  |     100
 pgbench_branches |      10

Step 2: run the benchmark

# -c 10: 10 concurrent clients
# -j 2: 2 threads (informal rule: c >= j; jobs process clients in parallel)
# -T 30: run for 30 seconds
# -P 5: print partial results every 5 seconds
pgbench -c 10 -j 2 -T 30 -P 5 pgbench_demo

Expected output (the numbers are illustrative — they vary by hardware):

pgbench (16.2)
starting vacuum...end.
progress: 5.0 s, 1245.0 tps, lat 8.034 ms stddev 5.124, 0 failed
progress: 10.0 s, 1320.4 tps, lat 7.578 ms stddev 4.850, 0 failed
progress: 15.0 s, 1305.2 tps, lat 7.660 ms stddev 4.901, 0 failed
progress: 20.0 s, 1334.6 tps, lat 7.490 ms stddev 4.783, 0 failed
progress: 25.0 s, 1322.0 tps, lat 7.564 ms stddev 4.825, 0 failed
progress: 30.0 s, 1310.8 tps, lat 7.628 ms stddev 4.862, 0 failed
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 10
query mode: simple
number of clients: 10
number of threads: 2
maximum number of tries: 1
duration: 30 s
number of transactions actually processed: 39419
number of failed transactions: 0 (0.000%)
latency average = 7.604 ms
latency stddev = 4.875 ms
initial connection time = 6.122 ms
tps = 1313.875422 (without initial connection time)

How to read the output

MetricWhat it means
tps (transactions per second)The throughput. How many "transactions" (each one is a group of TPC-B queries) the DB processes per second.
latency averageAverage latency per transaction, in ms. Careful: it's an average, not a percentile — pgbench does not report percentiles by default (a classic limitation of the tool).
latency stddevStandard deviation of the latency. A large stddev relative to the average suggests a noisy distribution.
initial connection timeHow long it took to establish the first connections. This time is NOT included in TPS if you use --initial-connection-time correctly.
number of failed transactionsHow many transactions failed (deadlocks, timeouts). Ideally 0.

Important limitation: pgbench and percentiles

pgbench does not report p95/p99 by default. It reports average and stddev.

How do you get percentiles?

Option 1: detailed latency log. Use --log so pgbench writes the latency of every transaction to a file. Then you process the file with awk or Python.

# --log: writes one file per client with detailed latencies
pgbench -c 10 -j 2 -T 30 --log pgbench_demo

# Generates pgbench_log.<pid>.<n> files with lines like:
#   0 1 1999 0 1784057292 2785
# The fields are:
#   client_id  transaction_no  time  script_no  time_epoch  time_us
#
# ⚠️ The `time` field (index 2) is the transaction latency in
#    MICROSECONDS, not milliseconds. That's why you divide by 1000.
#    (`time_epoch` + `time_us` are the end timestamp, not the latency.)

Option 2 (recommended): process it with Python to get real percentiles.

# percentiles_pgbench.py
import sys
import glob
import numpy as np

# Reads all the log files generated by --log
latencies = []
for filename in glob.glob("pgbench_log.*"):
    with open(filename) as f:
        for line in f:
            parts = line.split()
            # parts[2] is the latency in microseconds per the docs
            latencies.append(float(parts[2]) / 1000.0)  # → ms

if not latencies:
    print("No logs found. Did you run pgbench with --log?")
    sys.exit(1)

arr = np.array(latencies)
print(f"total transactions n: {len(arr)}")
print(f"p50:  {np.percentile(arr, 50):.2f} ms")
print(f"p95:  {np.percentile(arr, 95):.2f} ms")
print(f"p99:  {np.percentile(arr, 99):.2f} ms")
print(f"p99.9: {np.percentile(arr, 99.9):.2f} ms")
print(f"max:  {np.max(arr):.2f} ms")

Why pgbench's average without this is misleading: you already saw it in capsule 02. An average of 7.6ms with a stddev of 4.9ms can hide the fact that p99 is 200ms. Always pull percentiles from the log.


Custom SQL scripts: measuring your own queries

The TPC-B benchmark is useful as a reference point, but what's really useful is measuring your app's queries. pgbench lets you pass it a custom SQL script.

Example: simulating the /books?author=X endpoint

Imagine that in your FastAPI app, GET /books?author=tolkien runs this query:

SELECT b.id, b.title, b.published_year, a.name
FROM books b
JOIN authors a ON a.id = b.author_id
WHERE a.name = 'tolkien';

You want to know how fast PostgreSQL executes it in isolation, without your app on top.

Step 1: prepare a DB with data. (In capsule 08 you'll have the project's DB. Here we use a minimal one.)

-- setup.sql
CREATE TABLE IF NOT EXISTS authors (
  id SERIAL PRIMARY KEY,
  name TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS books (
  id SERIAL PRIMARY KEY,
  title TEXT NOT NULL,
  author_id INTEGER REFERENCES authors(id),
  published_year INTEGER
);

-- 5,000 authors
INSERT INTO authors (name)
SELECT 'author_' || g
FROM generate_series(1, 5000) g;

-- 100,000 books, distributed across the authors
INSERT INTO books (title, author_id, published_year)
SELECT
  'book_' || g,
  (random() * 4999 + 1)::int,
  1900 + (random() * 125)::int
FROM generate_series(1, 100000) g;

-- We set a known author to simulate the filter
UPDATE authors SET name = 'tolkien' WHERE id = 42;
psql -d pgbench_demo -f setup.sql

Step 2: create the SQL script for pgbench.

-- query_books_by_author.sql
SELECT b.id, b.title, b.published_year, a.name
FROM books b
JOIN authors a ON a.id = b.author_id
WHERE a.name = 'tolkien';

Step 3: run pgbench with your script.

# -f: use this SQL script instead of the default TPC-B
# -n: skip vacuum (vacuum only applies to the TPC-B tables)
# The rest is the same as the standard benchmark
pgbench -n -f query_books_by_author.sql -c 10 -j 2 -T 30 -P 5 --log pgbench_demo

Expected output (approx.):

pgbench (16.2)
progress: 5.0 s, 8200.0 tps, lat 1.215 ms stddev 0.512, 0 failed
progress: 10.0 s, 8350.4 tps, lat 1.196 ms stddev 0.498, 0 failed
...
transaction type: query_books_by_author.sql
scaling factor: 1
query mode: simple
number of clients: 10
number of threads: 2
duration: 30 s
number of transactions actually processed: 250300
latency average = 1.198 ms
latency stddev = 0.504 ms
tps = 8343.21 (without initial connection time)

And the percentiles with the Python script:

python percentiles_pgbench.py
# p50:  1.05 ms
# p95:  2.10 ms
# p99:  3.40 ms

Reading it: PostgreSQL runs the isolated query in ~1ms. If in your app the endpoint's HTTP latency is 800ms, the DB is not the bottleneck in this case — the problem lives in Python (probably in serialization, business logic or, most commonly, in an N+1 that fires many queries per request, not this single one).

Random variables in scripts

pgbench supports variables to keep every transaction from hitting the same row (which would cache artificially):

-- query_books_random.sql
\set author_id random(1, 5000)
SELECT b.id, b.title
FROM books b
WHERE b.author_id = :author_id;

This makes each transaction query a random author_id between 1 and 5000, simulating real load with varied cardinality. Without this, the cache hit rate rises artificially and the numbers lie in your favor.


Weights across multiple scripts

Your app probably doesn't run a single query — it runs a mix. You can pass several scripts with weights:

# Mix: 80% reads (weight 8), 20% writes (weight 2)
pgbench -n \
  -f script_read.sql@8 \
  -f script_write.sql@2 \
  -c 10 -j 2 -T 30 \
  pgbench_demo

Useful when you want to reproduce a realistic read-heavy or mixed workload (capsule 02).


Connection mode: simple vs extended vs prepared

pgbench supports three query-sending modes (-M):

ModeWhat it doesWhen to use it
simple (default)Sends each query as a SQL stringClosest to the behavior of psql or simple connections
extendedUses the extended query protocol (parse + bind + execute)Closest to the behavior of drivers that use parameterization
preparedPrepares once and reusesWhen your real app uses prepared statements (asyncpg does by default)
# For benchmarks that reflect what asyncpg does (Python's async driver)
pgbench -n -M prepared -f query.sql -c 10 -T 30 pgbench_demo

Important note for module 6: when you use PgBouncer in transaction mode, prepared statements don't work. If you benchmark with -M prepared a setup that will have PgBouncer transaction in production, your numbers won't reflect production. We'll look at this in detail when we get there.


How to read the output alongside your baselines

Let's put this in context. If your BENCHMARKS.md reports:

Endpoint /books?author=X (HTTP)
  p50: 180ms
  p95: 2,100ms
  p99: 8,400ms

And pgbench running the same query reports:

Direct query (pgbench)
  p50: 1ms
  p95: 2ms
  p99: 3ms

Immediate conclusion: the DB runs the query in ~1ms. Your API takes 180-8,400ms doing "that" query. The difference (hundreds to thousands of ms) is in your app, not in PostgreSQL.

Hypotheses (which you'll confirm in later modules):

  • An N+1 hidden by SQLAlchemy: the "query" you think runs once is actually running 50 times (module 4).
  • The connection pool is saturated and requests are waiting in line (module 6).
  • JSON serialization of large datasets in Python is expensive.
  • The planner is choosing a different plan in production than in pgbench because of different statistics (module 7).

Without pgbench, you couldn't separate "slow DB" from "slow app". That's the value of the tool.


Why does this matter in real work?

1. Validating that a version upgrade / hardware change doesn't break anything. Your team migrates from PostgreSQL 14 to 16. Before and after, you run pgbench with the same parameters. If TPS drops 30%, something's off (config, planner, hardware) — you investigate before deploying to prod.

2. Justifying/refuting "we need a bigger DB". Someone suggests "the app is slow, let's add more CPU to the RDS". Before spending thousands a month, you measure with pgbench how much throughput you can get out of the current instance. If it's at 30% CPU and your p99 is already high, the problem isn't CPU — it's slow queries or config.

3. Junior vs senior diagnosis. When someone says "Postgres is slow", the senior dev runs pgbench to get objective evidence. If pgbench says "this DB does 8,000 TPS without breaking a sweat", the problem lives somewhere else. It's a hypothesis-elimination tool.


Traps and common mistakes

Mistake 1 (conceptual): assuming the TPC-B benchmark's TPS is your app's TPS

Symptom: "pgbench says 5,000 TPS, my app should be able to handle 5,000 RPS."

Why it's wrong: TPC-B is a very specific synthetic workload (short transactions with PK lookups). Your real queries are almost never TPC-B — they're complex JOINs, filtered queries, aggregations. TPC-B's TPS tells you something about the DB's "raw health", not about your app's performance.

How to fix it: measure your real queries with custom scripts (-f), not the default TPC-B.

Mistake 2 (practical): running pgbench with -T 5 and reporting the result

Symptom: "I measured for 5 seconds, it does 1,200 TPS." Next measurement: 800 TPS. Third: 1,500.

Why it happens: very short runs don't let the system stabilize. The first second is dominated by connection warmup, planner JIT, cache misses. You need long runs (minimum 30s, ideally 60-120s) for the number to stabilize.

How to fix it: -T 60 minimum. And as you learned in capsule 03: multiple runs. The baseline formula applies to pgbench too.

Mistake 3 (conceptual): reporting latency average and forgetting percentiles

Symptom: "pgbench reported an average latency of 8ms, it's fine."

Why it's wrong: you already know from capsule 02 that the average can hide a horrible p99. pgbench doesn't report percentiles directly — you need --log and to process the file. If you report only the average, you're leaving 95% of the information on the table.

How to fix it: always run with --log and pull percentiles with a script. It's 30 extra seconds, and it gives you honest data.

Mistake 4 (practical): not using random variables

Symptom: your SQL script always queries the same row (WHERE id = 1). Your numbers show an unreal TPS of 50,000 because PostgreSQL caches that row perfectly.

Why it's wrong: production never hits the same row that obsessively. Your artificial cache hit rate biases the result.

How to fix it: use \set var random(min, max) to vary the parameters on each transaction. Measure with realistic cardinality.

Mistake 5 (conceptual): thinking pgbench replaces EXPLAIN ANALYZE

Symptom: "My query is slow. I'm going to run pgbench to understand why."

Why it's wrong: pgbench tells you how slow it is under concurrent load, not why. The "why" comes from EXPLAIN ANALYZE (module 2): which scan the planner chose, which buffers it read, where the cost is.

How to fix it: use pgbench to measure, use EXPLAIN ANALYZE to understand. They're complementary tools.


Exercises

Exercise 1: your first TPC-B benchmark

Initialize a database with -s 5 (half a million rows), run the TPC-B benchmark with 10 clients, 2 threads, for 60 seconds. Report TPS, average latency and stddev.

See solution
# Create a clean DB
createdb pgbench_ex1

# Initialize with scale factor 5
pgbench -i -s 5 pgbench_ex1

# Run the benchmark
pgbench -c 10 -j 2 -T 60 -P 10 pgbench_ex1

Example output:

progress: 10.0 s, 1850.4 tps, lat 5.405 ms stddev 3.221, 0 failed
progress: 20.0 s, 1875.6 tps, lat 5.331 ms stddev 3.108, 0 failed
...
latency average = 5.402 ms
latency stddev = 3.198 ms
tps = 1851.42

Analysis:

  • ~1,850 sustained TPS on typical laptop hardware.
  • Average latency ~5ms with stddev ~3ms — a relatively consistent distribution, but the average doesn't give you the p99.
  • If your app used TPC-B-like queries (PK lookups + small updates), this DB could handle ~1,850 TPS before saturating.

Your numbers will vary by hardware. What matters is that the output has this shape and that you understand each metric.

Exercise 2: extract percentiles from the log

Run the previous exercise with --log enabled and process the log to report the real p50, p95, p99.

See solution
# Same benchmark with --log
pgbench --log -c 10 -j 2 -T 60 pgbench_ex1

This creates pgbench_log.<pid>.<n> files (one per client).

# percentiles.py
import glob
import numpy as np

latencies = []
for filename in glob.glob("pgbench_log.*"):
    with open(filename) as f:
        for line in f:
            parts = line.split()
            # parts[2] is the latency in microseconds
            latencies.append(float(parts[2]) / 1000.0)  # → ms

arr = np.array(latencies)
print(f"n: {len(arr)}")
print(f"p50:  {np.percentile(arr, 50):.2f} ms")
print(f"p95:  {np.percentile(arr, 95):.2f} ms")
print(f"p99:  {np.percentile(arr, 99):.2f} ms")
print(f"p99.9: {np.percentile(arr, 99.9):.2f} ms")
print(f"max:  {np.max(arr):.2f} ms")

Example output:

n: 111080
p50:  4.50 ms
p95:  10.85 ms
p99:  18.20 ms
p99.9: 42.40 ms
max:  185.30 ms

Key insight: the average was 5.4ms, but the p99 is 18ms (3.4x the average) and the max reaches 185ms. If you only report the average, you hide this.

Exercise 3: measure a custom query

Create a users table with 50,000 rows. Design two SQL scripts:

  • script_pk.sql: a PK lookup with a random id.
  • script_seq.sql: a lookup on a column without an index.

Run pgbench for 30s with each script (10 clients) and compare TPS and latency. What difference do you see?

See solution

Setup:

-- setup_users.sql
DROP TABLE IF EXISTS users;
CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  email TEXT UNIQUE,
  name TEXT,
  created_at TIMESTAMP DEFAULT now()
);

INSERT INTO users (email, name)
SELECT
  'user_' || g || '@example.com',
  'name_' || g
FROM generate_series(1, 50000) g;
psql -d pgbench_demo -f setup_users.sql

PK lookup script:

-- script_pk.sql
\set user_id random(1, 50000)
SELECT id, email, name FROM users WHERE id = :user_id;

Seq scan script (filters on name, which has no index):

-- script_seq.sql
\set n random(1, 50000)
SELECT id, email FROM users WHERE name = 'name_' || :n;

Run and compare:

pgbench -n -f script_pk.sql -c 10 -j 2 -T 30 pgbench_demo
# latency average = 0.119 ms
# tps = 83716.6   (PK lookup, uses the primary key index)

pgbench -n -f script_seq.sql -c 10 -j 2 -T 30 pgbench_demo
# latency average = 2.788 ms
# tps = 3586.5    (full sequential scan on every query)

Confirm the second script really does a seq scan:

psql -d pgbench_demo -c "EXPLAIN (COSTS OFF) SELECT id, email FROM users WHERE name = 'name_123';"
             QUERY PLAN
-------------------------------------
 Seq Scan on users
   Filter: (name = 'name_123'::text)

Analysis:

  • PK lookup: ~83,700 TPS, latency ~0.12ms.
  • Sequential scan: ~3,600 TPS, latency ~2.8ms. ~23x slower.

The absolute numbers will vary quite a bit with your hardware (these are from a laptop with an SSD and local PostgreSQL). What does not vary is the order of magnitude of the difference: the index buys you more than an order of magnitude.

And notice something: 2.8ms to scan 50,000 rows sounds "fast". It is — at this scale. The problem is that a seq scan grows linearly with the table: if users had 5 million rows instead of 50 thousand, that query would go from ~2.8ms to hundreds of ms, while the PK lookup would stay at ~0.1ms (a B-tree index grows logarithmically). The gap widens with size. That's why a seq scan nobody notices today is the classic time bomb in an app that "suddenly" got slow as it grew.

Lesson: the absence of an index on a column you filter by is the difference between a fast API and a slow one. In module 3 you'll learn to design indexes the planner actually uses. For now, what matters is that pgbench gives you an objective number for what each design decision costs.

Exercise 4: separate a DB problem from an app problem

Your API has a GET /products/:id endpoint with p95=400ms. You bring up pgbench with the same query directly (PK lookup with a random id), 10 clients, 30s, and you get p95=2ms. Where is the problem and what would you investigate?

See solution

Diagnosis: the DB runs the query in 2ms at p95, but the API responds in 400ms. The difference (~398ms) lives in one of these layers (in order of likelihood):

  1. App logic before/after the query. Is it loading a YAML, calling an external service, validating something large with Pydantic?
  2. A hidden N+1. The "endpoint that runs one query" actually runs N queries per request. You'll see this in module 4.
  3. A saturated connection pool. If all the pool's connections are busy, the request waits. Module 6.
  4. Heavy JSON serialization. If you return large objects, json.dumps can take tens of ms.
  5. Slow middleware. Some middleware (auth, logging, tracing) may be adding latency.

What to investigate first:

  • Log time.perf_counter() before and after each step of the endpoint to locate where the time goes.
  • Look at the pg_stat_statements output (module 5) to confirm whether the app is firing 1 query or N.
  • Check the pool size and SQLAlchemy's waiting logs.

What you would NOT do right now: add indexes, change the planner, tune shared_buffers. The DB is not the problem — pgbench already proved it.

Exercise 5: vary concurrency to find the knee

Run the TPC-B benchmark with increasing concurrency: -c 1, -c 5, -c 10, -c 25, -c 50, -c 100 (30s each). Note TPS and average latency in each case. At what point does TPS stop scaling and latency explode?

See solution
for c in 1 5 10 25 50 100; do
  echo "=== c=$c ==="
  pgbench -c $c -j $((c < 4 ? 1 : 4)) -T 30 pgbench_ex1 \
    | grep -E "(tps|latency average)"
done

Example output (varies by hardware):

=== c=1 ===
latency average = 0.852 ms
tps = 1173.21

=== c=5 ===
latency average = 2.501 ms
tps = 1998.45

=== c=10 ===
latency average = 5.402 ms
tps = 1851.42

=== c=25 ===
latency average = 14.230 ms
tps = 1755.10

=== c=50 ===
latency average = 32.470 ms
tps = 1539.87

=== c=100 ===
latency average = 78.110 ms
tps = 1280.20

Analysis:

  • TPS rises from 1,173 → 1,998 (from c=1 to c=5). Concurrency helps.
  • TPS reaches a plateau around c=5 to c=10 (~1,800-2,000 TPS).
  • From c=25 onward, TPS starts to fall. Clients contend for shared resources (locks, IO, CPU).
  • Latency grows linearly with concurrency — more clients wait longer in line.

Conclusion: the "knee" (optimal operating point) is around c=5-10 for this DB on this hardware. Beyond that, adding more connections doesn't increase throughput — it only increases latency.

This is a preview of the pool sizing concept from module 6: it isn't about "the more the better", it's about finding the knee and sizing the pool just above it.

Exercise 6: document a benchmark in BENCHMARKS.md

Take the output of Exercise 5 and write the corresponding section of a BENCHMARKS.md. Include context, methodology, a table with the results and at least one observation.

See solution
## PostgreSQL raw capacity benchmark — TPC-B

### Context

- **Hardware:** MacBook Air M1, 16GB RAM, internal SSD
- **PostgreSQL:** 16.2 (local install via brew)
- **DB:** `pgbench_ex1`, scale factor 5 (~80MB on disk)
- **Other load:** none (Slack, browser closed)

### Methodology

- Tool: `pgbench` 16.2 with the TPC-B-like benchmark (default)
- Variable concurrency: c=1, 5, 10, 25, 50, 100
- Threads: max 4
- Duration: 30s per run
- A single run per concurrency level (preliminary — for a real baseline run 5x)

### Results

| Concurrency | TPS | Avg latency | Latency stddev |
|-------------|-----|-------------|----------------|
| c=1   | 1,173 | 0.85 ms  | 0.42 ms |
| c=5   | 1,998 | 2.50 ms  | 1.20 ms |
| c=10  | 1,851 | 5.40 ms  | 3.20 ms |
| c=25  | 1,755 | 14.23 ms | 8.50 ms |
| c=50  | 1,540 | 32.47 ms | 18.20 ms |
| c=100 | 1,280 | 78.11 ms | 45.10 ms |

### Observations

- Optimal throughput around c=5 (~2,000 TPS). More concurrency adds no capacity.
- At c=100, TPS falls 36% vs c=5 and avg latency grows 31x. A sign of saturation.
- Preliminary conclusion: sizing the connection pool above ~10 provides no benefit on this hardware with this workload.
- Missing: real percentiles (run with --log), multiple runs per level.

Important note: the real numbers vary by hardware. The structure is what matters.


Summary and next step

In this capsule you learned:

  • pgbench isolates PostgreSQL from the rest of the stack. It tells you how fast the DB is without contamination from your app.
  • The default TPC-B-like benchmark is useful as a generic baseline; the custom scripts (-f) are what's really valuable for measuring your queries.
  • pgbench reports average + stddev; always use --log and process the file to get real percentiles.
  • Random variables (\set var random(...)) prevent artificial cache hits.
  • The -M prepared mode is close to what asyncpg does, but it is not compatible with PgBouncer transaction mode (module 6).
  • When wrk reports slow and pgbench with the same query reports fast: the problem lives in your app, not in the DB.

Before moving on you should be able to:

  • Initialize and run the TPC-B benchmark with your parameters
  • Write a custom SQL script with random variables to measure one of your app's queries
  • Process the pgbench log to get real p50/p95/p99
  • Identify the concurrency "knee" by varying -c

Next capsule — wrk. You now know how to measure PostgreSQL in isolation. Now we go to the other half: measuring the full API with wrk. It's the tool you'll use for the capstone project's baseline in capsule 08. wrk is to HTTP what pgbench is to PostgreSQL: fast, simple, and it reports percentiles out of the box (better than pgbench in this respect).


Resources

  1. PostgreSQL Documentation — pgbench — the complete official reference with all the flags.
  2. PostgreSQL Wiki — Performance Optimization — a wiki with concrete tuning tips, many validated with pgbench.
  3. Greg Smith — PostgreSQL High Performance (book) — a classic reference with extensive pgbench examples.
  4. Bruce Momjian — "PostgreSQL Performance Tuning" (slides) — public presentations from the PostgreSQL core team.
  5. Daniel Westermann — "Tuning pgbench" — practical tips for getting more out of pgbench.
  6. PostgreSQL Source Code — pgbench tests directory — for the curious: the tool's implementation.
  7. Citus — "How to benchmark PostgreSQL with pgbench" — an applied article from the Citus / Microsoft team.

Module 1 — Database Performance & Query Tuning Guide