Module 2: EXPLAIN ANALYZE in Depth

Cost model and estimates: how the planner decides

Capsule description

Every time you run a query, PostgreSQL makes decisions: do I use this index or scan the whole table? do I do a Hash Join or a Nested Loop? do I parallelize or not? Those decisions are made by the query planner by comparing alternatives and choosing the lowest-cost one.

But cost is the most misinterpreted metric in the whole guide. Three typical confusions:

  1. "cost = milliseconds" — no. It's an arbitrary unit.
  2. "low cost always = fast query" — no. If the estimates are wrong, the cost is fiction.
  3. "the planner always gets it right" — no. It gets it wrong often due to stale statistics, non-uniform distributions, correlation between columns.

This capsule breaks down:

  • What cost is exactly (the base constants, how it's computed).
  • How the planner estimates rows (the most important piece; a cost is computed as a function of estimated rows).
  • How to detect estimation errors (the symptom you'll see a hundred times for the rest of your career).
  • What to do when the planner gets it wrong (a preview of module 7).

By the end, when someone says "the cost is low so it's fine", you'll know why that's sometimes false, and when to trust or distrust the estimates.


Mental model: cost is the planner's "internal currency"

Imagine you're the head chef and you have to decide between two recipes to serve a dish. You don't have time to cook both and compare. But you have a "complexity points" system based on experience:

  • Recipe A: requires 50 points (20 for chopping vegetables + 25 for slow cooking + 5 for plating).
  • Recipe B: requires 80 points (10 for marinating + 60 for the oven + 10 for saucing).

You choose A — fewer points. But the points aren't minutes. They're an internal metric you calibrate from historical averages. Sometimes you get it wrong (recipe A actually takes longer because today the vegetables are tough), but on average it helps you compare.

That's cost in PostgreSQL. An internal metric, calibrated from constants (which you can adjust), used to compare alternative plans, not to predict absolute time.

                       ┌────────────────────────────────────┐
                       │   PLANNER                          │
                       │                                    │
       Query  ────►   │   Generates N alternative plans    │
                       │                                    │
                       │   For each plan: computes cost     │
                       │   ├─ based on statistics           │
                       │   ├─ and constants (page_cost, etc)│
                       │                                    │
                       │   Chooses the lowest-cost plan     │
                       │                                    │
                       └─────────────┬──────────────────────┘
                                     │
                                     ▼
                       ┌────────────────────────────────────┐
                       │  EXECUTOR runs the chosen plan     │
                       └────────────────────────────────────┘

If the estimate is good, the chosen plan is the fastest. If the estimate is bad, the planner chooses a plan that seems cheap but runs slow. That's why the most important thing isn't the cost itself — it's the estimates that feed it (especially rows).


The constants that make up cost

PostgreSQL computes cost from a few configurable constants. These are in postgresql.conf (or you can see them with SHOW):

SHOW seq_page_cost;          -- 1.0 (default)
SHOW random_page_cost;       -- 4.0 (default)
SHOW cpu_tuple_cost;         -- 0.01 (default)
SHOW cpu_index_tuple_cost;   -- 0.005 (default)
SHOW cpu_operator_cost;      -- 0.0025 (default)
SHOW parallel_tuple_cost;    -- 0.1 (default)
SHOW parallel_setup_cost;    -- 1000.0 (default)
ConstantDefaultWhat it represents
seq_page_cost1.0Cost of reading a page (8KB) in sequential order. It's the base unit.
random_page_cost4.0Cost of reading a page in random order (more expensive because of the disk seek).
cpu_tuple_cost0.01CPU cost of processing a row.
cpu_index_tuple_cost0.005CPU cost of processing a row via an index.
cpu_operator_cost0.0025CPU cost of evaluating an operator (=, <, etc.).
parallel_tuple_cost0.1Cost of passing a row between parallel workers.
parallel_setup_cost1000.0Fixed cost of starting parallel workers.

The constants are calibrated for HDDs. On SSDs, random_page_cost should be ~1.1 (because SSDs have no seek penalty). If your DB runs on SSD and you didn't lower random_page_cost, the planner underestimates the value of indexes and sometimes chooses Sequential Scans when an Index Scan would be better.

-- For SSDs (recommended)
ALTER SYSTEM SET random_page_cost = 1.1;
SELECT pg_reload_conf();

For the cloud (RDS, Aurora, Cloud SQL, etc.) which is almost always SSD: lower it. It's a one-line change with a significant impact on the chosen plans.


How cost is computed: the Sequential Scan example

Let's take a Seq Scan on books (100,000 rows across ~810 pages):

Seq Scan on books  (cost=0.00..1620.00 rows=100000 width=22)

The cost is computed like this:

cost = (pages × seq_page_cost) + (rows × cpu_tuple_cost)
cost = (810 × 1.0) + (100000 × 0.01)
cost = 810 + 1000
cost = 1810

Hmm, the output says 1620. The difference is because cpu_tuple_cost is applied only to the rows that pass the filter (here there's no filter, so it's all of them). The exact number varies with the version, but the formula is pages + tuples, weighted by their respective constants.

Key point: the Seq Scan's cost is ~linear with the table's size (more rows → more pages → more cost).

The Index Scan example

Index Scan using idx_books_id on books  (cost=0.42..8.44 rows=1 width=22)

The cost is computed like this (simplified):

cost = (index startup) + (tree height × random_page_cost) + (rows × cpu_index_tuple_cost) + (heap fetch × random_page_cost)

For a PK lookup of 1 row: ~0.42 startup + a lookup in the B-tree (3-4 levels × random_page_cost) + 1 heap fetch.

Key point: the Index Scan's cost is logarithmic with the table's size (B-tree). That's why an Index Scan beats a Seq Scan on large tables with selective filters.

The comparison the planner makes

For SELECT * FROM books WHERE id = 12345:

  • Plan A: Seq Scan → cost ≈ 1620 (scan the 100k rows)
  • Plan B: Index Scan → cost ≈ 8 (use the PK index)

The planner chooses B. Cost ratio: 200x lower.

For SELECT * FROM books WHERE published_year > 1900 (a filter that matches ~99% of rows):

  • Plan A: Seq Scan → cost ≈ 1620
  • Plan B: Index Scan → cost ≈ 99,000 × random_page_cost / random_page_cost = much higher than A (each row is a random heap fetch).

The planner chooses A. Even though there's an index on published_year, the Seq Scan is cheaper because it's going to read the whole table anyway.

This is what you want the planner to do: compare costs and choose the lowest one. Capsule 05 goes deeper into these trade-offs.


How the planner estimates rows

Here comes the most important piece of the whole capsule. Cost is a function of estimated rows. If the rows are estimated well, the cost is reasonable and the chosen plan is the good one. If the rows are wrong, everything else falls apart.

How does it estimate rows? It uses statistics stored in the system, in the pg_stats and pg_statistic tables. Those statistics are generated/updated with:

  • ANALYZE (manual): you run it.
  • Autovacuum (automatic): PostgreSQL runs it periodically when it detects significant changes.

What the planner knows about each column

For each column of each table, PostgreSQL stores:

-- What the planner sees about books.published_year
SELECT
  attname,
  null_frac,           -- proportion of nulls
  avg_width,           -- average bytes
  n_distinct,          -- distinct values (negative = ratio)
  most_common_vals,    -- most frequent values
  most_common_freqs,   -- frequency of each one
  histogram_bounds,    -- histogram bins (for ranges)
  correlation          -- correlation between physical and logical order
FROM pg_stats
WHERE tablename = 'books' AND attname = 'published_year';

Example output:

attname             | published_year
null_frac           | 0
avg_width           | 4
n_distinct          | 125          ← 125 distinct values
most_common_vals    | {1923,1987,1956,...}
most_common_freqs   | {0.0085,0.0084,0.0083,...}
histogram_bounds    | {1900,1910,1920,...,2025}
correlation         | 0.0023      ← logical order doesn't correlate with physical order

With this, the planner estimates:

  • For WHERE published_year = 1987 → it looks up 1987 in most_common_vals, reads the freq (0.0084). Estimates: 100,000 × 0.0084 = ~840 rows. ✅
  • For WHERE published_year > 2000 → it uses histogram_bounds to estimate what fraction falls above 2000. If the bins are uniform, it estimates ~20% (the last bin spans 2020-2025, the second-to-last 2010-2020, etc.). 100,000 × 0.20 = ~20,000 rows.

When the planner gets the estimate wrong

Three typical cases:

Case 1: outdated statistics

You load 10M rows with COPY, don't run ANALYZE, run a query. The planner still thinks the table has 10K. It underestimates rows by orders of magnitude, chooses an Index Scan where a Seq Scan would be better.

Detect: look at last_analyze and last_autoanalyze in pg_stat_user_tables:

SELECT relname, n_live_tup, last_analyze, last_autoanalyze
FROM pg_stat_user_tables
WHERE relname = 'books';

If it was a long time ago or never, run ANALYZE:

ANALYZE books;

(Full detail in module 7.)

Case 2: non-uniform distribution that doesn't fit in most_common_vals

PostgreSQL stores the top-N most frequent values (default: 100). If your column has 10,000 unique values with a skewed distribution (1% of values cover 90% of the traffic), and the skewed values don't fit in the top 100, the planner uses the histogram — which assumes a uniform distribution. Result: it underestimates/overestimates.

Solution: raise default_statistics_target (default 100) so that more values are stored in most_common_vals:

ALTER TABLE books ALTER COLUMN published_year SET STATISTICS 1000;
ANALYZE books;

Case 3: correlation between columns the planner doesn't see

SELECT * FROM books WHERE category = 'Fiction' AND author_country = 'USA';

The planner assumes independence: selectivity(A AND B) = selectivity(A) × selectivity(B). If category='Fiction' selects 30% and author_country='USA' selects 40%, it estimates 12% (30% × 40%).

But if in reality 95% of Fiction books are from the USA, the real selectivity is ~28% (not 12%). The planner underestimates rows by 2x. It changes the chosen plan.

Solution (PG 10+): CREATE STATISTICS for multi-column correlation.

CREATE STATISTICS books_cat_country_corr (dependencies, ndistinct)
ON category, author_country FROM books;

ANALYZE books;

(Module 7 goes deeper into it.)


Detecting estimation errors: the number one symptom

Here's the rule you'll apply for the rest of your career:

At each node, compare rows=X (estimated) with rows=Y (actual). If the ratio is >10x in either direction, the statistics are wrong or there's a hidden correlation.

Clear example:

Index Scan using idx_status on orders  (cost=0.42..1500.00 rows=100 width=240) (actual time=0.245..823.412 rows=85000 loops=1)
  Index Cond: (status = 'pending')
  Buffers: shared hit=2400 read=180
  • Estimated: 100 rows with status='pending'.
  • Real: 85,000 rows.
  • Ratio: 850x. The planner thought "this brings back very little, the index is worth it". In reality it brought back a mass of rows and the plan becomes slow (each row is a random heap fetch).

Diagnosis:

  • Stale statistics for that table.
  • Or status='pending' covers most of the table and the planner assumes it's rare.

Immediate action:

  1. Note in HIPOTESIS.md: "850x row mismatch on orders.status — investigate statistics".
  2. (If urgent:) run ANALYZE orders and check whether the plan changes.
  3. (Module 7:) raise default_statistics_target for that column and/or create a partial index if status='pending' is always going to be a large volume.

Cost tells you what the planner chose — not whether it'll be fast

Classic example: two alternative plans for the same query.

Plan A (what the planner would choose with bad statistics):
  Nested Loop  (cost=0.42..500.00 rows=10 width=22) ← low cost
    -> Index Scan ... (rows=10 estimated, 10000 actual)
    -> Index Scan ... (loops=10000)
  Execution Time: 4500 ms

Plan B (what the planner would choose with correct statistics):
  Hash Join  (cost=2000.00..3000.00 rows=10000 width=22) ← high cost
    -> Seq Scan ...
    -> Hash ...
  Execution Time: 80 ms

Plan A has a lower cost (500 vs 3000). The planner chooses it. But the Execution Time is 4500ms — 56x slower than Plan B.

Why? Because Plan A's cost was computed assuming the Nested Loop would iterate over 10 rows. In reality it iterated over 10,000. The computed cost was fiction; the real cost was much higher.

This is why cost ≠ time. Cost is the planner's estimate; if the estimate is good, the time is proportional. If it's bad, it isn't.

How to force the planner to try another plan (for diagnosis)

PostgreSQL lets you disable scan/join types temporarily, to force the planner to choose something else and compare:

-- Disable Nested Loop
SET enable_nestloop = off;
EXPLAIN (ANALYZE, BUFFERS) <query>;
SET enable_nestloop = on;

-- Other useful options
SET enable_seqscan = off;       -- force an index
SET enable_indexscan = off;     -- force a seq scan
SET enable_hashjoin = off;
SET enable_mergejoin = off;

Caveat: these are diagnostic tools, not production ones. Never leave them enabled in postgresql.conf or in your app. They're for one session, to compare plans, diagnose what would have happened with another plan, and go back to normal.


Why does this matter in real work?

1. Diagnosing "the planner ignores my index".

You hear that phrase all the time. The reason is almost always: the planner estimated that the Index Scan would be more expensive than the Seq Scan (cost of the Index > cost of the Seq), because it overestimated rows. Without understanding cost and estimates, you can't diagnose this.

2. Tuning SSDs vs HDDs.

Lowering random_page_cost to 1.1 on SSDs is one of the most impactful and underused changes. Any serious conversation about cloud DB tuning goes through this.

3. "Do I need CREATE STATISTICS or not" decisions.

If a query with AND/OR over multiple columns has bad plans, the question isn't "let me throw another index at it" — it's "does the planner see a correlation between the columns?". If not, CREATE STATISTICS. That decision can only be made by understanding how the planner estimates.

4. Reading plans in interviews.

"Show plans with row mismatches and ask for a diagnosis" is a classic senior interview question. If you recognize "estimated=10, actual=10000, this is stale statistics", you mark as senior. If you only say "the query is bad", you mark as junior.


Traps and common mistakes

Mistake 1 (conceptual): cost is milliseconds

Symptom: "The cost is 1500, that's 1.5 seconds."

Why it's wrong: cost is an arbitrary planner unit based on seq_page_cost (default 1.0 = "one sequential page"). It's not ms, not μs, not anything external. It's only for comparing plans against each other.

How to fix it: get used to reading cost as "planner points for comparing". For time, you look at actual time or Execution Time.

Mistake 2 (operational): not running ANALYZE after a bulk load

Symptom: "I imported 10M rows with COPY, the queries are super slow and the planner ignores my indexes."

Why it happens: COPY doesn't update statistics. The planner still thinks the table has 0 (or the old count) rows. It estimates rows massively wrong, chooses bad plans.

How to fix it: after any bulk load, always ANALYZE:

COPY books FROM '...' CSV;
ANALYZE books;  ← not optional

(Module 7 goes deeper into autovacuum and ANALYZE.)

Mistake 3 (configuration): default random_page_cost on SSD

Symptom: "My DB runs on RDS / Aurora / cloud, but the planner keeps choosing Seq Scans where it should use indexes."

Why it happens: the default random_page_cost = 4.0 is calibrated for HDDs (where a random seek is ~4x more expensive than sequential). On SSD, the difference is minimal — random_page_cost should be ~1.1.

How to fix it:

ALTER SYSTEM SET random_page_cost = 1.1;
SELECT pg_reload_conf();

Re-run problematic queries. You'll see plans change (more Index Scans where there were Seq Scans before).

Mistake 4 (interpretation): estimated rows = the table's rows

Symptom: "The plan says rows=100,000, that table has exactly 100,000 rows."

Why it's sometimes wrong: the planner estimates based on statistics, which are approximate. The "100,000 estimated rows" may be 95k real, 110k real, or 1M real if the statistics are rotten. Without ANALYZE, the planner's figures are fiction.

How to fix it: always use EXPLAIN ANALYZE (not just EXPLAIN) to compare estimated vs actual.

Mistake 5 (subtle): thinking cost during planning affects the chosen plan

Symptom: "Planning takes 5ms, that seems like a lot to me."

Why it's sometimes wrong: Planning Time (5ms) and the computed costs are different things. The planner spends time computing costs for several alternative plans, then chooses the lowest-cost one. Planning time is CPU used by the planner; cost is the metric the planner produces.

How to fix it: if planning time is high, consider prepared statements (they cache the plan). If cost is high, look at which plan it chose and why.

Mistake 6 (conceptual): changing cost manually to "force" a plan

Symptom: "I'm going to set random_page_cost = 0.1 so it uses my index."

Why it's sometimes wrong: the costs are calibrated to reflect the reality of your hardware. Lowering random_page_cost artificially can make the planner choose indexes where a seq scan would be faster — because the cost doesn't reflect reality.

How to fix it: adjust the constants to the reality of your hardware (SSD → 1.1 - 1.5; HDD → 4.0). If you need to "force" a plan for diagnosis, use SET enable_*=off temporarily. If in production you need a specific plan stably, the solution is hints (not natively supported, but pg_hint_plan is an extension) or rewriting the query — not hacking the costs.


Exercises

Exercise 1: see your DB's constants

Connect to psql and show the 7 main constants of the cost model. Note which ones are at their default and which were tuned. If your DB runs on SSD, is random_page_cost well calibrated?

See solution
SELECT name, setting, unit, source
FROM pg_settings
WHERE name IN (
  'seq_page_cost',
  'random_page_cost',
  'cpu_tuple_cost',
  'cpu_index_tuple_cost',
  'cpu_operator_cost',
  'parallel_tuple_cost',
  'parallel_setup_cost'
)
ORDER BY name;

Typical output (defaults):

        name         | setting | unit |  source
---------------------+---------+------+-----------
 cpu_index_tuple_cost| 0.005   |      | default
 cpu_operator_cost   | 0.0025  |      | default
 cpu_tuple_cost      | 0.01    |      | default
 parallel_setup_cost | 1000    |      | default
 parallel_tuple_cost | 0.1     |      | default
 random_page_cost    | 4       |      | default
 seq_page_cost       | 1       |      | default

Analysis:

  • If source is default, it wasn't changed.
  • If random_page_cost is 4 and your DB runs on SSD (laptop with an internal SSD, RDS, Aurora, Neon, Supabase, Cloud SQL), it's poorly calibrated. It should be ~1.1.

Suggested change (if it's safe to do it in your environment):

ALTER SYSTEM SET random_page_cost = 1.1;
SELECT pg_reload_conf();

Re-run a few EXPLAINs of problematic queries — you'll likely see plans change.

Exercise 2: a column's statistics

Show the statistics the planner has about the published_year column of the books table (from the seeded database). How many most_common_vals does it store? What's the correlation? What does that number tell you?

See solution
SELECT
  attname,
  null_frac,
  avg_width,
  n_distinct,
  array_length(most_common_vals::text::text[], 1) as n_mcv,
  array_length(histogram_bounds::text::text[], 1) as n_hist,
  correlation
FROM pg_stats
WHERE tablename = 'books' AND attname = 'published_year';

Typical output:

   attname     | null_frac | avg_width | n_distinct | n_mcv | n_hist | correlation
---------------+-----------+-----------+------------+-------+--------+-------------
published_year |         0 |         4 |        125 |    62 |    101 |      0.0023

Reading:

  • null_frac=0: no nulls.
  • n_distinct=125: 125 unique values (1900-2025 + noise).
  • n_mcv=62: the planner stores the 62 most frequent values with their frequencies.
  • n_hist=101: the histogram has 100 bins (101 boundaries) for the non-MCV values.
  • correlation=0.0023: very low. It means the physical order of the rows (how they're on disk) doesn't correlate with the logical order of published_year. Physically close rows have random years.

Implication: an Index Scan on published_year that needs 1000 physically dispersed rows does 1000 random heap fetches. If correlation were ~1.0 (rows physically ordered by year), the same Index Scan would be almost sequential — much faster.

This justifies why a CLUSTER table USING idx is sometimes worth it, to physically reorganize the table by an index, especially for queries that scan ranges.

Exercise 3: synthetic row mismatch

Create a table with a skewed distribution where the planner gets it wrong. Verify the row mismatch.

See solution
-- Create a table with artificial skew
DROP TABLE IF EXISTS events;
CREATE TABLE events (
  id SERIAL PRIMARY KEY,
  status TEXT NOT NULL
);

-- 99,000 rows with status='processed'
INSERT INTO events (status)
SELECT 'processed' FROM generate_series(1, 99000);

-- 1,000 rows with status='pending'
INSERT INTO events (status)
SELECT 'pending' FROM generate_series(1, 1000);

-- Create an index on status
CREATE INDEX idx_events_status ON events(status);

-- IMPORTANT: do NOT run ANALYZE yet
-- Ask the planner
EXPLAIN ANALYZE SELECT * FROM events WHERE status = 'pending';

Output (without ANALYZE):

Seq Scan on events  (cost=0.00..1734.00 rows=500 width=18) (actual time=0.012..8.234 rows=1000 loops=1)
  Filter: (status = 'pending'::text)
  Rows Removed by Filter: 99000

The planner thinks status='pending' matches ~500 rows (default selectivity of 0.5% for columns without real statistics). Real: 1000 rows. A small mismatch because without ANALYZE the planner uses conservative heuristics.

-- Now run ANALYZE
ANALYZE events;

EXPLAIN ANALYZE SELECT * FROM events WHERE status = 'pending';

Output (with ANALYZE):

Index Scan using idx_events_status on events  (cost=0.42..47.21 rows=1000 width=18) (actual time=0.045..1.823 rows=1000 loops=1)
  Index Cond: (status = 'pending'::text)

Now yes: estimated 1000, real 1000. The planner switched to an Index Scan because it now "knows" that status='pending' is very selective.

Lesson: without ANALYZE, the planner is flying blind. Even for small tables, fresh statistics change the chosen plan.

Exercise 4: the effect of random_page_cost

Take a query that currently uses an Index Scan. Change random_page_cost artificially and observe the cost. How does it vary?

See solution
-- Setup: query with an index
CREATE INDEX IF NOT EXISTS idx_books_year ON books(published_year);

-- default random_page_cost
SHOW random_page_cost;
-- 4

EXPLAIN SELECT * FROM books WHERE published_year = 1987;
-- Index Scan using idx_books_year on books  (cost=0.42..28.45 rows=8 width=22)

-- We lower random_page_cost (ssd-friendly)
SET random_page_cost = 1.1;

EXPLAIN SELECT * FROM books WHERE published_year = 1987;
-- Index Scan using idx_books_year on books  (cost=0.29..7.83 rows=8 width=22)

-- We raise it artificially
SET random_page_cost = 20;

EXPLAIN SELECT * FROM books WHERE published_year = 1987;
-- Seq Scan on books  (cost=0.00..1870.00 rows=8 width=22)
--   Filter: (published_year = 1987)
-- ← The planner switches to a Seq Scan because the index became "too expensive"

-- Back to default
RESET random_page_cost;

Analysis:

  • With random_page_cost=4 (HDD default): Index Scan cost = 28.45 → the planner chooses the index.
  • With random_page_cost=1.1 (SSD): cost = 7.83 → the planner chooses the index (even more comfortably).
  • With random_page_cost=20 (extreme penalty): the Index Scan's cost rises so much that the planner chooses a Seq Scan.

Lesson: random_page_cost directly affects "when the planner uses indexes vs seq scans". Calibrating it correctly is one of the most impactful changes in module 7.

Caveat: don't play with this in production without understanding the impact on all your queries. Here it's a diagnostic exercise.

Exercise 5: diagnose the row mismatch

Capture the complete plan of this query (assume your DB has the books table):

EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM books WHERE published_year > 2010 AND title LIKE 'A%';

Identify: estimated vs actual rows at each node, and diagnose whether the planner is estimating well or badly.

See solution

Typical output:

Seq Scan on books  (cost=0.00..2370.00 rows=620 width=22) (actual time=0.024..12.123 rows=580 loops=1)
  Filter: ((published_year > 2010) AND (title ~~ 'A%'::text))
  Rows Removed by Filter: 99420
  Buffers: shared hit=810
Planning Time: 0.215 ms
Execution Time: 12.234 ms

Analysis:

  • Estimated: 620 rows. Actual: 580 rows. Ratio: ~0.94. Excellent estimate.
  • The planner assumed:
    • published_year > 2010 → ~12% of the table = 12,000 rows (this it did estimate from the histogram).
    • title LIKE 'A%' → ~5% of the table (assumes a uniform distribution of initials).
    • Combined AND: ~12% × 5% = 0.6%, or ~600 rows. Real: 580. A good estimate.

Diagnosis:

  • The planner estimates well here. The statistics are up to date. The distribution is reasonably uniform.
  • Bottleneck: the Seq Scan reads the whole table (100k) to return 580. That's 99.4% waste. A typical symptom of a missing index on one of the filter's columns.
  • Possible solutions (module 3):
    • An index on published_year (you already saw that it helps).
    • A composite index (published_year, title text_pattern_ops) so that LIKE works with a prefix.
    • A partial index WHERE published_year > 2010 (smaller, faster).

For now: you note "slow query — scans all of books, consider a combined index for module 3".

Exercise 6: force an alternative plan

Take a query that currently uses an Index Scan. Force the planner to use a Seq Scan with SET enable_indexscan=off and compare costs and times. Had the planner chosen well?

See solution
-- Make sure the index exists
CREATE INDEX IF NOT EXISTS idx_books_year ON books(published_year);
ANALYZE books;

-- Default plan (with the index)
EXPLAIN ANALYZE SELECT * FROM books WHERE published_year = 1987;
-- Index Scan using idx_books_year on books
--    (cost=0.42..28.45 rows=8 width=22)
--    (actual time=0.025..0.082 rows=8 loops=1)
-- Execution Time: 0.112 ms

-- Force a seq scan
SET enable_indexscan = off;
SET enable_bitmapscan = off;

EXPLAIN ANALYZE SELECT * FROM books WHERE published_year = 1987;
-- Seq Scan on books
--    (cost=0.00..1870.00 rows=8 width=22)
--    (actual time=0.014..7.812 rows=8 loops=1)
-- Execution Time: 7.845 ms

RESET enable_indexscan;
RESET enable_bitmapscan;

Comparison:

PlanCostExecution Time
Index Scan (default)28.450.112 ms
Seq Scan (forced)1870.007.845 ms

Analysis:

  • Cost ratio: 65x. Time ratio: 70x. Cost predicted reasonably well.
  • The planner had chosen well: with such a selective filter (8 of 100,000 = 0.008%), the index wins by a wide margin.

When a seq scan would win: if the filter were, for example, WHERE published_year > 1900 (matches 99% of the table), a Seq Scan would be cheaper than an Index Scan that has to do 99,000 random heap fetches. Try it:

EXPLAIN ANALYZE SELECT * FROM books WHERE published_year > 1900;
-- Seq Scan on books  (cost=0.00..2120.00 rows=99950 width=22) (actual time=0.012..7.823 rows=99950 loops=1)
-- ← The planner does choose a Seq Scan here, without needing to force it

Lesson: the planner makes a trade-off based on selectivity. Very selective filters → the index wins. Low-selectivity filters → the seq scan wins. This balance is what cost tries to capture.


Summary and next step

In this capsule you learned:

  • Cost is an arbitrary planner unit, calibrated by constants (seq_page_cost, random_page_cost, etc.). It's for comparing alternative plans, not for predicting time.
  • Cost is computed as a function of estimated rows. If the estimates are wrong, the cost is fiction and the planner chooses bad plans.
  • The planner estimates rows using pg_stats: most_common_vals, histogram_bounds, n_distinct, correlation.
  • Stale statistics (no ANALYZE or a lagging autovacuum) are the #1 cause of bad plans.
  • random_page_cost = 4.0 is calibrated for HDDs. On SSD it should be ~1.1. A simple change, a huge impact.
  • Row mismatch (estimated vs actual with a ratio >10x) is the most common symptom of problems: stale statistics, skewed distribution, correlation between columns.
  • SET enable_*=off is for forcing the planner to try another plan during diagnosis — never in production.

Before moving on you should be able to:

  • Read the 7 cost model constants in your DB and know what each does
  • Identify a row mismatch in any plan
  • Know when to run ANALYZE (after bulk loads)
  • Calibrate random_page_cost according to your storage (SSD vs HDD)

Next capsule — Sequential vs Index Scans. You now understand cost, estimates, and why the planner chooses each plan. Now we're going to go deeper into the scan types: why a Seq Scan sometimes beats an Index Scan, what an Index-Only Scan is and when it appears, what a Bitmap Heap Scan + Bitmap Index Scan is and when the planner combines them. It's the capsule that closes out "reading scan types with confidence" — the last step before diving into buffers and JIT in capsule 06.


Resources

  1. PostgreSQL Documentation — Planner / Optimizer — the official overview of the planner. Recommended reading before capsule 05.
  2. PostgreSQL Documentation — Statistics Used by the Planner — how the planner uses pg_stats, what it stores, how it combines it.
  3. PostgreSQL Documentation — Cost Constants — the official reference for the cost model constants.
  4. Tom Lane — "PostgreSQL Optimizer" (2007 talk, slides) — from the original author of the query planner. The concepts still apply 18 years later.
  5. Hubert "depesz" Lubaczewski — "Costing of plans" — part of depesz's series, focused on cost.
  6. Bruce Momjian — "Inside the PostgreSQL Query Optimizer" (slides) — from the core team, goes deeper into how cost is computed step by step.
  7. Markus Winand — "Use The Index, Luke!" — The Optimizer — a practical explanation of operations and costs oriented toward indexing.
  8. Citus — "How to read PostgreSQL EXPLAIN ANALYZE output" — a practical primer focused on cost and row mismatch.

Module 2 — Database Performance & Query Tuning Guide