6. Trade-offs and Decisions: Choosing the Right Index

Overview

Here you consolidate everything you've seen in the module: how to decide between exact and approximate search, when to use HNSW vs IVF, and which parameters to tune based on your use case. This is the most practical capsule in the module.

Decision matrix

Use case	Dataset size	Required latency	Required precision	Recommended index
Prototype/development	< 10K	Not critical	100%	Brute force
Small app	10K-100K	< 500ms	~99%	HNSW
Standard production	100K-10M	< 100ms	~99%	HNSW
Massive scale	> 10M	< 100ms	~95%	IVF or IVF-PQ
Academic/research use	Variable	Not critical	100%	Brute force

Decision flow

┌─ Is the dataset < 100K vectors?
│  └─ YES → Brute force (simple, fast at this size)
│  └─ NO → Next question
│
├─ Is 100% precision critical?
│  └─ YES → Brute force (there's no alternative)
│  └─ NO → Next question
│
├─ Is memory plentiful?
│  └─ YES → HNSW (better precision, more memory)
│  └─ NO → IVF or IVF-PQ (less memory)
│
└─ Is the dataset > 100M vectors?
   └─ YES → IVF-PQ (compression is necessary)
   └─ NO → HNSW (the sweet spot for most cases)

Full comparison

Metric	Brute force	HNSW	IVF	IVF-PQ
Precision	100%	~99%	~95%	~90%
Speed (1M)	1s	10ms	20ms	5ms
Memory	1x	2-3x	1.2x	0.1x
Build time	None	Hours	Minutes	Minutes
Max dataset	~100K	10M	100M	1B+
Inserts	Trivial	Expensive	Medium	Medium

Real use cases

Case 1: A startup with 50K documents

Requirements:

50,000 vectors (OpenAI 1536D)
Latency < 200ms
A tight budget

Decision: Brute force or basic HNSW

With 50K vectors, brute force is ~50ms → acceptable
If it grows to 500K → migrate to HNSW

Case 2: A company with 5M documents

Requirements:

5 million vectors
Latency < 50ms
High precision (~99%)

Decision: HNSW

Configuration: ef_construction=200, M=32, ef_search=100
Expected: ~15ms latency, ~99% precision

Case 3: A BigCo with 100M documents

Requirements:

100 million vectors
Latency < 100ms
A limited memory budget

Decision: IVF-PQ

Configuration: nlist=10000, nprobe=20, PQ compression=8x
Expected: ~30ms latency, ~93% precision, memory compressed 8x

Parameters by use case

High precision (99%+):

HNSW:

ef_construction = 400
M = 64
ef_search = 200

IVF:

nprobe = 100 (out of 1000 clusters)

Balanced (95-98% precision, fast):

HNSW:

ef_construction = 200
M = 32
ef_search = 100

IVF:

nprobe = 10 (out of 1000 clusters)

Maximum speed (90-95% precision):

HNSW:

ef_construction = 100
M = 16
ef_search = 50

IVF:

nprobe = 1 (out of 1000 clusters)

Additional considerations

1. Frequent inserts:

If you're constantly adding vectors:

IVF: Easier (reassign to a cluster)
HNSW: More expensive (a periodic rebuild is recommended)

2. Memory budget:

If memory is limited:

IVF-PQ: 8-16x compression
HNSW: Requires 2-3x the base memory

3. p99 latency:

If you need consistent latency (not just the average):

HNSW: More consistent
IVF: It can have outliers (if a query lands in a large cluster)

4. Multi-tenancy:

If multiple users share an index:

HNSW: Better isolation
IVF: It can suffer contention on popular clusters

Common mistakes

Mistake 1: Using brute force with > 100K vectors

Symptom: Slow queries (> 500ms)

Solution: Migrate to HNSW or IVF

Mistake 2: Configuring HNSW too aggressively

Symptom: A build time of days, excessive memory

Solution: Reduce ef_construction and M (a balance is better)

Mistake 3: nprobe=1 in IVF with > 1M vectors

Symptom: Very low precision (~85%)

Solution: Increase nprobe to 10-20

Mistake 4: No periodic rebuild

Symptom: Precision degrades over time (many inserts)

Solution: Rebuild the index every N inserts (e.g. every 100K)

Benchmarking tools

ann-benchmarks.com:

A comparison of multiple indexes
Public datasets
Precision vs speed charts

FAISS benchmarks:

Tests with different IVF configurations
Facebook's data

Vector database benchmarks:

Pinecone vs Weaviate vs Qdrant
p50, p95, p99 latency

Summary

Key points:

< 100K: Brute force (simple)
100K-10M: HNSW (the sweet spot)
> 10M: IVF or IVF-PQ (scalability)
Tune the parameters: Based on the precision vs speed you need
Monitor: Latency, precision, memory in production

Next capsule: 07-capstone-exercise-4.md — Designing a search strategy for use cases.