Module 2: How Vector Databases Work (Conceptual)

Capsule 07: HNSW vs IVF vs PQ comparison - Decision Framework

Consolidate your understanding of HNSW, IVF, and PQ with a practical decision framework for choosing the right algorithm based on your RAG system's requirements.

By the end of this capsule:

✅ You'll use a decision tree to choose an algorithm
✅ You'll compare benchmarks side by side
✅ You'll know the recommended configurations per scenario
✅ You'll plan a migration path (HNSW → IVF → IVF+PQ)

Estimated time: 8-10 minutes

🎯 Decision Tree: Which algorithm to use?

Decision framework

┌─────────────────────────────────────────┐
│ How many vectors do you have?           │
└─────────────────────────────────────────┘
                  │
       ┌──────────┴──────────┐
       │                     │
    < 1M                  > 1M
       │                     │
       │          ┌──────────┴──────────┐
       │          │                     │
       │       1M-5M                 > 5M
       │          │                     │
       │          │                     │
┌──────▼─────┐ ┌──▼────────┐ ┌─────────▼────────┐
│  HNSW      │ │  Accuracy  │ │   RAM critical?  │
│            │ │  >95%?     │ │                  │
│ ChromaDB   │ │            │ │  Yes       No    │
│ Weaviate   │ │  Yes   No  │ │   │         │    │
│ Qdrant     │ │   │     │  │ │  IVF+PQ   IVF   │
└────────────┘ │ HNSW  IVF │ │  Faiss    Faiss  │
               │            │ │                  │
               └────────────┘ └──────────────────┘

Key questions

1. How many vectors?

< 100K → HNSW (simple, ChromaDB)
100K - 1M → HNSW (managed, Pinecone/Weaviate)
1M - 5M → IVF or HNSW depending on memory
5M → IVF + PQ

2. Required accuracy?

95% → HNSW only
90-95% → IVF
85-90% → PQ or IVF+PQ
< 85% → Review the embeddings (upstream problem)

3. Available memory?

8 GB per 1M vectors → HNSW viable
2-4 GB per 1M vectors → IVF viable
< 1 GB per 1M vectors → Requires PQ

4. Required latency?

< 50ms → HNSW
< 100ms → IVF
< 200ms → IVF+PQ
200ms → Any (not critical)

5. Frequent updates?

Incremental (daily) → HNSW
Batch (weekly) → IVF or IVF+PQ
Batch (monthly) → Any

📊 Side-by-side comparison: Real scenarios

Scenario 1: Customer Support Chatbot

Requirements:

Vectors: 50K support articles
Accuracy: >95% (customer-facing)
Latency: <500ms (chatbot response)
Updates: Incremental (new articles daily)
Budget: Moderate

Comparison:

Metric	HNSW	IVF	IVF+PQ
Accuracy	98% ✅	93% ⚠️	88% ❌
Latency	12 ms ✅	25 ms ✅	30 ms ✅
Memory	2 GB ✅	0.8 GB ✅	0.4 GB ✅
Build time	3 min ✅	1 min ✅	5 min ✅
Incremental	✅	❌	❌

Decision: HNSW (ChromaDB)

Reason:

Accuracy is critical (customer-facing)
Memory is OK (2 GB is reasonable)
Incremental updates are essential
Excellent latency

Setup:

# ChromaDB with HNSW
collection = client.create_collection(
    name="support_articles",
    metadata={"hnsw:space": "cosine"}
)

Scenario 2: E-commerce Product Search

Requirements:

Vectors: 2M products
Accuracy: 90-95% (search relevance OK)
Latency: <200ms
Updates: Batch nightly (catalog changes little)
Budget: Limited (self-hosted)

Comparison:

Metric	HNSW	IVF	IVF+PQ
Accuracy	98% ✅	93% ✅	90% ✅
Latency	25 ms ✅	45 ms ✅	55 ms ✅
Memory	16 GB ❌	4 GB ✅	2 GB ✅
Build time	20 min ⚠️	8 min ✅	25 min ⚠️
Incremental	✅	❌ ⚠️	❌ ⚠️

Decision: IVF (Faiss self-hosted)

Reason:

93% accuracy is enough (search, not critical recommendations)
Memory is critical (16 GB HNSW is very expensive)
Batch updates are OK (nightly rebuild is viable)
Acceptable latency (45ms < 200ms)

Setup:

# Faiss IVF
import faiss
dimension = 1536
nlist = 1414  # sqrt(2M) ≈ 1414
index = faiss.IndexIVFFlat(
    faiss.IndexFlatL2(dimension),
    dimension,
    nlist
)
index.nprobe = 20  # Explore 20 clusters

Scenario 3: Internal Knowledge Base (Massive)

Requirements:

Vectors: 20M documents (Confluence + Slack + Jira + Docs)
Accuracy: 85-90% (internal, not critical)
Latency: <500ms
Updates: Batch weekly (rebuild index on the weekend)
Budget: Limited (self-hosted, limited RAM)

Comparison:

Metric	HNSW	IVF	IVF+PQ
Accuracy	98% ✅	92% ✅	88% ✅
Latency	60 ms ✅	80 ms ✅	90 ms ✅
Memory	160 GB ❌❌	40 GB ❌	12 GB ✅
Build time	180 min ❌	60 min ✅	120 min ⚠️
Incremental	✅	❌	❌

Decision: IVF + PQ (Faiss)

Reason:

Memory is critical (160 GB HNSW is impractical)
88% accuracy is acceptable (internal search)
Latency is OK (90ms < 500ms)
Batch updates are OK (weekly rebuild is viable)
Cost: $800/month (12 GB) vs $8000/month (160 GB) → 10x savings

Setup:

# Faiss IVF+PQ
dimension = 1536
nlist = 4472  # sqrt(20M) ≈ 4472
m = 8  # Sub-vectors
nbits = 8

index = faiss.IndexIVFPQ(
    faiss.IndexFlatL2(dimension),
    dimension,
    nlist,
    m,
    nbits
)
index.nprobe = 50  # Accuracy boost

Scenario 4: Legal Document Analysis

Requirements:

Vectors: 500K legal cases
Accuracy: >98% (critical - legal compliance)
Latency: <1s (research, not real-time)
Updates: Batch monthly (archived cases)
Budget: High (critical system)

Comparison:

Metric	HNSW	IVF	IVF+PQ
Accuracy	98% ✅	93% ❌	88% ❌
Latency	20 ms ✅	40 ms ✅	50 ms ✅
Memory	4 GB ✅	1 GB ✅	0.5 GB ✅
Build time	5 min ✅	2 min ✅	8 min ✅

Decision: HNSW (Weaviate managed)

Reason:

Accuracy is critical (legal = zero compromise)
Budget allows it (critical system)
Managed service (Weaviate) handles scale
Excellent latency

Setup:

# Weaviate with optimized HNSW
collection_config = {
    "vectorIndexType": "hnsw",
    "vectorIndexConfig": {
        "maxConnections": 64,  # High M for accuracy
        "efConstruction": 256,
        "ef": 200,  # High efSearch
    }
}

🎨 Recommended configurations

HNSW configurations per use case

Config A: High Accuracy (Legal, Medical)

{
    "hnsw:M": 64,  # More connections
    "hnsw:construction_ef": 400,  # Maximum build quality
    "hnsw:search_ef": 200,  # Maximum query quality
}

# Result: 99% accuracy, 30ms latency, 10 GB RAM (1M vecs)

Config B: Balanced (Default)

{
    "hnsw:M": 16,  # Balance
    "hnsw:construction_ef": 200,
    "hnsw:search_ef": 100,
}

# Result: 98% accuracy, 18ms latency, 6 GB RAM (1M vecs)

Config C: Fast Query

{
    "hnsw:M": 8,  # Fewer connections
    "hnsw:construction_ef": 100,
    "hnsw:search_ef": 50,
}

# Result: 95% accuracy, 10ms latency, 3 GB RAM (1M vecs)

IVF configurations per use case

Config A: High Accuracy

nlist = int(sqrt(n) * 1.5)  # More clusters
nprobe = int(nlist / 5)  # Explore 20% of clusters

# Example 1M vectors:
# nlist = 1500, nprobe = 300
# Result: 95% accuracy, 60ms latency

Config B: Balanced

nlist = int(sqrt(n))
nprobe = int(nlist / 10)

# Example 1M vectors:
# nlist = 1000, nprobe = 100
# Result: 93% accuracy, 40ms latency

Config C: Fast Query

nlist = int(sqrt(n) * 0.5)  # Fewer clusters
nprobe = int(nlist / 20)

# Example 1M vectors:
# nlist = 500, nprobe = 25
# Result: 90% accuracy, 25ms latency

IVF+PQ configurations per use case

Config A: High Accuracy (85-90%)

nlist = int(sqrt(n))
nprobe = int(nlist / 5)
m = 16  # More sub-vectors
nbits = 8

# Result: 89% accuracy, 80ms latency, 2x compression

Config B: Balanced

nlist = int(sqrt(n))
nprobe = int(nlist / 10)
m = 8
nbits = 8

# Result: 87% accuracy, 60ms latency, 6x compression

Config C: Maximum Compression

nlist = int(sqrt(n) * 0.8)
nprobe = int(nlist / 15)
m = 8
nbits = 4  # 4 bits (16 centroids)

# Result: 82% accuracy, 50ms latency, 12x compression

🚀 Migration Path: Scaling your RAG system

Stage 1: MVP (0-100K vectors)

Recommendation: HNSW (ChromaDB local)

# Simple setup
import chromadb
client = chromadb.Client()
collection = client.create_collection("docs")

# Metrics:
# - Accuracy: 98%
# - Latency: 15ms
# - Memory: 2 GB
# - Cost: $0 (self-hosted)

When to migrate: When you reach 100K vectors or 4 GB RAM.

Stage 2: Growth (100K-1M vectors)

Recommendation: HNSW (Pinecone/Weaviate managed)

# Pinecone managed
import pinecone
index = pinecone.Index("docs")

# Metrics:
# - Accuracy: 98%
# - Latency: 30ms (network)
# - Memory: Managed (no concern)
# - Cost: $70/month

When to migrate: When you reach 1M vectors or latency/downtime becomes critical.

Stage 3: Scale (1M-5M vectors)

Option A: HNSW (if budget allows)

# Weaviate managed with HNSW
# Cost: $200-500/month (16-32 GB RAM)
# Accuracy: 98%

Option B: IVF (if budget is limited)

# Faiss self-hosted with IVF
# Cost: $50-100/month (4-8 GB RAM)
# Accuracy: 93%

Decision: Accuracy critical → Option A. Cost critical → Option B.

Stage 4: Massive Scale (>5M vectors)

Recommendation: IVF + PQ (Faiss)

# Faiss IVF+PQ self-hosted
index = faiss.IndexIVFPQ(...)

# Metrics (10M vectors):
# - Accuracy: 88%
# - Latency: 80ms
# - Memory: 8 GB
# - Cost: $60/month

# vs HNSW:
# - Accuracy: 98%
# - Memory: 80 GB
# - Cost: $800/month ← 13x more expensive

Trade-off: 10% accuracy loss → 13x cost reduction.

🎯 Decision Scorecard

Decision matrix

Assign points (1-5) to each criterion based on your use case:

Criterion	Weight	HNSW	IVF	IVF+PQ
Accuracy >95%	×3	5	3	2
Latency <50ms	×2	5	3	3
Limited memory	×2	1	3	5
Incremental updates	×1	5	1	1
Fast build time	×1	2	4	2
Scale >5M vectors	×2	2	4	5

Example calculation (Customer Support):

HNSW:
  Accuracy: 5 × 3 = 15
  Latency: 5 × 2 = 10
  Memory: 1 × 2 = 2
  Incremental: 5 × 1 = 5
  Build: 2 × 1 = 2
  Scale: 2 × 2 = 4
  Total: 38 ✅ WINNER

IVF: Total: 34
IVF+PQ: Total: 35

Use this scorecard to evaluate your specific case.

✅ Comprehension checklist

Verify that you understood this capsule:

Which algorithm for 50K vectors, critical accuracy?
- Answer: HNSW (ChromaDB). High accuracy (98%), simple setup, reasonable memory (2 GB).
Which algorithm for 5M vectors, limited RAM?
- Answer: IVF+PQ (Faiss). 6-8x compression, 88% accuracy is acceptable, low cost.
When to migrate from HNSW to IVF?
- Answer: When >1M vectors, >16 GB memory required, or 90-95% accuracy is enough.
Which parameters to tune for higher accuracy in IVF?
- Answer: Increase nlist (more clusters) and nprobe (explore more clusters).
Main trade-off of PQ?
- Answer: 4-8x less memory vs 10-15% accuracy loss. Viable if 85-90% accuracy is enough.

If you answered 4-5/5 correctly → ✅ Ready for Capsule 08 (Summary + RAG)

🚀 Next step

In the module's last capsule, we'll consolidate everything and connect it with RAG systems.

Next capsule: 08 - Why this matters for RAG

You'll learn:

The impact of algorithms on RAG performance
The impact on costs (memory, compute, cloud)
The impact on accuracy (95% vs 100% - acceptable trade-off)
Module summary and transition to Module 3

Key: Applying everything you learned to real RAG architecture decisions.

Reading time: 8-10 minutes
Next: 08-why-rag-matters.md

← PreviousCapsule 06: PQ - Product Quantization Next →Capsule 08: Why this matters for RAG - Module Summary

Capsule 07: HNSW vs IVF vs PQ comparison - Decision Framework

🎯 Capsule objective

🎯 Decision Tree: Which algorithm to use?

Decision framework

Key questions

📊 Side-by-side comparison: Real scenarios

Scenario 1: Customer Support Chatbot

Scenario 2: E-commerce Product Search

Scenario 3: Internal Knowledge Base (Massive)

Scenario 4: Legal Document Analysis

🎨 Recommended configurations

HNSW configurations per use case

Config A: High Accuracy (Legal, Medical)

Config B: Balanced (Default)

Config C: Fast Query

IVF configurations per use case

Config A: High Accuracy

Config B: Balanced

Config C: Fast Query

IVF+PQ configurations per use case

Config A: High Accuracy (85-90%)

Config B: Balanced

Config C: Maximum Compression

🚀 Migration Path: Scaling your RAG system

Stage 1: MVP (0-100K vectors)

Stage 2: Growth (100K-1M vectors)

Stage 3: Scale (1M-5M vectors)

Stage 4: Massive Scale (>5M vectors)

🎯 Decision Scorecard

Decision matrix

✅ Comprehension checklist

🚀 Next step