Module 2: How Vector Databases Work (Conceptual)

Capsule 08: Why this matters for RAG - Module Summary

Inicia sesion para marcar tu progreso

Consolidate the Module 2 learnings and connect vector database architecture with practical decisions in RAG systems.

By the end of this capsule:

✅ You'll explain the impact of indexing algorithms on RAG performance
✅ You'll understand the impact on costs (10x difference)
✅ You'll justify accuracy vs cost trade-offs
✅ You'll be ready for Module 3 (Features for RAG)

Estimated time: 5-7 minutes

🏗️ Summary: The 3 layers + Algorithms

What you learned in this module

Module 2 in one image:

┌──────────────────────────────────────────────────┐
│  LAYER 1: INDEXING                               │
│                                                  │
│  HNSW: O(log n), 98% accuracy, 6 GB (1M vecs)  │
│  IVF:  O(sqrt(n)), 93% accuracy, 2 GB          │
│  PQ:   O(n) compressed, 88% accuracy, 1 GB     │
│                                                  │
│  Trade-off: Accuracy vs Speed vs Memory         │
└──────────────────────────────────────────────────┘
                      ↓
┌──────────────────────────────────────────────────┐
│  LAYER 2: QUERY ENGINE                           │
│  - ANN search using the Layer 1 index            │
│  - Metadata filtering (where clauses)            │
│  - Ranking and re-scoring                        │
└──────────────────────────────────────────────────┘
                      ↓
┌──────────────────────────────────────────────────┐
│  LAYER 3: STORAGE                                │
│  - Persist vectors + metadata                    │
│  - Durability (WAL, snapshots)                   │
│  - Replication (HA)                              │
└──────────────────────────────────────────────────┘

Key message: Understanding the internal architecture lets you debug, optimize, and make better decisions in RAG.

📊 Impact on RAG Performance

Retrieval is the bottleneck

A typical RAG pipeline:

User Query (100ms total)
    ↓
1. Embed query (OpenAI API)            10ms  (10%)
    ↓
2. Vector DB retrieval ← BOTTLENECK    60ms  (60%)
    ↓
3. LLM generation (OpenAI API)         30ms  (30%)
    ↓
Response

Key: Retrieval dominates 60% of the latency. Optimizing indexing = the biggest impact.

Comparison: Brute Force vs HNSW vs IVF vs PQ

Scenario: 1M vectors, 1536-dim, top-10 retrieval

Algorithm	Latency	Accuracy	% vs Requirement
Brute Force (numpy)	1500 ms	100%	❌ 3x slower (requirement: <500ms)
HNSW	18 ms	98%	✅ 27x faster than the requirement
IVF	40 ms	93%	✅ 12x faster than the requirement
IVF+PQ	60 ms	88%	✅ 8x faster than the requirement

Conclusion: Vector databases (with indexing) are ESSENTIAL for RAG in production.

The impact of accuracy loss

Question: Does 98% vs 88% accuracy matter in RAG?

Experiment:

Query: "How do I reset my password?"
Database: 100K support articles

HNSW (98% accuracy):
  Top 10 results: 9.8 relevant, 0.2 irrelevant
  
IVF (93% accuracy):
  Top 10 results: 9.3 relevant, 0.7 irrelevant
  
IVF+PQ (88% accuracy):
  Top 10 results: 8.8 relevant, 1.2 irrelevant

Does this affect the LLM's answer?

In practice: It depends on your threshold.

Case A: Top-3 context (critical)

HNSW: 2.94 relevant → ✅ Excellent
IVF: 2.79 relevant → ✅ Acceptable
IVF+PQ: 2.64 relevant → ⚠️ May fail

Case B: Top-10 context (majority voting)

HNSW: 9.8 relevant → ✅ Excellent
IVF: 9.3 relevant → ✅ Very good
IVF+PQ: 8.8 relevant → ✅ Enough

Conclusion: With top-10 retrieval, 88-93% accuracy is enough (the LLM can filter out noise).

💰 Impact on Costs

Memory cost (AWS)

Scenario: 10M vectors, 1536-dim

Algorithm	RAM required	Instance	Cost/month
HNSW	80 GB	r6i.4xlarge (128 GB)	$730/month
IVF	20 GB	r6i.xlarge (32 GB)	$180/month
IVF+PQ	8 GB	r6i.large (16 GB)	$90/month

Difference: 8x cost between HNSW and IVF+PQ.

Trade-off: 10% accuracy loss → 8x cost reduction.

Break-even analysis

Question: When is it worth sacrificing accuracy for cost?

Case A: Customer-facing chatbot (critical accuracy)

HNSW: $730/month
- Accuracy: 98%
- Potential revenue loss from bad answers: $0

ROI: HNSW is worth it

Case B: Internal search (accuracy not critical)

IVF+PQ: $90/month
- Accuracy: 88%
- Potential productivity loss: Minimal (users retry the query)

Savings: $640/month = $7680/year
ROI: IVF+PQ is worth it

Framework:

Customer-facing + revenue impact → HNSW
Internal + no revenue impact → IVF or IVF+PQ

🎯 Decision Framework for RAG

Decision matrix by scenario

Scenario A: RAG Chatbot (Customer-facing)

Requirements:
- Vectors: 100K-500K
- Accuracy: >95% (customer satisfaction is critical)
- Latency: <500ms
- Updates: Incremental (daily)

Algorithm: HNSW
DB: ChromaDB (MVP) → Pinecone/Weaviate (scale)

Scenario B: RAG Search (Internal)

Requirements:
- Vectors: 1M-5M
- Accuracy: 90-95% (internal, not critical)
- Latency: <1s
- Updates: Batch (weekly)

Algorithm: IVF
DB: Faiss (self-hosted) or Weaviate (managed)

Scenario C: RAG Analytics (Massive scale)

Requirements:
- Vectors: 10M-100M
- Accuracy: 85-90% (analytics, not real-time)
- Latency: <2s
- Updates: Batch (monthly)

Algorithm: IVF + PQ
DB: Faiss (self-hosted, optimized)

Decision checklist

Use this checklist before choosing an algorithm:

Dataset size
- < 1M → HNSW viable
- 1M-5M → IVF or HNSW (depending on RAM)
- 5M → IVF+PQ required
Accuracy requirement
- 95% → HNSW only
- 90-95% → IVF
- 85-90% → IVF+PQ
Latency requirement
- <50ms → HNSW
- <200ms → IVF
- <500ms → IVF+PQ
Budget
- High → HNSW (managed)
- Medium → IVF (self-hosted)
- Low → IVF+PQ (self-hosted)
Update pattern
- Incremental → HNSW
- Batch → IVF or IVF+PQ

If 4-5 criteria point to algorithm X → Use that algorithm.

🧠 Applied knowledge: Real cases

Case 1: Notion AI (Document Search)

Problem:

50M documents
Latency <100ms
Customer-facing (critical accuracy)

Solution (estimated):

Algorithm: HNSW (variant)
Architecture: Distributed HNSW (multiple shards)
Cost: $50K-100K/month in infra

Trade-off: High cost is acceptable (revenue justifies it).

Case 2: Perplexity AI (Web Search + RAG)

Problem:

100M+ web pages indexed
Latency <200ms
90% accuracy is enough (multi-source RAG)

Solution (estimated):

Algorithm: IVF (clustering by domain)
Architecture: Distributed shards by topic
Optimization: Metadata pre-filtering (domain, recency)

Trade-off: 93% vs 98% accuracy → It doesn't matter because it's multi-source (diversity > precision).

Case 3: ChatGPT Plugins (Enterprise RAG)

Problem:

Variable (100K-10M docs per customer)
Latency <500ms
Multi-tenant (isolation is critical)

Solution (estimated):

Algorithm: HNSW (per-tenant index)
Architecture: Pinecone (managed, native isolation)
Cost: $70-200/month per tenant

Trade-off: The managed service is more expensive but reduces ops overhead.

✅ Module 2 Summary

What you mastered

1. The 3-layer architecture:

✅ Layer 1 (Indexing): Build the search structure
✅ Layer 2 (Query Engine): Run the search + filters
✅ Layer 3 (Storage): Persist with durability

2. Indexing algorithms:

✅ HNSW: O(log n), 98% accuracy, best for <1M vectors
✅ IVF: O(sqrt(n)), 93% accuracy, best for 1M-5M vectors
✅ PQ: O(n) compressed, 88% accuracy, best for >5M vectors

3. Decision framework:

✅ Choose an algorithm based on requirements (accuracy, latency, memory, budget)
✅ Configure parameters (M, efConstruction, nlist, nprobe, m, nbits)
✅ Migration path (HNSW → IVF → IVF+PQ depending on scale)

4. Application to RAG:

✅ Optimize the retrieval bottleneck (60% of latency)
✅ Accuracy vs cost trade-off (10% accuracy → 8x cost savings)
✅ Configure per scenario (chatbot vs search vs analytics)

🎓 Unlocked skills

Now you can:

Debug performance issues

Problem: "Retrieval takes 500ms"

Analysis:
- Layer 1 (indexing): 450ms ← BOTTLENECK
- Layer 2 (query): 30ms
- Layer 3 (storage): 20ms

Solution:
- Rebuild the HNSW index with efConstruction=200 (vs 100)
- Result: 50ms retrieval (9x improvement)

Optimize costs

Problem: "ChromaDB consumes 64 GB RAM"

Analysis:
- 10M vectors × 6 KB = 60 GB
- HNSW overhead: 4 GB

Solution:
- Migrate to IVF+PQ
- Result: 12 GB RAM (5x reduction)
- Trade-off: 98% → 88% accuracy (acceptable for internal use)

Make architectural decisions

Question: "ChromaDB or Pinecone?"

Decision tree:
- < 1M vectors → ChromaDB (self-hosted, free)
- 1M-10M vectors → Pinecone (managed, $70-200/month)
- > 10M vectors → Faiss IVF+PQ (self-hosted, optimized)

Justification: Scale + ops overhead vs cost

🔗 Connection with the next modules

Module 3: Essential Features for RAG

Now that you understand HOW vector databases work internally, you'll learn WHAT features you need for RAG:

Topics:

Metadata filtering (where clauses)
Hybrid search (keyword + semantic)
Multi-tenancy (data isolation)
Batch operations (bulk insert/update)
Monitoring and observability

With your knowledge of the internal architecture, you'll know WHY these features are important and HOW they impact performance.

Modules 4-5: ChromaDB Hands-On

You'll implement a RAG system with ChromaDB:

Set up HNSW with optimized parameters
Configure metadata filtering
Benchmark performance (validate <50ms retrieval)

Your knowledge of HNSW will let you configure it intelligently (not just copy a tutorial).

Modules 6-8: Production Considerations

You'll scale the RAG system:

When to migrate from ChromaDB to Pinecone/Weaviate
Optimize costs (HNSW → IVF when >1M vectors)
Monitor accuracy degradation

Your knowledge of algorithms will let you make informed architecture decisions.

✅ Final module test

Confirm you mastered Module 2:

Question 1

Your RAG system has 500K vectors, accuracy must be >95%, latency <500ms, limited budget.
Which algorithm and DB would you use?

Solution

HNSW with ChromaDB (self-hosted)

Reason:

Accuracy: 98% ✅ (meets >95%)
Latency: 18ms ✅ (meets <500ms)
Memory: 3 GB ✅ (reasonable)
Cost: $0 self-hosted ✅ (limited budget)

Configuration:

collection = client.create_collection(
    name="docs",
    metadata={
        "hnsw:M": 16,
        "hnsw:construction_ef": 200,
        "hnsw:search_ef": 100,
    }
)

Question 2

Your retrieval takes 300ms. Profiling shows: Layer 1 (indexing) = 250ms, Layer 2 = 30ms, Layer 3 = 20ms.
What would you optimize?

Solution

Optimize Layer 1 (Indexing)

Actions:

Check the algorithm: HNSW or IVF?
If HNSW: Reduce efSearch (200 → 100)
If IVF: Reduce nprobe (100 → 50)
Check the dataset size: If >1M vectors, consider migrating HNSW → IVF

Justification:

Layer 1 dominates 83% of the latency (250ms / 300ms)
Layers 2-3 are reasonable (50ms)
Optimizing Layer 1 = the biggest ROI

Question 3

You have 20M vectors, limited RAM (16 GB), 88% accuracy is acceptable.
Which configuration would you use?

Solution

IVF + PQ with Faiss

import faiss

dimension = 1536
nlist = 4472  # sqrt(20M)
m = 8
nbits = 8

index = faiss.IndexIVFPQ(
    faiss.IndexFlatL2(dimension),
    dimension,
    nlist,
    m,
    nbits
)

index.nprobe = 50  # 1% of clusters

Result:

Memory: ~12 GB ✅ (fits in 16 GB)
Accuracy: ~88% ✅ (meets the requirement)
Latency: ~90ms ✅

Alternative (if RAM still isn't enough):

Reduce nbits = 4 (12x compression)
Accuracy drops to ~85%
Memory: ~6 GB If you answered 2-3/3 correctly → ✅ YOU MASTERED MODULE 2

🚀 Next step: Module 3

You already understand:

✅ WHY you need vector databases (Module 1)
✅ HOW they work internally (Module 2)

Now you'll learn:

🎯 WHAT features are essential for RAG (Module 3)

Module 3: Essential Features for RAG

Topics:

Metadata filtering (where clauses) → Why Layer 2 matters
Hybrid search (keyword + semantic) → Combining BM25 + vector search
Multi-tenancy (isolation) → Security in multi-customer RAG
Batch operations → Optimizing ingestion of 1M documents
Monitoring → Detecting accuracy degradation

Duration: 60-75 minutes (8 capsules)

Ready? Go to Module 3 - Essential Features for RAG

Reading time: 5-7 minutes
Next module: ../../module-03-essential-rag-features/en/01-module-introduction-3.md

← PreviousCapsule 07: HNSW vs IVF vs PQ comparison - Decision Framework Next →Module 3: Essential Features for RAG

Capsule 08: Why this matters for RAG - Module Summary

🎯 Capsule objective

🏗️ Summary: The 3 layers + Algorithms

What you learned in this module

📊 Impact on RAG Performance

Retrieval is the bottleneck

Comparison: Brute Force vs HNSW vs IVF vs PQ

The impact of accuracy loss

💰 Impact on Costs

Memory cost (AWS)

Break-even analysis

🎯 Decision Framework for RAG

Decision matrix by scenario

Scenario A: RAG Chatbot (Customer-facing)

Scenario B: RAG Search (Internal)

Scenario C: RAG Analytics (Massive scale)

Decision checklist

🧠 Applied knowledge: Real cases

Case 1: Notion AI (Document Search)

Case 2: Perplexity AI (Web Search + RAG)

Case 3: ChatGPT Plugins (Enterprise RAG)

✅ Module 2 Summary

What you mastered

🎓 Unlocked skills

🔗 Connection with the next modules

Module 3: Essential Features for RAG

Modules 4-5: ChromaDB Hands-On

Modules 6-8: Production Considerations

✅ Final module test

Question 1

Solution

Question 2

Solution

Question 3

Solution

🚀 Next step: Module 3