Module 2: How Vector Databases Work (Conceptual)
Module 2: How Vector Databases Work (Conceptual)
Module overview
In Module 1 it was established WHY you need vector databases: SQL/NoSQL don't understand high-dimensional geometry, numpy/pandas don't scale to millions of vectors, and RAG needs retrieval in milliseconds. The conclusion closes: you need a specialized tool. But the natural next step is to ask HOW? How do vector databases respond in 15ms when numpy over the same data takes 1.5 seconds? What do they do internally that makes them 100x faster?
The short answer: indexing algorithms — HNSW, IVF, PQ. These algorithms transform O(n) search (checking every vector) into O(log n) (jumping intelligently to the right group). They aren't magic — they're sophisticated data structures with very concrete trade-offs. And understanding them at a conceptual level is the difference between an AI Engineer who debugs when things fail and one who looks at the API as a black box hoping it works.
This module is 100% conceptual — no executable code. The pedagogical reason is deliberate: when you reach Module 4 and configure hnsw:M=32 in ChromaDB, I don't want it to be copy-paste. I want you to know that M=32 means "32 connections per node in the HNSW graph," to understand that more connections improve recall but consume more RAM, and to be able to defend the choice of that value to a Tech Lead.
By the end of this module you'll be able to:
- ✅ Explain the three-layer architecture of a vector database (indexing, query, storage) and how they connect
- ✅ Describe HNSW conceptually — a hierarchical navigable graph with long → short jumps, O(log n) complexity
- ✅ Describe IVF conceptually — k-means clustering that partitions the space, O(√n) approximate
- ✅ Describe PQ conceptually — compression of vectors into sub-vectors and codebooks, 4-8x memory reduction
- ✅ Compare HNSW vs IVF vs PQ in quantified trade-offs: accuracy (98% vs 93% vs 88%), memory (6 GB vs 2 GB vs 1 GB for 1M vectors)
- ✅ Use a decision framework to choose an algorithm based on scale, accuracy, memory, and budget requirements
- ✅ Connect each algorithm with its impact on RAG: why HNSW is the default for <10M vectors, when to migrate to IVF+PQ
Estimated module time: 2-3 hours (8 capsules).
Why understanding the internal machinery makes you a better AI Engineer
Imagine three scenarios where you'll be asked to make decisions:
Scenario 1: the slow query with no visible cause
Your production RAG takes 800ms for retrieval. The SLA is 200ms. Everyone on the team suspects something different:
- Frontend says: "it's the network, let's add client-side caching."
- Backend says: "it's the LLM, let's switch to a faster model."
- DevOps says: "it's the database, let's scale up vertically."
You know none of them touched the indexing. You know that ef_search=10 is the default and that with 1M vectors that sometimes gives pathologically slow queries. You raise ef_search to 50, measure, and it's fixed. It took you 30 minutes. The alternative would have been 2 weeks migrating to another DB with no guarantee.
Without understanding HNSW, you can't even name the parameter that fixes the problem.
Scenario 2: the decision to migrate (or not) when the dataset grows
A company wants to scale from 100K vectors to 5M in 6 months. The CTO asks you: "Can ChromaDB handle it or do we have to migrate to Pinecone?"
If you only know "ChromaDB is local, Pinecone is cloud," your answer will be intuition. If you understand that ChromaDB uses pure HNSW and that the RAM consumed by HNSW grows linearly with the number of vectors (~6 GB per 1M with dim=1536), you can calculate: 5M vectors × 6 GB / 1M = 30 GB of RAM for the index alone. A 32 GB machine won't handle it.
Your answer stops being "I think so" and becomes "no, we need to migrate to IVF+PQ, which reduces the RAM to ~5 GB for 5M, or move to a distributed vector DB like a Milvus/Qdrant cluster." That difference between intuition and numbers is what separates a junior AI Engineer from a senior one.
Scenario 3: the accuracy bug that looks like chance
Recall@10 drops from 95% to 78% for no apparent reason. The queries are the same, the dataset grew but didn't change in nature, the code didn't change. What happened?
If you understand that when ef_construction (HNSW's build parameter) is low, the built graph is suboptimal and inserting many new docs slowly degrades the index quality, you know that rebuilding the index with ef_construction=200 fixes it. It took you 45 minutes. Without that knowledge, debugging took 2 sprints and got attributed to "weird, must be the model."
Each of these scenarios happens. The only thing that changes is whether you're the one who solves them.
What you will NOT do in this module
To calibrate expectations: this module is conceptual. You're going to:
✅ Understand the idea behind HNSW (hierarchical graph navigation) ✅ Visualize how the index is built (layers with jumps) ✅ Compare the trade-offs of the algorithms without advanced math
What you will not do:
❌ Implement HNSW from scratch ❌ Derive the Navier-Stokes equations of IVF clustering ❌ Compute gradients for Product Quantization
Why? Because you're an AI Engineer, not an ML Engineer or a research scientist.
Analogy: an airplane pilot needs to understand aerodynamics conceptually — lift, drag, turbulence, angle of attack. They know that insufficient speed + a high angle = loss of lift, and so they avoid that combination. They don't need to derive the Navier-Stokes equations to fly well.
Your goal is to be a pilot of vector databases. When you use ChromaDB and configure HNSW, you'll know what happens. When something fails, you'll know where to look. When a colleague says "let's add PQ," you'll know whether it makes sense for your case. You don't need more than that.
The three algorithms you'll understand
To anchor what's coming, here's the panoramic view of the module's three algorithms, in a single table:
| Algorithm | Core idea | Search complexity | Typical accuracy | Memory (1M 1536-dim vectors) | Used by |
|---|---|---|---|---|---|
| HNSW (Hierarchical Navigable Small World) | Layered graph; long jumps at the top, short at the bottom | O(log n) | 98% | ~6 GB | ChromaDB, Weaviate, Qdrant, Pinecone |
| IVF (Inverted File Index) | Group vectors by proximity; search only in the nearby group | O(√n) approx | 93% | ~2 GB | Faiss, Milvus |
| PQ (Product Quantization) | Compress each vector into small pieces represented by codes | O(n) but fast | 88% | ~1 GB | Faiss compression, Milvus |
Pattern to notice: all three fight over the same balance — accuracy vs memory vs speed. HNSW maximizes accuracy at the cost of memory. IVF balances. PQ minimizes memory at the cost of accuracy. There's no "best" algorithm in the abstract — there's the right algorithm for your case.
Capsules 04, 05, and 06 go deep on each one. Capsule 07 compares them with real benchmarks. Capsule 08 closes by connecting everything with RAG.
Module map
| Capsule | Topic | Why it matters | Time |
|---|---|---|---|
| 01 (here) | Module introduction | Why understanding the internal machinery and what you'll learn | 5-8 min |
| 02 | Three-layer architecture | Indexing, query, storage — how they connect | 25-30 min |
| 03 | Indexing algorithms — overview | Brute force vs ANN; when each one matters | 25-30 min |
| 04 | HNSW in depth | The most used algorithm: hierarchical graph, parameters (M, ef) | 35-40 min |
| 05 | IVF clustering | When IVF wins over HNSW (large datasets, limited memory) | 25-30 min |
| 06 | PQ compression | When to accept 10% less accuracy for 6x less memory | 30-35 min |
| 07 | Algorithm comparison | Decision framework: HNSW vs IVF vs PQ vs combinations | 25-30 min |
| 08 | Why it matters for RAG | Connection with the rest of the guide | 15-20 min |
Total estimate: 3-4 hours. It's a conceptually dense module — read it across 2-3 sessions so it sinks in.
Connection with the rest of the guide
Module 1: Why Vector DBs
└─ You establish the need
↓
Module 2 (you're here): How they work
└─ You understand the internal machinery
↓
Module 3: Features for RAG
└─ You learn which features of the machinery matter for RAG
↓
Module 4: ChromaDB hands-on
└─ You configure HNSW with full information (not copy-paste)
↓
Modules 5-7: Landscape, decision, production
└─ You compare with judgment (HNSW vs Pinecone's custom, etc.)
↓
Module 8: Capstone project
└─ You apply everything in a real RAG system
The decisions you'll make later depend on what you understand here:
- In M4 you'll configure
hnsw:M=32andhnsw:construction_ef=200. If you understand HNSW (capsule 04), those numbers make sense. If not, they're magic. - In M5 you'll compare Pinecone vs ChromaDB vs Qdrant. Each one uses a different variant of the same family of algorithms. If you understand HNSW, you can evaluate the variants with judgment.
- In M7 you'll optimize costs. If your system has 5M vectors and costs $300/mo in RAM, you'll know that migrating to IVF+PQ brings it down to $50/mo at the cost of 5% recall — and you'll know whether that trade-off is worth it for your case.
Anticipatory view: three algorithms, one same trade-off
Before getting into the detail, notice how the three algorithms are solving the same problem with different compromises. You'll see this table in detail in capsule 07, but it's worth keeping in mind from the start:
Memory ←——————————————→ Accuracy
(minimum) (maximum)
│ │
PQ ─────────● │
│ HNSW │
IVF ────────────────────────● │
│ │
(1 GB │ (8 GB (12 GB
for 1M) │ for 1M) for 1M)
│
└───────────────────────────────────────→
Speed
│
(HNSW > IVF > PQ
in typical latency)
What defines which algorithm you use is not which one is "the best" but which is the hardest constraint of your system:
- Is memory the constraint? PQ. You accept 5-10% less accuracy in exchange for 6-8x less RAM.
- Is accuracy the constraint? HNSW with a high
M. You pay more RAM, you gain almost-perfect recall. - A balance between the two? IVF, or HNSW with a medium configuration.
When you reach production, you'll make this decision literally — and the correct answer depends on your hardware, your budget, and your SLA. That's the decision this module prepares you to make.
How I recommend reading this module
This is the most conceptually dense module of the guide. Some method recommendations:
1. Don't read it all in one go. Capsules 02-07 are dense. Better:
- Session 1 (1h): capsules 01 (this one) + 02 (three-layer architecture) + 03 (algorithm overview)
- Session 2 (1h): capsule 04 (HNSW in depth) — the most important
- Session 3 (1h): capsules 05 (IVF) + 06 (PQ)
- Session 4 (30min): capsules 07 (comparison) + 08 (connection with RAG)
2. Don't memorize formulas. If you find yourself copying equations into a notebook, you're missing the point. What matters are the mental models — what each algorithm does, how it differs, when to use it.
3. Come back to this intro between capsules. When you finish capsule 04 (HNSW), come back here and check whether you can now explain HNSW in 3 sentences. If not, re-read capsule 04 before moving on.
4. The IVF and PQ capsules are less critical. If your system uses ChromaDB or Pinecone, you'll use HNSW almost always. IVF and PQ are useful for huge datasets (10M+ vectors) or extreme memory constraints. If you're in an MVP with 100K vectors, you can skim them.
5. Capsule 07 (comparison) is the most practical. It's where all the pieces come together in an applicable decision framework. If you only have time for a second reading, that's the candidate.
What you do NOT need to have learned to start
Sometimes the biggest fear of a "conceptual" module is that it requires math you don't have. To anchor expectations — this is what you do NOT need to know to understand the module:
- ❌ Advanced linear algebra (eigenvectors, SVD, etc.)
- ❌ Calculus (gradients, derivatives)
- ❌ Probability and inferential statistics
- ❌ Formal graph theory
- ❌ Algorithm analysis (we'll use big-O notation, but conceptually)
What you do need:
- ✅ The basic concept of a vector and dimensions (Module 1 + Guide #5 AI Semantics + Guide #6 Embeddings already cover this)
- ✅ The idea that two vectors can be "near" or "far" in the space
- ✅ Familiarity with reading pseudo-code
If you finish a typical AI Engineering path module, you have more than enough. Don't get stuck on the math — the goal is intuition.
Self-check before moving on
Before continuing to capsule 02, make sure you can answer these questions (if not, re-read this intro):
- Why is this module conceptual instead of asking you to implement HNSW from scratch?
- Name three real situations where not understanding the internal machinery prevents you from making good decisions.
- Which of the three algorithms (HNSW, IVF, PQ) would prioritize minimal memory over accuracy?
Answers
-
Because your role as an AI Engineer is to use the tools with judgment, not to build them. Implementing HNSW from scratch would take you months and wouldn't make you better at your job. Understanding the algorithm conceptually takes you 40 minutes and lets you debug, optimize, and decide.
-
(a) A slow query with no visible cause — without understanding HNSW, you can't name
ef_search, the parameter that typically fixes the case. (b) The decision to migrate when the dataset grows — without understanding HNSW's RAM consumption, you can't calculate whether your current hardware handles the projected growth. (c) An accuracy bug that looks like chance — without understanding the role ofef_construction, you don't know that rebuilding the index with a better parameter fixes it. -
PQ (Product Quantization). You compress each vector to 1/4 to 1/8 of the original size at the cost of ~10% accuracy. It's the right algorithm when your bottleneck is RAM/storage and you can tolerate lower accuracy — typically datasets >10M vectors on modest machines.
Next step: Capsule 02
The next capsule opens the box: three-layer architecture — indexing, query engine, storage. Before getting into the specific algorithms (HNSW, IVF, PQ in capsules 04-06), you need the mental model of how the pieces connect. It's the foundation everything else is built on.
Estimated time: 5-8 minutes Next: 02-three-layer-architecture.md