Module 2: How Vector Databases Work (Conceptual)

Capsule 02: Vector Database Architecture (3 Layers)

Understand the 3 architectural layers of a vector database (Indexing, Query Engine, Storage) and how they interact to achieve efficient search over millions of vectors.

By the end of this capsule:

✅ You'll explain the 3 layers and their responsibility
✅ You'll understand the full flow: Insert → Index → Query → Retrieve
✅ You'll grasp why layer separation matters for performance

Estimated time: 10-12 minutes

🏗️ The 3 layers of a Vector Database

A vector database is NOT just "storing embeddings on disk." It's a 3-layer architecture designed for efficient search at scale.

Overview

┌─────────────────────────────────────────┐
│  LAYER 1: INDEXING                      │
│  Build a structure for fast             │
│  search (HNSW, IVF, PQ)                 │
└─────────────────────────────────────────┘
              ↓
┌─────────────────────────────────────────┐
│  LAYER 2: QUERY ENGINE                  │
│  Execute search on the index            │
│  (ANN search, filtering, ranking)       │
└─────────────────────────────────────────┘
              ↓
┌─────────────────────────────────────────┐
│  LAYER 3: STORAGE                       │
│  Persist vectors + metadata on disk     │
│  (durability, backups, replication)     │
└─────────────────────────────────────────┘

Key: Each layer has a specific responsibility. This allows optimizing each one independently.

Layer 1: Indexing (Index construction)

What does it do?

It builds a specialized data structure for fast search, transforming a disordered collection of vectors into a navigable graph/tree.

Why does it matter?

Without indexing, search is brute force O(n):

# Without an index (brute force)
for vector in database:  # ❌ 1M iterations
    similarity = cosine(query, vector)

With indexing, search is O(log n) or better:

# With an HNSW index (conceptual)
path = navigate_graph(query)  # ✅ ~100 jumps (not 1M)
candidates = get_neighbors(path)

Main algorithms

Algorithm	Type	Complexity	Used by
HNSW	Hierarchical navigable graph	O(log n)	ChromaDB, Weaviate, Qdrant
IVF	Clustering + buckets	O(sqrt(n))	Faiss, Milvus
PQ	Vector compression	O(n) compressed	Faiss, Milvus (addon)
ScaNN	Learned quantization	O(log n)	Google Research

We'll go deeper on each algorithm in Capsules 04-06.

Trade-offs

Build time vs Query time:

HNSW: Slow build (1M vectors = 5-10 min), fast Query (15ms)
IVF: Fast build (1M vectors = 2-3 min), medium Query (30ms)

Memory vs Accuracy:

HNSW: More memory (4-8 GB for 1M 1536-dim vectors), 98% accuracy
IVF+PQ: Less memory (1-2 GB), 88-93% accuracy

Layer 2: Query Engine (Search engine)

What does it do?

It executes search on the index using the appropriate algorithm + applies filters + ranking.

Responsibilities

ANN Search (Approximate Nearest Neighbors)
- Navigate the HNSW/IVF index
- Find the top-k candidates
Metadata filtering (crucial for RAG)
- Filter by date, author, category
- RAG example: Only search in "customer support" documents
Post-filtering ranking
- Re-score results with metadata
- Remove duplicates

Query flow

User query → Embed → ANN Search → Filter → Rank → Top-k results

Concrete RAG example:

# Query
query = "How do I reset my password?"
query_embedding = embed(query)  # [1536 dims]

# Layer 2: Query Engine executes
results = db.query(
    query_embedding=query_embedding,
    k=10,  # Top 10 candidates
    where={"category": "support", "language": "en"},  # Metadata filter
)

# Results: Top 10 relevant, filtered chunks

Trade-offs

Filtering before vs after ANN:

Pre-filtering (filter before search):

✅ Faster (searches a subset)
❌ Can fail if the subset is empty
Used by: Pinecone, Weaviate

Post-filtering (filter after search):

✅ Always returns results
❌ Slower (searches everything, then filters)
Used by: ChromaDB (default)

Decision: Depends on your RAG use case (filter volume, metadata cardinality).

Layer 3: Storage (Persistence)

What does it do?

It persists vectors + metadata on disk with guarantees of durability, consistency, and backups.

Responsibilities

Durability (don't lose data)
- Write-ahead log (WAL)
- Periodic snapshots
Replication (high availability)
- Master-replica setup
- Eventual consistency
Compression (reduce storage cost)
- Compress vectors on disk
- Decompress in memory for querying

Common implementations

Storage Backend	Used by	Characteristics
Embedded DB (SQLite, RocksDB)	ChromaDB	Local, single-node, easy setup
Object Storage (S3, GCS)	Pinecone	Cloud-native, infinitely scalable
Columnar Storage (Parquet)	Weaviate	Efficient for analytics
Distributed FS (HDFS, Ceph)	Milvus	Enterprise, multi-node

Trade-offs

In-memory vs Disk:

In-memory:

✅ Ultra-low latency (<5ms)
❌ Expensive (RAM > disk)
❌ Scale limit (RAM is finite)
Used by: Redis + RediSearch

Disk-based:

✅ Economical (disk is cheap)
✅ Infinitely scalable
❌ Higher latency (10-50ms)
Used by: ChromaDB, Pinecone, Milvus

Hybrid (most of production):

Hot data (recent) → RAM
Cold data (old) → Disk
Best of both worlds

🔄 Full flow: Insert → Query

Now that you know the 3 layers, let's see how they interact in real operations.

Operation 1: Insert (add vectors)

User                Layer 1         Layer 2         Layer 3
 │                  Indexing        Query Eng       Storage
 │
 ├─ insert(vec) ──→ Build index ──→ (no-op) ──────→ Persist to disk
 │                  (HNSW add)                       (append WAL)
 │                      │                                 │
 │                      └─────── Index built ────────────┘
 │                               (async)
 │
 └─ ACK inserted

Steps:

User inserts vector + metadata
Layer 1 (Indexing): Adds the vector to the HNSW index (updates the graph)
Layer 3 (Storage): Persists to disk (write-ahead log)
ACK to the user (operation complete)

Note: Building the index can be async (batch updates every N inserts).

Operation 2: Query (search for similar)

User                Layer 1         Layer 2         Layer 3
 │                  Indexing        Query Eng       Storage
 │
 ├─ query(vec) ────→ ANN search ──→ Apply filters ─→ Fetch metadata
 │                   (navigate      (where clause)   (from disk)
 │                    HNSW graph)        │                │
 │                       │               │                │
 │                       └───── Top-k candidates ────────┘
 │                                       │
 │                                 Re-rank + limit
 │                                       │
 └─ ← Results ────────────────────────────┘

Steps:

User sends a query vector
Layer 2 (Query Engine): Executes ANN search on the index (Layer 1)
Layer 1 (Indexing): Navigates the HNSW graph → returns candidates
Layer 2 (Query Engine): Applies metadata filters
Layer 3 (Storage): Fetches full metadata for the candidates
Layer 2 (Query Engine): Re-ranks + limits to top-k
Returns results to the user

Typical latency: 15-50ms (1M vectors, 1536-dim, HNSW)

🏭 Real example: ChromaDB architecture

Let's see how ChromaDB implements these 3 layers.

Layer 1: Indexing

Algorithm: HNSW (default)
Library: hnswlib (C++ binding)
Build strategy: Incremental (add vectors one by one)

Layer 2: Query Engine

ANN search: HNSW navigation
Filtering: Post-filtering (search first, filter afterward)
Ranking: Cosine similarity (default) or dot product

Layer 3: Storage

Backend: SQLite (embedded DB)
Durability: Write-ahead log (WAL)
Storage: SQLite database (metadata) + binary segments (HNSW index)

Advantage: The entire architecture in one process (no external server required).

Disadvantage: Single-node (not distributed). For scale → migrate to Pinecone/Weaviate.

📊 Benchmark: Impact of each layer

Let's measure each layer's latency in a typical query (1M vectors, 1536-dim, ChromaDB).

Layer	Operation	Latency	% of total
Layer 1	ANN search (HNSW navigate)	12 ms	60%
Layer 2	Metadata filtering	3 ms	15%
Layer 3	Fetch full metadata from disk	5 ms	25%
Total	End-to-end query	20 ms	100%

Insights:

Layer 1 (indexing) dominates latency → Optimizing the HNSW algorithm is critical
Layer 3 (storage) is 25% → Using an SSD (not HDD) helps significantly
Layer 2 (filtering) is cheap → You can filter aggressively without penalty

Comparison with brute force:

Method	Latency	Speedup
Brute force (numpy)	1500 ms	1x
Vector DB (HNSW)	20 ms	75x faster

🤔 Why layer separation matters

Advantage 1: Independent optimization

Without separation:

# ❌ Everything in one function (monolithic)
def search(query):
    results = brute_force_search(query)  # Slow
    filtered = apply_filters(results)
    return persist(filtered)

With separation:

# ✅ Each layer independently optimizable
def search(query):
    candidates = layer1.ann_search(query)  # Optimized: HNSW
    filtered = layer2.filter(candidates)   # Optimized: Bitmap index
    full = layer3.fetch(filtered)          # Optimized: Columnar storage
    return full

Result: You can change the Layer 1 algorithm (HNSW → IVF) without touching Layers 2-3.

Advantage 2: Horizontal scaling

Layer 1 (Indexing):

Scales with RAM (index in memory)
Requires a beefy machine (32-64 GB RAM)

Layer 3 (Storage):

Scales with disk (S3, GCS)
Can be separated to a storage cluster

Distributed architecture:

┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│  Query Node  │───→│  Index Node  │───→│ Storage Node │
│  (Layer 2)   │    │  (Layer 1)   │    │  (Layer 3)   │
└──────────────┘    └──────────────┘    └──────────────┘

Example: Pinecone uses a distributed architecture (Layers 1-3 on separate nodes).

Advantage 3: Configuration flexibility

You can tune each layer according to requirements:

Case A: Latency-critical (chatbot)

Layer 1: HNSW (high accuracy, 15ms)
Layer 2: Pre-filtering (reduces search space)
Layer 3: In-memory (Redis, <5ms)
Result: <20ms end-to-end

Case B: Cost-optimized (batch processing)

Layer 1: IVF (less memory, 50ms)
Layer 2: Post-filtering (simple)
Layer 3: Disk (S3, 100ms)
Result: <200ms end-to-end (but 10x cheaper)

✅ Comprehension checklist

Verify that you understood this capsule:

What are the 3 layers of a vector database?
- Answer: Indexing (build the index), Query Engine (execute search), Storage (persist data)
What does Layer 1 (Indexing) do?
- Answer: Builds a specialized data structure (HNSW, IVF, PQ) for O(log n) vs O(n) search
What does Layer 2 (Query Engine) do?
- Answer: Executes ANN search, applies metadata filters, re-ranks results
What does Layer 3 (Storage) do?
- Answer: Persists vectors + metadata with durability, replication, compression
Why does layer separation matter?
- Answer: It allows optimizing each layer independently, scaling horizontally, tuning trade-offs by use case

If you answered 4-5/5 correctly → ✅ Ready for Capsule 03 (Indexing Algorithms Overview)

🔗 Connection with RAG

How does this help in RAG?

When you build a RAG system, you need to understand the architecture to:

Debug performance

Problem: "My RAG takes 500ms for retrieval. Where's the bottleneck?"

Solution with layer knowledge:

Profile each layer:
- Layer 1 (indexing): 450ms → Problem here (poorly configured index)
- Layer 2 (query): 30ms
- Layer 3 (storage): 20ms
Fix: Rebuild the HNSW index with better parameters (efConstruction=200)

Optimize costs

Problem: "ChromaDB local consumes 64 GB RAM. Too expensive."

Solution with layer knowledge:

Layer 1: Switch HNSW → IVF (less memory)
Layer 3: Use disk storage instead of in-memory
Result: 16 GB RAM (4x reduction)

Make better decisions

Problem: "Migrate from ChromaDB to Pinecone?"

Comparison with architecture knowledge:

Layer	ChromaDB	Pinecone
Layer 1	HNSW (high accuracy)	Custom (optimized)
Layer 2	Post-filtering	Pre-filtering
Layer 3	SQLite (local)	S3 (cloud)
Scale	Single-node	Multi-node
Latency	20ms (local)	50ms (network)
Cost	Free (self-hosted)	$70/month

Informed decision: ChromaDB for MVP (<100K vectors), Pinecone for production (>1M vectors).

🚀 Next step

Now that you understand the 3-layer architecture, we'll go deeper into Layer 1 (Indexing).

Next capsule: 03 - Indexing Algorithms Overview

You'll learn:

Brute force O(n) vs indexing O(log n) (simple math)
Main algorithms: HNSW, IVF, PQ, ScaNN
Trade-offs: Accuracy vs Speed vs Memory
When to use each one

Why does it matter? Layer 1 dominates 60-70% of query latency. Optimizing indexing = the biggest impact on performance.

Reading time: 10-12 minutes
Next: 03-indexing-algorithms-overview.md

← PreviousModule 2: How Vector Databases Work (Conceptual)Next →Capsule 03: Indexing Algorithms - Overview

Capsule 02: Vector Database Architecture (3 Layers)

🎯 Capsule objective

🏗️ The 3 layers of a Vector Database

Overview

Layer 1: Indexing (Index construction)

What does it do?

Why does it matter?

Main algorithms

Trade-offs

Layer 2: Query Engine (Search engine)

What does it do?

Responsibilities

Query flow

Trade-offs

Layer 3: Storage (Persistence)

What does it do?

Responsibilities

Common implementations

Trade-offs

🔄 Full flow: Insert → Query

Operation 1: Insert (add vectors)

Operation 2: Query (search for similar)

🏭 Real example: ChromaDB architecture

Layer 1: Indexing

Layer 2: Query Engine

Layer 3: Storage

📊 Benchmark: Impact of each layer

🤔 Why layer separation matters

Advantage 1: Independent optimization

Advantage 2: Horizontal scaling

Advantage 3: Configuration flexibility

✅ Comprehension checklist

🔗 Connection with RAG

Debug performance

Optimize costs

Make better decisions

🚀 Next step