Module 1: Why Vector Databases for AI Engineers

When you DO need a vector database

Next →

Capsule overview

You've seen the limitations of SQL, NoSQL, and numpy. Now the critical question: When DO you need a vector database?

This capsule gives you clear, quantitative criteria to decide. It's not "always use a vector DB" or "never use numpy." It's: "IF you meet these 4 criteria, a vector DB is necessary. If not, alternatives may be enough."

By the end of this capsule, you'll be able to evaluate your RAG project in 5 minutes and decide whether you need a vector DB or not.

The 4 decision criteria

Criterion 1: Scale (# of vectors)

Simple rule:

<1K vectors:    numpy enough      (prototype)
1K-10K vectors: numpy acceptable  (development)
10K-100K:       gray zone          (evaluate latency)
>100K vectors:  vector DB needed   (production)

Why 100K is the inflection point:

  • numpy with 100K: ~150ms latency (acceptable limit)
  • numpy with 500K: ~750ms latency (unusable for RAG)
  • ChromaDB with 100K: ~8ms (excellent)
  • ChromaDB with 500K: ~12ms (excellent)

Quick calculation:

# Estimate the # of vectors in your project
docs = 50_000  # documents
chunks_per_doc = 3  # average chunks per doc
total_vectors = docs * chunks_per_doc

print(f"Total vectors: {total_vectors:,}")
# Output: Total vectors: 150,000

# Decision:
if total_vectors > 100_000:
    print("→ Vector DB needed")
else:
    print("→ numpy may be enough (evaluate latency)")

Criterion 2: Latency (<500ms retrieval)

Simple rule:

Latency requirement   | Option
----------------------|-------------------
No requirement        | numpy (batch processing)
<5s                   | numpy/SQL can work
<1s                   | Gray zone (evaluate)
<500ms                | Vector DB needed
<100ms                | Vector DB mandatory

Why <500ms is critical:

Total RAG latency:

Query embedding:  200ms  (OpenAI API)
Retrieval:        ???ms  (search over vectors)
Generation:     2,500ms  (GPT-4)
-----------------------------------
Total:         2,700ms + retrieval

To meet <3s total:

  • Retrieval must be: 3000ms - 2700ms = <300ms
  • Ideally <100ms (buffer for variability)

Benchmark your setup:

import numpy as np
import time

# Your current database
embeddings = np.load('your_embeddings.npy')  # Shape: (n_vectors, 1536)
query = np.random.randn(1536)

# Measure latency
start = time.time()
similarities = np.dot(embeddings, query)
top_5 = np.argsort(similarities)[-5:][::-1]
latency_ms = (time.time() - start) * 1000

print(f"Retrieval latency: {latency_ms:.1f}ms")

# Decision:
if latency_ms > 500:
    print("→ Vector DB needed (numpy too slow)")
elif latency_ms > 100:
    print("→ Gray zone (vector DB recommended if it will grow)")
else:
    print("→ numpy enough (for now)")

Criterion 3: Persistence (durable storage)

Simple rule:

Deployment type       | Persistence  | Option
----------------------|--------------|------------------
Jupyter notebook      | Not critical | numpy in memory
One-off script        | Not critical | numpy + .npy save
Service with uptime   | Critical     | Vector DB needed
Production API        | Critical     | Vector DB mandatory

Why persistence matters:

Without robust persistence (numpy):

  • Restart/crash = data loss OR 30-60s reload
  • Deployment = 30-60s downtime (reload embeddings)
  • Corruption risk (process interrupted during .save())

With persistence (Vector DB):

  • Restart/crash = <1ms reconnection (index persists)
  • Deployment = zero downtime (index already on disk)
  • ACID transactions (no corruption)

Evaluation:

Does your service need 99%+ uptime?
├─ YES → Vector DB needed
└─ NO → numpy can work

Can you tolerate 30-60s downtime on every deploy?
├─ YES → numpy can work
└─ NO → Vector DB needed

Do you have >10K vectors that take hours to generate?
├─ YES → Vector DB needed (you don't want to lose the work)
└─ NO → numpy can work (regenerating is fast)

Criterion 4: Concurrency (multiple users)

Simple rule:

# of concurrent users      | Option
---------------------------|-------------------
1 user (development)       | numpy enough
2-10 users                 | Gray zone
>10 concurrent users       | Vector DB needed
>100 users                 | Vector DB mandatory

Why concurrency matters:

numpy isn't thread-safe for writes:

import numpy as np
from threading import Thread

embeddings = np.load('embeddings.npy')

# ❌ Race condition: multiple users adding docs
def add_doc(user_id, doc_embedding):
    global embeddings
    embeddings = np.vstack([embeddings, doc_embedding])  # ❌ Not thread-safe
    np.save('embeddings.npy', embeddings)  # ❌ Corruption risk

# Multiple concurrent requests
threads = [Thread(target=add_doc, args=(i, emb)) for i, emb in enumerate(new_docs)]
for t in threads: t.start()

Vector DB thread-safe:

import chromadb

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_collection("docs")

# ✅ Thread-safe: multiple users can add concurrently
def add_doc(user_id, doc_embedding, doc_id):
    collection.add(embeddings=[doc_embedding], ids=[doc_id])  # ✅ Safe

# Multiple concurrent requests → No problems

Evaluation:

How many users will use your system simultaneously?
├─ 1-2 (development/demo) → numpy enough
├─ 3-10 (small team) → numpy with locks (complicated) OR vector DB (simple)
└─ >10 (production) → Vector DB needed

Do you need to add/update docs while others are querying?
├─ YES → Vector DB needed (handles read/write concurrently)
└─ NO → numpy can work (read-only)

Complete decision matrix

Combining the 4 criteria

ScaleLatencyPersistenceConcurrencyDecision
<10K>1sNot critical1 user✅ numpy enough
<10K<500msNot critical1-5 users⚠️ numpy/SQL (gray zone)
10K-100K<500msCritical>5 users⚠️ Vector DB recommended
>100K<500msCritical>10 users✅ Vector DB needed
>500K<100msCritical>50 users✅ Vector DB mandatory

Use cases: When you DO need one

Case 1: Tech support chatbot (enterprise)

Requirements:

  • 500,000 documentation articles (1.5M chunks)
  • 1,000 employees using it simultaneously
  • <3s total response (<500ms retrieval)
  • 99.9% uptime (24/7)

Evaluation:

  • Scale: 1.5M vectors → ✅ Vector DB needed
  • Latency: <500ms → ✅ Vector DB needed
  • Persistence: 99.9% uptime → ✅ Vector DB needed
  • Concurrency: 1,000 users → ✅ Vector DB needed

Decision: ✅ Vector DB needed (4/4 criteria met)

Recommendation:

  • Development: ChromaDB local (free, fast)
  • Production: Pinecone managed ($200-500/mo) or ChromaDB self-hosted on a robust server

Case 2: Q&A over legal documents (startup)

Requirements:

  • 50,000 legal documents (150K chunks)
  • 50 internal lawyers using it
  • <2s total response (<300ms retrieval)
  • 99% uptime (business hours)

Evaluation:

  • Scale: 150K vectors → ✅ Vector DB recommended
  • Latency: <300ms → ✅ Vector DB recommended
  • Persistence: 99% uptime → ✅ Vector DB recommended
  • Concurrency: 50 users → ✅ Vector DB recommended

Decision: ✅ Vector DB recommended (4/4 criteria)

Recommendation:

  • ChromaDB self-hosted (free, enough for 150K vectors)
  • Or Pinecone ($70-150/mo) if they prefer managed

Case 3: Multi-tenant RAG system (SaaS)

Requirements:

  • 100 clients, each with 10K-50K documents
  • Total: 1M-5M vectors (aggregated)
  • <100ms retrieval (competitive)
  • Multi-tenancy (data isolation per client)
  • 99.99% uptime (contractual SLA)

Evaluation:

  • Scale: 1M-5M vectors → ✅ Vector DB mandatory
  • Latency: <100ms → ✅ Vector DB mandatory
  • Persistence: 99.99% uptime → ✅ Vector DB mandatory
  • Concurrency: 100s-1000s users → ✅ Vector DB mandatory
  • Plus: Multi-tenancy → ✅ Vector DB with a native feature

Decision: ✅ Vector DB mandatory + managed preferred

Recommendation:

  • Pinecone managed ($500-2000/mo) - the best option for multi-tenant SaaS
  • Or Weaviate Cloud ($300-1000/mo) - flexible, GraphQL API
  • NOT ChromaDB self-hosted (requires manual multi-tenancy handling)

Case 4: Research paper search (personal)

Requirements:

  • 100,000 scientific papers (300K chunks)
  • 1 user (researcher)
  • <1s retrieval (exploratory analysis)
  • Local on a laptop (privacy)

Evaluation:

  • Scale: 300K vectors → ⚠️ Gray zone (numpy takes ~500ms)
  • Latency: <1s → ⚠️ numpy could work (~500ms acceptable for exploration)
  • Persistence: Local laptop → ⚠️ Not critical (you tolerate reload)
  • Concurrency: 1 user → ✅ numpy enough

Decision: ⚠️ Gray zone (2/4 criteria don't favor a vector DB)

Options:

  1. numpy + pgvector (local Postgres):

    • Moderate setup (install Postgres + pgvector)
    • 50-80ms retrieval (enough for exploration)
    • Better persistence than raw numpy

  2. ChromaDB local:

    • pip install chromadb (simple)
    • 15-30ms retrieval (excellent)
    • Native persistence

  3. Raw numpy:

    • pip install numpy (simplest)
    • 500ms retrieval (acceptable for 1 user exploring)
    • No persistence (60s reload after restart)

Recommendation: ChromaDB local (best simplicity/performance balance)

Clear signs that you need a vector DB

IF you answer "YES" to 3+ of these:

  1. ✅ You have >100K vectors (or you'll grow to that)
  2. ✅ You need <500ms retrieval latency
  3. ✅ You need >99% uptime (production service)
  4. ✅ You have >10 concurrent users
  5. ✅ You need metadata filtering (category, date, author)
  6. ✅ You need to add/update docs dynamically
  7. ✅ You need robust backup/restore
  8. ✅ You need monitoring (latency, throughput, usage)

→ A vector DB is needed

IF you answer "NO" to all of them:

  • You only have <1K vectors
  • Latency doesn't matter (batch processing)
  • You don't need uptime (one-off script)
  • A single user (development)
  • You don't need advanced features
  • You won't add docs dynamically
  • You don't need backup/monitoring

→ numpy is enough

Summary

What you learned:

  1. ✅ Criterion 1: Scale - >100K vectors → vector DB needed
  2. ✅ Criterion 2: Latency - <500ms → vector DB recommended, <100ms → mandatory
  3. ✅ Criterion 3: Persistence - uptime >99% → vector DB needed
  4. ✅ Criterion 4: Concurrency - >10 users → vector DB needed
  5. ✅ Decision matrix - Combine the criteria to decide

Typical cases:

  • Production RAG (enterprise): 4/4 criteria → Vector DB needed
  • Prototype/MVP: 0-1/4 criteria → numpy enough
  • Gray zone: 2/4 criteria → Evaluate trade-offs (simplicity vs performance)

Why it matters:

  • Don't over-engineer: If numpy meets the requirements, use it (simplicity)
  • Don't under-engineer: If you need a vector DB, don't waste time with numpy (performance)
  • An informed decision = time saved

Next capsule: Now that you know when you DO need a vector DB, you'll see when you DON'T (cases where alternatives are better).

Additional resources

  1. Choosing a Vector Database - Decision framework
  2. ChromaDB vs Pinecone - Self-hosted vs managed
  3. RAG at Scale - Production considerations
  4. Vector DB Benchmarks - Performance comparisons
  5. Multi-tenant RAG - SaaS use case
  6. Vector Database Comparison - Landscape overview

Reading time: 6-8 minutes
Next: 06-when-you-do-not-need-a-vector-database.md

← PreviousWhy numpy/pandas don't scale to productionNext →When you DON'T need a vector database