Module 1: Why Vector Databases for AI Engineers

When you DON'T need a vector database

Next →

Capsule overview

Not every project needs a vector database. Sometimes numpy is enough. Sometimes SQL with pgvector is better. Sometimes you don't need embeddings at all.

This capsule saves you from over-engineering. Seeing developers spend 2 weeks learning Pinecone when they had 5K vectors (numpy would take 8ms) is common. This capsule prevents that mistake.

You'll learn clear signs that you do NOT need a vector DB and which alternatives to use instead.

Sign #1: You have <10K vectors

The inflection point

numpy performance at different scales:

    1K vectors:    2-3ms    ✅ Excellent (better than vector DB overhead)
    5K vectors:    8-10ms   ✅ Excellent
   10K vectors:   15-20ms   ✅ Acceptable
   50K vectors:   75-100ms  ⚠️ Gray zone
  100K vectors:  150-200ms  ❌ Slow

Decision rule:

def should_use_vector_db(n_vectors):
    if n_vectors < 10_000:
        return False, "numpy is simpler and fast enough"
    elif n_vectors < 100_000:
        return "maybe", "evaluate latency requirements"
    else:
        return True, "vector DB needed for performance"

Case: MVP prototype with 2K documents

Setup:

  • Startup building an internal documentation chatbot
  • 2,000 docs (6,000 chunks)
  • 10 employees using it
  • Prototype to validate the concept

With numpy:

import numpy as np
from openai import OpenAI

# Load embeddings (6K vectors)
embeddings = np.load('embeddings.npy')  # Shape: (6000, 1536)

# Query
client = OpenAI()
query_emb = client.embeddings.create(
    input="how to deploy?",
    model="text-embedding-3-small"
).data[0].embedding

# Search (brute force)
similarities = np.dot(embeddings, query_emb)
top_5 = np.argsort(similarities)[-5:][::-1]

# Latency: ~10ms ✅ Excellent

Why is numpy enough?

  • Scale: 6K vectors → 10ms latency (excellent)
  • Simplicity: pip install numpy (vs learning the ChromaDB API)
  • Cost: $0 (vs Pinecone $70/mo)
  • Prototype: If the concept doesn't work, you didn't waste time on vector DB setup

Decision: ✅ numpy is the better option (don't over-engineer)

Sign #2: There's no latency requirement

Batch processing vs real-time

If your use case is:

  • Offline analysis (no users waiting)
  • ETL pipelines (process overnight)
  • Research (unhurried exploration)
  • Clustering (once, not repeated)

→ Latency doesn't matter. numpy is enough.

Case: Clustering 500K scientific papers

Setup:

  • Research: Group 500K papers into thematic clusters
  • No users (offline analysis)
  • Can take hours (no rush)

With numpy (brute force):

import numpy as np
from sklearn.cluster import KMeans

# Load embeddings (500K vectors)
embeddings = np.load('papers_embeddings.npy')  # Shape: (500000, 1536)

# Clustering (will take ~30-60 minutes, but it's done only once)
kmeans = KMeans(n_clusters=100, random_state=42)
clusters = kmeans.fit_predict(embeddings)

# Analyze clusters
for cluster_id in range(100):
    papers_in_cluster = np.where(clusters == cluster_id)[0]
    print(f"Cluster {cluster_id}: {len(papers_in_cluster)} papers")

Why is numpy enough?

  • No users waiting (offline)
  • Latency doesn't matter (it can take 1 hour)
  • It runs once (you don't need to optimize for repetition)

Decision: ✅ numpy is the better option (a vector DB would be unnecessary overhead)

Sign #3: Local development/experimentation

A single developer iterating

If you're:

  • Prototyping different chunking strategies
  • Experimenting with embedding models
  • Iterating on prompt engineering
  • Developing on a local laptop

→ numpy is faster to iterate with

Case: A developer experimenting with RAG

Setup:

  • 1 developer testing different strategies
  • Dataset: 8K documents (24K chunks)
  • Fast iteration (change, re-run, evaluate)

With numpy:

# Experimentation cycle:
# 1. Generate embeddings → save .npy (5 minutes)
# 2. Test retrieval → numpy search (20ms)
# 3. Evaluate results → adjust strategy
# 4. Repeat

# Total iteration time: 10-15 minutes

# With a vector DB:
# 1. Generate embeddings
# 2. Setup ChromaDB/Pinecone (first time: 30 min)
# 3. Insert embeddings (10 minutes)
# 4. Test retrieval
# 5. Evaluate
# 6. Change strategy → re-insert embeddings (10 min)
# 7. Repeat

# Total iteration time: 25-30 minutes (2x slower)

Why is numpy better for experimentation?

  • No setup overhead (just pip install)
  • Fast changes (reload .npy < 1 second)
  • You don't need to learn a new API (focus on experimentation)

Decision: ✅ numpy for development, migrate to a vector DB only when the strategy is validated and you're going to production

Sign #4: You don't need embeddings at all

Keyword search can be enough

Cases where keyword search >> semantic search:

  1. Exact search of technical terms:

    • "ERROR-404" → keyword search is perfect
    • Semantic search: "error not found" (not exact)

  2. Proper nouns (people, places, products):

    • "John Smith" → keyword searches exactly that
    • Embeddings: "John" and "Smith" are poorly semanticized

  3. Codes / IDs:

    • "ORDER-12345" → keyword search finds a unique match
    • Embeddings: vectorizes "order twelve thousand..." (useless)

  4. SQL queries / code:

    • "SELECT * FROM users WHERE age > 30" → keyword searches the exact syntax
    • Semantic search: understands the concept but loses the syntax

Case: Searching a support knowledge base

Setup:

  • Troubleshooting articles with error codes
  • Users search for "ERROR-404", "CONNECTION-TIMEOUT", etc.
  • They need an exact match, not a semantic one

With keyword search (Elasticsearch):

POST /articles/_search
{
  "query": {
    "match": {
      "content": {
        "query": "ERROR-404",
        "operator": "and"
      }
    }
  }
}

Latency: 10-50ms (excellent)
Accuracy: 100% (exact match)

With semantic search (embeddings + vector DB):

# Query: "ERROR-404"
# Embedding: [0.12, -0.34, 0.56, ...]
# Results: Docs about "errors", "404", but also "403", "500"
# Accuracy: 60-70% (not exact)

Decision: ✅ Keyword search is better (you do NOT need embeddings or a vector DB)

When to combine: Hybrid search (keyword + semantic) in Module 3

Sign #5: Static dataset (doesn't grow)

Read-only datasets

If your dataset:

  • Doesn't change (historical, archive)
  • Is generated once (no updates)
  • Doesn't get new docs added

→ precomputed numpy can be optimal

Case: Wikipedia archive (2023 snapshot)

Setup:

  • Wikipedia dump from 2023 (6M articles)
  • Static dataset (doesn't change)
  • Read-only (search, not writing)

Optimal strategy:

# 1. Generate embeddings (once, offline)
# Takes: 10-20 hours (6M docs × 3 chunks = 18M vectors)

# 2. Save in an optimized format
np.save('wikipedia_embeddings.npy', embeddings)  # 110GB

# 3. For queries:
#    Option A: Load EVERYTHING into RAM (if you have 128GB+ RAM)
embeddings = np.load('wikipedia_embeddings.npy')  # Takes 5-10 min, but only once
# Then: queries at 50-100ms (ultra-optimized brute force in RAM)

#    Option B: FAISS with PQ (compression)
import faiss
index = faiss.read_index('wikipedia_faiss.index')
# Queries: 200-300ms, memory: 30GB (vs 110GB)

Vector DB or numpy?

  • Vector DB: Unnecessary overhead (the dataset doesn't grow, you don't need writes)
  • numpy/FAISS: Optimal (one-time setup, fast queries)

Decision: ✅ numpy/FAISS (specialized), NOT a vector DB (over-engineering)

Sign #6: Zero budget (student/hobby)

Learning at no cost

If:

  • Student learning RAG
  • Hobby project (no revenue)
  • You can't spend $70/mo (Pinecone)
  • Local laptop (no server)

→ numpy or ChromaDB local (free)

Case: A student building a RAG portfolio

Setup:

  • Student building a chatbot over their class notes
  • 3,000 notes (9,000 chunks)
  • Only they use it (1 user)
  • Budget: $0

Options:

  1. numpy: Free, 15ms retrieval, enough for 9K vectors ✅
  2. ChromaDB local: Free, 5ms retrieval, more features ✅
  3. Pinecone: $70/mo, 3ms retrieval, unnecessary ❌

Decision: ✅ numpy or ChromaDB local (zero cost, enough for a portfolio)

When to migrate to Pinecone: If the project becomes a startup with funding (then $70/mo is reasonable)

Decision tree: Do you need a vector DB?

Flowchart

How many vectors do you have?
├─ <10K → numpy enough ✅
├─ 10K-100K → Do you need <100ms latency?
│   ├─ NO → numpy enough ✅
│   └─ YES → Vector DB recommended ⚠️
└─ >100K → Vector DB needed ✅

Do you have a latency requirement?
├─ NO (batch/offline) → numpy enough ✅
└─ YES (<500ms) → How many vectors?
    ├─ <10K → numpy enough ✅
    └─ >10K → Vector DB needed ✅

Multiple concurrent users?
├─ NO (1 user) → numpy enough ✅
└─ YES (>10 users) → How many vectors?
    ├─ <10K → numpy with locks ⚠️
    └─ >10K → Vector DB needed ✅

Does the dataset grow dynamically?
├─ NO (static) → numpy/FAISS enough ✅
└─ YES (docs added) → How often?
    ├─ Rarely (1x/month) → reloadable numpy ✅
    └─ Frequently (daily) → Vector DB needed ✅

Budget available?
├─ $0 → numpy or ChromaDB local ✅
├─ <$100/mo → ChromaDB self-hosted ✅
└─ >$100/mo → Pinecone managed ✅

Summary

When you DON'T need a vector DB:

  1. ✅ <10K vectors → numpy enough (15ms latency)
  2. ✅ No latency requirement → numpy batch processing
  3. ✅ 1 user (development) → numpy for fast iteration
  4. ✅ Keyword search is enough → Elasticsearch, not embeddings
  5. ✅ Static dataset → precomputed numpy/FAISS
  6. ✅ $0 budget → numpy or ChromaDB local

Alternatives to a vector DB:

  • numpy: <10K vectors, development, prototype
  • SQL + pgvector: You already use Postgres, <100K vectors
  • Elasticsearch: Hybrid search (keyword + semantic)
  • FAISS: Static dataset, offline processing

Why it matters:

  • Don't over-engineer: If numpy meets the requirements, it saves time and complexity
  • Don't under-engineer: If you need a vector DB, don't waste time with inadequate numpy
  • Start simple (numpy) → Scale when necessary (vector DB)

Next capsule: Full trade-offs: simplicity vs performance vs cost (quantitative comparison).

Additional resources

  1. When NOT to Use Vector Databases - Anti-patterns
  2. NumPy for Semantic Search - Simple approach
  3. FAISS for Static Datasets - Offline optimization
  4. Keyword vs Semantic Search - When to use each one
  5. pgvector Guide - SQL alternative
  6. Cost Optimization for AI - Budget considerations

Reading time: 6-8 minutes
Next: 07-trade-offs-simplicity-vs-performance.md

← PreviousWhen you DO need a vector databaseNext →Trade-offs: Simplicity vs Performance vs Cost