Module 1: Why Vector Databases for AI Engineers

Trade-offs: Simplicity vs Performance vs Cost

Next →

Capsule overview

Every technical decision involves trade-offs. There's no absolute "best option." There's the "best option FOR your context."

This capsule gives you quantitative comparisons (not just qualitative) of numpy vs SQL+pgvector vs a dedicated Vector DB. Real latency, real memory, real cost, real setup time.

By the end, you'll be able to build your own trade-off matrix and defend your decision with data.

The 3 trade-off dimensions

1. Simplicity (setup time + learning curve)

numpy:

pip install numpy
# Done. 10 seconds.

SQL + pgvector:

# 1. Install Postgres (if you don't have it)
brew install postgresql  # Mac: 5-10 min
# Or apt-get on Linux, installer on Windows

# 2. Start Postgres
brew services start postgresql

# 3. Install pgvector
git clone https://github.com/pgvector/pgvector.git
cd pgvector
make
make install  # 5-10 min

# 4. Enable the extension
psql -U postgres -c "CREATE EXTENSION vector;"

# Total: 30-45 minutes the first time

ChromaDB:

pip install chromadb
# Done. 30 seconds.

Pinecone:

pip install pinecone-client

# In addition:
# 1. Create an account at pinecone.io
# 2. Get an API key
# 3. Create an index (via UI or API)
# 4. Configure the environment

# Total: 15-20 minutes the first time

2. Performance (latency + throughput)

Real benchmark (1M vectors, 1536D, M1 Pro 16GB):

Metricnumpypgvector+HNSWChromaDBPinecone
Latency (p50)1,500ms60ms15ms8ms
Latency (p95)1,650ms95ms25ms15ms
Throughput0.6 QPS15 QPS50 QPS100+ QPS
Concurrent users15-1020-50100+

Conclusions:

  • numpy: 100x slower than ChromaDB
  • pgvector: 4x slower than ChromaDB
  • ChromaDB: Competitive with Pinecone for <5M vectors
  • Pinecone: Best performance for >5M vectors + high concurrency

3. Cost ($ per month)

Benchmark (1M vectors, 100 users, 1000 queries/day):

OptionSetupMonthlyNotes
numpy$0$0Requires a server with enough RAM
pgvector$0$0-50Existing or new Postgres server
ChromaDB self$0$20-100Server (DigitalOcean $20/mo, AWS t3.medium $30/mo)
Pinecone$0$70-200Starter $70, Standard $150, Enterprise $200+

Hidden cost: Developer time:

  • numpy: Simple but requires manual optimization (1-2 weeks)
  • pgvector: Moderate setup + optimization (1 week)
  • ChromaDB: Fast setup, minor optimization (2-3 days)
  • Pinecone: Plug & play, no optimization (1 day)

ROI calculation:

# Developer cost: $50/hr (junior) to $150/hr (senior)

# numpy optimization: 80 hours × $100/hr = $8,000
# pgvector setup + opt: 40 hours × $100/hr = $4,000
# ChromaDB setup: 16 hours × $100/hr = $1,600
# Pinecone setup: 8 hours × $100/hr = $800

# Pinecone monthly: $150/mo
# ChromaDB monthly: $50/mo (server)

# Break-even Pinecone vs ChromaDB:
# ($8,000 - $1,600) / ($150 - $50) = 64 months

# But... Pinecone performance > ChromaDB for >5M vectors
# And managed (no maintenance) is worth $$

Detailed comparison by dimension

Latency (ms) by scale

Vectors   | numpy    | pgvector | ChromaDB | Pinecone
----------|----------|----------|----------|----------
1K        | 2ms  ✅ | 5ms      | 3ms      | 4ms
10K       | 15ms ✅ | 12ms     | 5ms      | 5ms
50K       | 75ms    | 30ms     | 8ms  ✅  | 7ms ✅
100K      | 150ms   | 55ms     | 12ms ✅  | 9ms ✅
500K      | 750ms   | 180ms    | 20ms ✅  | 12ms ✅
1M        | 1,500ms | 350ms    | 30ms     | 15ms ✅
5M        | 7,500ms | 1,800ms  | 120ms    | 40ms ✅
10M       | OOM     | 3,600ms  | 250ms    | 80ms ✅

✅ = Best in its category

Key insight:

  • numpy: Only competitive with <10K vectors
  • pgvector: Competitive with <100K vectors
  • ChromaDB: Optimal for 100K-5M vectors
  • Pinecone: Optimal for >1M vectors (especially >5M)

Memory (GB RAM required)

Vectors   | numpy | pgvector | ChromaDB | Pinecone
----------|-------|----------|----------|----------
1K        | 0.01  | 0.02     | 0.05     | N/A (managed)
10K       | 0.06  | 0.12     | 0.15     | N/A
100K      | 0.6   | 1.2      | 0.8      | N/A
1M        | 6     | 12       | 3        | N/A
5M        | 30    | 60       | 12       | N/A
10M       | 60    | 120      | 25       | N/A

Note: pgvector uses more memory because it keeps the index + the original data
      ChromaDB compresses with HNSW (less memory than brute force)

Key insight:

  • numpy: ~6GB per million vectors (everything in RAM)
  • pgvector: ~12GB per million (SQL overhead)
  • ChromaDB: ~3GB per million (compressed HNSW + disk)
  • Pinecone: Not relevant (managed, scales automatically)

Throughput (queries per second, 10 concurrent users)

Vectors   | numpy | pgvector | ChromaDB | Pinecone
----------|-------|----------|----------|----------
100K      | 6 QPS | 18 QPS   | 80 QPS   | 100+ QPS
1M        | 0.6   | 3 QPS    | 30 QPS   | 100+ QPS
5M        | 0.1   | 0.5 QPS  | 8 QPS    | 100+ QPS

Note: numpy collapses with concurrency (not thread-safe for writes)

Key insight:

  • numpy: Not designed for concurrency (single-threaded effectively)
  • pgvector: Moderate (Postgres handles concurrency but with overhead)
  • ChromaDB: Good (handles 20-50 concurrent users)
  • Pinecone: Excellent (handles 100+ users, auto-scaling)

Setup time (first time)

Option         | Install | Config | Learn API | Total
---------------|---------|--------|-----------|-------
numpy          | 10s     | 0      | 1h        | ~1h
pgvector       | 30min   | 15min  | 2h        | ~3h
ChromaDB       | 30s     | 5min   | 2h        | ~2.5h
Pinecone       | 1min    | 10min  | 2h        | ~2h

Note: "Learn API" = time to first working query (not mastery)

Ongoing maintenance (hours/month)

Option         | Monitoring | Updates | Scaling | Backups | Total
---------------|------------|---------|---------|---------|-------
numpy          | 0          | 0       | Manual  | Manual  | 4-8h
pgvector       | 2h         | 1h      | Manual  | Auto    | 6-10h
ChromaDB self  | 3h         | 1h      | Manual  | Manual  | 8-12h
Pinecone       | 0          | 0       | Auto    | Auto    | 0h

Note: Maintenance assuming 1M vectors, 100 users

Key insight:

  • numpy: Low maintenance (but requires manual intervention to scale)
  • pgvector: Moderate (Postgres maintenance)
  • ChromaDB self-hosted: Moderate-High (server maintenance + DB)
  • Pinecone managed: Zero maintenance ($$$ buys time)

Final decision matrix

Scorecard (1-10, 10 = best)

DimensionnumpypgvectorChromaDBPinecone
Setup simplicity10497
API simplicity10688
Performance (<100K)8799
Performance (>1M)13710
Cost (self-hosted)10980
Cost (managed)10806
Concurrency26810
Features (filter, etc)15910
Maintenance85510
Learning curve10677
TOTAL70597077

Interpretation:

  • numpy: Excellent for <10K vectors, prototype, development (simplicity wins)
  • pgvector: Moderate across the board (reasonable compromise if you already use Postgres)
  • ChromaDB: Excellent balance for 100K-5M vectors, self-hosted
  • Pinecone: Best performance + zero maintenance (worth $$$ in production >1M vectors)

Recommendations by scenario

Scenario 1: Pre-seed startup (budget $0)

Requirements: 50K docs, 20 users, <500ms latency

Option A (simplest): numpy

  • Pros: $0 setup, $0 monthly, 75ms latency (acceptable)
  • Cons: Doesn't scale to >100K, manual handling

Option B (more scalable): ChromaDB self-hosted

  • Pros: $20/mo (DigitalOcean), 10ms latency, scales to 1M+
  • Cons: Requires server management

Recommendation: ChromaDB (a $20/mo investment is worth it vs numpy's limitations)

Scenario 2: Company with 1M docs

Requirements: 1M docs, 500 users, <100ms latency, 99.9% uptime

Option A: ChromaDB self-hosted

  • Pros: $100/mo, 30ms latency, full control
  • Cons: Requires DevOps (1-2 days/month)

Option B: Pinecone managed

  • Pros: $150/mo, 15ms latency, zero maintenance, auto-scaling
  • Cons: +$50/mo vs ChromaDB

Recommendation: Pinecone (the $50/mo difference << the value of 1-2 DevOps days/month = $1,600)

Scenario 3: Multi-tenant SaaS (10M aggregated vectors)

Requirements: 100 clients, 10M vectors total, <50ms latency, 99.99% uptime

Option A: ChromaDB self-hosted

  • Pros: $300-500/mo (cluster), full control
  • Cons: High maintenance (4-8 days/month), 250ms latency with 10M

Option B: Pinecone enterprise

  • Pros: $500-1000/mo, 80ms latency, auto-scaling, SLA
  • Cons: Vendor lock-in

Recommendation: Pinecone Enterprise (performance + SLA + zero maintenance critical for SaaS)

Summary

Key trade-offs:

  1. ✅ Simplicity: numpy (10s) > ChromaDB (2.5h) > Pinecone (2h) > pgvector (3h)
  2. ✅ Performance: Pinecone (15ms) > ChromaDB (30ms) > pgvector (350ms) > numpy (1500ms) @ 1M vectors
  3. ✅ Cost self: numpy ($0) = pgvector ($0) > ChromaDB ($50) >> Pinecone (N/A)
  4. ✅ Cost managed: ChromaDB (N/A) < Pinecone ($150) << pgvector cloud ($200+)
  5. ✅ Maintenance: Pinecone (0h) > numpy (4h) > pgvector (8h) = ChromaDB (8h) / month

Decision framework:

  • <10K vectors: numpy (simplicity)
  • 10K-100K: ChromaDB self (balance)
  • 100K-1M: ChromaDB self or Pinecone (evaluate budget vs maintenance)
  • >1M: Pinecone managed (performance + zero maintenance)

Why it matters:

  • There's no absolute "best," there's "best for YOUR context"
  • Trade-offs are quantitative (latency, cost, time), not just qualitative
  • An informed decision with data > intuition

Next capsule: Summary of the full module + transition to Module 2 (how vector DBs work internally).

Additional resources

  1. Vector Database Benchmarks - Performance comparisons
  2. ChromaDB vs Pinecone - Detailed comparison
  3. Cost Optimization for AI - Budget considerations
  4. pgvector Performance Tuning - Optimization guide
  5. When to Use Managed vs Self-hosted - Decision framework
  6. ROI Calculator for Vector DBs - Pinecone calculator

Reading time: 6-8 minutes
Next: 08-summary-and-transition.md

← PreviousWhen you DON'T need a vector databaseNext →Module 1 Summary and Transition