Benchmarking and Costs: ChromaDB vs Pinecone

Capsule description

By now you have a RAG system on Pinecone with namespaces, typed filters and multi-tenant isolation. But before declaring the migration a success and closing the ticket, you need to answer a question your boss, your CFO or your co-founder is going to ask: is this really worth what it costs?

Migrating without benchmarks is a decision based on opinion. "Pinecone is faster" without numbers is marketing. "ChromaDB felt slow" without measuring it is perception. Sustainable infrastructure decisions are made with data: percentiled latencies, sustained throughput, an estimated monthly cost, and degradation under load.

In this capsule you're going to build a reproducible benchmark that measures both systems under the same conditions, compute realistic costs with Pinecone's serverless calculator, and produce a one-page executive report that closes the "do we migrate or not?" discussion with evidence, not with opinion.

The final deliverable is a script you can run at any time (when the prices change, when the volume grows, when a competitor appears) to revalidate the decision.

What to measure and why

Four dimensions count for the decision, in order of importance:

The classic anti-pattern: measuring only the average latency (avg). Your p99 can be 4× the average, and the median looks "fine" while 1% of your users suffer a 3-second wait. Averages for the board's dashboard; percentiles for engineering decisions.

Designing a reproducible benchmark

A benchmark that matters meets five requirements:

1. The same dataset indexed in both systems (same vectors, same metadata)
2. The same set of queries (ideally real queries from the logs, not synthetic ones)
3. The same retrieval configuration (the same top_k, the same filters)
4. The same network conditions (run from the same client, ideally close to Pinecone)
5. Enough samples (a minimum of 500 queries; ideally 2000+)

from dataclasses import dataclass

@dataclass
class BenchmarkConfig:
    name: str
    dataset_size: int
    queries: list[dict]
    top_k: int = 10
    warmup_runs: int = 50
    measure_runs: int = 1000

    def __post_init__(self):
        assert len(self.queries) >= self.measure_runs, "Need enough queries"

The warmup_runs are critical: the first queries pay for cold caches, new TCP connections, JIT warmup. Discard them.

A benchmark script with correct percentiles

import time
import statistics
from typing import Callable

def benchmark_retriever(
    name: str,
    run_query: Callable[[dict], list],
    queries: list[dict],
    warmup: int = 50,
) -> dict:
    # The discarded warmup
    for q in queries[:warmup]:
        run_query(q)

    latencies_ms = []
    errors = 0
    for q in queries[warmup:]:
        start = time.perf_counter()
        try:
            run_query(q)
            latencies_ms.append((time.perf_counter() - start) * 1000)
        except Exception:
            errors += 1

    latencies_ms.sort()
    n = len(latencies_ms)
    return {
        "name": name,
        "samples": n,
        "errors": errors,
        "p50": latencies_ms[int(0.50 * n)],
        "p90": latencies_ms[int(0.90 * n)],
        "p95": latencies_ms[int(0.95 * n)],
        "p99": latencies_ms[int(0.99 * n)],
        "avg": statistics.mean(latencies_ms),
        "stdev": statistics.stdev(latencies_ms) if n > 1 else 0,
    }

Details that look minor but matter:

• time.perf_counter() instead of time.time(): monotonic, high resolution, unaffected by NTP.
• Errors are counted separately, not averaged in with the successful latencies (if Pinecone has 5% errors and ChromaDB has 0%, the averages lie).
• Multiply by 1000 at insertion time (not at the end) to avoid a loss of precision.

Sustained throughput under concurrency

Sequential latency doesn't predict behavior under load. A system can have p95=200ms with 1 client and p95=4000ms with 50 concurrent clients.

import asyncio
from time import perf_counter

async def measure_throughput(
    async_query: Callable,
    queries: list[dict],
    concurrency: int,
    duration_sec: int = 300,
):
    sem = asyncio.Semaphore(concurrency)
    latencies = []
    completed = 0

    async def one_request(q):
        nonlocal completed
        async with sem:
            start = perf_counter()
            await async_query(q)
            latencies.append((perf_counter() - start) * 1000)
            completed += 1

    deadline = perf_counter() + duration_sec
    tasks = []
    idx = 0
    while perf_counter() < deadline:
        tasks.append(asyncio.create_task(one_request(queries[idx % len(queries)])))
        idx += 1
        if len(tasks) >= concurrency * 10:
            await asyncio.gather(*tasks)
            tasks = []
    if tasks:
        await asyncio.gather(*tasks)

    elapsed = duration_sec
    return {
        "concurrency": concurrency,
        "qps": completed / elapsed,
        "p95_ms": sorted(latencies)[int(0.95 * len(latencies))],
        "completed": completed,
    }

Run this test at concurrency=[1, 10, 50, 100] and plot QPS vs latency. The point where p95 starts to spike is your real capacity.

Cost estimation: Pinecone serverless

Pinecone serverless charges on three dimensions (check pinecone.io/pricing for up-to-date prices; this is as of March 2026):

Useful approximations:

• 1 vector of 1536 dims with small metadata ≈ 8 KB → 1M vectors ≈ 8 GB → ~$2.6/month of storage
• 1 typical query with top_k=10 ≈ 1-3 read units (it scales with the namespace's size)
• 1 upsert ≈ 1 write unit per vector

def estimate_pinecone_monthly(
    vectors_total: int,
    avg_metadata_kb: float,
    queries_per_month: int,
    avg_read_units_per_query: float,
    upserts_per_month: int,
    prices: dict | None = None,
) -> dict:
    p = prices or {
        "storage_gb_month": 0.33,
        "read_units_million": 8.25,
        "write_units_million": 2.00,
    }
    storage_gb = (vectors_total * (8 + avg_metadata_kb)) / (1024 * 1024)
    storage_cost = storage_gb * p["storage_gb_month"]
    read_cost = (queries_per_month * avg_read_units_per_query / 1_000_000) * p["read_units_million"]
    write_cost = (upserts_per_month / 1_000_000) * p["write_units_million"]
    total = storage_cost + read_cost + write_cost
    return {
        "storage_gb": round(storage_gb, 2),
        "storage_cost": round(storage_cost, 2),
        "read_cost": round(read_cost, 2),
        "write_cost": round(write_cost, 2),
        "total_monthly_usd": round(total, 2),
    }

A realistic example for a B2B SaaS:

estimate_pinecone_monthly(
    vectors_total=2_000_000,         # 2M chunks indexed
    avg_metadata_kb=1.5,             # ~1.5 KB of metadata each
    queries_per_month=15_000_000,    # 15M queries/month
    avg_read_units_per_query=2.0,    # top_k=10 with a filter
    upserts_per_month=200_000,       # 200K new chunks/month
)
# {'storage_gb': 18.13, 'storage_cost': 5.98, 'read_cost': 247.50, 'write_cost': 0.40, 'total_monthly_usd': 253.88}

Compare that against a self-hosted ChromaDB: an m6i.2xlarge EC2 instance ($200/month) + EBS storage ($30/month) + your operational time. Once you include oncall, backups, scaling and monitoring, Pinecone's $250/month starts to look cheap.

Quality: making sure the migration doesn't degrade the results

Latency and cost are worth nothing if Pinecone returns worse documents. Measure recall@k against a ground truth:

def recall_at_k(retriever, queries_with_truth: list[tuple[str, set[str]]], k: int = 10) -> float:
    hits = 0
    total = 0
    for query, expected_doc_ids in queries_with_truth:
        results = retriever(query, top_k=k)
        retrieved_ids = {r["id"] for r in results}
        hits += len(expected_doc_ids & retrieved_ids)
        total += len(expected_doc_ids)
    return hits / total if total else 0.0

What to expect in a well-executed migration (same chunks, same embeddings): recall_chroma ≈ recall_pinecone ± 1%. If the difference is larger, there's a bug in the migration (probably badly copied metadata or duplicate IDs).

An example comparison table

The real result of a benchmark over 5,000 queries on a dataset of 1.2M vectors:

The interpretation: Pinecone cuts p99 by 4×, supports 6× more concurrency, maintains the same quality, and costs marginally more in pure infrastructure. Once you add the operational cost (one engineer spending 5 hours/week on ChromaDB is ~$2,000/month), Pinecone is clearly cheaper.

Connection with the Production RAG project

Your final module project must include a BENCHMARK.md report with:

1. The setup: hardware, dataset size, the exact configuration
2. The latency results: a table with p50/p95/p99 for ChromaDB and Pinecone
3. The throughput results: a QPS vs latency plot at several concurrency levels
4. The cost estimate: at 6 months, with the projected growth
5. Recall@10: validation that the quality didn't degrade
6. The recommendation: migrate/don't migrate/migrate partially, with the reasons
7. A reproducible script: to run when the conditions change

This is the deliverable you defend in front of non-technical stakeholders.

Troubleshooting

Problem 1: "Inconsistent latencies between runs"

The cause: variable network conditions, noisy neighbors in the cloud, GC pauses.
The fix: run the benchmark 3 times at different hours and report the median of the runs. Exclude extreme outliers only if you have evidence (GC logs, CPU metrics).

Problem 2: "Pinecone seems slower on my laptop"

The cause: network latency dominating. Your laptop in us-west is ~100ms from the Pinecone cluster; a local ChromaDB is 0ms away.
The fix: run the benchmark from the region where production will run (an EC2 in the same region as the Pinecone index). Without that, you're comparing the network against a local disk.

Problem 3: "The real costs exploded vs the estimate"

The cause: the real queries have a higher top_k than you estimated, or the filter forces more read units.
The fix: review the logs of your real queries (don't estimate, measure). Pinecone exposes usage in the dashboard; correlate it against your application logs.

Problem 4: "Throughput drops off after 5 minutes"

The cause: rate limiting on Pinecone serverless's free/starter plan, or a saturated client connection pool.
The fix: check the plan; the starter plan has low limits. For Python clients, configure pool_threads appropriately and consider a keep-alive client pool.

Problem 5: "Recall@k drops on Pinecone"

The typical cause: the metadata got truncated, or the IDs aren't unique during the migration.
The fix: validate with index.fetch(ids=[...]) that the document exists and has the right metadata. Compare N random documents between ChromaDB and Pinecone byte by byte.

Exercises

Exercise 1: Computing percentiles correctly

Implement a function that takes latencies and returns p50, p95, p99 without using external libraries. Watch out for the index calculation when the array is small.

def percentile(values: list[float], p: float) -> float:
    if not values:
        return 0.0
    sorted_vals = sorted(values)
    k = (len(sorted_vals) - 1) * p
    f = int(k)
    c = min(f + 1, len(sorted_vals) - 1)
    if f == c:
        return sorted_vals[f]
    return sorted_vals[f] * (c - k) + sorted_vals[c] * (k - f)

def all_percentiles(latencies: list[float]) -> dict:
    return {
        "p50": percentile(latencies, 0.50),
        "p95": percentile(latencies, 0.95),
        "p99": percentile(latencies, 0.99),
    }

Exercise 2: A benchmark with 1000 queries

Build a complete benchmark that generates a JSON report with every metric for both systems.

import json

def full_benchmark(chroma_query, pinecone_query, queries):
    chroma_result = benchmark_retriever("chromadb", chroma_query, queries)
    pinecone_result = benchmark_retriever("pinecone", pinecone_query, queries)
    return {
        "chromadb": chroma_result,
        "pinecone": pinecone_result,
        "improvement": {
            "p95_speedup": round(chroma_result["p95"] / pinecone_result["p95"], 2),
            "p99_speedup": round(chroma_result["p99"] / pinecone_result["p99"], 2),
        },
    }

report = full_benchmark(my_chroma_retriever, my_pinecone_retriever, test_queries[:1000])
with open("benchmark.json", "w") as f:
    json.dump(report, f, indent=2)

Exercise 3: A 12-month cost estimate with growth

Assume 15% monthly growth in queries and vectors. Estimate the monthly cost at month 12.

def project_12_month_cost(initial_vectors, initial_queries, monthly_growth_rate=0.15):
    projections = []
    vectors = initial_vectors
    queries = initial_queries
    for month in range(1, 13):
        cost = estimate_pinecone_monthly(
            vectors_total=vectors,
            avg_metadata_kb=1.5,
            queries_per_month=queries,
            avg_read_units_per_query=2.0,
            upserts_per_month=int(vectors * 0.05),
        )
        projections.append({"month": month, "vectors": vectors, "queries": queries, **cost})
        vectors = int(vectors * (1 + monthly_growth_rate))
        queries = int(queries * (1 + monthly_growth_rate))
    return projections

projections = project_12_month_cost(1_000_000, 5_000_000)
print(f"Month 1: ${projections[0]['total_monthly_usd']}")
print(f"Month 12: ${projections[-1]['total_monthly_usd']}")
print(f"Year total: ${sum(p['total_monthly_usd'] for p in projections)}")

Exercise 4: An automated decision with explicit criteria

Create a function that produces a textual recommendation based on clear thresholds.

def migration_recommendation(benchmark_report: dict, monthly_cost: float, ops_cost: float = 2000) -> dict:
    chroma = benchmark_report["chromadb"]
    pinecone = benchmark_report["pinecone"]
    p95_speedup = chroma["p95"] / pinecone["p95"]
    p99_speedup = chroma["p99"] / pinecone["p99"]
    total_pinecone = monthly_cost
    total_chroma = monthly_cost * 0.7 + ops_cost  # infra + ops time

    reasons = []
    if p99_speedup >= 2:
        reasons.append(f"p99 improves {p99_speedup:.1f}×")
    if total_pinecone < total_chroma:
        reasons.append(f"Total cost ${total_pinecone:.0f}/month vs ${total_chroma:.0f}/month")
    if not reasons:
        return {"decision": "Don't migrate", "reasons": ["No clear advantage"]}
    return {
        "decision": "Migrate",
        "reasons": reasons,
        "monthly_savings": round(total_chroma - total_pinecone, 2),
    }

Exercise 5: Generating a QPS vs latency plot

Use matplotlib to visualize throughput vs latency at multiple concurrency levels.

import matplotlib.pyplot as plt

async def concurrency_curve(async_query, queries, levels=[1, 10, 25, 50, 100, 200]):
    results = []
    for c in levels:
        r = await measure_throughput(async_query, queries, concurrency=c, duration_sec=120)
        results.append(r)
    return results

def plot_curve(chroma_results, pinecone_results):
    fig, ax = plt.subplots(figsize=(8, 5))
    ax.plot([r["qps"] for r in chroma_results], [r["p95_ms"] for r in chroma_results],
            "o-", label="ChromaDB")
    ax.plot([r["qps"] for r in pinecone_results], [r["p95_ms"] for r in pinecone_results],
            "s-", label="Pinecone")
    ax.set_xlabel("Throughput (QPS)")
    ax.set_ylabel("p95 latency (ms)")
    ax.set_title("Latency vs Throughput under concurrency")
    ax.legend()
    ax.grid(True, alpha=0.3)
    plt.savefig("throughput_vs_latency.png", dpi=150)

Summary

• Benchmarking turns intuitions into defensible, data-backed decisions
• Measure p95 and p99, not just the average: the tail defines the user's experience
• Throughput under concurrency matters more than sequential latency for production
• The total cost includes infrastructure + operational time, not just the cloud invoice
• Pinecone serverless charges for storage + read units + write units; estimate it with the official calculator
• Validate recall@k to make sure the migration doesn't degrade the quality
• The final deliverable is a reproducible BENCHMARK.md with a versioned script

Additional resources

1. Pinecone Pricing - Official serverless and pod-based prices.
2. Pinecone Performance Monitoring - Operational metrics in the dashboard.
3. Measuring Latency Percentiles - Why the average is misleading.
4. Locust - A professional framework for load testing.
5. Pinecone Read Unit Calculator - How read units are computed.

Created: March 13, 2026
Version: 2.0