Module 4: ChromaDB Setup and Configuration

Capsule 06: Query Optimization — where you win or lose real latency

Capsule description

In the previous capsules you learned to insert data into ChromaDB efficiently and to filter by metadata. Now comes the other half of the equation: how to make queries fast, predictable, and of consistent quality.

Query latency isn't a single number. It's a statistical distribution — most queries are fast, but a small percentage is very slow. If you report only the average, you hide the problem. If you only optimize the average, the worst 5% of your users suffer. This capsule teaches you to measure correctly (p50/p95/p99), to understand which parameters move the needle at each percentile, and to optimize with judgment instead of copying configurations from tutorials.

When you finish, you'll know why your query that's fast in development becomes slow in production, what to raise and what to lower to hit a specific SLA, and which traps turn "optimization" into "silent regression".

By the end of this capsule, you'll be able to:

✅ Measure latency correctly with p50/p95/p99 percentiles and understand what each one tells you
✅ Identify the four parameters that move latency: n_results, metadata filtering, ef_search, dataset size
✅ Calculate the n_results sweet spot for your case (RAG vs search)
✅ Configure ef_search (HNSW search-time parameter) to balance latency vs accuracy
✅ Design a reproducible benchmark that validates changes before deploying
✅ Anticipate how latency changes when the dataset grows from 10K to 1M vectors

Estimated time: 35-45 minutes

Why measuring the average lies to you

The question "how long does a query take?" has three different answers, and each tells a story:

p50 (median): the latency in the typical case. Half of the queries are faster than this.
p95: the latency of the worst 5% of queries. What 1 in 20 users suffers.
p99: the latency of the worst 1% of queries. Tail latency — the pathological case that defines your SLA.

Imagine two systems with the same average of 50ms:

System A: latencies = [40, 45, 48, 50, 52, 55, 60] ms
  average = 50ms, p50 = 50ms, p99 = 60ms
  → Stable, predictable distribution

System B: latencies = [10, 15, 20, 25, 30, 50, 200] ms
  average = 50ms, p50 = 25ms, p99 = 200ms
  → Half super fast, but the 1% is 4x slower

If your SLA says "p95 < 100ms", system B fails while system A passes comfortably. If you looked only at the average, you'd consider them equivalent.

Why this matters especially in vector search: HNSW has variable latency. Most queries navigate the graph efficiently, but some require more backtracking and can be 5-10x slower. The average hides the pathological case.

Measuring correctly

# benchmark_query_latency.py
import chromadb
from chromadb.utils import embedding_functions
import os
import time
import statistics

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small"
)
client = chromadb.PersistentClient(path="./chroma_query_bench")
collection = client.get_or_create_collection(
    name="bench",
    embedding_function=openai_ef
)

# Make sure you have data (e.g.: 10K docs previously ingested)
print(f"Collection size: {collection.count()}")

# Varied queries (simulate real traffic)
queries = [
    "How do I configure HNSW for production?",
    "What is the difference between cosine and L2 distance?",
    "How does batch ingestion work in ChromaDB?",
    # ... ideally 50-100 real or synthetic queries
] * 25  # repeat to get 75 measurements

# Warm-up: the first queries are slower (cold cache)
for q in queries[:10]:
    collection.query(query_texts=[q], n_results=10)

# Real measurement
latencies = []
for query in queries:
    start = time.perf_counter()
    collection.query(query_texts=[query], n_results=10)
    elapsed_ms = (time.perf_counter() - start) * 1000
    latencies.append(elapsed_ms)

# Percentiles
latencies.sort()
n = len(latencies)
p50 = latencies[n // 2]
p95 = latencies[int(n * 0.95)]
p99 = latencies[int(n * 0.99)]
mean = statistics.mean(latencies)
stdev = statistics.stdev(latencies)

print(f"\nLatency benchmark ({n} queries):")
print(f"  Mean:   {mean:.1f}ms ± {stdev:.1f}ms")
print(f"  p50:    {p50:.1f}ms")
print(f"  p95:    {p95:.1f}ms")
print(f"  p99:    {p99:.1f}ms")
print(f"  Worst:  {latencies[-1]:.1f}ms")

Typical output (10K docs, OpenAI embeddings):

Latency benchmark (75 queries):
  Mean:   142.3ms ± 38.5ms
  p50:    128.5ms
  p95:    198.2ms
  p99:    245.7ms
  Worst:  287.0ms

Reading: the average (142ms) isn't what you typically see — the p50 says the typical case is 128ms. The p99 (245ms) is almost 2x the p50, normal for HNSW. If your SLA is "p95 < 200ms", you're at the limit — a bit more load and you start missing the SLA.

Critical note: the latency includes the time to generate the query embedding with OpenAI (~80-150ms in this example). If you want to measure only the search in ChromaDB, you must pre-compute the embeddings and pass query_embeddings= instead of query_texts=.

The four parameters that move latency

Parameter 1: `n_results` (top-K)

How many results you ask for. More results = more HNSW work + more data to serialize.

# Benchmark: latency vs n_results
for n in [1, 5, 10, 20, 50, 100]:
    latencies = []
    for q in queries:
        start = time.perf_counter()
        collection.query(query_embeddings=[pre_computed_embedding], n_results=n)
        latencies.append((time.perf_counter() - start) * 1000)
    p95 = sorted(latencies)[int(len(latencies) * 0.95)]
    print(f"n_results={n:3d}: p95 = {p95:.1f}ms")

Typical output:

n_results=  1: p95 = 12.3ms
n_results=  5: p95 = 14.8ms
n_results= 10: p95 = 17.2ms
n_results= 20: p95 = 24.1ms
n_results= 50: p95 = 38.6ms
n_results=100: p95 = 71.4ms

Pattern: latency grows sub-linearly with n_results for reasonable values (1-20), and worsens quickly for large values (50+).

Recommendation by case:

Use case	Typical `n_results`
RAG with LLM (passing context to the model)	3-10
Search interfaces (showing results to the user)	10-20
Re-ranking pipeline (retrieve a lot, filter later)	50-100
Recommendations (multi-criteria)	20-50

Common trap: "more results = better just in case". False. In RAG, passing 50 chunks to the LLM (a) raises the generation cost 5x, (b) puts irrelevant information into the context that distracts the model (lost in the middle), (c) doubles the retrieval time. The sweet spot for most RAG is n_results=5.

Parameter 2: metadata filtering (where clauses)

As we saw in M04/04, filtering by metadata reduces the search space and speeds up queries. But the effect depends on the type of filter and on how much it filters.

# No filter (searches in 10K docs)
no_filter = collection.query(query_embeddings=[emb], n_results=10)

# Aggressive filter (reduces to ~500 candidate docs)
heavy_filter = collection.query(
    query_embeddings=[emb],
    n_results=10,
    where={"category": "support"}  # assumes 5% of docs are "support"
)

# Very aggressive filter (reduces to ~50 docs)
very_heavy_filter = collection.query(
    query_embeddings=[emb],
    n_results=10,
    where={"category": "support", "language": "es", "version": "2.3"}
)

Typical result:

Filter	Search space	p95
No filter	10,000	17ms
`category="support"`	~500	6ms
`category="support" AND language="es" AND version="2.3"`	~50	3ms

But there's a pathological case: if your filter is too restrictive and leaves fewer candidates than n_results, the query gets slower:

# Filter that leaves only 5 candidates, but you ask for n_results=10
result = collection.query(
    query_embeddings=[emb],
    n_results=10,
    where={"super_specific_id": "xyz123"}  # only 1 doc matches
)
# HNSW has to scan the whole space to find enough matches
# that satisfy the filter → it can be SLOWER than no filter

Rule: the filter should leave at least 5-10x the requested n_results for HNSW to work efficiently. If your filter is ultra-specific, consider get() with a direct filter instead of query().

Parameter 3: `ef_search` (HNSW runtime parameter)

ef_search controls how many HNSW graph nodes each query visits. Higher = more exhaustive search = better recall = slower.

# Configure ef_search at runtime (dynamically modifies accuracy/latency)
collection_with_high_ef = client.get_or_create_collection(
    name="high_ef_collection",
    embedding_function=openai_ef,
    metadata={
        "hnsw:space": "cosine",
        "hnsw:search_ef": 200  # default is 10 — much more exhaustive
    }
)

# Compare
configs = [
    ("ef_search=10 (default)", 10),
    ("ef_search=50", 50),
    ("ef_search=100", 100),
    ("ef_search=200", 200),
]

for name, ef in configs:
    coll = client.get_or_create_collection(
        name=f"bench_ef_{ef}",
        embedding_function=openai_ef,
        metadata={"hnsw:space": "cosine", "hnsw:search_ef": ef}
    )
    # Assumes data already loaded
    latencies = [time_query(coll, q) for q in queries]
    recall = measure_recall(coll, eval_set)  # evaluation function
    p95 = sorted(latencies)[int(len(latencies) * 0.95)]
    print(f"{name}: p95={p95:.1f}ms, recall@10={recall:.2%}")

Typical output:

ef_search=10 (default): p95=8.2ms,   recall@10=88%
ef_search=50:           p95=15.6ms,  recall@10=94%
ef_search=100:          p95=24.3ms,  recall@10=97%
ef_search=200:          p95=41.5ms,  recall@10=99%

Pattern: ef_search has diminishing returns. From 10 → 50 recall improves 6%; from 100 → 200 it improves only 2% but doubles the latency.

Recommendation:

Typical production: ef_search=50-100. The accuracy/latency sweet spot.
Demo / prototype: default (10) is fine.
Critical recall (medical, legal): 200+, justify the latency.

Parameter 4: dataset size

The last thing you control — and the most important in the long run. HNSW latency grows logarithmically with the number of vectors, not linearly. That's good (10x docs ≠ 10x latency), but it isn't free.

Dataset size	Typical p95 (no filter, ef_search=10)
1,000	2-5ms
10,000	8-15ms
100,000	18-35ms
1,000,000	40-80ms
10,000,000	80-150ms

Operational implication: if you build a RAG today with 10K docs and project growth to 1M in 6 months, the base latency will rise from 10ms to 50ms. That can break your SLA. Anticipating it means:

Design the SLA with margin for growth (if the final target is <100ms with 1M, define <50ms with 100K to have margin).
Consider migration to distributed vector DBs (Pinecone, Weaviate cluster) when you cross 5-10M vectors.

Optimization: the correct order

Don't optimize blindly. The sequence that gives the most performance for the least effort:

1. First, measure

Before touching anything, run the latency and recall benchmark over your real dataset. Without a baseline, you can't know whether you "optimized" or "regressed".

2. Then, lower `n_results` to the minimum needed

There's almost always margin here. If your RAG uses n_results=20 out of inertia, try 5. If the answer quality doesn't drop, you already won 30-50% latency for free.

3. Then, add metadata filtering

If you have filters that reduce to 10-30% of the dataset, apply them. That reduces the search space dramatically. If you don't have useful metadata, consider adding it (category, date, source) — it's ingestion work but it saves latency forever.

4. Then, tune `ef_search`

If you need more recall (queries fail to find relevant docs you know exist), raise ef_search. If latency is the problem, lower it. Measure both metrics — accuracy AND latency — before and after.

5. If all of the above isn't enough, consider hardware or a different vector DB

More RAM lets you load the full HNSW index into memory (vs paging to disk).
SSD vs HDD makes a noticeable difference for large datasets.
If you exceed 10M vectors and latency matters, ChromaDB won't scale well. Migrate to Pinecone or distributed Qdrant.

Traps and common mistakes

Trap 1: measuring without warm-up

The mistake: the first benchmark, the first query includes the cost of loading the HNSW index from disk to RAM. The measurement is 5-10x slower than steady state.

Symptom: latencies from the first benchmark are much higher than the second.

How to prevent it: run 5-10 "warm-up" queries before starting to measure. Those queries aren't counted.

# Warm-up
for q in queries[:10]:
    collection.query(query_texts=[q], n_results=10)

# Real measurement (discarding warm-up)
latencies = []
for q in queries:
    start = time.perf_counter()
    collection.query(query_texts=[q], n_results=10)
    latencies.append((time.perf_counter() - start) * 1000)

Trap 2: mixing embedding latency with search latency

The mistake: you measure collection.query(query_texts=["..."]) and see p95=180ms. You assume ChromaDB is slow.

Symptom: you think the search in ChromaDB takes 180ms, when in reality ChromaDB takes 15ms and the other 165ms are the OpenAI API call to embed the query.

How to prevent it: separate the two measurements.

# Pre-compute the embedding once
query_embedding = openai_client.embeddings.create(
    model="text-embedding-3-small",
    input=query_text
).data[0].embedding

# Measure only the ChromaDB cost
start = time.perf_counter()
collection.query(query_embeddings=[query_embedding], n_results=10)
chromadb_latency = (time.perf_counter() - start) * 1000

Trap 3: optimizing latency by breaking recall

The mistake: you lower ef_search aggressively to improve p95, without measuring the impact on accuracy.

Symptom: p95 improves from 30ms to 8ms 🎉. But recall drops from 95% to 78% — the system responds fast with worse results. Users report "the bot doesn't find things I know are there".

How to prevent it: always measure both metrics together on an eval set. A valid optimization is one that improves one without destroying the other.

# Eval set: queries with labeled relevant docs
EVAL_SET = [
    {"query": "How to configure HNSW?", "relevant_doc_ids": ["doc_42", "doc_87"]},
    # ... 30+ entries
]

def measure_recall_at_k(collection, eval_set, k=10):
    hits = 0
    total = 0
    for item in eval_set:
        results = collection.query(query_texts=[item["query"]], n_results=k)
        retrieved_ids = set(results['ids'][0])
        relevant_ids = set(item["relevant_doc_ids"])
        if retrieved_ids & relevant_ids:
            hits += len(retrieved_ids & relevant_ids)
        total += len(relevant_ids)
    return hits / total

# Before and after each change
print(f"Latency p95: {p95:.1f}ms")
print(f"Recall@10:  {measure_recall_at_k(collection, EVAL_SET):.2%}")

Trap 4: an ultra-specific filter makes the query slower

The mistake: you assume more filter = faster. You apply a filter that leaves only 1-2 valid candidates.

Symptom: queries with an ultra-specific filter are slower than without a filter, counterintuitively.

Why it happens: HNSW has to scan more nodes to find enough that satisfy the filter (target = n_results).

How to prevent it: if your filter leaves fewer candidates than n_results × 5, consider using get() with a direct filter:

# Instead of a query with an ultra-specific filter
results = collection.query(
    query_embeddings=[emb],
    n_results=10,
    where={"unique_field": "specific_value"}
)

# Better: direct get if there are only a few matches
results = collection.get(
    where={"unique_field": "specific_value"},
    include=['documents', 'metadatas']
)
# It's not semantic search, but it's much faster when the filter is very restrictive

Trap 5: optimizing the average without looking at the tails

The mistake: you report "p50 improved 20%, deploy". But you didn't look at p99.

Symptom: p50 drops from 50ms to 40ms. But p99 rises from 200ms to 800ms. The 1% of your users is suffering 4x more, and the average hides it.

How to prevent it: always report at least p50, p95, p99. The change is validated if all improve or hold. If any regresses, revert.

Trap 6: irreproducible benchmarks

The mistake: you run a benchmark, note the result, change something, run again. The two measurements have different runs, different queries, different datasets.

Symptom: "I think it improved but I'm not sure". Decisions based on intuition.

How to prevent it:

Same fixed eval set in every benchmark.
Same dataset (snapshot if necessary).
Same machine, same load level (not while something else is running).
Multiple executions (3-5 runs), report the median of the medians.

def run_benchmark_n_times(collection, queries, n_runs=3):
    """Runs the benchmark N times and reports robust statistics."""
    all_p50, all_p95, all_p99 = [], [], []
    for run in range(n_runs):
        latencies = sorted([time_query(collection, q) for q in queries])
        all_p50.append(latencies[len(latencies) // 2])
        all_p95.append(latencies[int(len(latencies) * 0.95)])
        all_p99.append(latencies[int(len(latencies) * 0.99)])

    print(f"p50: {statistics.median(all_p50):.1f}ms (variation {min(all_p50):.0f}-{max(all_p50):.0f})")
    print(f"p95: {statistics.median(all_p95):.1f}ms (variation {min(all_p95):.0f}-{max(all_p95):.0f})")
    print(f"p99: {statistics.median(all_p99):.1f}ms (variation {min(all_p99):.0f}-{max(all_p99):.0f})")

Applied exercise

Scenario: a team that operates a RAG in production calls you. Symptoms:

Current p50: 180ms (acceptable)
Current p95: 450ms (breaks the 300ms SLA)
Current p99: 850ms (terrible)
Current configuration: ChromaDB with 250K vectors, OpenAI embeddings, n_results=15, no metadata filtering, ef_search default
The team says "we need a faster vector DB, we have to migrate to Pinecone"

Question: without migrating DB, propose three specific optimizations you would try in order, justify the expected impact of each one, and what you would measure to validate them.

Solution

Problem analysis:

Before proposing optimizations, identify what causes the high latency. The suspects:

Overhead of the OpenAI query embedding: 80-150ms of the p50 is probably this. If you verify it (separating query embedding from search), you confirm that ChromaDB itself takes ~30-80ms.
n_results=15 is high for typical RAG — especially if it's then passed to the LLM, where 5-7 chunks usually suffice.
No metadata filtering: it always searches the 250K vectors.
ef_search default (10): fast but recall may be poor, and it doesn't directly affect the current p95 unless it's one of the secondary factors.

Optimization 1: lower n_results from 15 to 5

Hypothesis: most RAG doesn't need 15 chunks. Lowering to 5 reduces latency ~30% (based on the pattern observed in benchmarks).

Expected impact:

p50: 180ms → ~140ms
p95: 450ms → ~320ms
Cost to validate: 30 minutes. Test it on the eval set, measure recall@5 vs recall@15. If recall doesn't drop more than 3-5%, deploy.

Risk: some queries need more context. Mitigate with re-ranking or expanding the top-k only when the first pass doesn't find enough.

Metric to validate:

p50/p95/p99 before and after
Recall@5 over the eval set (should stay >90% of the current recall@15)
LLM answer quality over 50 eval queries (subjective, A/B comparison)

Optimization 2: add metadata filtering

Hypothesis: if queries have identifiable context (document type, language, department), filtering reduces the search space 5-20x.

Implementation:

Inventory what metadata the chunks currently have. If they don't have enough, add it (re-ingest with category, source_type, language, date_range).
Identify queries from the production log and group them by intent (are they "support"? "billing"? "general"?). If there's obvious classification, filter.
Pre-classify queries with a simple model (even an LLM with a short prompt) to assign a category before retrieval.

Expected impact (if the filter reduces to 30-50K candidates, ~20% of the dataset):

p50: 140ms → ~110ms
p95: 320ms → ~220ms (enters the 300ms SLA)
p99: 850ms → ~400ms

Metric to validate:

p50/p95/p99 with and without the filter, over the same eval set
Recall should improve or hold: filtering well removes noise
% of queries that can be filtered (coverage)

Optimization 3: tune ef_search

Hypothesis: if after the two previous optimizations latency is good but recall is mediocre, raising ef_search can improve it. If latency is still high and recall is more than enough, lowering it can provide margin.

How to decide:

If current recall@5 > 92%: rich dataset, lowering ef_search can squeeze out a bit more latency without touching quality.
If current recall@5 < 85%: raising ef_search to 50-100 improves quality at the cost of latency (which is already good after optimizations 1-2).

Metric to validate:

Pareto plot: latency (X axis) vs recall (Y axis) with different ef_search. Choose the point that meets the SLA with the best recall.

Plan summary:

Optimization	Effort	Risk	Expected p95 impact
1. n_results 15→5	30 min	Low (measurable in eval)	-30%
2. Metadata filtering	4-8 hrs (ingest + classification)	Medium (requires metadata)	-30% additional
3. Tune ef_search	1 hr	Low (reversible)	±10% (depends on direction)

Total expected: p95 from 450ms → 200-250ms. Within the SLA without migrating.

If after the 3 optimizations it still doesn't get there:

Verify the real overhead of the OpenAI embedding (probably 80-120ms of the p95). If it's that, consider caching query embeddings (LRU over the most common queries).
Consider local embeddings with a GPU for queries (lower and more predictable latency than the API).
Only then evaluate migration to another DB.

Argument to the team: "Migrating to Pinecone is a 4-6 week project with a recurring cost of $$/month. The three optimizations I propose are 1-2 weeks with zero cost. Let's try first. If we don't hit the SLA, the migration stays as Plan B with data to justify it."

Summary and next step

What you learned:

Measure latency with percentiles (p50/p95/p99), not averages. The average hides tail latency, which is what defines your SLA.
Four parameters move query latency: n_results, metadata filtering, ef_search, and dataset size.
Optimize in order: first lower n_results to the minimum needed, then add metadata filtering, then tune ef_search, and last change hardware or DB.
Always measure latency and recall together. Optimizing one by destroying the other isn't optimization.
Warm-up matters. The first 5-10 queries of any benchmark are outliers and shouldn't be counted.
An ultra-specific filter can be slower than no filter — if it leaves fewer candidates than n_results × 5, use a direct get() instead of query().

Checkpoint: before moving on, you should be able to:

Design a reproducible benchmark that reports p50/p95/p99 over your dataset.
Justify the n_results choice for a case (RAG vs search vs re-ranking).
Diagnose high latency in order (n_results → filtering → ef_search → hardware) without jumping to "I need another DB".

Next capsule: 07 — Persistence and durability.

You just learned to make queries fast. But what happens to your data when you restart the process? What happens if the machine crashes during a write? How do you back up and restore without losing vectors? Capsule 07 covers persistence and durability — the operational aspects that distinguish a prototype (that gets wiped on restart) from a reliable system.

Resources

ChromaDB — Querying Collections — Query and parameter reference
HNSW Algorithm Paper — Malkov & Yashunin — Section 4.2 explains ef_search in depth
Latency vs Throughput in Vector Databases (Pinecone Blog) — Analysis of latency patterns
Lost in the Middle: How Language Models Use Long Contexts — Why lowering n_results in RAG improves quality
Percentiles: Why Averages Lie (Brendan Gregg) — On the importance of p99 in systems
hyperfine — Command-line Benchmarking Tool — Reference for rigorous benchmarks

Estimated time: 35-45 minutes Next: 07-persistence-durability.md

← PreviousCapsule 05: Batch Ingestion in ChromaDB — from concept to an operable pipeline Next →Capsule 07: Persistence and Durability — what happens when something breaks

Capsule 06: Query Optimization — where you win or lose real latency

Capsule description

Why measuring the average lies to you

Measuring correctly

The four parameters that move latency

Parameter 1: n_results (top-K)

Parameter 2: metadata filtering (where clauses)

Parameter 3: ef_search (HNSW runtime parameter)

Parameter 4: dataset size

Optimization: the correct order

1. First, measure

2. Then, lower n_results to the minimum needed

3. Then, add metadata filtering

4. Then, tune ef_search

5. If all of the above isn't enough, consider hardware or a different vector DB

Traps and common mistakes

Trap 1: measuring without warm-up

Trap 2: mixing embedding latency with search latency

Trap 3: optimizing latency by breaking recall

Trap 4: an ultra-specific filter makes the query slower

Trap 5: optimizing the average without looking at the tails

Trap 6: irreproducible benchmarks

Applied exercise

Summary and next step

Resources

Parameter 1: `n_results` (top-K)

Parameter 3: `ef_search` (HNSW runtime parameter)

2. Then, lower `n_results` to the minimum needed

4. Then, tune `ef_search`