4. Query Processing: From Query to Results

Overview

Query processing is the online phase: turning the user's query into relevant results. It covers embedding the query, kNN search, reranking, and latency optimization.

---

The query processing flow

User Query
    ↓
1. Query embedding (the OpenAI API)
    ↓
2. kNN search (the Vector DB)
    ↓
3. Metadata filtering (optional)
    ↓
4. Reranking (optional)
    ↓
5. Top-K results

---

Step 1: Query embedding

Input: The user's query (text)
Output: A 1536D vector (or 768D, 3072D depending on the model)

Example:

Query: "how to implement RAG with Pinecone"
→ The OpenAI API (text-embedding-3-small)
→ [0.23, -0.45, 0.12, ..., -0.34]

Latency: 50-200ms (the OpenAI API)

---

Step 2: kNN search

Input: The query embedding
Output: The top-K candidates (k=100 is typical)

A Pinecone example:

Query embedding: [0.23, -0.45, ...]
Index: "production-docs"
Top-K: 100
Filters: {"source": "rag-guide.pdf"}

→ Returns the 100 chunks with the highest cosine

Latency: 10-50ms (with HNSW)

---

Step 3: Metadata filtering

What is it?
Filtering the results by metadata BEFORE or AFTER the kNN.

Pre-filtering (before the kNN):

1. Filter by metadata: source = "rag-guide.pdf"
2. Run kNN only on that subset

Advantage: Faster (it searches a subset).
Disadvantage: If the filter is too restrictive, you get few results.

Post-filtering (after the kNN):

1. kNN returns the top-100
2. Filter the top-100 by metadata

Advantage: It guarantees K results.
Disadvantage: It can discard good candidates.

---

Step 4: Reranking

Why rerank?

The problem:

kNN returns the top-100 based on cosine
→ But cosine is the similarity of the embeddings
→ It does NOT consider the specific query-document interaction

The solution: A cross-encoder

Model: ms-marco-MiniLM (specialized in reranking)

For each (query, document) pair:
  → A relevance score [0-1]
  → Reorder the top-100 by this score
  → Return the final top-10

Example:

Query: "how to use Pinecone"

kNN top-3:
1. Doc A: "Pinecone is a vector database..." (cosine: 0.89)
2. Doc B: "A step-by-step guide to using Pinecone" (cosine: 0.87)
3. Doc C: "Pinecone vs Weaviate comparison" (cosine: 0.86)

Cross-encoder:
1. Doc B: Score 0.95 (most relevant: it's a usage guide)
2. Doc A: Score 0.75 (a general definition)
3. Doc C: Score 0.60 (a comparison, not direct usage)

Final ranking:
1. Doc B ✅
2. Doc A
3. Doc C

Latency: 100-300ms (the Cohere Rerank API)

---

Step 5: Returning the results

Typical output:

{
  "query": "how to use Pinecone",
  "results": [
    {
      "id": "chunk-456",
      "score": 0.95,
      "text": "A guide to using Pinecone...",
      "metadata": {
        "source": "rag-guide.pdf",
        "page": 12
      }
    },
    {
      "id": "chunk-123",
      "score": 0.75,
      "text": "Pinecone is a vector database...",
      "metadata": {
        "source": "intro.pdf",
        "page": 3
      }
    }
  ],
  "latency_ms": 320
}

---

Trade-offs

kNN only (no reranking):

• ✅ Low latency (60-250ms)
• ❌ Limited precision

kNN + reranking:

• ✅ Better precision (reordered with a cross-encoder)
• ❌ Higher latency (160-550ms)

Decision: Use reranking if precision > speed (e.g. scientific search). Skip it if speed is critical (e.g. live chat).

---

Latency optimization

Technique 1: Cache the embeddings of frequent queries

Query: "what is RAG"
→ Check the cache
→ If it exists: use the cached embedding (latency: 0ms)
→ If NOT: generate the embedding (latency: 50-200ms)

Impact: 50-200ms saved on repeated queries.

---

Technique 2: Reduce top-K

Top-100 vs Top-50
→ A faster kNN (fewer candidates to evaluate)
→ A faster reranking (fewer cross-encoder evaluations)

Trade-off: Fewer candidates → lower recall.

---

Technique 3: Use selective reranking

If the query is simple (1-3 words):
→ kNN only

If the query is complex (a long question):
→ kNN + reranking

---

A complete example

Query: "the difference between HNSW and IVF"

Step 1: Embedding

The OpenAI API → [0.23, -0.45, ...]
Latency: 120ms

Step 2: kNN

Pinecone.query(
  vector=[0.23, -0.45, ...],
  top_k=100,
  filter={"tags": "indexes"}
)
→ 100 candidates
Latency: 30ms

Step 3: Reranking

The Cohere Rerank API:
  query="the difference between HNSW and IVF"
  documents=[top-100]
→ The top-10 reordered
Latency: 200ms

Step 4: Return

The top-10 results
Total latency: 120 + 30 + 200 = 350ms

---

Summary

Key points:

• Query processing: Embedding → kNN → Reranking
• Typical latency: 160-550ms (with reranking)
• Reranking: A cross-encoder improves precision
• Optimization: Caching, reducing top-K, selective reranking

---

Next capsule: 05-ranking-strategies.md — Score fusion, RRF, MMR.