Module 6: Designing Search Systems

6. Evaluating Search Quality

Next →

Overview

How do you know whether your search system is any good? This capsule covers the standard metrics: precision, recall, MRR (Mean Reciprocal Rank), NDCG (Normalized Discounted Cumulative Gain), and how to create an evaluation set.

Base concepts

Ground truth (the expected answer):
For each test query, you define which documents ARE relevant.

Example:

Query: "how to use Pinecone"

Ground truth (the relevant documents):
- Doc 12 (a setup guide)
- Doc 34 (a basic tutorial)
- Doc 56 (an FAQ)

NOT relevant documents:
- Doc 78 (a Pinecone vs Weaviate comparison)
- Doc 90 (Pinecone pricing)

Metric 1: Precision

Definition:
Of the documents returned, how many are relevant?

Formula:

Precision@K = (relevant documents in the top-K) / K

Example:

Query: "how to use Pinecone"
Ground truth: [Doc 12, Doc 34, Doc 56]

The system returns the top-5:
1. Doc 12 ✅
2. Doc 78 ❌
3. Doc 34 ✅
4. Doc 90 ❌
5. Doc 56 ✅

Precision@5 = 3/5 = 0.60 (60%)

Interpretation:

  • Precision@5 = 1.0 → All of the top-5 are relevant (perfect)
  • Precision@5 = 0.0 → None are relevant (terrible)

Metric 2: Recall

Definition:
Of all the relevant documents, how many were returned?

Formula:

Recall@K = (relevant documents in the top-K) / (total relevant)

Example (the same case):

Ground truth: 3 relevant documents [Doc 12, Doc 34, Doc 56]
The system returns the top-5: [12, 78, 34, 90, 56]

Relevant documents returned: 3 (Doc 12, 34, 56)

Recall@5 = 3/3 = 1.0 (100%)

If the system had returned only the top-3:

Top-3: [Doc 12, Doc 78, Doc 34]
Relevant ones returned: 2 (Doc 12, 34)

Recall@3 = 2/3 ≈ 0.67 (67%)

The precision vs recall trade-off

The scenario:

  • Return few results → high precision, low recall
  • Return many results → high recall, low precision

Example:

Ground truth: [Doc A, Doc B, Doc C]

System 1 (top-3): [Doc A, Doc B, Doc X]
  Precision@3 = 2/3 ≈ 0.67
  Recall@3 = 2/3 ≈ 0.67

System 2 (top-10): [Doc A, Doc B, Doc C, Doc X, Doc Y, ...]
  Precision@10 = 3/10 = 0.30
  Recall@10 = 3/3 = 1.0

The trade-off: More results → more recall, less precision.

Metric 3: MRR (Mean Reciprocal Rank)

Definition:
The position of the first relevant document (averaged across multiple queries).

Formula:

RR = 1 / (the position of the first relevant one)

MRR = the average of RR across all the queries

Example:

Query 1: The first relevant one is at position 1
  RR1 = 1/1 = 1.0

Query 2: The first relevant one is at position 3
  RR2 = 1/3 ≈ 0.33

Query 3: The first relevant one is at position 2
  RR3 = 1/2 = 0.5

MRR = (1.0 + 0.33 + 0.5) / 3 ≈ 0.61

Interpretation:

  • MRR = 1.0 → The first result is always relevant (perfect)
  • MRR = 0.5 → The first relevant one is at position 2 on average
  • MRR < 0.3 → A poor system

Metric 4: NDCG (Normalized Discounted Cumulative Gain)

Why NDCG?
Precision/recall don't consider order. NDCG penalizes relevant items in low positions.

Formula (simplified):

DCG@K = Σ (relevance_i / log2(position_i + 1))

NDCG@K = DCG@K / IDCG@K

IDCG (Ideal DCG): The DCG if the results were in perfect order.

Example:

Query: "Python tutorial"
Ground truth relevance (0-3):
- Doc A: 3 (highly relevant)
- Doc B: 2 (relevant)
- Doc C: 1 (somewhat relevant)
- Doc D: 0 (not relevant)

The system returns:
1. Doc B (relevance: 2)
2. Doc D (relevance: 0)
3. Doc A (relevance: 3)

DCG@3:
  = 2/log2(2) + 0/log2(3) + 3/log2(4)
  = 2/1 + 0 + 3/2
  = 2 + 0 + 1.5
  = 3.5

IDCG@3 (the ideal order: A, B, C):
  = 3/log2(2) + 2/log2(3) + 1/log2(4)
  = 3/1 + 2/1.58 + 1/2
  ≈ 3 + 1.26 + 0.5
  = 4.76

NDCG@3 = 3.5 / 4.76 ≈ 0.735

Interpretation:

  • NDCG = 1.0 → Perfect order
  • NDCG = 0.735 → Good (Doc A should have been first)

Creating an evaluation set

Step 1: Define the test queries

Query 1: "how to use Pinecone"
Query 2: "the difference between HNSW and IVF"
Query 3: "what is RAG"
...
(20-100 typical queries)

Step 2: Labeling (manual)

For each query:
  - Run the current system
  - Review the top-20 results
  - Label each as relevant (1) or not (0)
  - Save the ground truth

Step 3: Compute the metrics

For each query:
  - Run the system
  - Compare against the ground truth
  - Compute Precision@K, Recall@K, MRR, NDCG

Average the metrics across all the queries

Benchmarking

Before the changes:

Precision@5: 0.65
Recall@10: 0.80
MRR: 0.55
NDCG@10: 0.70

After adding reranking:

Precision@5: 0.78 (+13%)
Recall@10: 0.80 (unchanged)
MRR: 0.68 (+13%)
NDCG@10: 0.82 (+12%)

Conclusion: Reranking improves precision and ranking without affecting recall.

Summary

Key points:

  • Precision: How many of the returned documents are relevant?
  • Recall: How many of the relevant documents were returned?
  • MRR: The position of the first relevant one
  • NDCG: The quality of the ranking (order matters)
  • Evaluation set: 20-100 labeled queries

Next capsule: 07-case-studies.md — RAG, e-commerce, support.

← Previous5. Advanced Ranking StrategiesNext →7. Search System Case Studies