Module 4: Evaluation and Chunking Strategies

Retrieval Metrics: Measuring RAG Performance

Next →

Capsule overview

"Does my RAG system work?" Without metrics, you only have intuition. Retrieval metrics (nDCG, MRR, Recall@K, Precision@K) let you objectively measure how well your system finds relevant documents.

In this capsule you'll learn the main retrieval metrics, how to calculate them in Python, what each score means, and when to use each metric. You'll also implement a complete evaluator.

By the end, you'll be able to evaluate and optimize your RAG systems with rigorous metrics.

Main metrics

1. Recall@K

Definition:

Of all the relevant documents, what percentage is in the top-K results?

Formula:

Recall@K = (Relevant docs in top-K) / (Total relevant docs)

Example:

# Query: "Python installation"
# Total relevant docs: [doc_3, doc_7, doc_12]  (3 relevant)
# Top-5 retrieved: [doc_1, doc_3, doc_5, doc_7, doc_9]

# Relevant docs in top-5: [doc_3, doc_7]  (2 of 3)
Recall@5 = 2 / 3 = 0.67 (67%)

Interpretation:

  • Recall@5 = 1.0 → All relevant docs are in the top-5 ✅
  • Recall@5 = 0.67 → Missing 1 relevant doc
  • Recall@5 = 0.0 → No relevant doc in the top-5 ❌

2. Precision@K

Definition:

Of the top-K results, what percentage is relevant?

Formula:

Precision@K = (Relevant docs in top-K) / K

Example:

# Top-5 retrieved: [doc_1, doc_3, doc_5, doc_7, doc_9]
# Relevant docs: [doc_3, doc_7]  (2 of 5)

Precision@5 = 2 / 5 = 0.40 (40%)

Interpretation:

  • Precision@5 = 1.0 → All top-5 are relevant ✅
  • Precision@5 = 0.40 → 40% of the top-5 are relevant
  • Precision@5 = 0.0 → None of the top-5 is relevant ❌

3. F1@K

Definition:

The harmonic mean of Recall@K and Precision@K.

Formula:

F1@K = 2 × (Precision@K × Recall@K) / (Precision@K + Recall@K)

Example:

Recall@5 = 0.67
Precision@5 = 0.40

F1@5 = 2 × (0.40 × 0.67) / (0.40 + 0.67) = 0.50

Interpretation: A balance between Recall and Precision.

4. MRR (Mean Reciprocal Rank)

Definition:

The position of the FIRST relevant document.

Formula:

RR = 1 / position_of_first_relevant
MRR = average of RR over multiple queries

Example:

# Query 1:
# Top-5: [doc_1, doc_3✓, doc_5, doc_7✓, doc_9]
# First relevant: position 2
RR_1 = 1/2 = 0.50

# Query 2:
# Top-5: [doc_2, doc_4, doc_6✓, doc_8, doc_10]
# First relevant: position 3
RR_2 = 1/3 = 0.33

# Query 3:
# Top-5: [doc_1✓, doc_3, doc_5, doc_7, doc_9]
# First relevant: position 1
RR_3 = 1/1 = 1.0

MRR = (0.50 + 0.33 + 1.0) / 3 = 0.61

Interpretation:

  • MRR = 1.0 → Always a relevant doc in position 1 ✅
  • MRR = 0.5 → On average in position 2
  • MRR < 0.2 → Relevant docs very far down ❌

5. nDCG@K (Normalized Discounted Cumulative Gain)

Definition:

Like Recall but it penalizes relevant docs that are further down in the ranking.

Formula:

DCG@K = Σ (relevance_i / log2(position_i + 1))
nDCG@K = DCG@K / IDCG@K  (normalized 0-1)

Example:

# Top-5 retrieved: [doc_1, doc_3, doc_5, doc_7, doc_9]
# Relevance scores:
#   doc_1: 0 (not relevant)
#   doc_3: 1 (relevant)
#   doc_5: 0
#   doc_7: 1 (relevant)
#   doc_9: 0

# DCG@5:
DCG = (0/log2(2)) + (1/log2(3)) + (0/log2(4)) + (1/log2(5)) + (0/log2(6))
    = 0 + 0.63 + 0 + 0.43 + 0
    = 1.06

# IDCG@5 (ideal ranking: all relevant at the start):
IDCG = (1/log2(2)) + (1/log2(3)) + ...
     = 1.63

# nDCG@5:
nDCG = 1.06 / 1.63 = 0.65

Interpretation:

  • nDCG@5 = 1.0 → Perfect ranking ✅
  • nDCG@5 = 0.65 → Acceptable ranking
  • nDCG@5 < 0.3 → Bad ranking ❌

Implementation in Python

The RetrievalEvaluator class:

import numpy as np
from typing import List, Dict

class RetrievalEvaluator:
    """Evaluator for retrieval metrics"""
    
    def __init__(self, k: int = 5):
        """
        Args:
            k: Number of docs to consider (top-K)
        """
        self.k = k
    
    def recall_at_k(self, retrieved: List[int], relevant: List[int]) -> float:
        """
        Recall@K
        
        Args:
            retrieved: IDs of retrieved docs (ordered by score)
            relevant: IDs of relevant docs (ground truth)
        
        Returns:
            Recall score [0, 1]
        """
        if not relevant:
            return 0.0
        
        retrieved_k = set(retrieved[:self.k])
        relevant_set = set(relevant)
        
        intersection = retrieved_k & relevant_set
        
        return len(intersection) / len(relevant_set)
    
    def precision_at_k(self, retrieved: List[int], relevant: List[int]) -> float:
        """Precision@K"""
        retrieved_k = set(retrieved[:self.k])
        relevant_set = set(relevant)
        
        intersection = retrieved_k & relevant_set
        
        return len(intersection) / self.k if self.k > 0 else 0.0
    
    def f1_at_k(self, retrieved: List[int], relevant: List[int]) -> float:
        """F1@K"""
        precision = self.precision_at_k(retrieved, relevant)
        recall = self.recall_at_k(retrieved, relevant)
        
        if precision + recall == 0:
            return 0.0
        
        return 2 * (precision * recall) / (precision + recall)
    
    def mrr(self, retrieved: List[int], relevant: List[int]) -> float:
        """Mean Reciprocal Rank (for 1 query)"""
        for i, doc_id in enumerate(retrieved):
            if doc_id in relevant:
                return 1.0 / (i + 1)
        
        return 0.0
    
    def ndcg_at_k(self, retrieved: List[int], relevant: List[int]) -> float:
        """
        nDCG@K (binary relevance: 1 or 0)
        
        Args:
            retrieved: retrieved IDs (ordered)
            relevant: relevant IDs
        """
        relevant_set = set(relevant)
        
        # DCG (Discounted Cumulative Gain)
        dcg = 0.0
        for i, doc_id in enumerate(retrieved[:self.k]):
            relevance = 1 if doc_id in relevant_set else 0
            dcg += relevance / np.log2(i + 2)  # i+2 because i starts at 0
        
        # IDCG (Ideal DCG - all relevant at the start)
        idcg = 0.0
        for i in range(min(len(relevant), self.k)):
            idcg += 1.0 / np.log2(i + 2)
        
        if idcg == 0:
            return 0.0
        
        return dcg / idcg
    
    def evaluate_all(self, retrieved: List[int], relevant: List[int]) -> Dict:
        """Calculate all metrics"""
        return {
            'recall@k': self.recall_at_k(retrieved, relevant),
            'precision@k': self.precision_at_k(retrieved, relevant),
            'f1@k': self.f1_at_k(retrieved, relevant),
            'mrr': self.mrr(retrieved, relevant),
            'ndcg@k': self.ndcg_at_k(retrieved, relevant)
        }

# Usage
evaluator = RetrievalEvaluator(k=5)

# Example
retrieved = [1, 3, 5, 7, 9, 11, 13]  # Retrieved IDs (ordered by score)
relevant = [3, 7, 12]  # Relevant IDs (ground truth)

metrics = evaluator.evaluate_all(retrieved, relevant)

print("Retrieval Metrics:")
for metric, value in metrics.items():
    print(f"  {metric}: {value:.3f}")

Output:

Retrieval Metrics:
  recall@k: 0.667  (2 of 3 relevant in top-5)
  precision@k: 0.400  (2 of 5 retrieved are relevant)
  f1@k: 0.500  (balance)
  mrr: 0.500  (first relevant in position 2)
  ndcg@k: 0.498  (ranking can improve: doc_12 never appears)

Evaluate multiple queries

Batch evaluation:

def evaluate_multiple_queries(evaluator, test_cases):
    """
    Evaluate multiple queries
    
    Args:
        test_cases: List of dicts with keys 'retrieved' and 'relevant'
    
    Returns:
        Dict with average metrics
    """
    all_metrics = []
    
    for case in test_cases:
        metrics = evaluator.evaluate_all(case['retrieved'], case['relevant'])
        all_metrics.append(metrics)
    
    # Average
    avg_metrics = {}
    for key in all_metrics[0].keys():
        avg_metrics[key] = np.mean([m[key] for m in all_metrics])
    
    return avg_metrics

# Test cases
test_cases = [
    {
        'query': 'Python installation',
        'retrieved': [1, 3, 5, 7, 9],
        'relevant': [3, 7, 12]
    },
    {
        'query': 'List comprehensions',
        'retrieved': [2, 4, 6, 8, 10],
        'relevant': [4, 6]
    },
    {
        'query': 'Django tutorial',
        'retrieved': [1, 2, 3, 4, 5],
        'relevant': [1, 3]
    }
]

evaluator = RetrievalEvaluator(k=5)
avg_metrics = evaluate_multiple_queries(evaluator, test_cases)

print("Average metrics (3 queries):")
for metric, value in avg_metrics.items():
    print(f"  {metric}: {value:.3f}")

Output:

Average metrics (3 queries):
  recall@k: 0.889
  precision@k: 0.400
  f1@k: 0.548
  mrr: 0.667
  ndcg@k: 0.704

When to use each metric

Recall@K:

✅ RAG systems (you want to capture ALL relevant docs)
✅ When missing a relevant doc is costly
✅ Example: Legal search (you CAN'T miss relevant cases)

Typical target: Recall@5 > 0.85

Precision@K:

✅ When noise (irrelevant docs) is costly
✅ Limited LLM context window (you only want what's relevant)
✅ Example: Chatbot (concise answers)

Typical target: Precision@3 > 0.80

F1@K:

✅ Balance between Recall and Precision
✅ When both matter equally
✅ A single metric to compare systems

Typical target: F1@5 > 0.75

MRR:

✅ When only the FIRST result matters
✅ Example: FAQ search (the user only sees #1)
✅ Web search (Google)

Typical target: MRR > 0.8

nDCG@K:

✅ When ranking matters (position of relevant docs)
✅ RAG with multiple retrieved docs
✅ The most complete metric (it considers position)

Typical target: nDCG@5 > 0.85

Exercises

Exercise 1: Calculate Recall@K

retrieved = [1, 2, 3, 4, 5]
relevant = [3, 6, 9]

# What is Recall@5?
See solution 
retrieved_k = set(retrieved[:5])  # {1, 2, 3, 4, 5}
relevant_set = set(relevant)       # {3, 6, 9}

intersection = retrieved_k & relevant_set  # {3}

recall = len(intersection) / len(relevant_set)
# recall = 1 / 3 = 0.33

print(f"Recall@5: {recall:.2f}")  # 0.33

Interpretation: Only 1 of 3 relevant docs is in the top-5.

Exercise 2: Calculate nDCG@5

retrieved = [1, 3, 5, 7, 9]  # Retrieved IDs
relevant = [3, 7]             # Relevant IDs

# What is nDCG@5?
See solution 
import numpy as np

# DCG
dcg = 0.0
for i, doc_id in enumerate(retrieved):
    relevance = 1 if doc_id in relevant else 0
    dcg += relevance / np.log2(i + 2)

# doc_1: 0/log2(2) = 0
# doc_3: 1/log2(3) = 0.63
# doc_5: 0/log2(4) = 0
# doc_7: 1/log2(5) = 0.43
# doc_9: 0/log2(6) = 0
# dcg = 1.06

# IDCG (2 relevant at the start)
idcg = 1/np.log2(2) + 1/np.log2(3)
# idcg = 1.0 + 0.63 = 1.63

# nDCG
ndcg = dcg / idcg
print(f"nDCG@5: {ndcg:.3f}")  # 0.651

Summary

What you learned:

  • ✅ Recall@K: % of relevant in top-K
  • ✅ Precision@K: % of top-K that are relevant
  • ✅ F1@K: Recall/Precision balance
  • ✅ MRR: Position of the first relevant
  • ✅ nDCG@K: Ranking-aware (the best metric)

Key concepts:

  1. nDCG@K is the best general metric (it considers ranking)
  2. Recall is critical for RAG (capture all relevant)
  3. MRR for single-result systems (FAQ)

Additional resources

  1. Information Retrieval Metrics - Microsoft Research
  2. nDCG Explained - Wikipedia
  3. Evaluation Metrics - Stanford NLP

In the next capsule

Capsule 05: Create Evaluation Datasets

You'll learn:

  • Evaluation dataset formats
  • Synthetic data generation
  • Human annotation
  • Quality assurance

From metrics to evaluation datasets.

Module 4 - Embeddings Deep Dive Guide Retrieval metrics: measuring RAG objectively

← PreviousSemantic Chunking: Respecting Document StructureNext →Creating Evaluation Datasets for RAG