Creating Evaluation Datasets for RAG

Quick overview

An evaluation dataset lets you objectively measure your RAG system. Without one, you only have intuition. With one, you have data to optimize chunking, models, and retrieval.

You'll learn evaluation dataset formats, synthetic data generation, human annotation guidelines, and quality assurance. By the end, you'll be able to create robust datasets.

---

Evaluation Dataset Format

Basic structure:

{
  "queries": [
    {
      "id": "q1",
      "text": "How to install Python?",
      "relevant_doc_ids": [3, 7, 12],
      "difficulty": "easy"
    }
  ],
  "corpus": [
    {"id": 3, "text": "Download Python from python.org..."},
    {"id": 7, "text": "Use package manager: apt install python3..."}
  ]
}

---

Synthetic Data Generation

Method 1: GPT-4 to generate queries

from openai import OpenAI
import os
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def generate_queries_for_doc(doc_text: str, n_queries: int = 3):
    """Generate synthetic queries for a document"""
    prompt = f"""Given this document, generate {n_queries} questions that this document would answer.

Document:
{doc_text}

Generate questions in JSON format:
{{"queries": ["question 1", "question 2", ...]}}
"""
    
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7
    )
    
    # Parse JSON response
    import json
    result = json.loads(response.choices[0].message.content)
    return result['queries']

# Example
doc = "Python is a programming language. Install it from python.org."
queries = generate_queries_for_doc(doc, n_queries=3)
print(queries)
# ["How to install Python?", "Where to download Python?", "What is Python?"]

---

Human Annotation

Guidelines for annotators:

# Annotation Guidelines

## Task
For each query, mark the relevant documents (on a 1-5 scale).

## Relevance scale:
- 0: Not relevant
- 1: Marginally relevant
- 2: Relevant
- 3: Highly relevant

## Examples:
Query: "How to install Python?"

Doc A: "Download Python from python.org" → 3 (highly relevant)
Doc B: "Python history: created in 1991" → 1 (marginally relevant)
Doc C: "JavaScript tutorial" → 0 (not relevant)

---

Summary

What you learned:

• ✅ Evaluation dataset format
• ✅ Synthetic data with GPT-4
• ✅ Human annotation guidelines

Optimal: Combine synthetic + human annotation.

---

In the next capsule

Capsule 06: A/B Testing Chunking Strategies

You'll learn to compare strategies systematically.

---

Module 4 - Embeddings Deep Dive Guide Evaluation datasets: the foundation for optimization