Crear Evaluation Datasets para RAG

Descripción breve

Un evaluation dataset permite medir objetivamente tu sistema RAG. Sin él, solo tienes intuición. Con él, tienes datos para optimizar chunking, modelos, y retrieval.

Aprenderás formatos de eval datasets, synthetic data generation, human annotation guidelines, y quality assurance. Al final, podrás crear datasets robustos.

---

Formato de Evaluation Dataset

Estructura básica:

{
  "queries": [
    {
      "id": "q1",
      "text": "How to install Python?",
      "relevant_doc_ids": [3, 7, 12],
      "difficulty": "easy"
    }
  ],
  "corpus": [
    {"id": 3, "text": "Download Python from python.org..."},
    {"id": 7, "text": "Use package manager: apt install python3..."}
  ]
}

---

Synthetic Data Generation

Método 1: GPT-4 para generar queries

from openai import OpenAI
import os
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def generate_queries_for_doc(doc_text: str, n_queries: int = 3):
    """Generar queries sintéticas para documento"""
    prompt = f"""Given this document, generate {n_queries} questions that this document would answer.

Document:
{doc_text}

Generate questions in JSON format:
{{"queries": ["question 1", "question 2", ...]}}
"""
    
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7
    )
    
    # Parse JSON response
    import json
    result = json.loads(response.choices[0].message.content)
    return result['queries']

# Ejemplo
doc = "Python is a programming language. Install it from python.org."
queries = generate_queries_for_doc(doc, n_queries=3)
print(queries)
# ["How to install Python?", "Where to download Python?", "What is Python?"]

---

Human Annotation

Guidelines para anotadores:

# Annotation Guidelines

## Tarea
Para cada query, marca los documentos relevantes (1-5 en escala).

## Escala de relevancia:
- 0: No relevante
- 1: Marginalmente relevante
- 2: Relevante
- 3: Altamente relevante

## Ejemplos:
Query: "How to install Python?"

Doc A: "Download Python from python.org" → 3 (altamente relevante)
Doc B: "Python history: created in 1991" → 1 (marginalmente relevante)
Doc C: "JavaScript tutorial" → 0 (no relevante)

---

Resumen

Qué aprendiste:

• ✅ Formato de eval datasets
• ✅ Synthetic data con GPT-4
• ✅ Human annotation guidelines

Optimal: Combinar synthetic + human annotation.

---

En la siguiente cápsula

Cápsula 06: A/B Testing Chunking Strategies

Aprenderás a comparar estrategias sistemáticamente.

---

Módulo 4 - Embeddings Deep Dive Guide Evaluation datasets: base para optimización