The Golden Dataset and Ground Truth for RAG

Capsule description

There's an uncomfortable truth about evaluating RAG systems that most tutorials don't stress enough: the quality of your evaluation depends more on the dataset than on the framework. You can use a perfect RAGAS setup, GPT-4o as the judge, exquisitely calibrated metrics — and if your golden dataset is trivial or doesn't represent your real traffic, you'll get beautiful metrics that don't predict production quality.

A golden dataset is your system's quality contract. It's the versioned set of queries with known correct answers (ground truth) and expected relevant documents, against which you measure any change to the system. If it's badly designed, your green CI/CD is theater.

In this capsule you'll learn to build a golden dataset that actually predicts production quality: with demographic representativeness of the real traffic, a calibrated difficulty distribution, ground truth validated by two people, versioning in git, and a documented maintenance process.

By the end you'll have 50-100 golden queries that are your most valuable asset for continuous evaluation. It costs more effort than any other part of the guide. It's worth more than any other part.

The problem with purely synthetic datasets

The natural temptation is to generate the dataset with an LLM: "give me 100 questions about our domain". It works as a starting point but it has three problems that destroy its usefulness:

1. A non-representative distribution: the LLM generates "clean", well-formed questions. Your real traffic has typos, abbreviations, one-word queries, Spanglish queries, passive-aggressive queries aimed at your system.
2. Circular ground truth: if the LLM generates the question and then another LLM answers it from the docs, you're measuring how consistent the LLM is with itself, not how correct the answer is.
3. The model's biases: the LLM avoids the areas where it's weak. Your synthetic dataset under-represents exactly the queries where your system most needs evaluation.

A practical rule: at minimum 50% of your golden dataset must be sampled from real queries (with anonymization if needed). The other 50% can be synthetic to fill coverage gaps.

The anatomy of a golden dataset record

A record must capture everything needed to evaluate retrieval AND generation. The minimum structure:

from pydantic import BaseModel, Field
from typing import Literal

Difficulty = Literal["easy", "medium", "hard"]
QueryType = Literal["factoid", "reasoning", "multi_hop", "ambiguous", "out_of_scope"]

class GoldenRecord(BaseModel):
    id: str = Field(..., description="A stable ID, in the format q_001")
    query: str = Field(..., description="The query exactly as a user would write it")
    ground_truth_answer: str = Field(..., description="The validated correct answer")
    expected_sources: list[str] = Field(..., description="Doc IDs that must appear in the retrieval")
    difficulty: Difficulty
    query_type: QueryType
    category: str = Field(..., description="The domain: security, deployment, etc")
    annotated_by: str = Field(..., description="Who created this record")
    validated_by: str | None = None  # the second reviewer
    notes: str | None = None  # context for future editors

Why each field matters:

• query exactly as a user would write it: including typos, abbreviations, casual phrasing if that's what your real traffic looks like
• expected_sources lets you compute retrieval's precision@k and recall@k, not just evaluate the final answer
• difficulty enables segmented reports ("did the system improve on the hard queries?")
• query_type lets you catch specific regressions ("multi_hop dropped 15% but factoid held")
• annotated_by and validated_by enforce that the ground truth passed through two pairs of eyes
• notes captures context that gets lost with team turnover

A calibrated difficulty distribution

Not all queries are equal. A dataset that's 90% factoid (direct lookups) inflates the metrics and masks the real problems. A suggested distribution for a 50-query dataset:

Why each category:

• Factoid: the easy case; if it fails here the system is broken
• Reasoning: requires combining context; it measures chunking quality
• Multi-hop: requires multiple docs; it measures hybrid search and query expansion
• Ambiguous: it measures how well the system asks for clarification or makes a reasonable inference
• Out-of-scope: it measures whether the system knows how to abstain ("I don't have that information") instead of hallucinating

A healthy system doesn't maximize the score on factoid; it keeps a balanced score across every category.

The process for creating the dataset

Building 50 queries with real quality takes roughly 1-2 days of work. The process:

Step 1: Sample real queries (4 hours)

def sample_real_queries(logs_path: str, n: int = 100) -> list[str]:
    import random
    with open(logs_path) as f:
        all_queries = [line.strip() for line in f if line.strip()]
    # stratify by length so we don't bias toward short queries
    short = [q for q in all_queries if len(q) < 50]
    medium = [q for q in all_queries if 50 <= len(q) < 150]
    long = [q for q in all_queries if len(q) >= 150]
    sampled = (
        random.sample(short, min(40, len(short)))
        + random.sample(medium, min(40, len(medium)))
        + random.sample(long, min(20, len(long)))
    )
    return sampled[:n]

Anonymize names, sensitive IDs and personal data before versioning.

Step 2: Annotate the ground truth (8 hours)

For each query, one person writes the correct answer by consulting the source documents. Important:

• Cite the sources (expected_sources) — don't assume, verify
• Be concise but complete — the correct answer isn't an elaborate paragraph, it's the minimum that's correct
• Mark the difficulty honestly — "easy for me" can be "hard for the system"

Step 3: Cross-validation (4 hours)

A second person reviews every record and fills in validated_by. If they disagree, they discuss it. If they can't reach agreement, the record gets dropped or marked as an "edge_case" and handled separately.

Step 4: Synthetic generation for the gaps (2 hours)

If after sampling you notice you're missing queries of a certain type (e.g. out-of-scope), generate them with an LLM and review each one manually.

def generate_out_of_scope_queries(domain: str, n: int = 5) -> list[str]:
    prompt = f"""Generate {n} questions a user might ask that fall outside 
    the domain of {domain}. They should be plausible but not answerable with the product's 
    documentation. One per line, no numbering."""
    return llm_generate(prompt).split("\n")[:n]

Versioning and maintaining the dataset

GOLDEN_DATASET_META = {
    "name": "advanced-rag-golden",
    "version": "v1.2.0",
    "created_at": "2026-03-13",
    "last_updated": "2026-04-15",
    "size": 75,
    "distribution": {
        "factoid": 22,
        "reasoning": 20,
        "multi_hop": 15,
        "ambiguous": 8,
        "out_of_scope": 10,
    },
    "categories": ["security", "deployment", "api", "troubleshooting", "architecture"],
    "schema_version": "1.0",
    "annotators": ["maria@team.com", "luis@team.com"],
}

The versioning rules (semver, adapted):

• MAJOR (v2.0.0): changes that invalidate comparison with earlier versions (e.g. changing the difficulty criteria)
• MINOR (v1.x.0): adding new queries; comparable, but with care
• PATCH (v1.0.x): fixing typos in existing queries; directly comparable

Every release of the system gets evaluated against a specific version of the dataset. You document it in the CHANGELOG: "Release X.Y.Z evaluated with golden v1.2.0, scores [...]".

Quarterly maintenance:

• 20% of the queries get replaced with new ones from real traffic
• Queries the system solves perfectly no longer inform anything; rotate them out
• Queries where the system persistently fails stay in as "regression tests"

Connection with the final project

Your Advanced RAG System must include:

• golden_dataset/v1.0.0.json versioned in git
• golden_dataset/META.json with the meta structure shown above
• golden_dataset/PROCESS.md documenting how it was created and how it's maintained
• An explicit reference in CI: "the evaluation runs against golden v1.0.0"

golden_dataset/
├── v1.0.0.json           # the initial 50 queries
├── v1.1.0.json           # 60 queries (added 10 multi-hop)
├── v1.2.0.json           # current
├── META.json
├── PROCESS.md
└── CHANGELOG.md

Old versions are kept so you can reproduce historical comparisons.

Troubleshooting

Problem 1: "Perfect metrics but the system fails in production"

The cause: a trivial dataset, too much factoid, not representative of the real traffic.
The fix: sample real logs and replace 30% of the dataset. If your metrics drop, that was it. If they hold, add more hard and multi-hop queries.

Problem 2: "Inconsistent ground truth between annotators"

The cause: vague criteria, or annotators with different criteria.
The fix: document explicit criteria in PROCESS.md. For example: "ground truth is 200 words maximum", "always cite verified expected_sources". Calibrate with a joint session on 5 queries before starting.

Problem 3: "Every run gives incomparable results"

The cause: the dataset changes with no versioning.
The fix: freeze the versions in git. Any change bumps the version. CI uses an explicit version, not "latest".

Problem 4: "The ground truth goes stale when the docs change"

The cause: the source documentation changed but the ground truth wasn't updated.
The fix: a review trigger: any PR touching docs/ requires validating the golden dataset. Keep a mapping of expected_sources → last_validated_at.

Problem 5: "The dataset is too small to catch subtle regressions"

The cause: 20 queries don't have the statistical power for 2-3% differences.
The fix: grow it to 100-200 queries. For differences <2% you need >300. Beyond that, the returns diminish.

Exercises

Exercise 1: Designing a distribution for your domain

For a RAG system over Kubernetes documentation, design the golden dataset's distribution (60 queries total) with justification.

distribution = {
    "factoid": 15,        # "what is a Pod?", "kubelet's default port"
    "reasoning": 18,      # "why a Deployment vs a StatefulSet?"
    "multi_hop": 12,      # "differences between Service ClusterIP, NodePort, LoadBalancer in terms of exposure"
    "ambiguous": 8,       # "how do I scale this" (no context)
    "out_of_scope": 7,    # "how do I configure AWS RDS?" (outside the K8s docs)
}
total = sum(distribution.values())  # 60

categories = {
    "core_concepts": 15,    # Pods, Deployments, Services
    "networking": 12,
    "storage": 8,
    "security": 10,
    "operations": 15,
}

Exercise 2: A complete schema validator

Implement a function that validates a full golden dataset and reports specific issues.

def validate_dataset(records: list[dict]) -> dict:
    issues = []
    seen_ids = set()
    for r in records:
        if "id" not in r or not r["id"]:
            issues.append(f"Missing id: {r}")
            continue
        if r["id"] in seen_ids:
            issues.append(f"Duplicate id: {r['id']}")
        seen_ids.add(r["id"])
        if not r.get("validated_by"):
            issues.append(f"{r['id']}: not validated by second person")
        if not r.get("expected_sources"):
            issues.append(f"{r['id']}: missing expected_sources")
        if r.get("difficulty") not in ["easy", "medium", "hard"]:
            issues.append(f"{r['id']}: invalid difficulty {r.get('difficulty')}")
        if len(r.get("ground_truth_answer", "")) < 10:
            issues.append(f"{r['id']}: ground truth too short, suspicious")
    distribution = {}
    for r in records:
        qt = r.get("query_type", "unknown")
        distribution[qt] = distribution.get(qt, 0) + 1
    return {
        "valid": len(issues) == 0,
        "issues": issues,
        "size": len(records),
        "distribution": distribution,
    }

Exercise 3: A sampling pipeline from the logs

Implement a function that samples queries from the logs while anonymizing sensitive data.

import re
import random

def anonymize(query: str) -> str:
    # Email
    query = re.sub(r"[\w.+-]+@[\w-]+\.[\w.-]+", "<EMAIL>", query)
    # IPs
    query = re.sub(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", "<IP>", query)
    # UUIDs
    query = re.sub(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b", "<UUID>", query)
    # Long numbers (potential IDs/phones)
    query = re.sub(r"\b\d{8,}\b", "<NUM>", query)
    return query

def sample_for_golden(logs: list[str], n: int = 50, min_length: int = 10) -> list[str]:
    filtered = [q for q in logs if len(q) >= min_length]
    sampled = random.sample(filtered, min(n, len(filtered)))
    return [anonymize(q) for q in sampled]

Exercise 4: A coverage report

Implement a function that reports the gaps in the dataset (under-represented categories).

def coverage_report(records: list[dict], target_distribution: dict) -> dict:
    actual = {}
    for r in records:
        key = r.get("query_type", "unknown")
        actual[key] = actual.get(key, 0) + 1

    gaps = {}
    for query_type, target_count in target_distribution.items():
        current = actual.get(query_type, 0)
        if current < target_count:
            gaps[query_type] = {
                "current": current,
                "target": target_count,
                "missing": target_count - current,
            }
    return {
        "total_records": len(records),
        "actual_distribution": actual,
        "gaps": gaps,
        "coverage_complete": len(gaps) == 0,
    }

Summary

• The golden dataset is your evaluation system's most valuable strategic asset
• At minimum 50% must be sampled from real traffic, not 100% synthetic
• A calibrated distribution: factoid + reasoning + multi-hop + ambiguous + out-of-scope
• Every record needs expected_sources to evaluate retrieval, not just the final answer
• Cross-validation by two annotators before versioning
• Semver versioning: MAJOR breaks comparability, MINOR adds, PATCH corrects
• Quarterly maintenance: 20% of the queries get rotated with new traffic

Additional resources

1. RAGAS Test Data Generation - Complementary synthetic generation.
2. OpenAI Evals - Building Custom Evals - Dataset patterns.
3. DVC - Data Version Control - Versioning large datasets.
4. Promptfoo Test Datasets - An alternative structure.
5. Anthropic Evaluation Guide - Best practices.

Created: March 13, 2026
Version: 2.0