RAG Evaluation Fundamentals

Capsule description

Before choosing metrics or configuring tools, you need a clear mental framework of what you're evaluating and why. Without that framework, you'll end up measuring the things that are easy to measure instead of the things that matter, and you'll celebrate improvements that never translate into a better user experience.

A RAG system has two stages that must be evaluated separately: retrieval (which documents get recovered) and generation (which answer gets produced). If you only measure the final answer, you don't know whether the problem is that you retrieved bad documents or that the LLM ignored good ones. If you only measure retrieval, you don't know whether the answer actually addresses what the user asked.

This capsule builds that framework: what to measure at each stage, which metrics to use, when to combine them, and how to design your logging so evaluation is possible tomorrow without rebuilding anything.

By the end you'll have clarity on what kind of evaluation your case needs (not everyone needs the same) and what data you should start recording today even if you don't have the RAGAS pipeline implemented yet.

The fundamental rule: separate retrieval and generation

Imagine a user asks "what's the vacation policy for employees with less than a year?" and gets an incorrect answer. Without separating the stages, all you know is "the system failed". With a clear separation, you can diagnose it:

Every case has a completely different fix. You can't diagnose without separating them.

What to evaluate in retrieval

The retriever produces an ordered list of documents. The questions are:

• Are the documents relevant to the query?
• Do they cover the evidence needed to answer?
• Is the ranking stable across different query types (technical, search-like, conversational)?

The standard metrics:

def precision_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    top_k = retrieved_ids[:k]
    if not top_k:
        return 0.0
    return sum(1 for d in top_k if d in relevant_ids) / k

def recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    if not relevant_ids:
        return 1.0
    top_k = set(retrieved_ids[:k])
    return len(top_k & relevant_ids) / len(relevant_ids)

def mrr(retrieved_ids: list[str], relevant_ids: set[str]) -> float:
    for idx, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in relevant_ids:
            return 1.0 / idx
    return 0.0

A practical rule for your system: measure recall@20 (what goes into the re-ranker) and precision@5 (what reaches the LLM). If recall@20 is low, improve the retriever. If recall@20 is high but precision@5 is low, improve the re-ranker.

What to evaluate in generation

The generator takes the context + the query and produces an answer. The questions are:

• Is the answer supported by the context (grounding)?
• Does the answer address the original question (relevancy)?
• Is the answer factually correct (when there's ground truth)?

The standard metrics (we'll work through them thoroughly in capsule 03):

A critical point: faithfulness ≠ correctness. An answer can be faithful to the context (everything it says is in the docs) but incorrect (the docs were wrong). And it can be correct but not faithful (the LLM knew it from training, not from the context). You need both.

Manual vs automated evaluation

There's a spectrum between 100% manual evaluation (a human reads and judges) and 100% automated (an unsupervised LLM-as-judge). Each extreme has problems:

A practical rule: combine both at different frequencies:

• Every PR: 10 queries with the LLM-as-judge (a smoke test, 30 seconds)
• Every night: 100 queries with the LLM-as-judge (the full set)
• Every release: 20 queries with a human review (calibrating the judge)
• Every quarter: 50 new queries annotated by hand (refreshing the golden dataset)

Without the periodic human calibration, your LLM-judge will drift and stop catching real problems.

Designing logging so evaluation is possible

To evaluate tomorrow, you have to record today. Your pipeline should persist every query with enough detail:

from pydantic import BaseModel
from datetime import datetime

class RAGTrace(BaseModel):
    trace_id: str
    timestamp: datetime
    tenant_id: str
    query: str
    expanded_queries: list[str] | None = None  # M03 query expansion
    retrieved_docs: list[dict]                  # [{"id": ..., "score": ..., "content": ...}]
    reranked_docs: list[dict] | None = None     # M04 reranking
    final_context: list[str]                    # what actually went to the LLM
    answer: str
    latency_ms: dict                            # {"retrieval": 80, "rerank": 120, "generation": 850}
    model: str
    user_feedback: str | None = None            # thumbs up/down if you capture it

Why each field matters:

• retrieved_docs and reranked_docs kept separate: to diagnose where precision breaks (case C above)
• final_context: to reproduce exactly what the LLM saw (don't assume, record)
• latency_ms per stage: to correlate quality with performance (sometimes bad answers are truncated timeouts)
• user_feedback: free ground truth. Every thumbs-down is a candidate query for your golden dataset.

Connection with the final project

Your Advanced RAG System must explicitly separate three layers of metrics:

1. Retrieval metrics: precision@5, recall@20, MRR over the golden dataset
2. Generation metrics: faithfulness, answer relevancy, correctness with RAGAS
3. System metrics: p95 latency, cost per query, error rate

When you report the improvements in the final README, you'll be able to say things like:

"M3 query expansion improved recall@20 from 0.72 → 0.84 (+17%). M4 cross-encoder re-ranking improved precision@5 from 0.65 → 0.78 (+20%). The complete system has faithfulness 0.91 vs a simple RAG baseline's 0.74."

Without separating the layers, all you can say is "the system improved" — a useless claim.

Troubleshooting

Problem 1: "One metric summarizes everything"

The cause: excessive simplification, typically "answer correctness" as the single metric.
The fix: define at least 4 complementary metrics (2 retrieval + 2 generation). A single metric hides the trade-offs and lets you optimize one dimension at the expense of the others.

Problem 2: "I don't know whether retrieval or generation is failing"

The cause: you didn't separate the stages in the logging or in the metrics.
The fix: record retrieved_docs, final_context and answer in every trace. Compute the retrieval and generation metrics separately. When something fails, look at retrieval first (the most common root cause).

Problem 3: "Evaluation is slow and expensive"

The cause: running 1000 queries with an LLM-judge on every PR costs money and time.
The fix: structure it in layers: smoke (10 queries per PR), full (100 nightly), exhaustive (1000 weekly). Use a cheap model (gpt-4o-mini) for the judge except in critical pre-release evaluations.

Problem 4: "The metrics go up but the users complain"

The cause: the golden dataset doesn't represent the real traffic; it measures the easy cases.
The fix: sample real queries from the logs (with consent/anonymization) and turn them into the golden dataset. Refresh 20% of the dataset every quarter with queries that currently fail.

Problem 5: "The LLM-judge gives inconsistent scores"

The cause: temperature > 0 or vague prompts.
The fix: temperature=0, prompts with explicit criteria and examples. Calibrate monthly by comparing the judge against human evaluation over 20 queries; if the correlation drops below 0.7, the judge needs a re-prompt.

Exercises

Exercise 1: Define the minimum metric set for your system

For a technical-support RAG system (specific queries, ground truth available), define exactly 5 metrics with justification.

metrics_plan = {
    "retrieval": {
        "recall_at_20": "Guarantees the reranker has material; losing relevant docs here is unrecoverable",
        "precision_at_5": "What reaches the LLM; noise here damages the answers",
    },
    "generation": {
        "faithfulness": "The most common bug in technical support is hallucinating undocumented steps",
        "answer_relevancy": "Rambling answers frustrate users who are looking for an action",
        "correctness": "We have ground truth for resolved tickets; use it",
    },
}

Exercise 2: Design the RAG logging structure

Define a complete Pydantic schema to record every query the system handles, with enough fields for retrospective evaluation.

from pydantic import BaseModel
from datetime import datetime

class RetrievedDoc(BaseModel):
    doc_id: str
    score: float
    content_snippet: str
    metadata: dict

class RAGTrace(BaseModel):
    trace_id: str
    timestamp: datetime
    tenant_id: str
    query: str
    expanded_queries: list[str] = []
    retrieved_docs: list[RetrievedDoc]
    reranked_docs: list[RetrievedDoc] = []
    final_context: list[str]
    answer: str
    latency_ms: dict
    model: str
    cost_usd: float
    user_feedback: int | None = None  # 1, -1 or None

Exercise 3: A layered evaluation plan

Design an evaluation plan with different frequencies based on the speed/coverage/cost trade-off.

evaluation_plan = {
    "smoke": {
        "queries": 10,
        "frequency": "every_pr",
        "metrics": ["faithfulness", "answer_relevancy"],
        "judge_model": "gpt-4o-mini",
        "max_runtime_min": 2,
        "blocking": True,  # the PR can't merge if it drops
    },
    "full": {
        "queries": 100,
        "frequency": "nightly",
        "metrics": ["recall_at_20", "precision_at_5", "faithfulness", "relevancy", "correctness"],
        "judge_model": "gpt-4o-mini",
        "max_runtime_min": 15,
        "blocking": False,  # it alerts but doesn't block
    },
    "exhaustive": {
        "queries": 500,
        "frequency": "weekly",
        "metrics": "all",
        "judge_model": "gpt-4o",  # a stronger judge
        "max_runtime_min": 60,
        "blocking": False,
    },
    "human_calibration": {
        "queries": 20,
        "frequency": "monthly",
        "method": "human_review",
        "purpose": "validates that the LLM-judge hasn't drifted",
    },
}

Exercise 4: Diagnosis by separating the layers

A query returned an incorrect answer. Design a diagnostic script that says whether the problem was retrieval or generation.

def diagnose_failure(trace: RAGTrace, ground_truth: str, relevant_doc_ids: set[str]) -> str:
    retrieved_relevant = any(d.doc_id in relevant_doc_ids for d in trace.retrieved_docs)
    final_context_has_answer = any(
        ground_truth.lower()[:50] in ctx.lower() for ctx in trace.final_context
    )

    if not retrieved_relevant:
        return "RETRIEVAL_FAIL: no relevant document was retrieved"
    if not final_context_has_answer:
        return "RERANK_FAIL: the relevant doc was retrieved but filtered out of the final context"
    if final_context_has_answer:
        return "GENERATION_FAIL: the context contained the answer, the LLM didn't use it"
    return "UNKNOWN"

Summary

• RAG evaluation requires separating retrieval from generation; without the separation you can't diagnose anything
• Retrieval measures relevance and coverage: precision@k, recall@k, MRR, NDCG
• Generation measures grounding and usefulness: faithfulness, answer relevancy, correctness
• Faithfulness ≠ correctness: you need both, because they measure different things
• Combine automated evaluation (for volume) with manual (for monthly calibration)
• Design your logging today so that evaluation is possible tomorrow with no rework
• Structure it in layers: smoke (PR), full (nightly), exhaustive (weekly), human (monthly)

Additional resources

1. RAGAS Introduction - The framework's fundamentals.
2. Evaluating RAG Systems - Pinecone - A practical guide.
3. OpenAI Evals Guide - Designing evaluations.
4. Information Retrieval Evaluation - Wikipedia - The classic IR metrics.
5. Eugene Yan on Evals - An exhaustive synthesis.
6. LangSmith Eval Docs - Operational patterns.

Created: March 13, 2026
Version: 2.0