Core RAGAS Metrics

Capsule description

RAGAS (Retrieval Augmented Generation Assessment) is the open evaluation framework that has become the de facto standard for RAG systems. Its value isn't that it has the single best metric — it's that it offers a coherent set of metrics designed specifically for the RAG architecture, computed with an LLM-as-judge in a reproducible pipeline.

In this capsule you'll learn RAGAS's four core metrics — faithfulness, answer relevancy, context precision and context recall — understand what each one measures mechanically (how the LLM-judge arrives at the score), interpret the results with judgment (what a 0.85 vs a 0.90 means), and map each metric to a concrete action when it drops below threshold.

By the end you'll have the vocabulary and the intuition to talk about RAG quality in defensible terms, not in invented metrics or gut feelings. When a stakeholder asks "why do you say the system improved?", you'll be able to point at specific numbers with a clear interpretation.

The LLM-as-judge paradigm

The four RAGAS metrics share a common mechanism: an LLM (the "judge") evaluates the system's answer against structured criteria. That has three important implications:

1. The metrics aren't deterministic without temperature=0. Even at temperature=0, different models produce slightly different scores.
2. The judge needs to be as capable or more than the model being evaluated. Judging gpt-4o with gpt-3.5 produces noise.
3. The scores are comparative, not absolute. A faithfulness of 0.87 doesn't mean "87% correct"; it means "higher than the 0.80 you had before".

from openai import OpenAI

client = OpenAI()

def llm_judge(prompt: str, model: str = "gpt-4o-mini") -> str:
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        temperature=0,  # critical for reproducibility
        seed=42,        # improves consistency between runs
    )
    return response.choices[0].message.content

1) Faithfulness: is the answer supported by the context?

What it measures: what fraction of the claims (assertions) in the answer can be inferred from the retrieved context.

How it's computed (the internal mechanics):

1. The judge breaks the answer down into individual atomic claims
2. For each claim, the judge evaluates whether the context supports it
3. Score = (supported claims) / (total claims)

# A conceptual example of how RAGAS computes faithfulness

def faithfulness_score(answer: str, context: list[str]) -> float:
    claims = extract_claims_with_llm(answer)
    if not claims:
        return 1.0
    supported = 0
    context_text = "\n".join(context)
    for claim in claims:
        if claim_is_supported_by(claim, context_text):
            supported += 1
    return supported / len(claims)

Interpretation:

When faithfulness drops with no obvious cause:

• The context got silently truncated (the LLM's token limit)
• The prompt doesn't explicitly instruct "answer only from the context"
• The model is too small and "fills in" from its training
• The chunks are too short and lose the relevant context

2) Answer Relevancy: does the answer address the question?

What it measures: how direct, complete and focused the answer is with respect to the original query.

How it's computed:

1. The judge generates N hypothetical questions the answer could be answering
2. Each hypothetical question is compared semantically with the original query (cosine similarity of the embeddings)
3. Score = the average of the similarities

The intuition: if the answer addresses the query well, "reverse-engineered questions" from the answer should look like the original query.

def answer_relevancy_score(query: str, answer: str, n: int = 5) -> float:
    hypothetical_questions = generate_questions_from_answer_with_llm(answer, n=n)
    query_embedding = embed(query)
    similarities = [
        cosine_sim(query_embedding, embed(hq))
        for hq in hypothetical_questions
    ]
    return sum(similarities) / len(similarities)

Interpretation:

Typical causes of low relevancy:

• An ambiguous query that the LLM interpreted differently
• A prompt that asks for an "exhaustive answer" → long answers with information nobody asked for
• Context that doesn't contain the answer → the LLM rambles or abstains badly

3) Context Precision: are the relevant documents up front?

What it measures: how early in the context the genuinely useful documents for answering appear.

Why it matters: LLMs have a positional bias ("lost in the middle"). Documents at the end of the context tend to get ignored even when they're relevant. If your retriever found the right doc but put it at position 9 of 10, you'll have high faithfulness but poor answers.

How it's computed:

def context_precision_score(query: str, context_docs: list[str], ground_truth: str) -> float:
    relevant_flags = [
        is_useful_for_answering(query, doc, ground_truth)
        for doc in context_docs
    ]
    if not any(relevant_flags):
        return 0.0
    # Precision@k weighted by position
    weighted_sum = 0.0
    relevant_count = 0
    for k, is_relevant in enumerate(relevant_flags, start=1):
        if is_relevant:
            relevant_count += 1
            weighted_sum += relevant_count / k
    return weighted_sum / sum(relevant_flags)

The practical interpretation: a low context precision is a signal that you need more aggressive re-ranking, or better calibration of the existing re-ranker.

4) Context Recall: does the context contain what's needed?

What it measures: what fraction of the ground truth can be derived from the retrieved context.

How it's computed:

1. The judge breaks the ground truth down into atomic statements
2. For each statement, it evaluates whether it's covered by some document in the context
3. Score = (covered statements) / (total statements)

def context_recall_score(context: list[str], ground_truth: str) -> float:
    gt_statements = extract_statements_with_llm(ground_truth)
    if not gt_statements:
        return 1.0
    context_text = "\n".join(context)
    covered = sum(
        1 for stmt in gt_statements
        if statement_supported_by(stmt, context_text)
    )
    return covered / len(gt_statements)

Interpretation:

• High recall + low faithfulness: the context has the answer but the LLM hallucinates. A prompt or model bug.
• Low recall + high faithfulness: the LLM is being honest about incomplete context. A retrieval bug.
• Low recall + low faithfulness: a double problem. The LLM hallucinates over incomplete context. Critical.
• High recall + high faithfulness: a healthy system.

This 2x2 matrix is RAGAS's most useful diagnostic tool.

The implementation with RAGAS

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

def run_ragas_evaluation(samples: list[dict]) -> dict:
    """
    samples: a list of dicts with the keys:
        - question: str
        - answer: str (the system's answer)
        - contexts: list[str] (the retrieved documents)
        - ground_truth: str (the correct answer)
    """
    dataset = Dataset.from_list(samples)
    result = evaluate(
        dataset=dataset,
        metrics=[
            faithfulness,
            answer_relevancy,
            context_precision,
            context_recall,
        ],
    )
    return {
        "faithfulness": float(result["faithfulness"]),
        "answer_relevancy": float(result["answer_relevancy"]),
        "context_precision": float(result["context_precision"]),
        "context_recall": float(result["context_recall"]),
    }

Combined interpretation and the diagnostic matrix

Connection with the final project

Your Advanced RAG System must report these four metrics as a table in the README:

| Metric | Simple RAG baseline | Advanced RAG | Gain |
|---------|---------------------|--------------|--------|
| Faithfulness | 0.74 | 0.91 | +23% |
| Answer Relevancy | 0.81 | 0.89 | +10% |
| Context Precision | 0.62 | 0.84 | +35% |
| Context Recall | 0.71 | 0.88 | +24% |

This table is the quantitative proof that the guide's techniques are worth what it costs to implement them. Without it, everything you built is theory.

Troubleshooting

Problem 1: "High faithfulness but the answers feel bad"

The cause: the answer is faithful to the context, but the context is trivial or incomplete.
The fix: check the context recall (it's probably low). The problem is retrieval, not generation. Improve the chunking, hybrid search or expansion.

Problem 2: "High context recall but low faithfulness"

The cause: the LLM has the answer available but hallucinates anyway.
The fix: an explicit prompt: "answer only with information from the context; if it isn't there, say 'I can't find that information'". Consider a more capable model if it persists.

Problem 3: "Unstable metrics between runs"

The cause: temperature > 0, a changing dataset, a varying judge model.
The fix: fix temperature=0, seed=42, a git-versioned dataset, a fixed judge model (gpt-4o-mini in production, gpt-4o for the quarterly validation).

Problem 4: "Low context precision even though there are relevant documents"

The cause: the re-ranker isn't prioritizing well, or there's no re-ranker.
The fix: introduce or calibrate M04's cross-encoder. If there's already a re-ranker, check that it's operating over a large enough top_k from the retriever.

Problem 5: "The cost of evaluation is prohibitive"

The cause: you run every metric over 500 queries with gpt-4o.
The fix: use gpt-4o-mini as the judge (10× cheaper, similar quality for 90% of cases). Reserve gpt-4o for pre-release evaluation. Cut down to 100 queries if that's enough to catch the regressions.

Exercises

Exercise 1: Configuring the 4 core metrics

Implement a function that evaluates a dataset and returns every metric in JSON-serializable form.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset
import json

def evaluate_rag(samples: list[dict]) -> dict:
    dataset = Dataset.from_list(samples)
    result = evaluate(
        dataset=dataset,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    )
    scores = {
        "faithfulness": float(result["faithfulness"]),
        "answer_relevancy": float(result["answer_relevancy"]),
        "context_precision": float(result["context_precision"]),
        "context_recall": float(result["context_recall"]),
    }
    return scores

scores = evaluate_rag(my_samples)
print(json.dumps(scores, indent=2))

Exercise 2: Defining calibrated thresholds

Define initial thresholds and a function that evaluates whether a set of scores meets the release criteria.

THRESHOLDS = {
    "faithfulness": 0.85,
    "answer_relevancy": 0.80,
    "context_precision": 0.75,
    "context_recall": 0.80,
}

def passes_release_gate(scores: dict, thresholds: dict = THRESHOLDS) -> tuple[bool, list[str]]:
    failures = []
    for metric, min_value in thresholds.items():
        if scores.get(metric, 0) < min_value:
            failures.append(f"{metric}: {scores[metric]:.3f} < {min_value}")
    return len(failures) == 0, failures

ok, failures = passes_release_gate(scores)
if not ok:
    print("Release blocked:")
    for f in failures:
        print(f"  - {f}")

Exercise 3: An automatic diagnosis with the 2x2 matrix

Implement the diagnostic function that classifies the problem by faithfulness × recall.

def diagnose(scores: dict) -> dict:
    f = scores["faithfulness"]
    r = scores["context_recall"]

    if f >= 0.85 and r >= 0.80:
        return {"status": "healthy", "action": "monitor"}
    if f >= 0.85 and r < 0.80:
        return {
            "status": "retrieval_gap",
            "action": "improve retriever (chunking/hybrid/filters)",
            "priority": "high",
        }
    if f < 0.85 and r >= 0.80:
        return {
            "status": "hallucination",
            "action": "fix prompt to enforce grounding; consider stronger model",
            "priority": "critical",
        }
    return {
        "status": "compound_failure",
        "action": "fix retrieval first, then generation",
        "priority": "critical",
    }

print(diagnose(scores))

Exercise 4: A baseline vs advanced comparison report

Create a function that compares two sets of scores and generates a tabular report in markdown.

def compare_systems(baseline: dict, advanced: dict) -> str:
    metrics = ["faithfulness", "answer_relevancy", "context_precision", "context_recall"]
    lines = ["| Metric | Baseline | Advanced | Gain |", "|---------|----------|----------|--------|"]
    for m in metrics:
        b = baseline[m]
        a = advanced[m]
        improvement = ((a - b) / b) * 100 if b > 0 else 0
        lines.append(f"| {m.replace('_', ' ').title()} | {b:.3f} | {a:.3f} | +{improvement:.1f}% |")
    return "\n".join(lines)

print(compare_systems(baseline_scores, advanced_scores))

Summary

• The RAGAS core: faithfulness, answer relevancy, context precision, context recall
• Every metric uses an LLM-as-judge with temperature=0 and a fixed seed for reproducibility
• Faithfulness = grounding (is the answer in the context?)
• Answer Relevancy = usefulness (does it address the query?)
• Context Precision = order (is the relevant material up front?)
• Context Recall = coverage (does the context contain the answer?)
• The 2×2 matrix (faithfulness × recall) classifies problems into 4 quadrants with different actions
• Thresholds turn metrics into release quality gates

Additional resources

1. RAGAS Metrics Documentation - The formal definitions and examples.
2. RAG Evaluation Patterns - Pinecone - Practical cases.
3. Lost in the Middle - Stanford - Positional bias in LLMs.
4. Prompting Guide for Groundedness - Best practices.
5. LangSmith Evaluation - An alternative to RAGAS.

Created: March 13, 2026
Version: 2.0