Module 12: LangSmith and Production

Evaluation: Datasets and Evaluators

Capsule overview

"Is the answer good?" is not evaluation. It's an opinion with no criteria, no repeatability, and no scale. Real evaluation uses specific criteria (relevance, completeness, accuracy, format), reference answers (ground truth), and automated measurement you can run every time you change a prompt, upgrade a model, or add a feature.

LangSmith gives you the whole framework: datasets with questions and expected answers, built-in evaluators for common criteria, custom evaluators in Python for your own logic, and LLM-as-judge for criteria that need reasoning. This capsule teaches you to use all of them — and to trust none of them blindly.

Why evaluation matters

Without automated evaluation, you're guessing. And guessing doesn't scale:

Scenario without evaluation:
  1. You change the agent's system prompt
  2. You try 3 queries by hand
  3. "Looks good" → deploy
  4. Users report that the answers got worse
  5. Revert → try something else → 3 queries → "looks good" → deploy
  6. An infinite loop of blind iteration

Scenario with evaluation:
  1. You change the agent's system prompt
  2. You run the evaluation dataset (50 queries with expected answers)
  3. Relevance score: 0.85 → 0.72 (down)
  4. Completeness score: 0.78 → 0.82 (up)
  5. Informed decision: the change improved completeness but hurt relevance
  6. Adjust → re-evaluate → deploy when the numbers are good

Evaluation turns agent development from a subjective art into a measurable discipline.

Building an evaluation dataset

A dataset is a collection of examples: each example has an input (the query), an expected output (the reference), and optionally metadata:

from dotenv import load_dotenv
load_dotenv()

from langsmith import Client

client = Client()

dataset_name = "research-assistant-eval"

dataset = client.create_dataset(
    dataset_name=dataset_name,
    description="Evaluation dataset for the Research Assistant. "
                "Research queries with reference answers.",
)

examples = [
    {
        "input": {"query": "What is RAG and how is it implemented?"},
        "output": {
            "reference": "RAG (Retrieval-Augmented Generation) combines a retrieval system "
                         "(which finds relevant documents) with a generative model (which produces "
                         "the answer). It's implemented with: 1) a vector store to index documents, "
                         "2) an embedding model to turn text into vectors, 3) a retriever to "
                         "find similar documents, and 4) an LLM to generate the final answer "
                         "using the retrieved documents as context."
        },
        "metadata": {"difficulty": "easy", "topic": "RAG"},
    },
    {
        "input": {"query": "Compare fine-tuning vs RAG: when to use each one"},
        "output": {
            "reference": "Fine-tuning modifies the model's weights to specialize it for a domain. "
                         "RAG adds external context without modifying the model. Use fine-tuning when: "
                         "you need to change the model's style/format, you have stable training "
                         "data, and the knowledge doesn't change often. Use RAG when: the data "
                         "changes often, you need to cite sources, and you want control over what "
                         "information the model uses. Many systems combine both."
        },
        "metadata": {"difficulty": "medium", "topic": "architecture"},
    },
    {
        "input": {"query": "How does the attention mechanism work in transformers?"},
        "output": {
            "reference": "The attention mechanism lets the model weigh the relative importance "
                         "of each token in the sequence. It works with three matrices: Query (Q), Key (K), "
                         "and Value (V). The attention score is computed as softmax(QK^T/√d_k)V, where "
                         "d_k is the dimension of the keys. Multi-head attention runs this process "
                         "multiple times in parallel with different linear projections, capturing "
                         "different kinds of relationships between tokens."
        },
        "metadata": {"difficulty": "hard", "topic": "transformers"},
    },
]

for example in examples:
    client.create_example(
        dataset_id=dataset.id,
        inputs=example["input"],
        outputs=example["output"],
        metadata=example["metadata"],
    )

print(f"Dataset created: '{dataset_name}'")
print(f"Examples: {len(examples)}")
for ex in examples:
    print(f"  - [{ex['metadata']['difficulty']}] {ex['input']['query'][:50]}...")
# Expected output:
# Dataset created: 'research-assistant-eval'
# Examples: 3
#   - [easy] What is RAG and how is it implemented?...
#   - [medium] Compare fine-tuning vs RAG: when to use each one...
#   - [hard] How does the attention mechanism work in transfo...

What makes a dataset good

Aspect	Bad	Good
Size	3 examples	30-50 examples minimum
Diversity	All the queries look alike	Covers easy/medium/hard and different topics
Reference	"Yes" / "No"	A complete answer with the key points you expect
Metadata	None	Difficulty, topic, expected_format
Upkeep	Created once and forgotten	Updated with every new kind of query

The 4 evaluation criteria

"Is it good?" is not a criterion. These are the four criteria you need:

1. Relevance: does the answer address the question?

If you ask "What is RAG?" and the answer talks about fine-tuning, it isn't relevant. It doesn't matter that it's a correct answer — it doesn't answer what was asked.

2. Completeness: does it cover every aspect?

If you ask "Compare RAG vs fine-tuning" and the answer only explains RAG without mentioning fine-tuning, it's incomplete. The answer can be relevant and correct, but it doesn't cover everything that was asked.

3. Accuracy: are the facts right?

If the answer says "GPT-4 has 100 trillion parameters" or "RAG was invented in 2023," the facts are wrong. The answer can be relevant and complete, and still be factually wrong.

4. Format: is the structure the one that was asked for?

If you asked for "a list of 5 bullets" and the answer is a narrative paragraph, the format is wrong. The answer can be relevant, complete, and correct, but not in the expected format.

An answer can be:
  ✅ Relevant + ✅ Complete + ✅ Accurate + ✅ Format    → Excellent
  ✅ Relevant + ❌ Incomplete + ✅ Accurate + ✅ Format  → Needs work
  ✅ Relevant + ✅ Complete + ❌ Inaccurate + ✅ Format  → Dangerous (looks good but has errors)
  ❌ Irrelevant + ... + ... + ...                        → Total failure

Each criterion is its own evaluator. Don't blend them.

Custom evaluators: Python functions

The simplest evaluator is a Python function that takes the agent's output and returns a score:

from dotenv import load_dotenv
load_dotenv()

from langsmith import Client
from langsmith.evaluation import evaluate

client = Client()


def relevance_evaluator(run, example) -> dict:
    """Evaluate whether the answer is relevant to the question."""
    output = run.outputs.get("response", "") if run.outputs else ""
    query = example.inputs.get("query", "")

    query_keywords = set(query.lower().split())
    response_lower = output.lower()

    stop_words = {"?", "how", "what", "is", "and", "of", "in", "the", "a", "an", "to", "does", "with", "for"}
    relevant_keywords = query_keywords - stop_words

    if not relevant_keywords:
        return {"key": "relevance", "score": 0.5}

    matches = sum(1 for kw in relevant_keywords if kw in response_lower)
    score = matches / len(relevant_keywords) if relevant_keywords else 0

    return {"key": "relevance", "score": min(score, 1.0)}


def completeness_evaluator(run, example) -> dict:
    """Evaluate whether the answer covers the points in the reference."""
    output = run.outputs.get("response", "") if run.outputs else ""
    reference = example.outputs.get("reference", "") if example.outputs else ""

    if not reference:
        return {"key": "completeness", "score": 0.5}

    ref_sentences = [s.strip() for s in reference.split(".") if len(s.strip()) > 10]
    if not ref_sentences:
        return {"key": "completeness", "score": 0.5}

    output_lower = output.lower()
    covered = 0
    for sentence in ref_sentences:
        key_words = [w for w in sentence.lower().split() if len(w) > 4]
        if key_words:
            matches = sum(1 for w in key_words if w in output_lower)
            if matches / len(key_words) > 0.3:
                covered += 1

    score = covered / len(ref_sentences) if ref_sentences else 0
    return {"key": "completeness", "score": min(score, 1.0)}


def format_evaluator(run, example) -> dict:
    """Evaluate whether the answer has a structured format."""
    output = run.outputs.get("response", "") if run.outputs else ""

    has_bullets = any(line.strip().startswith(("-", "•", "*", "1.", "2.")) for line in output.split("\n"))
    has_headers = any(line.strip().startswith("#") for line in output.split("\n"))
    has_paragraphs = len([p for p in output.split("\n\n") if p.strip()]) > 1
    min_length = len(output) > 100

    format_score = sum([has_bullets, has_headers, has_paragraphs, min_length]) / 4
    return {"key": "formatting", "score": format_score}


test_output = "RAG combines retrieval with generation. It uses vector stores to index documents."
test_query = "What is RAG?"
test_reference = "RAG (Retrieval-Augmented Generation) combines a retrieval system with a generative model."

class MockRun:
    def __init__(self, outputs):
        self.outputs = outputs

class MockExample:
    def __init__(self, inputs, outputs):
        self.inputs = inputs
        self.outputs = outputs

mock_run = MockRun({"response": test_output})
mock_example = MockExample({"query": test_query}, {"reference": test_reference})

print("=== MANUAL EVALUATION ===")
print(f"Query: {test_query}")
print(f"Output: {test_output}")
print(f"Reference: {test_reference[:60]}...")
print()

for evaluator in [relevance_evaluator, completeness_evaluator, format_evaluator]:
    result = evaluator(mock_run, mock_example)
    bar = "█" * int(result["score"] * 10)
    print(f"  {result['key']:<15} {result['score']:.2f} {bar}")
# Expected output:
# === MANUAL EVALUATION ===
# Query: What is RAG?
# Output: RAG combines retrieval with generation. It uses vector stores to index documents.
# Reference: RAG (Retrieval-Augmented Generation) combines a retrieval sy...
#
#   relevance       0.50 █████
#   completeness    0.67 ██████
#   formatting      0.25 ██

LLM-as-judge: using one model to evaluate another

For criteria that need reasoning (is the answer factually correct? is the tone appropriate?), a Python evaluator isn't enough. You need an LLM doing the evaluating:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model


def llm_relevance_judge(query: str, response: str, reference: str) -> dict:
    """Use an LLM to score relevance on a 1-5 scale."""
    judge_model = init_chat_model("openai:gpt-4.1-mini")

    judge_prompt = f"""You are an expert evaluator. Score the RELEVANCE of the answer to the question.

Question: {query}

Answer to evaluate:
{response}

Reference answer:
{reference}

Relevance criterion: Does the answer directly address what was asked? Are the topics it mentions pertinent?

Reply with a JSON object ONLY:
{{"score": <1-5>, "reasoning": "<brief explanation>"}}

Scale:
1 = Completely irrelevant
2 = Partly relevant but drifts off topic
3 = Relevant but padded with unnecessary information
4 = Very relevant
5 = Perfectly relevant"""

    result = judge_model.invoke(judge_prompt)
    content = result.content.strip()

    import json
    try:
        if "```" in content:
            content = content.split("```")[1].replace("json", "").strip()
        parsed = json.loads(content)
        return {
            "key": "llm_relevance",
            "score": parsed["score"] / 5.0,
            "comment": parsed.get("reasoning", ""),
        }
    except (json.JSONDecodeError, KeyError):
        return {"key": "llm_relevance", "score": 0.5, "comment": f"Parse error: {content[:50]}"}


def llm_accuracy_judge(query: str, response: str, reference: str) -> dict:
    """Use an LLM to score factual accuracy."""
    judge_model = init_chat_model("openai:gpt-4.1-mini")

    judge_prompt = f"""You are an expert evaluator. Score the FACTUAL ACCURACY of the answer.

Question: {query}

Answer to evaluate:
{response}

Reference answer (ground truth):
{reference}

Criterion: Are the facts it states correct? Are there any false or misleading claims?

Reply with a JSON object ONLY:
{{"score": <1-5>, "reasoning": "<explanation>", "factual_errors": ["<error 1>", "<error 2>"]}}

Scale:
1 = Multiple serious factual errors
2 = Some factual errors
3 = Mostly correct with minor imprecisions
4 = Correct with one minor imprecision
5 = Factually flawless"""

    result = judge_model.invoke(judge_prompt)
    content = result.content.strip()

    import json
    try:
        if "```" in content:
            content = content.split("```")[1].replace("json", "").strip()
        parsed = json.loads(content)
        return {
            "key": "llm_accuracy",
            "score": parsed["score"] / 5.0,
            "comment": parsed.get("reasoning", ""),
            "errors": parsed.get("factual_errors", []),
        }
    except (json.JSONDecodeError, KeyError):
        return {"key": "llm_accuracy", "score": 0.5, "comment": f"Parse error: {content[:50]}", "errors": []}


query = "What is RAG?"
response = "RAG is Retrieval-Augmented Generation. It combines document search with text generation. It was proposed by Facebook AI Research in 2020."
reference = "RAG (Retrieval-Augmented Generation) combines a retrieval system with a generative model. It was proposed by Lewis et al. in 2020."

relevance = llm_relevance_judge(query, response, reference)
accuracy = llm_accuracy_judge(query, response, reference)

print(f"Query: {query}")
print(f"Response: {response[:80]}...")
print(f"\n=== LLM-AS-JUDGE RESULTS ===")
print(f"  Relevance: {relevance['score']:.2f} — {relevance['comment']}")
print(f"  Accuracy:  {accuracy['score']:.2f} — {accuracy['comment']}")
if accuracy.get("errors"):
    print(f"  Errors: {accuracy['errors']}")
# Expected output:
# Query: What is RAG?
# Response: RAG is Retrieval-Augmented Generation. It combines document se...
#
# === LLM-AS-JUDGE RESULTS ===
#   Relevance: 1.00 — The answer directly addresses what RAG is...
#   Accuracy:  0.80 — Mostly correct, the origin is right...

Calibrating the LLM-as-judge: don't trust it blindly

An LLM judge has predictable biases. The most common one: it's too generous. It tends to hand out 4s and 5s even when the answer has problems. You need calibration:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
import json


def judge_with_calibration(query: str, response: str, reference: str) -> dict:
    """An LLM judge with calibration examples baked into the prompt."""
    model = init_chat_model("openai:gpt-4.1-mini")

    prompt = f"""You are a STRICT evaluator. Score the quality of the answer.

CALIBRATION EXAMPLES (so you calibrate your scale):

Example 1 (Score: 5/5 — Perfect):
  Q: "What is machine learning?"
  A: "Machine learning is a branch of artificial intelligence where systems learn patterns
     from data without being explicitly programmed. It uses algorithms like regression, decision trees,
     and neural networks. It's applied in image recognition, NLP, and recommendations."
  → Relevant, complete, factual, well structured.

Example 2 (Score: 2/5 — Poor):
  Q: "What is machine learning?"
  A: "It's a computer thing where they learn on their own."
  → Relevant but extremely incomplete and imprecise.

Example 3 (Score: 1/5 — Unacceptable):
  Q: "What is machine learning?"
  A: "Python is a very popular programming language."
  → Completely irrelevant.

NOW EVALUATE:
  Q: "{query}"
  A: "{response}"
  Reference: "{reference}"

Reply with JSON ONLY: {{"score": <1-5>, "reasoning": "<explanation>"}}
Be STRICT. A score of 4 or 5 requires real excellence."""

    result = model.invoke(prompt)
    content = result.content.strip()

    try:
        if "```" in content:
            content = content.split("```")[1].replace("json", "").strip()
        parsed = json.loads(content)
        return {"score": parsed["score"] / 5.0, "reasoning": parsed.get("reasoning", "")}
    except (json.JSONDecodeError, KeyError):
        return {"score": 0.5, "reasoning": f"Parse error: {content[:80]}"}


test_cases = [
    {
        "query": "What is RAG?",
        "response": "RAG is an AI technique.",
        "reference": "RAG combines retrieval with generation using vector stores and LLMs.",
        "expected": "low (incomplete)",
    },
    {
        "query": "What is RAG?",
        "response": "RAG (Retrieval-Augmented Generation) combines a retrieval system that finds "
                    "relevant documents in a vector store with an LLM that generates answers using "
                    "those documents as context. It's implemented with embeddings, vector databases "
                    "like Chroma or Pinecone, and a generative model.",
        "reference": "RAG combines retrieval with generation using vector stores and LLMs.",
        "expected": "high (complete and detailed)",
    },
    {
        "query": "What is RAG?",
        "response": "Python is an interpreted programming language.",
        "reference": "RAG combines retrieval with generation.",
        "expected": "very low (irrelevant)",
    },
]

print("=== JUDGE CALIBRATION ===\n")
for i, tc in enumerate(test_cases):
    result = judge_with_calibration(tc["query"], tc["response"], tc["reference"])
    bar = "█" * int(result["score"] * 10)
    print(f"Test {i+1} (expected: {tc['expected']}):")
    print(f"  Response: {tc['response'][:60]}...")
    print(f"  Score: {result['score']:.2f} {bar}")
    print(f"  Reasoning: {result['reasoning'][:80]}")
    print()
# Expected output:
# === JUDGE CALIBRATION ===
#
# Test 1 (expected: low (incomplete)):
#   Response: RAG is an AI technique....
#   Score: 0.40 ████
#   Reasoning: The answer is relevant but extremely incomplete...
#
# Test 2 (expected: high (complete and detailed)):
#   Response: RAG (Retrieval-Augmented Generation) combines a retrieval sys...
#   Score: 0.90 █████████
#   Reasoning: Complete, factual, and well-structured answer...
#
# Test 3 (expected: very low (irrelevant)):
#   Response: Python is an interpreted programming language....
#   Score: 0.20 ██
#   Reasoning: Completely irrelevant to the question...

Calibration rules

✅ Put calibration examples in the judge's prompt: one good (5/5), one mediocre (2-3/5), and one bad (1/5)
✅ Ask it to be strict explicitly: "A 4 or 5 requires real excellence"
✅ Include ground truth whenever you can: without a reference, the judge can't evaluate accuracy
❌ Don't rely on a single criterion: use several evaluators (relevance + completeness + accuracy)
❌ Don't assume the judge is right: validate against human-scored examples periodically

Running an evaluation with LangSmith

LangSmith wires datasets and evaluators into one complete evaluation workflow:

from dotenv import load_dotenv
load_dotenv()

from langsmith import Client
from langsmith.evaluation import evaluate
from langchain.chat_models import init_chat_model
import json

client = Client()


def research_agent(inputs: dict) -> dict:
    """Stand-in for the Research Assistant answering a query."""
    model = init_chat_model("openai:gpt-4.1-mini")
    response = model.invoke(
        f"Answer concisely and technically:\n\n{inputs['query']}"
    )
    return {"response": response.content}


def relevance_eval(run, example) -> dict:
    """Score relevance with an LLM judge."""
    output = run.outputs.get("response", "") if run.outputs else ""
    query = example.inputs.get("query", "")

    model = init_chat_model("openai:gpt-4.1-mini")
    result = model.invoke(
        f"Is the following answer relevant to the question? "
        f"Reply with a JSON object: {{\"score\": <0.0-1.0>, \"reasoning\": \"...\"}}\n\n"
        f"Question: {query}\n\nAnswer: {output}"
    )

    try:
        content = result.content.strip()
        if "```" in content:
            content = content.split("```")[1].replace("json", "").strip()
        parsed = json.loads(content)
        return {"key": "relevance", "score": float(parsed.get("score", 0.5))}
    except (json.JSONDecodeError, KeyError, ValueError):
        return {"key": "relevance", "score": 0.5}


def length_eval(run, example) -> dict:
    """Score whether the answer has a reasonable length (>50 characters)."""
    output = run.outputs.get("response", "") if run.outputs else ""
    score = 1.0 if len(output) > 50 else len(output) / 50.0
    return {"key": "adequate_length", "score": score}


def has_structure_eval(run, example) -> dict:
    """Score whether the answer has structure (bullets, paragraphs, etc.)."""
    output = run.outputs.get("response", "") if run.outputs else ""
    has_bullets = any(line.strip().startswith(("-", "•", "*", "1.")) for line in output.split("\n"))
    has_paragraphs = len([p for p in output.split("\n\n") if p.strip()]) > 1
    score = 0.5 * has_bullets + 0.5 * has_paragraphs
    return {"key": "structure", "score": score}


dataset_name = "research-assistant-eval"

try:
    existing = client.read_dataset(dataset_name=dataset_name)
except Exception:
    existing = client.create_dataset(dataset_name=dataset_name)
    examples = [
        {"input": {"query": "What is RAG?"}, "output": {"reference": "RAG combines retrieval with generation."}},
        {"input": {"query": "How does attention work?"}, "output": {"reference": "Attention uses Q, K, V matrices."}},
        {"input": {"query": "Compare fine-tuning vs RAG"}, "output": {"reference": "Fine-tuning modifies the model, RAG adds context."}},
    ]
    for ex in examples:
        client.create_example(dataset_id=existing.id, inputs=ex["input"], outputs=ex["output"])

results = evaluate(
    research_agent,
    data=dataset_name,
    evaluators=[relevance_eval, length_eval, has_structure_eval],
    experiment_prefix="research-eval-v1",
)

print(f"\n=== EVALUATION RESULTS ===")
print(f"Experiment: research-eval-v1")
print(f"Dataset: {dataset_name}")
print(f"\n→ Open LangSmith to see the detailed per-example results")
print(f"→ Each example has scores for relevance, adequate_length, and structure")
# Expected output:
# === EVALUATION RESULTS ===
# Experiment: research-eval-v1
# Dataset: research-assistant-eval
#
# → Open LangSmith to see the detailed per-example results
# → Each example has scores for relevance, adequate_length, and structure

Reading the results: distributions and patterns

Individual scores matter, but patterns matter more:

from dotenv import load_dotenv
load_dotenv()

import random

random.seed(42)
results = []
for i in range(20):
    results.append({
        "query": f"Query {i+1}",
        "relevance": random.uniform(0.6, 1.0),
        "completeness": random.uniform(0.3, 0.9),
        "accuracy": random.uniform(0.7, 1.0),
        "formatting": random.uniform(0.2, 1.0),
    })

print("=== SCORE DISTRIBUTION ===\n")

for metric in ["relevance", "completeness", "accuracy", "formatting"]:
    scores = [r[metric] for r in results]
    avg = sum(scores) / len(scores)
    min_s = min(scores)
    max_s = max(scores)
    low_count = sum(1 for s in scores if s < 0.5)

    bar = "█" * int(avg * 20)
    print(f"  {metric:<15} avg={avg:.2f} min={min_s:.2f} max={max_s:.2f} low(<0.5)={low_count} {bar}")

failures = [r for r in results if r["relevance"] < 0.5 or r["accuracy"] < 0.5]
print(f"\n=== PATTERN ANALYSIS ===")
print(f"  Total examples: {len(results)}")
print(f"  Failures (relevance<0.5 OR accuracy<0.5): {len(failures)}")

if failures:
    print(f"  Failed queries:")
    for f in failures:
        print(f"    - {f['query']}: relevance={f['relevance']:.2f}, accuracy={f['accuracy']:.2f}")
else:
    print(f"  ✅ No critical failures detected")

weakest = min(
    ["relevance", "completeness", "accuracy", "formatting"],
    key=lambda m: sum(r[m] for r in results) / len(results)
)
print(f"\n  Weakest criterion: {weakest}")
print(f"  → Focus optimization efforts here")
# Expected output:
# === SCORE DISTRIBUTION ===
#
#   relevance       avg=0.82 min=0.62 max=0.99 low(<0.5)=0 ████████████████
#   completeness    avg=0.58 min=0.31 max=0.88 low(<0.5)=6 ███████████
#   accuracy        avg=0.87 min=0.71 max=1.00 low(<0.5)=0 █████████████████
#   formatting      avg=0.59 min=0.22 max=0.98 low(<0.5)=5 ███████████
#
# === PATTERN ANALYSIS ===
#   Total examples: 20
#   Failures (relevance<0.5 OR accuracy<0.5): 0
#   ✅ No critical failures detected
#
#   Weakest criterion: completeness
#   → Focus optimization efforts here

What to look for in the results

✅ Average score per criterion: which criterion is the weakest? Focus your optimization there
✅ Distribution: an average of 0.7 can mean "everything around 0.7" (consistent) or "half at 0.9, half at 0.5" (inconsistent)
✅ Failures by query type: do the hard queries fail more? Does a specific topic score low?
✅ Changes between versions: the same evaluation before and after a prompt change

Evaluation as a continuous practice

Evaluation isn't a final step. It's part of the development workflow:

Development cycle with evaluation:

1. You change something (prompt, model, tool, logic)
     ↓
2. You run the evaluation dataset
     ↓
3. You compare scores against the previous version
     ↓
4. Did it improve? → Deploy
   Did it get worse? → Revert and iterate
   Mixed? (better at X, worse at Y) → Informed decision
     ↓
5. Repeat

Every experiment in LangSmith has a name (experiment_prefix). Name them after the change so you can compare:

research-eval-v7-baseline
research-eval-v7-new-prompt
research-eval-v7-gpt4.1-mini
research-eval-v7-with-reranking

Troubleshooting

Problem 1: "The LLM judge gives everything a high score"

Symptom: Every example scores 4-5/5 even when the answers are mediocre.

Cause: LLM judges have a natural generosity bias.

Fix: Put calibration examples in the judge's prompt (one good, one mediocre, one bad). Ask it explicitly to be strict. Add the instruction "A score of 5 is rare and requires exceptional excellence."

Problem 2: "My custom evaluator's scores aren't consistent"

Symptom: The same answer gets different scores on different runs.

Cause: The LLM judge isn't deterministic. With temperature > 0, the answers vary.

Fix: Use temperature=0 for the judge model. If there's still variation, run the judge 3 times and take the median.

Problem 3: "The dataset is too small to mean anything"

Symptom: 5 examples don't give you confidence in the scores.

Cause: Results from small samples are noisy.

Fix: Aim for 30-50 examples minimum. Cover different difficulty levels, topics, and expected formats. Add examples every time you find a new kind of query in production.

Problem 4: "I don't know how to write reference answers"

Symptom: You have the questions but not the expected answers.

Cause: Creating ground truth is manual work and takes expertise.

Fix: Start with answers generated by a strong model (GPT-4.1) and review them by hand. They don't have to be perfect — they have to capture the key points you expect in a good answer.

Problem 5: "evaluate() takes forever"

Symptom: Evaluating 50 examples takes 10+ minutes.

Cause: Each example runs the agent plus the evaluators. With LLM-as-judge, that's several LLM calls per example.

Fix: Use gpt-4.1-mini for the judge (faster and cheaper). LangSmith runs evaluations in parallel by default. For very large datasets, consider evaluating a representative subset.

Exercises

Exercise 1: Build an evaluation dataset (Easy)

Create a dataset in LangSmith with 5 examples on AI topics. Each example should have: a query, a reference answer, and metadata (difficulty, topic). Confirm the dataset shows up in the dashboard.

See solution

from dotenv import load_dotenv
load_dotenv()

from langsmith import Client

client = Client()

dataset = client.create_dataset(
    dataset_name="ai-basics-eval",
    description="Evaluation of basic AI knowledge",
)

examples = [
    {"input": {"query": "What is a transformer?"}, "output": {"reference": "A transformer is a neural network architecture based on the self-attention mechanism, proposed in 2017 by Vaswani et al."}, "metadata": {"difficulty": "easy", "topic": "architecture"}},
    {"input": {"query": "What is fine-tuning?"}, "output": {"reference": "Fine-tuning is the process of re-training a pre-trained model on a specific dataset to adapt it to a particular task."}, "metadata": {"difficulty": "easy", "topic": "training"}},
    {"input": {"query": "Explain the bias-variance tradeoff"}, "output": {"reference": "The bias-variance tradeoff describes the tension between simple models (high bias, low variance) and complex ones (low bias, high variance). The goal is to find the sweet spot."}, "metadata": {"difficulty": "medium", "topic": "ML-theory"}},
    {"input": {"query": "How does RLHF work?"}, "output": {"reference": "RLHF (Reinforcement Learning from Human Feedback) trains a reward model on human preferences, then uses PPO to optimize the LLM against that reward model."}, "metadata": {"difficulty": "hard", "topic": "training"}},
    {"input": {"query": "Compare GPT-4 vs Claude on capabilities"}, "output": {"reference": "GPT-4 and Claude are competing LLMs. GPT-4 stands out at reasoning and coding. Claude stands out at long context and instruction following. Both support multimodal input."}, "metadata": {"difficulty": "medium", "topic": "models"}},
]

for ex in examples:
    client.create_example(
        dataset_id=dataset.id,
        inputs=ex["input"],
        outputs=ex["output"],
        metadata=ex["metadata"],
    )

print(f"Dataset created: 'ai-basics-eval'")
print(f"Examples: {len(examples)}")
for ex in examples:
    print(f"  [{ex['metadata']['difficulty']:>6}] {ex['input']['query']}")
# Expected output:
# Dataset created: 'ai-basics-eval'
# Examples: 5
#   [  easy] What is a transformer?
#   [  easy] What is fine-tuning?
#   [medium] Explain the bias-variance tradeoff
#   [  hard] How does RLHF work?
#   [medium] Compare GPT-4 vs Claude on capabilities

Exercise 2: Build a length evaluator (Easy)

Write an evaluator that checks whether the answer is between 50 and 500 characters. Score 1.0 if it's in range, 0.5 if it's slightly outside, 0.0 if it's far too short or far too long. Try it on 3 answers of different lengths.

See solution

from dotenv import load_dotenv
load_dotenv()


def length_range_evaluator(response: str, min_len: int = 50, max_len: int = 500) -> dict:
    """Evaluate whether the answer falls in an acceptable length range."""
    length = len(response)

    if min_len <= length <= max_len:
        score = 1.0
        comment = f"Perfect length: {length} chars"
    elif length < min_len:
        score = max(0, length / min_len)
        comment = f"Too short: {length} chars (minimum: {min_len})"
    else:
        overshoot = (length - max_len) / max_len
        score = max(0, 1.0 - overshoot)
        comment = f"Too long: {length} chars (maximum: {max_len})"

    return {"key": "length_range", "score": round(score, 2), "comment": comment}


test_responses = [
    "AI is great.",
    "Machine learning is a branch of artificial intelligence that lets systems learn from data. It's used in image recognition, NLP, and recommendations.",
    "A" * 800,
]

print("=== LENGTH EVALUATOR TEST ===\n")
for i, response in enumerate(test_responses):
    result = length_range_evaluator(response)
    bar = "█" * int(result["score"] * 10)
    print(f"Test {i+1}: '{response[:50]}{'...' if len(response) > 50 else ''}'")
    print(f"  Score: {result['score']:.2f} {bar}")
    print(f"  {result['comment']}")
    print()
# Expected output:
# === LENGTH EVALUATOR TEST ===
#
# Test 1: 'AI is great.'
#   Score: 0.28 ██
#   Too short: 14 chars (minimum: 50)
#
# Test 2: 'Machine learning is a branch of artificial intellig...'
#   Score: 1.00 ██████████
#   Perfect length: 165 chars
#
# Test 3: 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA...'
#   Score: 0.40 ████
#   Too long: 800 chars (maximum: 500)

Exercise 3: LLM-as-judge with calibration (Medium)

Build an LLM judge that scores completeness. Put 3 calibration examples in the prompt (complete, partial, incomplete). Try it on 3 answers of different completeness and check that the scores reflect real quality.

See solution

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
import json


def completeness_judge(query: str, response: str, reference: str) -> dict:
    """A calibrated LLM judge for completeness."""
    model = init_chat_model("openai:gpt-4.1-mini", temperature=0)

    prompt = f"""You are a STRICT evaluator of completeness.

CALIBRATION:

Example 1 (5/5 — Complete):
  Q: "What is ML?"
  Reference: "ML is a branch of AI that learns from data. It includes supervised, unsupervised, and reinforcement learning."
  A: "ML is a branch of artificial intelligence. The systems learn patterns from data.
     Three main types: supervised learning (with labels), unsupervised (without labels),
     and reinforcement learning (reward signals)."
  → Covers every point in the reference.

Example 2 (3/5 — Partial):
  Q: "What is ML?"
  Reference: "ML is a branch of AI that learns from data. It includes supervised, unsupervised, and reinforcement learning."
  A: "ML is artificial intelligence that learns from data. The most common type is supervised."
  → Covers the definition but only one of the three types.

Example 3 (1/5 — Incomplete):
  Q: "What is ML?"
  Reference: "ML is a branch of AI that learns from data. It includes supervised, unsupervised, and reinforcement learning."
  A: "It's a computing topic."
  → Covers none of the specific points in the reference.

EVALUATE:
  Q: "{query}"
  Reference: "{reference}"
  A: "{response}"

Reply with JSON ONLY: {{"score": <1-5>, "reasoning": "<explanation>", "covered": ["<covered point>"], "missing": ["<missing point>"]}}"""

    result = model.invoke(prompt)
    content = result.content.strip()

    try:
        if "```" in content:
            content = content.split("```")[1].replace("json", "").strip()
        parsed = json.loads(content)
        return {
            "score": parsed["score"] / 5.0,
            "reasoning": parsed.get("reasoning", ""),
            "covered": parsed.get("covered", []),
            "missing": parsed.get("missing", []),
        }
    except (json.JSONDecodeError, KeyError):
        return {"score": 0.5, "reasoning": f"Parse error", "covered": [], "missing": []}


query = "What is RAG and how is it implemented?"
reference = "RAG combines retrieval with generation. It's implemented with vector stores, embeddings, a retriever, and an LLM."

test_responses = [
    ("RAG (Retrieval-Augmented Generation) combines a retriever that finds documents in a vector store "
     "with an LLM that generates answers. It's implemented with embeddings for indexing, a vector database "
     "like Chroma, a retriever to search, and a model like GPT-4 to generate.", "complete"),
    ("RAG is a technique that uses search to improve an LLM's answers.", "partial"),
    ("It's an AI acronym.", "incomplete"),
]

print("=== COMPLETENESS JUDGE (calibrated) ===\n")
for response, expected in test_responses:
    result = completeness_judge(query, response, reference)
    bar = "█" * int(result["score"] * 10)
    print(f"[Expected: {expected}]")
    print(f"  Response: {response[:70]}...")
    print(f"  Score: {result['score']:.2f} {bar}")
    print(f"  Reasoning: {result['reasoning'][:80]}")
    if result["missing"]:
        print(f"  Missing: {result['missing']}")
    print()
# Expected output:
# === COMPLETENESS JUDGE (calibrated) ===
#
# [Expected: complete]
#   Response: RAG (Retrieval-Augmented Generation) combines a retriever that finds do...
#   Score: 1.00 ██████████
#   Reasoning: Covers every point: retrieval + generation, vector stores, embeddings...
#
# [Expected: partial]
#   Response: RAG is a technique that uses search to improve an LLM's answers....
#   Score: 0.50 █████
#   Reasoning: Covers the general idea but says nothing about implementation...
#   Missing: ['vector stores', 'embeddings', 'retriever']
#
# [Expected: incomplete]
#   Response: It's an AI acronym....
#   Score: 0.20 ██
#   Reasoning: Covers none of the points in the reference...

Exercise 4: Run a full evaluation with evaluate() (Medium)

Create a dataset of 3 examples, implement 2 evaluators (one Python, one LLM-as-judge), and run LangSmith's evaluate(). Print the results and the link to the experiment in the dashboard.

See solution

from dotenv import load_dotenv
load_dotenv()

from langsmith import Client
from langsmith.evaluation import evaluate
from langchain.chat_models import init_chat_model
import json

client = Client()

dataset_name = "eval-exercise-4"
try:
    ds = client.read_dataset(dataset_name=dataset_name)
except Exception:
    ds = client.create_dataset(dataset_name=dataset_name)
    examples = [
        {"input": {"query": "What is an LLM?"}, "output": {"reference": "An LLM is a large language model trained on massive amounts of text to generate and understand natural language."}},
        {"input": {"query": "What are embeddings?"}, "output": {"reference": "Embeddings are numeric representations of text in a vector space where similar texts sit close together."}},
        {"input": {"query": "What is prompt engineering?"}, "output": {"reference": "Prompt engineering is the design of instructions to get better answers out of an LLM."}},
    ]
    for ex in examples:
        client.create_example(dataset_id=ds.id, inputs=ex["input"], outputs=ex["output"])


def my_agent(inputs: dict) -> dict:
    model = init_chat_model("openai:gpt-4.1-mini")
    response = model.invoke(f"Answer in 2-3 sentences:\n\n{inputs['query']}")
    return {"response": response.content}


def python_length_eval(run, example) -> dict:
    output = run.outputs.get("response", "") if run.outputs else ""
    score = 1.0 if 50 < len(output) < 500 else 0.5
    return {"key": "length_ok", "score": score}


def llm_quality_eval(run, example) -> dict:
    output = run.outputs.get("response", "") if run.outputs else ""
    query = example.inputs.get("query", "")
    reference = example.outputs.get("reference", "") if example.outputs else ""

    model = init_chat_model("openai:gpt-4.1-mini", temperature=0)
    result = model.invoke(
        f"Score the quality from 0.0 to 1.0. Reply with JSON ONLY: {{\"score\": <float>}}\n\n"
        f"Question: {query}\nReference: {reference}\nAnswer: {output}"
    )
    try:
        content = result.content.strip()
        if "```" in content:
            content = content.split("```")[1].replace("json", "").strip()
        parsed = json.loads(content)
        return {"key": "llm_quality", "score": float(parsed.get("score", 0.5))}
    except (json.JSONDecodeError, KeyError, ValueError):
        return {"key": "llm_quality", "score": 0.5}


results = evaluate(
    my_agent,
    data=dataset_name,
    evaluators=[python_length_eval, llm_quality_eval],
    experiment_prefix="eval-exercise-4-v1",
)

print(f"✅ Evaluation complete")
print(f"Dataset: {dataset_name}")
print(f"Experiment: eval-exercise-4-v1")
print(f"\n→ Open LangSmith → Datasets → '{dataset_name}' → view results")
# Expected output:
# ✅ Evaluation complete
# Dataset: eval-exercise-4
# Experiment: eval-exercise-4-v1
#
# → Open LangSmith → Datasets → 'eval-exercise-4' → view results

Exercise 5: Detect judge bias (Advanced)

Build a bias test: send the LLM judge an answer that's clearly mediocre but beautifully written, and an answer that's correct but badly formatted. Compare the scores. Does the judge punish formatting or accuracy more? Write down what you find.

See solution

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
import json


def biased_judge_test(query: str, response: str, reference: str, label: str) -> dict:
    model = init_chat_model("openai:gpt-4.1-mini", temperature=0)

    prompt = f"""Score the overall quality of this answer from 1 to 5.

Question: {query}
Reference: {reference}
Answer: {response}

Reply with JSON: {{"score": <1-5>, "reasoning": "<explanation>"}}"""

    result = model.invoke(prompt)
    content = result.content.strip()

    try:
        if "```" in content:
            content = content.split("```")[1].replace("json", "").strip()
        parsed = json.loads(content)
        return {
            "label": label,
            "score": parsed["score"],
            "reasoning": parsed.get("reasoning", ""),
        }
    except (json.JSONDecodeError, KeyError):
        return {"label": label, "score": 0, "reasoning": "parse error"}


query = "What is machine learning?"
reference = "Machine learning is a branch of AI where systems learn from data. It includes supervised, unsupervised, and reinforcement learning."

test_cases = [
    {
        "response": "Machine learning, commonly abbreviated as ML, represents a fascinating "
                    "intersection between computational statistics and artificial intelligence. "
                    "It is a field that has revolutionized multiple industries and continues to evolve "
                    "at an impressive pace in the modern technological era.",
        "label": "well-written-but-vague",
        "description": "Beautifully written but vague (no types mentioned, not technical)",
    },
    {
        "response": "ml = AI subsection. learns from data. types: supervised (labels), "
                    "unsupervised (no labels), reinforcement (rewards). uses: image recognition, "
                    "nlp, recommendations. key algorithms: linear regression, decision trees, neural nets.",
        "label": "ugly-but-accurate",
        "description": "Badly formatted but technically complete and correct",
    },
    {
        "response": "Machine learning is a branch of artificial intelligence that lets systems "
                    "learn patterns from data without explicit programming. The three main types "
                    "are: supervised, unsupervised, and reinforcement learning.",
        "label": "balanced-good",
        "description": "Well written AND correct (control)",
    },
]

print("=== JUDGE BIAS TEST ===\n")
results = []
for tc in test_cases:
    result = biased_judge_test(query, tc["response"], reference, tc["label"])
    results.append(result)
    print(f"[{tc['label']}] — {tc['description']}")
    print(f"  Score: {result['score']}/5")
    print(f"  Reasoning: {result['reasoning'][:80]}")
    print()

print("=== BIAS ANALYSIS ===")
vague = next(r for r in results if r["label"] == "well-written-but-vague")
accurate = next(r for r in results if r["label"] == "ugly-but-accurate")
control = next(r for r in results if r["label"] == "balanced-good")

print(f"  Well-written but vague: {vague['score']}/5")
print(f"  Ugly but accurate:     {accurate['score']}/5")
print(f"  Balanced (control):    {control['score']}/5")

if vague["score"] >= accurate["score"]:
    print(f"\n  ⚠️  BIAS DETECTED: The judge favors polished prose over technical accuracy.")
    print(f"  → The vague answer ({vague['score']}/5) scored >= the precise one ({accurate['score']}/5)")
    print(f"  → Fix: add explicit accuracy criteria to the judge's prompt")
else:
    print(f"\n  ✅ The judge seems to prioritize accuracy over style.")
# Expected output:
# === JUDGE BIAS TEST ===
#
# [well-written-but-vague] — Beautifully written but vague
#   Score: 3/5
#   Reasoning: Well-written but lacks specific technical details...
#
# [ugly-but-accurate] — Badly formatted but correct
#   Score: 4/5
#   Reasoning: Covers all key points accurately despite poor formatting...
#
# [balanced-good] — Control
#   Score: 5/5
#   Reasoning: Complete, accurate, and well-structured...
#
# === BIAS ANALYSIS ===
#   Well-written but vague: 3/5
#   Ugly but accurate:     4/5
#   Balanced (control):    5/5
#
#   ✅ The judge seems to prioritize accuracy over style.

Summary

In this capsule you learned:

"Is it good?" is not evaluation. Real evaluation uses specific criteria: relevance (does it address the question?), completeness (does it cover every aspect?), accuracy (are the facts right?), and format (is the structure the one that was asked for?). Each criterion is its own evaluator
Evaluation datasets are collections of examples with a query, a reference answer, and metadata. A good dataset has 30-50 examples, covers different difficulties and topics, and gets updated continuously
Custom evaluators in Python are functions that take the agent's output and return a score. They're fast, deterministic, and perfect for simple criteria (length, format, keywords)
LLM-as-judge uses one model to evaluate another. It's powerful for criteria that need reasoning (relevance, accuracy), but it has biases: it tends to be too generous, it can miss factual errors, and it isn't deterministic
Calibrating the judge is mandatory: put calibration examples in the prompt (good, mediocre, bad), tell it to be strict, and validate against human-scored examples periodically
LangSmith's evaluate() runs a whole dataset through several evaluators and produces an experiment with detailed results. Name your experiments after the change so you can compare them
Evaluation is a continuous practice: run it with every prompt change, every model upgrade, every new feature. The numbers tell you whether you improved or regressed — no more guessing

Next capsule: Token Tracking and Cost Control — how to know exactly what each operation of your agent costs, set up per-user rate limiting, and make business decisions based on real cost.

Additional resources

LangSmith — Evaluation Quickstart — Quickstart for building datasets and running evaluations
LangSmith — How to create and manage datasets — Managing datasets with the SDK
LangSmith — Custom Evaluators — How to build custom evaluators in Python
LangSmith — LLM-as-Judge — Guide to implementing LLM-as-judge with calibration
LangSmith — Compare experiments — Comparing results across versions
Judging LLM-as-Judge — Research Paper — Paper on the biases and limits of LLM-as-judge (MT-Bench)

Module 12 — LangChain & LangGraph: From Chains to Agents

← PreviousVisual Agent Debugging Next →Token Usage Tracking and Prompt Caching