1. Introduction: If You Don't Measure, You Don't Improve

Overview

The problem: "looks good" is not a metric. Most prompt engineers evaluate on vibes. Evaluation as an engineering discipline: objective metrics, benchmarks, regression testing, A/B testing.

The Problem: Evaluation by Intuition

When you build an LLM system, you probably do something like this:

1. You write a prompt
2. You test 3-5 cases by hand
3. "Looks good" → you deploy
4. Weeks later: "Hey, why is this failing?"

This is the standard workflow in most LLM projects. And it's a huge problem.

Why "Looks Good" Doesn't Work

Problem 1: Confirmation bias
→ You test the cases you expect to work
→ You don't test the hard or unusual ones
→ Your sample of 5 examples isn't representative

Problem 2: You have no baseline
→ You change the prompt next week
→ Did it improve? Did it get worse? You have no data to tell

Problem 3: No regression detection
→ You fix a bug in the prompt
→ Without noticing, you break something that used to work
→ Your users find out before you do

Problem 4: Failure doesn't scale linearly
→ 5 cases = "it works"
→ 1000 real cases = 8% fail silently
→ In production, that 8% is unhappy users

Why Evaluate

Systematic prompt evaluation isn't academic bureaucracy. It's the difference between an LLM system that works in production and one that fails in invisible ways.

Concrete Benefits

• Reproducibility: Knowing for sure whether a change improves or degrades the system
• Regression: Automatically catching a prompt that stopped working
• Objective comparison: Zero-shot vs few-shot, prompt A vs B, model X vs model Y
• Confidence at deploy: Not shipping without knowing the system passes its metrics
• Informed debugging: When something fails, knowing exactly what failed and in what percentage

A Real Example: What Metrics Buy You

Scenario: Support ticket classification system

Without metrics:
- "The classifier seems to work"
- Deploy to production
- 3 weeks later: a customer reports that urgent tickets are being categorized as "low priority"
- You dig in: the problem was there from day 1 for tickets with certain patterns

With metrics:
- Golden set of 200 tickets with the correct categories
- Baseline accuracy: 94%
- After changing the prompt: accuracy drops to 87%
- Regression caught BEFORE the deploy
- You fix it before it reaches real users

Evaluation vs Vibes

The fundamental difference between systematic evaluation and evaluation by intuition:

The Three Layers of Evaluation

A complete evaluation framework has three layers:

Layer 1: Automated Metrics

Computed in code, with no human in the loop:

• Accuracy: Prediction == ground truth
• Format compliance: Does the output have the expected format (JSON, etc.)?
• BLEU/ROUGE: For text generation, compare against a reference

Layer 2: LLM-as-Judge

One LLM evaluates another LLM's output:

• Faithfulness: Is the output faithful to the input? Does it make things up?
• Relevance: Does the output answer the question?
• Quality: Multi-criteria rubric (correctness, clarity, completeness)

Layer 3: Human Evaluation

For critical decisions and calibration:

• Validating golden sets
• Reviewing samples from the LLM-as-judge
• Final deploy decisions

Layer 1 (Automated)   ← Fast, cheap, frequent (on every commit)
Layer 2 (LLM-as-Judge) ← Pricier, less frequent (on every PR or daily)
Layer 3 (Human)       ← Expensive, periodic (weekly/monthly, or at milestones)

Types of Evaluation

Offline Evaluation

You evaluate the prompt against a fixed dataset before deploying.

from openai import OpenAI

client = OpenAI()

def evaluate_offline(prompt_template: str, golden_set: list[dict]) -> dict:
    """
    Offline evaluation: before the deploy.
    Compares the prompt's outputs against the golden set's expected_outputs.
    """
    results = []

    for example in golden_set:
        # Build the full message
        prompt = prompt_template.format(input=example["input"])

        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=0  # Determinism for reproducibility
        )

        output = response.choices[0].message.content
        expected = example["expected_output"]

        results.append({
            "input": example["input"],
            "output": output,
            "expected": expected,
            "correct": output.strip().lower() == expected.strip().lower()
        })

    accuracy = sum(r["correct"] for r in results) / len(results)
    return {"accuracy": accuracy, "results": results}

Online Evaluation

You evaluate the prompt in production with real users:

• Sampling of real traffic
• A/B testing with users
• Implicit feedback (clicks, corrections, abandonment)

Offline → Before the deploy (always)
Online  → After the deploy (for continuous refinement)

When to Evaluate

Evaluation has to be continuous, not a one-off event:

DEVELOPMENT:
│
├── When you create a new prompt → Evaluate against the golden set
│
├── When you modify a prompt → Regression test vs baseline
│
├── When you change the model (gpt-4o-mini → gpt-4o) → Compare metrics
│
├── Periodically (weekly/monthly) → Catch gradual drift
│
└── Before every deploy → Full checklist

PRODUCTION:
│
├── Daily → Sample of real requests with LLM-as-judge
│
└── When you get negative feedback → Investigate with metrics

Key Terminology for This Module

Before moving on, make sure you understand these terms:

Module 7 Roadmap

Tools in the Ecosystem

For context, these are the existing tools in the LLM evaluation ecosystem:

In this module we'll build our own framework to understand the fundamentals. That lets you pick up any tool in the ecosystem later with a deep understanding of what it's doing.

The Cost of Not Evaluating

One last reflection before we go deeper:

LLM system in production without evaluation:
→ Takes weeks or months to notice quality degradation
→ Can't prove an improvement is real (or that it isn't a regression)
→ Every prompt change is a leap of faith
→ Users find the bugs before the team does
→ Impossible to answer: "how well does this actually work?"

LLM system with evaluation:
→ Catches regressions in minutes (in CI/CD)
→ Can measure and prove improvements objectively
→ Every deploy ships with data, not hope
→ Real-time dashboard of production quality
→ Can answer: "Accuracy 94%, faithfulness 0.91, p95 < 2.1s"

Systematic evaluation is what separates production LLM projects from prototypes. This module gives you the tools to build it.

Exercises

Exercise 1: Diagnose your current system

Think about the last LLM system you built or worked on. Answer:

1. Did you have a golden set of examples?
2. Did you know the system's accuracy?
3. Could you catch regressions automatically?

Exercise 2: Your first minimum viable golden set

Create a golden set of 10 examples for a basic sentiment classifier:

# Your task: create this structure
golden_set = [
    # 7 happy path (clearly positive/negative phrases)
    # 2 edge cases (neutral, ambiguous)
    # 1 adversarial (trying to confuse the classifier)
]

golden_set = [
    # Happy path - positive
    {"input": "I love this product, it exceeded my expectations", "expected_output": "POSITIVE"},
    {"input": "Excellent customer service, highly recommended", "expected_output": "POSITIVE"},
    {"input": "The quality is incredible for the price", "expected_output": "POSITIVE"},
    {"input": "It arrived fast and in perfect condition", "expected_output": "POSITIVE"},

    # Happy path - negative
    {"input": "Terrible experience, I don't recommend it", "expected_output": "NEGATIVE"},
    {"input": "It broke the first time I used it, very disappointing", "expected_output": "NEGATIVE"},
    {"input": "The service was awful and the wait was horrible", "expected_output": "NEGATIVE"},

    # Edge cases
    {"input": "The product arrived, I used it once", "expected_output": "NEUTRAL"},
    {"input": "It's not bad, but it isn't the best either", "expected_output": "NEUTRAL"},

    # Adversarial
    {"input": "It's not bad, but it has serious quality problems", "expected_output": "NEGATIVE"},
]

Exercise 3: Compute the cost of not measuring

If you have a system that processes 10,000 requests/day and 8% fail silently:

1. How many users are affected per day?
2. If each user is worth $10/month in LTV, what is it worth to catch this problem in 1 day vs 30 days?

requests_per_day = 10_000
failure_rate = 0.08
affected_users_per_day = requests_per_day * failure_rate  # 800

user_ltv = 10  # $10/month ≈ $0.33/day

# Cost of catching it late:
# 30 days × 800 users × $0.33/day = $7,920 in eroded LTV
# Not counting reputation damage, churn, etc.

cost_30_days = 30 * affected_users_per_day * (user_ltv / 30)
print(f"Affected users: {affected_users_per_day}/day")
print(f"Cost of not catching it for 30 days: ${cost_30_days:,.0f}")
# → $7,920 — and that's the conservative estimate

Summary

• Evaluation: Objective metrics, not vibes — the difference between prototypes and production systems
• Three layers: Automated (fast), LLM-as-judge (deep), human (quality)
• Golden sets: Inputs + expected outputs = the foundation of all evaluation
• Regression: Automatically catching things that break
• A/B testing: Comparing versions with statistical data, not intuition
• Timing: Evaluate always: on creation, on modification, before deploy, in production

Module Prerequisites

Before continuing, make sure you're comfortable with:

• Basic Python: Functions, dictionaries, lists, f-strings
• OpenAI SDK: from openai import OpenAI, client.chat.completions.create()
• JSON parsing: json.loads(), json.dumps() to handle structured outputs
• Pydantic (recommended): To validate golden set and result schemas
• Basic statistics: Mean, percentage, standard deviation — you don't need to be a statistician, but you should be able to compute accuracy and understand what statistical significance means

If you're coming from Module 06, you already have everything you need. The pipelines you built there are exactly what you'll be evaluating here.

Frequently Asked Questions

Do I need a huge golden set to get started? No. 50-100 well-chosen examples are enough to catch meaningful regressions. What matters is that they cover happy paths, edge cases and adversarial examples. Capsule 04 teaches you how to build them systematically.

Is LLM-as-judge reliable? It's surprisingly good at evaluating general quality, but it has known biases (it prefers longer answers, it can be self-congratulatory). Capsule 03 covers those biases and how to mitigate them. Best practice is to use LLM-as-judge as the middle layer and validate it periodically with human evaluation.

Can I wire this into CI/CD? Yes, and you should. Capsule 05 shows how to create regression tests that run automatically in your CI/CD pipeline. A prompt change doesn't ship if the metrics fall below the baseline.

Additional resources

1. OpenAI Evals — OpenAI's evaluation framework
2. LangSmith — Observability platform for LLMs
3. Ragas — Evaluation built specifically for RAG systems
4. HELM: Holistic Evaluation of Language Models — Comprehensive benchmark
5. Evaluation of LLMs (Survey) — Academic paper on evaluation