2. Evaluation Metrics

Description

Accuracy, faithfulness, relevance, coherence, completeness, format compliance. Automatic metrics vs LLM-based metrics. BLEU and ROUGE for text generation. When to use each type and how to combine them into a final score.

Metric Taxonomy

Before implementing metrics, you need to understand which kind of metric you need:

Accuracy

The simplest, most direct metric. It only works when there's a single "correct" answer.

Basic Implementation

def accuracy(predictions: list[str], ground_truth: list[str]) -> float:
    """Simple accuracy: prediction == ground truth (exact)."""
    if not predictions:
        return 0.0
    return sum(p == g for p, g in zip(predictions, ground_truth)) / len(predictions)

With Normalization

Exact matching often fails on capitalization, extra spaces, or punctuation. Normalization avoids false negatives:

import re

def normalize(text: str) -> str:
    """Normalize for comparison: lowercase, strip whitespace, remove punctuation."""
    text = text.lower().strip()
    text = re.sub(r'[^\w\s]', '', text)
    text = re.sub(r'\s+', ' ', text)
    return text

def normalized_accuracy(predictions: list[str], ground_truth: list[str]) -> float:
    """Accuracy with normalization — reduces false negatives."""
    return sum(
        normalize(p) == normalize(g)
        for p, g in zip(predictions, ground_truth)
    ) / len(predictions)

# Example:
preds = ["POSITIVE", " positive ", "Positive."]
truths = ["POSITIVE", "POSITIVE", "POSITIVE"]

print(accuracy(preds, truths))             # 0.33 — only the first one matches
print(normalized_accuracy(preds, truths))  # 1.0 — they all normalize to "positive"

Accuracy by Category (Breakdown)

For classifiers, the global accuracy can hide problems in specific categories:

from collections import defaultdict

def accuracy_by_category(
    predictions: list[str],
    ground_truth: list[str]
) -> dict[str, dict]:
    """
    Calculates accuracy broken down by category.
    Useful for spotting that the model fails on a specific class.
    """
    stats = defaultdict(lambda: {"total": 0, "correct": 0})
    
    for pred, truth in zip(predictions, ground_truth):
        stats[truth]["total"] += 1
        if normalize(pred) == normalize(truth):
            stats[truth]["correct"] += 1
    
    return {
        cat: {
            "accuracy": data["correct"] / data["total"],
            "total": data["total"],
            "correct": data["correct"]
        }
        for cat, data in stats.items()
    }

# Usage:
preds = ["POS", "NEG", "POS", "NEG", "POS"]
truths = ["POS", "NEG", "NEG", "NEG", "POS"]

breakdown = accuracy_by_category(preds, truths)
# Output: {"POS": {"accuracy": 1.0, "total": 2}, "NEG": {"accuracy": 0.67, "total": 3}}

Precision, Recall and F1

For binary or multi-class classification, accuracy alone can be misleading (especially with imbalanced classes).

def precision_recall_f1(
    predictions: list[str],
    ground_truth: list[str],
    positive_class: str
) -> dict[str, float]:
    """
    Calculates precision, recall and F1 for a specific class.
    
    Precision: Of what I predicted as positive, how much was really positive?
    Recall: Of what was really positive, how much did I predict as positive?
    F1: Harmonic mean of precision and recall
    """
    tp = sum(p == positive_class and g == positive_class
             for p, g in zip(predictions, ground_truth))
    fp = sum(p == positive_class and g != positive_class
             for p, g in zip(predictions, ground_truth))
    fn = sum(p != positive_class and g == positive_class
             for p, g in zip(predictions, ground_truth))
    
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    
    return {"precision": precision, "recall": recall, "f1": f1}

# Example with imbalanced classes:
# If your dataset has 90% negatives and 10% positives,
# a model that always says "NEGATIVE" has accuracy=0.90
# but recall=0.0 for the positive class — useless

Faithfulness

Is the output faithful to the input? Does the model invent information (hallucination)?

This metric is critical for summarization, extraction, or RAG systems.

from openai import OpenAI

client = OpenAI()

def evaluate_faithfulness(input_text: str, output: str) -> float:
    """
    Evaluates whether the output only contains information from the input.
    Returns 1.0 if it's faithful, 0.0 if there are hallucinations.
    
    Uses LLM-as-judge for this evaluation.
    """
    prompt = f"""You are a faithfulness evaluator.

Your task: determine whether the OUTPUT contains information that is NOT in the INPUT.

INPUT:
{input_text}

OUTPUT:
{output}

Instructions:
- If the OUTPUT only uses information explicitly present in the INPUT: answer "1"
- If the OUTPUT invents, assumes, or adds information NOT in the INPUT: answer "0"
- Answer only with the number "0" or "1". Nothing else.

Evaluation:"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=5
    )
    
    result = response.choices[0].message.content.strip()
    return float("1" in result)


def evaluate_faithfulness_detailed(input_text: str, output: str) -> dict:
    """
    Detailed version: it also explains what was made up.
    """
    prompt = f"""You are a faithfulness evaluator.

INPUT:
{input_text}

OUTPUT:
{output}

Evaluate in JSON format:
{{
  "score": 0 or 1,
  "faithful": true or false,
  "hallucinated_claims": ["list of claims that are not in the input"],
  "rationale": "brief explanation"
}}"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        response_format={"type": "json_object"}
    )
    
    import json
    return json.loads(response.choices[0].message.content)


# Usage example:
document = "The company, founded in 2010, has 500 employees and operates in Mexico."
bad_summary = "The company, founded in 2010, has 500 employees and operates in Mexico and across Latin America."
good_summary = "The company, established in 2010, has 500 employees and works in Mexico."

print(evaluate_faithfulness(document, bad_summary))   # 0.0 — it invented "across Latin America"
print(evaluate_faithfulness(document, good_summary))  # 1.0 — it only uses info from the document

Relevance

Does the output answer the question or task it was given?

def evaluate_relevance(question: str, output: str, context: str = "") -> float:
    """
    Evaluates whether the output is relevant and answers the question.
    Scale: 0.0 (irrelevant) to 1.0 (perfectly relevant)
    """
    context_str = f"\nAdditional context: {context}" if context else ""
    
    prompt = f"""You are a relevance evaluator.

QUESTION/TASK:
{question}{context_str}

ANSWER:
{output}

Evaluate how well the ANSWER addresses the QUESTION/TASK.
Answer with a number between 0 and 10 (no extra text):
- 0-3: Irrelevant or off topic
- 4-6: Partially relevant
- 7-9: Relevant but incomplete
- 10: Perfectly relevant and complete

Score (0-10):"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=5
    )
    
    try:
        score_raw = response.choices[0].message.content.strip()
        score = float(score_raw) / 10.0  # Normalize to 0-1
        return max(0.0, min(1.0, score))
    except ValueError:
        return 0.5  # Default if it can't parse


# Example:
question = "What is the capital of France?"
correct_answer = "The capital of France is Paris."
incorrect_answer = "France is a country in Western Europe known for its cuisine."

print(evaluate_relevance(question, correct_answer))    # ~1.0
print(evaluate_relevance(question, incorrect_answer))  # ~0.3

Coherence and Completeness

For longer texts, you need to evaluate coherence (it flows well) and completeness (it covers everything needed).

def evaluate_coherence_completeness(
    task: str,
    output: str,
    completeness_criteria: list[str]
) -> dict[str, float]:
    """
    Evaluates the coherence and completeness of the output.
    
    completeness_criteria: List of aspects the output must cover.
    """
    criteria_str = "\n".join(f"- {c}" for c in completeness_criteria)
    
    prompt = f"""Evaluate the following text along two dimensions.

ORIGINAL TASK:
{task}

TEXT TO EVALUATE:
{output}

COMPLETENESS CRITERIA (it must cover all of them):
{criteria_str}

Answer in JSON:
{{
  "coherence_score": 0-10,
  "completeness_score": 0-10,
  "covered_criteria": ["list of criteria it DOES cover"],
  "missing_criteria": ["list of criteria it does NOT cover"],
  "coherence_issues": ["list of flow or structure problems"]
}}"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        response_format={"type": "json_object"}
    )
    
    import json
    result = json.loads(response.choices[0].message.content)
    
    return {
        "coherence": result["coherence_score"] / 10.0,
        "completeness": result["completeness_score"] / 10.0,
        "covered_criteria": result.get("covered_criteria", []),
        "missing_criteria": result.get("missing_criteria", []),
        "coherence_issues": result.get("coherence_issues", [])
    }

Format Compliance

Does the output meet the requested format? This metric can be verified programmatically.

import json
import re

def evaluate_format_compliance(output: str, expected_format: str) -> dict[str, any]:
    """
    Verifies that the output meets the requested format.
    
    Supported formats: "json", "json_array", "bullet_list", "numbered_list",
                       "markdown_table", "yaml", "email"
    """
    
    result = {"compliant": False, "format": expected_format, "error": None}
    
    if expected_format == "json":
        try:
            parsed = json.loads(output)
            result["compliant"] = True
            result["parsed"] = parsed
        except json.JSONDecodeError as e:
            result["error"] = str(e)
    
    elif expected_format == "json_array":
        try:
            parsed = json.loads(output)
            result["compliant"] = isinstance(parsed, list)
            if not result["compliant"]:
                result["error"] = "The JSON is not an array"
        except json.JSONDecodeError as e:
            result["error"] = str(e)
    
    elif expected_format == "bullet_list":
        lines = [l.strip() for l in output.strip().split("\n") if l.strip()]
        bullet_lines = [l for l in lines if l.startswith(("-", "*", "•"))]
        result["compliant"] = len(bullet_lines) >= 2
        result["bullet_count"] = len(bullet_lines)
        if not result["compliant"]:
            result["error"] = f"Only {len(bullet_lines)} bullets, expected 2+"
    
    elif expected_format == "numbered_list":
        lines = [l.strip() for l in output.strip().split("\n") if l.strip()]
        numbered = [l for l in lines if re.match(r'^\d+[\.\)]', l)]
        result["compliant"] = len(numbered) >= 2
        result["item_count"] = len(numbered)
    
    elif expected_format == "markdown_table":
        result["compliant"] = "|" in output and "---" in output
        if not result["compliant"]:
            result["error"] = "No markdown table detected (|...|...---...)"
    
    return result


# With Pydantic for JSON with a specific schema:
from pydantic import BaseModel, ValidationError
from typing import Optional

class SentimentOutput(BaseModel):
    sentiment: str  # "POSITIVE", "NEGATIVE", "NEUTRAL"
    confidence: float  # 0.0 - 1.0
    explanation: Optional[str] = None

def evaluate_schema_compliance(output: str, schema_class: type[BaseModel]) -> dict:
    """Validates the output against a Pydantic schema."""
    try:
        data = json.loads(output)
        validated = schema_class(**data)
        return {"compliant": True, "parsed": validated.model_dump()}
    except json.JSONDecodeError as e:
        return {"compliant": False, "error": f"Invalid JSON: {e}"}
    except ValidationError as e:
        return {"compliant": False, "error": f"Invalid schema: {e}"}

BLEU and ROUGE

Reference metrics for text generation. They compare the output against a reference answer.

BLEU (Bilingual Evaluation Understudy)

It measures how many n-grams from the output show up in the reference. Originally for translation.

from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
import nltk

def calculate_bleu(reference: str, candidate: str, n: int = 4) -> float:
    """
    Calculates the BLEU score between reference and candidate.
    
    n: maximum n-gram (typically 1-4)
    Returns a float 0.0-1.0
    """
    ref_tokens = reference.lower().split()
    cand_tokens = candidate.lower().split()
    
    # Weights for n-grams (1-gram, 2-gram, etc.)
    weights = [1/n] * n
    
    # SmoothingFunction to avoid 0 when there are n-grams with no matches
    smoothing = SmoothingFunction().method1
    
    score = sentence_bleu(
        [ref_tokens],
        cand_tokens,
        weights=weights,
        smoothing_function=smoothing
    )
    return score


# BLEU on a full dataset:
def bleu_dataset(references: list[str], candidates: list[str]) -> dict[str, float]:
    scores = [calculate_bleu(r, c) for r, c in zip(references, candidates)]
    return {
        "bleu_mean": sum(scores) / len(scores),
        "bleu_min": min(scores),
        "bleu_max": max(scores)
    }

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

More recall-oriented. It measures how many tokens from the reference show up in the output.

from rouge_score import rouge_scorer

def calculate_rouge(reference: str, candidate: str) -> dict[str, float]:
    """
    Calculates ROUGE-1, ROUGE-2 and ROUGE-L.
    
    ROUGE-1: unigram overlap
    ROUGE-2: bigram overlap  
    ROUGE-L: longest common subsequence
    """
    scorer = rouge_scorer.RougeScorer(
        ['rouge1', 'rouge2', 'rougeL'],
        use_stemmer=True
    )
    
    scores = scorer.score(reference, candidate)
    
    return {
        "rouge1_f1": scores['rouge1'].fmeasure,
        "rouge2_f1": scores['rouge2'].fmeasure,
        "rougeL_f1": scores['rougeL'].fmeasure,
        "rouge1_precision": scores['rouge1'].precision,
        "rouge1_recall": scores['rouge1'].recall,
    }


# Example:
ref = "The central bank raised interest rates to 5% to control inflation."
good_candidate = "The central bank increased rates to 5% with the goal of reducing inflation."
bad_candidate = "Rates were modified by the monetary authorities."

print(calculate_rouge(ref, good_candidate))
# {'rouge1_f1': ~0.6, 'rouge2_f1': ~0.4, 'rougeL_f1': ~0.5}

print(calculate_rouge(ref, bad_candidate))
# {'rouge1_f1': ~0.3, 'rouge2_f1': ~0.1, 'rougeL_f1': ~0.2}

When to use BLEU vs ROUGE

Composite Score: Combining Metrics

In production, you rarely use a single metric. The ideal is a composite score:

def composite_score(
    accuracy: float,
    faithfulness: float,
    relevance: float,
    format_compliance: float,
    weights: dict[str, float] | None = None
) -> dict[str, float]:
    """
    Weighted composite score.
    
    The default weights are equal, but in your use case
    faithfulness might matter more (RAG, summarization)
    or format_compliance might be critical (structured output).
    """
    if weights is None:
        weights = {
            "accuracy": 0.30,
            "faithfulness": 0.25,
            "relevance": 0.25,
            "format": 0.20
        }
    
    assert abs(sum(weights.values()) - 1.0) < 0.001, "The weights must add up to 1.0"
    
    composite = (
        accuracy * weights["accuracy"] +
        faithfulness * weights["faithfulness"] +
        relevance * weights["relevance"] +
        format_compliance * weights["format"]
    )
    
    return {
        "composite": composite,
        "breakdown": {
            "accuracy": accuracy,
            "faithfulness": faithfulness,
            "relevance": relevance,
            "format": format_compliance
        },
        "interpretation": (
            "EXCELLENT" if composite >= 0.90 else
            "GOOD" if composite >= 0.80 else
            "ACCEPTABLE" if composite >= 0.70 else
            "NEEDS_WORK"
        )
    }


# Example:
result = composite_score(
    accuracy=0.94,
    faithfulness=0.88,
    relevance=0.91,
    format_compliance=1.0,
    # Specific weights for a RAG system where faithfulness is critical
    weights={"accuracy": 0.25, "faithfulness": 0.40, "relevance": 0.25, "format": 0.10}
)
print(result)
# {'composite': 0.919, 'interpretation': 'EXCELLENT', ...}

Metrics Dashboard in Production

import json
from datetime import datetime
from pathlib import Path

class MetricsDashboard:
    """Class for tracking metrics over time."""
    
    def __init__(self, storage_path: str = "metrics_history.jsonl"):
        self.storage_path = Path(storage_path)
    
    def record(self, prompt_name: str, version: str, metrics: dict) -> None:
        """Records a metrics snapshot with a timestamp."""
        entry = {
            "timestamp": datetime.now().isoformat(),
            "prompt_name": prompt_name,
            "version": version,
            **metrics
        }
        with open(self.storage_path, "a") as f:
            f.write(json.dumps(entry) + "\n")
    
    def history(self, prompt_name: str, last_n: int = 10) -> list[dict]:
        """Returns the last N entries for a prompt."""
        if not self.storage_path.exists():
            return []
        
        entries = []
        with open(self.storage_path) as f:
            for line in f:
                r = json.loads(line)
                if r["prompt_name"] == prompt_name:
                    entries.append(r)
        
        return entries[-last_n:]
    
    def detect_trend(self, prompt_name: str, metric: str) -> str:
        """Detects whether a metric is improving, degrading, or stable."""
        history = self.history(prompt_name, last_n=5)
        
        if len(history) < 2:
            return "INSUFFICIENT_DATA"
        
        values = [h.get(metric, 0) for h in history]
        delta = values[-1] - values[0]
        
        if delta > 0.05:
            return "IMPROVING ↑"
        elif delta < -0.05:
            return "DEGRADING ↓"
        else:
            return "STABLE →"


# Usage:
dashboard = MetricsDashboard()
dashboard.record(
    prompt_name="ticket_classifier",
    version="v1.2",
    metrics={"accuracy": 0.94, "faithfulness": 0.88, "format_compliance": 1.0}
)

Comparison: Automatic vs LLM-based Metrics

Troubleshooting

Problem 1: Artificially inflated accuracy

Symptom: Accuracy of 0.98 but the system fails in production.

Cause: Ambiguous ground truth or a biased golden set (easy cases only).

Solution:

# Audit the golden set
def audit_golden_set(golden_set: list[dict]) -> dict:
    """Detects possible problems in a golden set."""
    categories = {}
    for ex in golden_set:
        cat = ex.get("expected_output", "UNKNOWN")
        categories[cat] = categories.get(cat, 0) + 1
    
    total = len(golden_set)
    distribution = {k: v/total for k, v in categories.items()}
    
    # Detect imbalance
    max_cat = max(distribution.values())
    alert = max_cat > 0.7  # One category dominates
    
    return {
        "total_examples": total,
        "distribution": distribution,
        "imbalance_alert": alert,
        "recommendation": "Balance the categories" if alert else "OK"
    }

Problem 2: Inconsistent LLM-as-judge

Symptom: The same output gets 7/10 one day and 5/10 the next.

Cause: Temperature > 0, or ambiguous rubrics.

Solution:

# Always temperature=0 for the judge
# Use rubrics with concrete examples (anchoring)
RUBRIC_WITH_EXAMPLES = """
Evaluate faithfulness (0-5):
- 5: The output only contains information from the input. Example: [example of a faithful output]
- 3: The output contains a reasonable inference. Example: [borderline example]
- 0: The output invents data. Example: [example of a hallucination]
"""

Problem 3: Format compliance with false positives

Symptom: The model returns valid JSON but with the wrong structure.

Cause: Validating only the JSON syntax, not the schema.

Solution: Use Pydantic or JSON Schema for structural validation:

import jsonschema

SCHEMA = {
    "type": "object",
    "properties": {
        "sentiment": {"type": "string", "enum": ["POSITIVE", "NEGATIVE", "NEUTRAL"]},
        "confidence": {"type": "number", "minimum": 0, "maximum": 1}
    },
    "required": ["sentiment", "confidence"]
}

def validate_json_schema(output: str, schema: dict) -> dict:
    try:
        data = json.loads(output)
        jsonschema.validate(data, schema)
        return {"valid": True, "data": data}
    except json.JSONDecodeError as e:
        return {"valid": False, "error": f"Invalid JSON: {e}"}
    except jsonschema.ValidationError as e:
        return {"valid": False, "error": f"Invalid schema: {e.message}"}

Problem 4: BLEU/ROUGE very low even though the output is good

Symptom: Semantically correct output but BLEU = 0.2.

Cause: BLEU/ROUGE are sensitive to lexical differences even when the meaning is the same.

Solution: Complement them with BERTScore or an LLM-as-judge for relevance:

# If BLEU < threshold, evaluate with an LLM before rejecting
def evaluate_with_fallback(reference: str, candidate: str) -> dict:
    bleu = calculate_bleu(reference, candidate)
    
    if bleu >= 0.4:
        return {"score": bleu, "method": "bleu"}
    
    # Fall back to an LLM if BLEU is low (possible valid paraphrase)
    relevance = evaluate_relevance(reference, candidate)
    return {"score": relevance, "method": "llm_relevance", "bleu": bleu}

Exercises

Exercise 1: Implement accuracy with normalization

Implement an accuracy function that normalizes the predictions and ground truths before comparing. It must handle: uppercase, extra spaces, and trailing punctuation.

import re

def normalized_accuracy(predictions: list[str], ground_truths: list[str]) -> float:
    """Accuracy with full normalization."""
    def normalize(text: str) -> str:
        text = text.lower().strip()
        text = re.sub(r'[^\w\s]', '', text)  # Remove punctuation
        text = re.sub(r'\s+', ' ', text)     # Extra spaces
        return text
    
    correct = sum(
        normalize(p) == normalize(g)
        for p, g in zip(predictions, ground_truths)
    )
    return correct / len(predictions)

# Test
preds = ["POSITIVE", " positive ", "Positive!", "positive."]
truths = ["POSITIVE"] * 4
assert normalized_accuracy(preds, truths) == 1.0
print("✓ Normalized accuracy works")

Exercise 2: Evaluate the faithfulness of a summary

Take the following document and two summaries (one faithful, one with hallucinations). Implement the faithfulness evaluation and verify that it correctly detects which is which.

document = """
The Apollo 11 project landed on the Moon on July 20, 1969.
Astronauts Neil Armstrong and Buzz Aldrin walked on the lunar surface.
Michael Collins stayed in orbit in the command module.
"""

faithful_summary = "Apollo 11 reached the Moon in July 1969, with Armstrong and Aldrin on the surface and Collins in orbit."
hallucinated_summary = "Apollo 11 reached the Moon in July 1969. The three astronauts walked the lunar surface together."

from openai import OpenAI
client = OpenAI()

def evaluate_faithfulness(input_text: str, output: str) -> float:
    prompt = f"""Evaluate whether the OUTPUT only contains information from the INPUT.
INPUT: {input_text}
OUTPUT: {output}
Answer "1" (faithful) or "0" (invents information). Only the number."""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=5
    )
    return float("1" in response.choices[0].message.content.strip())

faithful_score = evaluate_faithfulness(document, faithful_summary)
hallucinated_score = evaluate_faithfulness(document, hallucinated_summary)

print(f"Faithful summary: {faithful_score}")                    # Should be 1.0
print(f"Summary with hallucination: {hallucinated_score}")      # Should be 0.0
# The hallucination: "the three astronauts walked together" — Collins never walked

Exercise 3: Composite score for a classifier

Implement a composite score for a sentiment classifier that evaluates accuracy, format compliance and relevance:

import json
from openai import OpenAI
client = OpenAI()

def evaluate_full_classifier(
    prompt_template: str,
    golden_set: list[dict]
) -> dict:
    """Evaluates a classifier with multiple metrics."""
    
    results = {
        "accuracy_scores": [],
        "format_scores": [],
        "total_examples": len(golden_set)
    }
    
    for example in golden_set:
        prompt = prompt_template.format(text=example["input"])
        
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
            response_format={"type": "json_object"}
        )
        
        output = response.choices[0].message.content
        
        # Format compliance
        try:
            data = json.loads(output)
            format_ok = "sentiment" in data
            results["format_scores"].append(1.0 if format_ok else 0.0)
            
            # Accuracy
            if format_ok:
                pred = data["sentiment"].upper()
                truth = example["expected_output"].upper()
                results["accuracy_scores"].append(1.0 if pred == truth else 0.0)
        except json.JSONDecodeError:
            results["format_scores"].append(0.0)
            results["accuracy_scores"].append(0.0)
    
    accuracy = sum(results["accuracy_scores"]) / len(results["accuracy_scores"])
    format_compliance = sum(results["format_scores"]) / len(results["format_scores"])
    composite = accuracy * 0.6 + format_compliance * 0.4
    
    return {
        "accuracy": accuracy,
        "format_compliance": format_compliance,
        "composite": composite
    }

Exercise 4: Spot the problematic class

Given the following classification result, identify which class has the worst performance:

predictions  = ["POS", "NEG", "POS", "NEU", "NEG", "POS", "NEG", "NEU", "POS", "NEG"]
ground_truth = ["POS", "NEG", "NEG", "NEU", "NEG", "POS", "POS", "NEU", "POS", "POS"]

from collections import defaultdict

def accuracy_by_category(predictions, ground_truth):
    stats = defaultdict(lambda: {"total": 0, "correct": 0})
    for pred, truth in zip(predictions, ground_truth):
        stats[truth]["total"] += 1
        if pred == truth:
            stats[truth]["correct"] += 1
    
    return {
        cat: {
            "accuracy": data["correct"] / data["total"],
            "total": data["total"]
        }
        for cat, data in stats.items()
    }

predictions  = ["POS", "NEG", "POS", "NEU", "NEG", "POS", "NEG", "NEU", "POS", "NEG"]
ground_truth = ["POS", "NEG", "NEG", "NEU", "NEG", "POS", "POS", "NEU", "POS", "POS"]

result = accuracy_by_category(predictions, ground_truth)
for cat, stats in sorted(result.items(), key=lambda x: x[1]["accuracy"]):
    print(f"{cat}: {stats['accuracy']:.1%} ({stats['total']} examples)")

# NEG: 50.0% — problematic class
# NEU: 100.0% — works well
# POS: 80.0% — acceptable
# → The NEG class has problems, it needs more examples or a prompt tweak

Exercise 5: Compare the BLEU of two prompts

Given this golden set of summaries, which prompt produces better summaries according to BLEU?

from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
from openai import OpenAI

client = OpenAI()

references = [
    "The stock market rose 2% driven by the technology sector.",
    "The company reported record earnings of 500 million for the quarter.",
]

prompt_a = "Summarize in one sentence: {text}"
prompt_b = "Summarize the following text in one concise sentence, using the key words from the original: {text}"

texts = [
    "Stocks in the stock market performed positively today with an increase of two percent, led mainly by companies in the technology sector that reported good results.",
    "The company announced its quarterly financial results showing unprecedented earnings that reached five hundred million dollars, beating every analyst expectation.",
]

def average_bleu(prompt_template: str, texts: list, references: list) -> float:
    smoothing = SmoothingFunction().method1
    scores = []
    for text, ref in zip(texts, references):
        prompt = prompt_template.format(text=text)
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=0
        )
        output = response.choices[0].message.content
        score = sentence_bleu([ref.split()], output.split(), smoothing_function=smoothing)
        scores.append(score)
    return sum(scores) / len(scores)

bleu_a = average_bleu(prompt_a, texts, references)
bleu_b = average_bleu(prompt_b, texts, references)
print(f"Prompt A BLEU: {bleu_a:.3f}")
print(f"Prompt B BLEU: {bleu_b:.3f}")
print(f"Winner: {'A' if bleu_a > bleu_b else 'B'}")

Summary

• Accuracy: For classification and QA with a fixed answer. Always normalize.
• Precision/Recall/F1: For imbalanced classification or when recall matters
• Faithfulness: Catch hallucinations. Critical for RAG and summaries
• Relevance: Does the output answer the question? Always evaluate with an LLM
• Format compliance: For structured output. Use Pydantic or JSON Schema
• BLEU/ROUGE: For generation with a reference. Complement with semantic metrics
• Composite score: Combine metrics with weights based on the use case

Additional resources

1. G-Eval: NLG Evaluation using GPT-4 — Paper on LLM-as-judge
2. BERTScore — BERT-based semantic metric
3. ROUGE in Python (rouge-score) — Official library
4. NLTK BLEU — BLEU implementation
5. Ragas Metrics — Metrics for RAG
6. OpenAI Cookbook: Evaluations — Official guide