3. LLM-as-Judge

Description

Using an LLM to evaluate another LLM's outputs. Detailed rubrics, scoring scales, comparison-based evaluation. LLM-judge biases and how to mitigate them. When to trust the LLM's judgment.

The Problem It Solves

For metrics like faithfulness, relevance or quality, there is no deterministic function that computes them. You can't compare strings and detect hallucinations. You can't parse a JSON and know whether it's "relevant".

LLM-as-judge is the answer: you use an LLM (the "judge") to evaluate the output of another LLM (the "evaluated model").

LLM-as-judge flow:

Input → [Evaluated model] → Output
                               ↓
            Input + Output → [LLM Judge] → Score + Rationale

When to use LLM-as-judge vs automatic metrics

Principles of a Good LLM-Judge

For the LLM's judgment to be trustworthy, the judge's prompt must:

1. Be specific: Clear criteria, not "evaluate the quality"
2. Be calibrated: Include examples of each level (anchoring)
3. Be reproducible: Temperature=0, always the same model
4. Be consistent: Same scale, same criteria across every evaluation
5. Be explainable: The judge must justify its score

Rubric-Based Scoring

The rubric is the criteria document you hand the judge. The quality of the rubric determines the quality of the judgment.

Simple Rubric (Global Score)

from openai import OpenAI

client = OpenAI()

RUBRIC_SIMPLE = """
Evaluate the answer on a scale from 1 to 5:
1 = Completely incorrect or irrelevant
2 = Mostly incorrect with a few useful elements
3 = Partially correct but incomplete or with errors
4 = Mostly correct with minor incorrect details
5 = Completely correct, complete and relevant

Answer only with the number (1-5). Nothing else.
"""

def llm_judge_simple(question: str, answer: str) -> int:
    """Simple judge: score from 1 to 5."""
    prompt = f"""{RUBRIC_SIMPLE}

Question: {question}
Answer: {answer}

Score:"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=5
    )
    
    try:
        score_text = response.choices[0].message.content.strip()
        return int(score_text[0])  # Only the first digit
    except (ValueError, IndexError):
        return 3  # Default if it can't parse

Multi-Criteria Rubric (Detailed)

For deeper evaluations, break it down into dimensions:

RUBRIC_DETAILED = """
Evaluate the answer along 3 dimensions (0-2 points each):

1. CORRECTNESS (0-2):
   - 0: Factually incorrect information
   - 1: Mostly correct with one minor error
   - 2: Completely correct and verifiable

2. COMPLETENESS (0-2):
   - 0: Doesn't answer what was asked
   - 1: Partially answers, important information is missing
   - 2: Answers everything that was asked

3. CLARITY (0-2):
   - 0: Confusing, hard to understand
   - 1: Understandable but could be clearer
   - 2: Very clear and well organized

TOTAL POSSIBLE: 6 points

Answer EXACTLY in this format:
CORRECTNESS: X/2
COMPLETENESS: X/2
CLARITY: X/2
TOTAL: X/6
RATIONALE: [One sentence explaining the evaluation]
"""

def llm_judge_detailed(
    question: str,
    answer: str,
    ground_truth: str = ""
) -> dict:
    """Multi-criteria judge with a rationale."""
    
    gt_str = f"\nReference answer (ground truth): {ground_truth}" if ground_truth else ""
    
    prompt = f"""{RUBRIC_DETAILED}

Question: {question}{gt_str}
Answer to evaluate: {answer}

Evaluation:"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=200
    )
    
    raw = response.choices[0].message.content
    return parse_detailed_rubric(raw)


def parse_detailed_rubric(raw: str) -> dict:
    """Parses the output of the detailed rubric."""
    import re
    
    result = {
        "correctness": 0,
        "completeness": 0,
        "clarity": 0,
        "total": 0,
        "max": 6,
        "normalized_score": 0.0,
        "rationale": "",
        "raw": raw
    }
    
    # Extract scores
    correctness_match = re.search(r'CORRECTNESS:\s*(\d)/2', raw)
    completeness_match = re.search(r'COMPLETENESS:\s*(\d)/2', raw)
    clarity_match = re.search(r'CLARITY:\s*(\d)/2', raw)
    total_match = re.search(r'TOTAL:\s*(\d)/6', raw)
    rationale_match = re.search(r'RATIONALE:\s*(.+)', raw)
    
    if correctness_match:
        result["correctness"] = int(correctness_match.group(1))
    if completeness_match:
        result["completeness"] = int(completeness_match.group(1))
    if clarity_match:
        result["clarity"] = int(clarity_match.group(1))
    if total_match:
        result["total"] = int(total_match.group(1))
    else:
        result["total"] = result["correctness"] + result["completeness"] + result["clarity"]
    
    result["normalized_score"] = result["total"] / 6.0
    
    if rationale_match:
        result["rationale"] = rationale_match.group(1).strip()
    
    return result

Rubric with Anchoring (Calibration Examples)

Anchoring is the most effective technique for reducing the judge's variance:

RUBRIC_WITH_ANCHORING = """
Evaluate the FAITHFULNESS of a summary with respect to the original text (0-5).

CALIBRATION EXAMPLES:

Score 5 — The summary only uses information from the original text:
  Original: "Apple was founded in 1976 in California. Steve Jobs was a co-founder."
  Summary: "Apple was founded in 1976 in California, with Steve Jobs as a co-founder."
  → Score: 5 ✓

Score 3 — The summary makes reasonable but unverifiable inferences:
  Original: "Apple was founded in 1976. In the 80s it launched the Mac."
  Summary: "Apple, founded in 1976, was a pioneer in personal computers with the Mac."
  → Score: 3 (the inference of "pioneer" is not in the text)

Score 1 — The summary invents information:
  Original: "Apple was founded in 1976 in California."
  Summary: "Apple, founded in 1976 in California, is currently the most valuable company in the world."
  → Score: 1 (there is no info about "most valuable in the world")

Now evaluate:
Original: {original}
Summary: {summary}

Answer ONLY: "Score: X/5" where X is your evaluation.
"""

def judge_with_anchoring(original: str, summary: str) -> float:
    """LLM judge with anchoring for maximum consistency."""
    prompt = RUBRIC_WITH_ANCHORING.format(original=original, summary=summary)
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=20
    )
    
    raw = response.choices[0].message.content
    
    import re
    match = re.search(r'Score:\s*(\d)', raw)
    if match:
        return int(match.group(1)) / 5.0
    return 0.5

Comparison-Based Evaluation

Instead of scoring in absolute terms, compare two outputs directly. It's more reliable for detecting which one is better.

def compare_responses(
    question: str,
    resp_a: str,
    resp_b: str,
    criteria: str = "correctness, completeness and clarity"
) -> dict:
    """
    Compares two answers and determines which is better.
    
    Returns: "A", "B", or "TIE"
    """
    prompt = f"""Compare these two answers to the same question.
Evaluate based on: {criteria}

Question: {question}

Answer A:
{resp_a}

Answer B:
{resp_b}

Which answer is better?
- Answer "A" if Answer A is clearly better
- Answer "B" if Answer B is clearly better
- Answer "TIE" if they are equivalent in quality

Answer ONLY with: A, B, or TIE"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=10
    )
    
    result = response.choices[0].message.content.strip().upper()
    
    # Normalize the answer
    if "TIE" in result:
        winner = "TIE"
    elif result.startswith("A"):
        winner = "A"
    elif result.startswith("B"):
        winner = "B"
    else:
        winner = "TIE"  # Conservative default
    
    return {
        "winner": winner,
        "raw": result,
        "resp_a": resp_a[:100] + "...",
        "resp_b": resp_b[:100] + "..."
    }


def tournament_evaluation(
    question: str,
    answers: dict[str, str]
) -> dict[str, int]:
    """
    Round-robin tournament between N answers.
    Every pair is compared, the winner takes a point.
    
    answers: {"name": "answer_text"}
    """
    from itertools import combinations
    
    points = {name: 0 for name in answers}
    
    for (name_a, resp_a), (name_b, resp_b) in combinations(answers.items(), 2):
        result = compare_responses(question, resp_a, resp_b)
        
        if result["winner"] == "A":
            points[name_a] += 1
        elif result["winner"] == "B":
            points[name_b] += 1
        else:  # TIE
            points[name_a] += 0.5
            points[name_b] += 0.5
    
    return dict(sorted(points.items(), key=lambda x: x[1], reverse=True))


# Example: Compare 3 versions of a prompt
question = "How does machine learning work?"
answers = {
    "zero_shot": "Machine learning is...",
    "few_shot": "Machine learning learns...",
    "cot": "To understand machine learning, first...",
}

ranking = tournament_evaluation(question, answers)
print("Ranking:", ranking)
# Example: {"cot": 2, "few_shot": 1, "zero_shot": 0}

LLM-Judge Biases and How to Mitigate Them

The LLM-judge is not impartial. It has known biases that can skew the results.

Bias 1: Length Bias (Verbosity Bias)

The judge tends to prefer longer answers, assuming more text = more information.

Example:
Answer A: "Python is better for ML because of its library ecosystem."
Answer B: "Python is a reasonable option for ML, although there are also other options like R and Julia that have their own strengths depending on the specific use case under consideration."

Without bias correction: The judge prefers B (longer)
With bias correction: A may be better (more concise and direct)

Mitigation:

RUBRIC_ANTI_LENGTH_BIAS = """
IMPORTANT: Evaluate the CONTENT, not the length.
A short, precise answer is BETTER than a long answer with filler.
Actively penalize answers that add unnecessary words.

Quality criterion: does every sentence add useful information? If not, lower the score.
"""

def judge_without_length_bias(question: str, answer: str) -> float:
    """Judge with explicit instructions against length bias."""
    prompt = f"""{RUBRIC_ANTI_LENGTH_BIAS}

Evaluate (0-10): how well does this answer address the question?

Question: {question}
Answer: {answer}

Score (0-10):"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=5
    )
    
    try:
        return float(response.choices[0].message.content.strip()) / 10.0
    except ValueError:
        return 0.5

Bias 2: Position Bias

In comparison-based evaluation, the judge tends to prefer the first answer (when both are similar).

Mitigation: evaluate in both directions and average:

def compare_without_position_bias(
    question: str,
    resp_a: str,
    resp_b: str
) -> dict:
    """
    Evaluates A vs B and B vs A, then combines the results.
    Removes the position bias.
    """
    # Direct comparison: A first
    result_ab = compare_responses(question, resp_a, resp_b)
    
    # Reverse comparison: B first (positions swapped)
    result_ba = compare_responses(question, resp_b, resp_a)
    
    # Interpretation: in result_ba, "A" (first slot) is resp_b, "B" is resp_a
    winner_ab = result_ab["winner"]
    winner_ba_in_original = {
        "A": "B",  # In BA, "A" is resp_b → resp_b won → "B" in the original
        "B": "A",  # In BA, "B" is resp_a → resp_a won → "A" in the original
        "TIE": "TIE"
    }[result_ba["winner"]]
    
    # Consolidate
    if winner_ab == winner_ba_in_original:
        # Both evaluations agree → reliable result
        final_winner = winner_ab
        confidence = "HIGH"
    elif winner_ab == "TIE" or winner_ba_in_original == "TIE":
        # One is a tie → use the one that isn't
        final_winner = winner_ab if winner_ba_in_original == "TIE" else winner_ba_in_original
        confidence = "MEDIUM"
    else:
        # Contradiction → conservative tie
        final_winner = "TIE"
        confidence = "LOW (possible position bias detected)"
    
    return {
        "winner": final_winner,
        "confidence": confidence,
        "result_ab": winner_ab,
        "normalized_result_ba": winner_ba_in_original
    }

Bias 3: Self-Enhancement Bias

If the judge is the same model that generated the answers, it tends to prefer its own style.

Problem: You evaluate gpt-4o-mini outputs with gpt-4o-mini
→ The judge favors answers that sound like gpt-4o-mini

Mitigation:

• Use a different model as judge (e.g. Claude judging GPT)
• If that's not possible, use a stronger model as judge (gpt-4o judging gpt-4o-mini)
• Evaluate with multiple judges and average

Bias 4: Sycophancy

The judge can be a people-pleaser: if you include the "correct answer" in the prompt, it will lean toward a high score regardless of the actual quality.

Mitigation:

# BAD: It contaminates the judgment
contaminated_prompt = f"""
The correct answer is: {ground_truth}
Evaluate whether this answer is correct: {output}
"""

# GOOD: Ground truth as reference, not as "the correct answer"
clean_prompt = f"""
Reference context (to verify facts): {ground_truth}
Evaluate the quality of this answer without considering whether it matches the reference exactly:
{output}
"""

Evaluation with Multiple Judges

For more reliability, use multiple calls or multiple models:

def judge_with_consensus(
    question: str,
    answer: str,
    n_judges: int = 3
) -> dict:
    """
    Evaluates N times and computes consensus.
    Reduces variance caused by residual randomness.
    
    Note: even though we use temperature=0, there can be minimal variation.
    In more recent models, temperature 0 is not 100% deterministic.
    """
    scores = []
    
    for i in range(n_judges):
        prompt = f"""Evaluate (1-10) how well the answer addresses the question.
Only the number.

Question: {question}
Answer: {answer}

Score:"""
        
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=0
        )
        
        try:
            score = float(response.choices[0].message.content.strip())
            scores.append(score / 10.0)
        except ValueError:
            pass
    
    if not scores:
        return {"score": 0.5, "variance": 0, "n": 0}
    
    mean = sum(scores) / len(scores)
    variance = sum((s - mean) ** 2 for s in scores) / len(scores)
    
    return {
        "score": mean,
        "variance": variance,
        "scores": scores,
        "n": len(scores),
        "confidence": "HIGH" if variance < 0.01 else "MEDIUM" if variance < 0.04 else "LOW"
    }

Validating the LLM-Judge

The judge needs to be evaluated too. How well does it correlate with human evaluation?

def validate_human_judge_correlation(
    human_scores: list[float],
    llm_scores: list[float]
) -> dict:
    """
    Calculates the Pearson and Spearman correlation between human and LLM evaluations.
    
    Interpretation:
    > 0.8: Excellent — the LLM-judge is reliable
    0.6-0.8: Good — use with caution
    < 0.6: Poor — revisit the rubric
    """
    from scipy import stats
    
    pearson_r, pearson_p = stats.pearsonr(human_scores, llm_scores)
    spearman_r, spearman_p = stats.spearmanr(human_scores, llm_scores)
    
    return {
        "pearson_r": pearson_r,
        "pearson_p": pearson_p,
        "spearman_r": spearman_r,
        "spearman_p": spearman_p,
        "interpretation": (
            "EXCELLENT" if pearson_r > 0.8 else
            "GOOD" if pearson_r > 0.6 else
            "POOR — revisit the rubric"
        )
    }


# Validation process:
# 1. Take 50-100 examples
# 2. Evaluate with humans (gold standard)
# 3. Evaluate with the LLM-judge
# 4. Compute the correlation
# 5. If correlation > 0.7: the LLM-judge is reliable for this kind of task

Production Patterns

Pattern 1: Judge as a service

class LLMJudge:
    """Reusable judge for multiple metrics."""
    
    def __init__(self, model: str = "gpt-4o-mini"):
        self.client = OpenAI()
        self.model = model
    
    def score(
        self,
        dimension: str,
        question: str,
        answer: str,
        context: str = ""
    ) -> dict:
        """
        dimension: "faithfulness" | "relevance" | "quality" | "safety"
        """
        rubrics = {
            "faithfulness": "Does the output contain only information from the input? 0=invents, 1=faithful",
            "relevance": "Does the output answer the question? 0=irrelevant, 10=perfectly relevant",
            "quality": "Is the answer useful, correct and well written? 1-5",
            "safety": "Is the output safe and appropriate? 0=unsafe, 1=safe"
        }
        
        if dimension not in rubrics:
            raise ValueError(f"Unsupported dimension: {dimension}")
        
        ctx_str = f"\nContext: {context}" if context else ""
        
        prompt = f"""{rubrics[dimension]}

Question: {question}{ctx_str}
Answer: {answer}

Answer only with the number:"""
        
        response = self.client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
            max_tokens=10
        )
        
        raw = response.choices[0].message.content.strip()
        
        try:
            score = float(raw.split()[0])
        except (ValueError, IndexError):
            score = 0.5
        
        return {
            "dimension": dimension,
            "score": score,
            "raw": raw
        }
    
    def evaluate_all(self, question: str, answer: str, input_text: str = "") -> dict:
        """Evaluates across every dimension and returns a summary."""
        results = {}
        
        for dim in ["faithfulness", "relevance", "quality", "safety"]:
            r = self.score(dim, question, answer, context=input_text)
            results[dim] = r["score"]
        
        # Normalize to 0-1
        def normalize(score, dim):
            if dim == "faithfulness": return score  # already 0-1
            if dim == "relevance": return score / 10.0
            if dim == "quality": return (score - 1) / 4.0
            if dim == "safety": return score  # already 0-1
            return score
        
        return {dim: normalize(score, dim) for dim, score in results.items()}


# Usage:
judge = LLMJudge()
scores = judge.evaluate_all(
    question="What is Python?",
    answer="Python is an interpreted, high-level programming language.",
    input_text="Python is a language created by Guido van Rossum in 1991."
)
print(scores)

Troubleshooting

Problem 1: Judge too generous (score always high)

Symptom: The judge gives 4-5/5 to almost everything.

Cause: An uncalibrated rubric, or the model tends to be positive.

Solution:

# Add an explicit calibration instruction:
CALIBRATION = """
IMPORTANT: Use the FULL scale.
- I expect 20% of answers to get 1-2 (poor)
- 30% to get 3 (acceptable)
- 40% to get 4 (good)
- Only the most exceptional 10% to get 5

Be critical. A "complete" answer with minor errors is a 3, not a 5.
"""

Problem 2: Inconsistent score across runs

Symptom: The same input gets different scores on consecutive runs.

Cause: Temperature > 0, or an ambiguous prompt.

Solution:

# 1. Force temperature=0
# 2. Use a more structured output format
# 3. If the variance persists, average N runs

def stable_score(question, answer, n=3):
    scores = []
    for _ in range(n):
        r = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"Evaluate (1-5): {answer}"}],
            temperature=0
        )
        try:
            scores.append(int(r.choices[0].message.content.strip()[0]))
        except:
            pass
    return sum(scores) / len(scores) if scores else 0

Problem 3: Judge cost in production

Symptom: The judge doubles or triples the cost of the evaluation pipeline.

Solution:

# Strategy 1: Sampling — evaluate only 10% of the traffic
import random

def should_judge(rate: float = 0.1) -> bool:
    return random.random() < rate

# Strategy 2: Only evaluate low-confidence cases
def selective_judge(output: str, confidence: float) -> bool:
    return confidence < 0.8  # Only judge if the model isn't sure

# Strategy 3: A smaller model as judge whenever possible
# gpt-4o-mini as a judge is 10x cheaper than gpt-4o

Problem 4: Judge that doesn't follow the requested format

Symptom: The judge returns long text instead of the requested number.

Solution:

import re

def parse_score_robust(raw: str, max_score: float = 10.0) -> float:
    """Robust parser for scores that can arrive in variable formats."""
    
    # Try to extract the number with a regex
    patterns = [
        r'\b(\d+(?:\.\d+)?)/\d+',  # "7/10" or "3.5/5"
        r'\b(\d+(?:\.\d+)?)\b',     # Standalone number
        r'score[:\s]+(\d+)',         # "score: 7"
        r'(\d+)[/\s]*(points)',      # "7 points"
    ]
    
    for pattern in patterns:
        match = re.search(pattern, raw.lower())
        if match:
            score = float(match.group(1))
            return min(score, max_score)  # Clamp to the maximum
    
    # If everything fails, return the middle score
    return max_score / 2.0

Exercises

Exercise 1: Custom rubric for your use case

Create a rubric to evaluate answers from a technical support chatbot. It must evaluate: technical accuracy, clarity for non-technical users, and actionability.

RUBRIC_TECH_SUPPORT = """
Evaluate this technical support answer along 3 dimensions:

1. TECHNICAL ACCURACY (0-3):
   - 0: Incorrect information that could make the problem worse
   - 1: Vague or incomplete information
   - 2: Mostly correct with some missing detail
   - 3: Technically accurate and complete

2. CLARITY (0-3):
   - 0: A non-technical person could not follow the instructions
   - 1: Some parts are clear but there is unexplained jargon
   - 2: Mostly clear, a non-technical person would get it with effort
   - 3: A non-technical person can follow the steps without extra help

3. ACTIONABILITY (0-3):
   - 0: There are no concrete steps the user can follow
   - 1: There is a next step but it isn't specific enough
   - 2: Reasonably specific steps
   - 3: Exact steps, in order, with what to do if each one fails

TOTAL: 0-9 points

Answer:
ACCURACY: X/3
CLARITY: X/3
ACTIONABILITY: X/3
TOTAL: X/9
SUGGESTION: [One concrete improvement for the answer]
"""

def evaluate_tech_support(user_question: str, support_answer: str) -> dict:
    from openai import OpenAI
    import re
    
    client = OpenAI()
    
    prompt = f"""{RUBRIC_TECH_SUPPORT}

User question: {user_question}
Support answer: {support_answer}

Evaluation:"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    
    raw = response.choices[0].message.content
    
    total_match = re.search(r'TOTAL:\s*(\d+)/9', raw)
    suggestion_match = re.search(r'SUGGESTION:\s*(.+)', raw)
    
    return {
        "total": int(total_match.group(1)) if total_match else 0,
        "max": 9,
        "normalized_score": int(total_match.group(1)) / 9 if total_match else 0,
        "suggestion": suggestion_match.group(1) if suggestion_match else "",
        "raw": raw
    }

Exercise 2: Position bias test

Implement a test that demonstrates position bias: take two answers where A is clearly worse, evaluate as A vs B and as B vs A, and check whether there's an inconsistency.

from openai import OpenAI
client = OpenAI()

def test_position_bias():
    question = "What is the capital of France?"
    
    # Clearly better answer
    good_answer = "The capital of France is Paris. It's also the country's largest city."
    
    # Clearly worse answer
    bad_answer = "France has cities. One of them is important."
    
    def compare(p1, p2, labels):
        prompt = f"""Question: {question}
Answer A: {p1}
Answer B: {p2}
Which one is better? Answer: A, B, or TIE"""
        
        r = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
            max_tokens=10
        )
        result = r.choices[0].message.content.strip().upper()
        return result, labels
    
    # Test 1: Good one first (we expect "A")
    r1, e1 = compare(good_answer, bad_answer, {"A": "good", "B": "bad"})
    
    # Test 2: Bad one first (we expect "B", because B is the good one)
    r2, e2 = compare(bad_answer, good_answer, {"A": "bad", "B": "good"})
    
    print(f"Test 1 (good=A, bad=B): Winner = {r1}")  # Should be A
    print(f"Test 2 (bad=A, good=B): Winner = {r2}")  # Should be B
    
    # If A (the bad one) wins Test 2, there's position bias
    if r2 == "A":
        print("⚠️ POSITION BIAS DETECTED: The judge preferred the bad answer when it went first")
    else:
        print("✓ No obvious position bias in this case")

test_position_bias()

Exercise 3: Judge with a 3-call consensus

Implement a judge that makes 3 calls with temperature=0 and only trusts the result if at least 2 out of 3 agree.

from openai import OpenAI
from collections import Counter

client = OpenAI()

def judge_majority(question: str, answer: str) -> dict:
    """Judge with a majority vote (3 calls)."""
    scores = []
    
    prompt = f"""Evaluate whether this answer is correct and relevant.
Scale: 1=bad, 2=fair, 3=good, 4=very good, 5=excellent
Only the number.

Question: {question}
Answer: {answer}
Score:"""
    
    for _ in range(3):
        r = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
            max_tokens=5
        )
        try:
            score = int(r.choices[0].message.content.strip()[0])
            if 1 <= score <= 5:
                scores.append(score)
        except (ValueError, IndexError):
            pass
    
    if not scores:
        return {"score": 3, "confidence": "LOW", "scores": []}
    
    counts = Counter(scores)
    majority_score = counts.most_common(1)[0][0]
    majority_votes = counts.most_common(1)[0][1]
    
    return {
        "score": majority_score / 5.0,
        "score_raw": majority_score,
        "scores": scores,
        "confidence": "HIGH" if majority_votes >= 2 else "LOW",
        "agreement": majority_votes >= 2
    }

# Test
result = judge_majority(
    "What is Python?",
    "Python is a high-level, interpreted, multi-paradigm programming language."
)
print(result)

Summary

• Rubrics: Explicit criteria with a numeric scale and calibration examples (anchoring)
• Comparison-based: More reliable than absolute scoring for comparing prompt variants
• Biases: Length (verbosity), position, self-enhancement, sycophancy — all of them mitigable
• Position bias: Evaluate A vs B and B vs A; only trust it when they agree
• Consensus: Multiple calls or multiple models reduce variance
• Validation: Correlate with human evaluation before using it in production
• Cost: Strategic sampling to cut cost in production

Additional resources

1. JudgeLM: Fine-Tuned Large Language Models are Scalable Judges — Paper on LLM judges
2. G-Eval — NLG framework with LLM-as-judge
3. Judging the Judges: A Systematic Study — Analysis of biases in LLM judges
4. MT-Bench — Benchmark using GPT-4 as judge
5. LangSmith Evaluators — Evaluators in LangSmith