Module 3: The Eval as a Fitness Function

3. The eval-set and the score

Next →

Overview

By the end of this lesson you'll have built the base tool of the whole module: the eval-set and the score it produces. In lesson 2 you saw the change of shape —from exact equality to a property criterion, and from a boolean per case to an aggregate score— with three runs of a single question. Here you formalize it: an eval-set isn't one case, it's a set of cases, each with its input and its success criterion; running it against the component means grading each case; and the result is a score —the fraction of cases that passed their criterion—. You're going to see Mercado's support agent eval-set executed case by case: ten frequent questions, each with the phrase its answer must contain, run against a version of the agent, with the detail of which pass and which fail, and the final score: 9 out of 10, 0.90. When you finish, you'll have in your hands the piece the four remaining lessons operate on.

This matters because the score is the module's currency. Lesson 4's gate compares a score against a threshold; lesson 5's regression detection compares the score of a change against a baseline's; lesson 6's CI gate puts the score into an exit code. Everything rests on this number, and this number doesn't appear by magic: it comes from a very concrete structure —cases with a criterion— run in a very concrete way —each case is graded, the results are aggregated—. If you understand the anatomy well here, the rest of the module is applying this piece in different contexts. And there's a reproducibility reason: unlike "testing by eye" —which gives a different result depending on who looks and which cases they pick—, a run eval-set produces the same score every time, because the cases and the criterion are fixed. That's what makes it fit for a gate: a gate needs a reliable number, not an impression.

Connection with the module: this lesson is the one that manufactures the raw material. Lesson 2 told you why you need a score; this one gives you how it's built. What follows consumes it: lesson 4 puts a threshold on it and turns it into a gate; lesson 5 compares scores between versions to catch regressions; lesson 6 puts it into the pipeline. Here also appears, marked clearly, the module's hard boundary: you'll see the eval-set comes given —ten questions already chosen, with their criterion already defined— and that how those cases and that criterion are chosen (coverage, representativeness, bias) is eval design, AI Engineering's. Our work begins when the eval-set already exists.

The analogy: the standardized exam with its answer key

Go back to the standardized exam millions of students take on the same day. Look at it in parts, because each part is a piece of the eval-set. The exam has many questions —not just one—, and each question carries its correct answer in the teacher's key. When a student answers it, a grader goes question by question: it compares the student's answer against that question's key and marks right or wrong. At the end, it doesn't report "the student got question 7 right"; it reports a grade: "85 out of 100." That grade is a single number that summarizes the performance on the whole exam.

Translate each part. The whole exam —the set of questions with their keys— is the eval-set. Each question with its key is a case (an input with its success criterion). Grading question by question is running the component against each case and applying the criterion. And the final grade —"85 out of 100", or 0.85— is the score. Notice three properties the exam makes clear and the eval-set inherits. First: the grade is objective and reproducible —two graders with the same key give the same grade—. Second: it's an aggregate —it doesn't matter how well the student worded a single question, it matters how many they got right in total—. Third: a grade neither passes nor fails on its own —you need a threshold, "you pass with 60," which is lesson 4's gate—. The exam produces the grade; the threshold decides what to do with it. This lesson builds the exam and gets the grade; the next puts the threshold.

Worked example: the support agent's eval-set, case by case

We're going to build Mercado's support agent eval-set and run it against a version of the agent, seeing the case-by-case detail and the final score. The eval-set is ten frequent questions, each with the key phrase (must_contain) its correct answer must contain. The version of the agent we test answers nine of the ten well —it fails exactly the coupon one—, so you see a realistic score, not a lab 1.00.

Remember the boundary: these ten questions and their criteria come given. Why these ten and not others, whether they cover what matters well, whether the contains criterion is right for each —all of that is eval design, AI Engineering's—. Here the eval-set already exists and we run it.

# Lesson 03 — the anatomy of the eval-set and the score
# Everything SIMULATED. Zero network, zero API, zero keys. Deterministic output.

# --- THE EVAL-SET: each case = input (question) + success criterion (must_contain) ---
# BOUNDARY NOTE: how these cases and criteria are CHOSEN is AI Engineering.
# Here the eval-set ALREADY exists; our job is to run it and get the score.
EVAL_SET = [
    {"id": "q1",  "question": "where is my order",         "must_contain": "tracking"},
    {"id": "q2",  "question": "how do i return a product",  "must_contain": "return"},
    {"id": "q3",  "question": "how long does shipping take", "must_contain": "3 to 5 days"},
    {"id": "q4",  "question": "can i pay in installments",  "must_contain": "installments"},
    {"id": "q5",  "question": "the product arrived broken", "must_contain": "refund"},
    {"id": "q6",  "question": "how do i change my address", "must_contain": "profile"},
    {"id": "q7",  "question": "i didn't get my invoice",    "must_contain": "email"},
    {"id": "q8",  "question": "i want to cancel my order",  "must_contain": "cancel"},
    {"id": "q9",  "question": "the coupon doesn't work",    "must_contain": "expiration"},
    {"id": "q10", "question": "how do i contact a seller",  "must_contain": "messages"},
]

# The correct answer ("gold") per case; each contains the criterion phrase.
GOLD = {
    "q1":  "You can see the tracking of your order in your profile.",
    "q2":  "For a return, go to your order and click Return.",
    "q3":  "Standard shipping takes 3 to 5 days.",
    "q4":  "Yes, you can pay in installments with no interest by card.",
    "q5":  "We're sorry about that; you can request a refund from the order.",
    "q6":  "Change your address in the Profile, Addresses section.",
    "q7":  "We'll resend the invoice to your account email.",
    "q8":  "You can cancel the order if it hasn't shipped yet.",
    "q9":  "Check the coupon's expiration; it may have already expired.",
    "q10": "Message the seller from the Messages section.",
}
POOR_ANSWER = "Sorry, I don't have information about that."

def make_agent(competent_ids):
    # LLM STUB: answers well the cases in competent_ids; for the rest,
    # a poor answer that fails the criterion. Deterministic, no real API.
    def agent(question, case_id):
        return GOLD[case_id] if case_id in competent_ids else POOR_ANSWER
    return agent

def criterion_contains(output, must_contain):
    # The success criterion: the output contains the expected phrase.
    return must_contain.lower() in output.lower()

def run_eval(agent):
    # Runs the agent against EACH case, grades, and aggregates into a score.
    passed, detail = 0, []
    for case in EVAL_SET:
        out = agent(case["question"], case["id"])
        ok = criterion_contains(out, case["must_contain"])
        passed += ok
        detail.append((case["id"], ok))
    total = len(EVAL_SET)
    return passed, total, passed / total, detail

# The version under test: competent in 9 of 10 (fails q9, the coupon).
ALL_IDS = {c["id"] for c in EVAL_SET}
agent = make_agent(ALL_IDS - {"q9"})

passed, total, score, detail = run_eval(agent)
print(f"{'case':<6}{'must_contain':<16}{'result'}")
for case in EVAL_SET:
    out = agent(case["question"], case["id"])
    ok = criterion_contains(out, case["must_contain"])
    print(f"{case['id']:<6}{case['must_contain']:<16}{'PASS' if ok else 'FAIL'}")
print(f"\npassed = {passed}/{total}   score = {score:.2f}")

What to expect. When you run it:

case  must_contain    result
q1    tracking        PASS
q2    return          PASS
q3    3 to 5 days     PASS
q4    installments    PASS
q5    refund          PASS
q6    profile         PASS
q7    email           PASS
q8    cancel          PASS
q9    expiration      FAIL
q10   messages        PASS

passed = 9/10   score = 0.90

Here's the eval-set run, and here's the complete anatomy in view. Read it on two levels, which is the key of the lesson.

The case level. Each row is a graded case: the question was passed to the agent, the response was compared against the criterion, and PASS or FAIL came out. Nine rows say PASS —the agent responded with the key phrase— and one, q9 (the coupon), says FAIL —the agent gave the poor answer "I don't have information," which doesn't contain "expiration"—. This level is diagnostic: it tells you exactly what failed. If tomorrow you want to fix the agent, you know the problem is in the coupon questions, not in the shipping ones. The per-case detail is your map of where it hurts.

The score level. The last line aggregates the ten cases into a single number: passed = 9/10, score = 0.90. This level is decisional: it's the number the gate will compare against a threshold. Notice what the aggregate did: it turned ten binary verdicts into a degree —0.90— that summarizes the component's quality in a comparable figure. If another version got 0.60, you'd instantly know it's worse, even without looking at any individual case. The score is what makes two versions of a probabilistic component comparable.

The two levels coexist and serve different things. The score decides (does it pass the gate?); the detail diagnoses (what do I fix?). A good eval system reports both: the number for the gate and the list of failures for the team. But for the rest of the module, the protagonist is the score —0.90—, because it's what the gate consumes.

Going deeper: the anatomy, the aggregation, and the boundary

The three pieces of the eval-set. An eval-set has exactly three pieces, and it's worth naming them so as not to confuse them. First, the cases: the inputs you test the component with (the ten questions). Second, the success criterion: the rule that decides whether the response to a case is right (here, contains(must_contain)). Third, the aggregation: how you combine the per-case verdicts into a score (here, the fraction that passes). Change any of the three and the eval changes. In this module we vary mostly the criterion (lesson 7, different types) and always use the simplest aggregation —the fraction that passes, or pass-rate—; the choice of cases comes given.

Why the pass-rate is the default aggregation (and not the only one). Aggregating by "fraction of cases that pass" is the simplest form and the one we'll use throughout the module: each case counts equally, it passes or it doesn't, and the score is how many passed out of the total. It's intuitive and enough for a gate. But it isn't the only way to aggregate, and it's worth knowing: you could weight cases (the critical ones count more), average a continuous score per case (when the criterion gives a degree, not a binary —lesson 7's LLM-as-judge gives 0.4, not just 0 or 1—), or require that no critical case fail (a veto, not an average). Which aggregation to use is, again, eval design (AI Engineering). Here we use pass-rate because it's the one that makes the gate's mechanics most transparent.

Why the eval-set is reproducible and "by eye" isn't. The value of this structure, versus reviewing responses by hand, is that it produces the same score every time. The cases are fixed (always the same ten questions), the criterion is fixed (always contains), and the aggregation is fixed (always pass-rate). Run the eval today, tomorrow, on your machine or in CI: if the component didn't change, the score is identical. "Testing by eye" doesn't have that property —each person looks at different cases, with a different mental criterion, and gets a different impression—. A gate needs a number you can trust to make a deploy decision, and only a fixed structure gives it. This is the deep reason the module insists on the eval-set: it's not that "by eye" never works, it's that it isn't reproducible, and a gate without reproducibility isn't a gate.

The boundary, again, precisely. All this lesson does is run a given eval-set and aggregate a score. What it does NOT do —and is AI Engineering's— is decide the eval-set's content: which questions to include so they cover the cases that matter and not just the easy ones, how many suffice for the score to be reliable, how to avoid the bias of putting in only cases you already know the agent answers well (which would inflate the score without measuring anything), and which success criterion truly captures each task's quality. A badly designed eval-set gives a score that looks like a measurement but isn't —an exam with only trivial questions passes anyone—. Recognizing that the eval-set can be badly designed is part of knowing how to use it; designing it well is the other guide. This module assumes a reasonable eval-set and concentrates on what follows: turning its score into a gate.

Common mistakes

Confusing the per-case detail with the score (a level mistake). What happens: the team looks at the PASS/FAIL list and makes decisions case by case —"let's fix q9 and done"— without looking at the aggregate score, or the reverse, looks only at the score and loses the diagnosis of what failed. Why it happens: the two levels coexist in the same output and it's easy to stay on one. How to spot it: if you can't say at once "the score is 0.90" and "what fails is the coupon question," you're using only one level. How to fix it: use each level for its purpose —the score for the gate (does the deploy pass?), the detail for debugging (what do I fix?)—; both are part of the eval's report.

An eval-set with only easy cases (a design mistake, on the boundary). What happens: the eval-set is filled with questions the agent already answers well, the score comes out very high (0.98) and the team concludes the quality is excellent —when in reality it tested no hard case—. Why it happens: it's natural to put in cases we "know work" and it gives the satisfaction of a high score. How to spot it: if your eval-set never gives an intermediate or low score, suspect it's too easy, not that your component is perfect. How to fix it: this is eval design (AI Engineering) —coverage, hard cases, representativeness—; but at this module's level, the warning sign is an eval-set that always passes (you'll see it as the "useless fitness function" in lesson 4).

Aggregating badly: mixing cases that don't count equally (an aggregation mistake). What happens: the eval-set mixes trivial cases with critical cases and averages them equally with pass-rate, so that failing a critical case (a badly explained refund) weighs the same as failing a trivial one, and the score doesn't reflect the real risk. Why it happens: simple pass-rate treats all cases as equal, which is convenient but not always correct. How to spot it: if a serious failure and a minor one move the score exactly the same, your aggregation doesn't distinguish what matters. How to fix it: the aggregation can weight or veto (eval design, AI Engineering); in this module we use pass-rate for simplicity, but knowing it isn't the only way to aggregate.

Exercises

Exercise 1 — Compute the score. An eval-set has 8 cases. When run against a version of the agent, the detail is: PASS, PASS, FAIL, PASS, PASS, PASS, FAIL, PASS. (a) What's the score with pass-rate aggregation? (b) If the gate's threshold were 0.80, would it pass? (c) What does the detail tell you that the score alone doesn't?

See solution
  • (a) Score: 6 PASS of 8 cases = 6/8 = 0.75.
  • (b) Does it pass the 0.80 threshold? No: 0.75 < 0.80, the gate fails (deploy blocked). The score falls just below the threshold.
  • (c) What the detail adds: the score (0.75) tells you how much fails, but the detail tells you what fails —cases 3 and 7—. To fix the component you need the detail: you know the problem is in those two concrete cases, not spread across the whole eval-set. The score decides (it doesn't pass the gate); the detail guides the fix (look at cases 3 and 7). The two levels, each for its job.

Exercise 2 — Change the criterion, change the score. Take case q3 ("how long does shipping take?", must_contain = "3 to 5 days"). The agent responds: "Shipping usually arrives fairly fast, in a few days." Give the verdict of this case under two criteria: (a) contains("3 to 5 days") and (b) contains("days"). Explain why the eval-set's score depends not only on the component, but on the chosen criterion.

See solution
  • (a) contains("3 to 5 days"): the response "in a few days" does not contain "3 to 5 days" → FAIL. The criterion demands the specific fact (the range), and the response was vague.
  • (b) contains("days"): the response does contain "days" → PASS. The criterion only demands that "days" be mentioned, and the response does, even though it doesn't give the correct range.

The moral: the same response from the same component gives FAIL with one criterion and PASS with another. That means the eval-set's score doesn't measure only how good the component is —it measures how good it is according to the criterion you chose—. A loose criterion (contains("days")) approves a vague response and inflates the score; a strict criterion (contains("3 to 5 days")) demands the correct fact. That's why choosing the criterion well is so important —and it's eval design, AI Engineering—. In this module the criterion comes given; but understanding that the score depends on it is key to not blindly trusting a high number.

Exercise 3 — What's from the eval as a gate and what from its design? You have to put a quality gate on Mercado's semantic search. For each task, say whether it's from this module (use the eval-set as a gate) or from the boundary (design the eval-set), and why. (a) Running the search's eval-set and computing its score. (b) Choosing 100 queries representative of the real traffic for the eval-set. (c) Comparing the score against a 0.85 threshold to decide the deploy. (d) Defining that the success criterion be "the correct product is among the top three results".

See solution
  • (a) Run the eval-set and compute the score → this module. It's executing the tool and producing the number. This lesson's mechanics.
  • (b) Choose the 100 representative queries → boundary (AI Engineering). It's eval-set design: which cases compose it so it covers the real traffic without bias. Not from here.
  • (c) Compare the score against the threshold → this module. It's turning the score into a deploy decision: the gate (lesson 4).
  • (d) Define the success criterion ("among the top three") → boundary (AI Engineering). It's designing what makes a response count as correct for this task. Choosing the metric is eval design.

The rule that separates: building the eval-set —its cases and its criterion— (b, d) is AI Engineering; running it and using its score to decide (a, c) is from here. This module begins when the eval-set already exists.

Summary and next step

In this lesson you built the module's base tool: the eval-set and the score. With the standardized-exam analogy you saw its three pieces —the cases (the questions), the success criterion (the answer key), and the aggregation (the final grade)— and the two properties it inherits: it's objective and reproducible (same eval-set, same score every time), and it's an aggregate (it matters how many cases pass, not the wording of a single one). You executed Mercado's support agent eval-set case by case and saw the two levels of its output: the per-case detail (nine PASS and one FAIL on the coupon), which diagnoses what fails, and the aggregate score (9/10 = 0.90), which decides —the number the gate consumes—. And you marked the boundary precisely: running and aggregating is from here; choosing the cases and the criterion is eval design, AI Engineering's.

Before moving on you should be able to: name the three pieces of an eval-set (cases, criterion, aggregation); explain the difference between the case level (diagnosis) and the score level (decision); justify why an eval-set is reproducible and "by eye" isn't; and recognize that the score depends on the criterion, not only on the component.

What follows is giving that score power. So far it's just a number —0.90— that describes the component but decides nothing. In lesson 4 you're going to put a threshold on it and turn it into a gate: the heart of the module. You're going to execute the eval_gate over two versions of the agent —one that passes (0.90 ≥ 0.80 → deploy allowed, in green) and another that regressed (0.60 < 0.80 → deploy blocked, in red)— and you're going to see why that gate is, exactly, a fitness function for the quality of a probabilistic component. It's the step from "I have a score" to "the score governs the deploy."

Resources

← Previous2. Why you can't assert a probabilistic componentNext →4. The eval as a quality gate (fitness function)