Module 3: The Eval as a Fitness Function

1. Module introduction: the eval as a fitness function

Overview

By the end of this lesson you'll understand the idea that holds up the whole module, and that is the most counterintuitive of the entire guide: a probabilistic component isn't tested with an exact assert, but with an eval-set that produces a score, and that score against a threshold is a gate that governs its deploy. In module 1 you learned that an LLM doesn't give the same output twice, and in module 2 you set it latency and cost budgets. We closed that module with a promise: every AI feature passes three gates —latency, cost, and quality—. Module 2 gave you the first two. This is the third, and it's special because the other two are measured by a stopwatch and a counter, while this one measures something a classic test can't touch: whether the component's response is good enough.

This matters because, without this gate, AI quality is an opinion. "It looks good" isn't a test: it's not reproducible, it doesn't scale, and it doesn't survive a prompt change made on a Friday afternoon. The question "is the response good?" seems impossible to automate —how do you verify something written differently each time?—, and that apparent impossibility is exactly where many teams give up and leave quality to sight and smell. The module's answer is a change in the shape of the test: instead of comparing an output against an exact value, you define an eval-set —a set of cases, each with its success criterion— and run it against the component. The result isn't a "passed / didn't pass" per case, but a score: the fraction of cases that meet the criterion. And that score, compared against an explicit threshold, becomes a binary and verifiable gate —the same jump that in module 2 turned "fast" and "cheap" into budget gates—.

Connection with the module: this lesson is the map, not the territory. Here you build nothing yet; you understand why the six lessons that follow go in the order they go. First the problem: lesson 2 shows why the exact assert breaks against a probabilistic component and what replaces it. Then the tool: lesson 3 defines the anatomy of the eval-set —cases, criterion, score— and runs it. With the score in hand, lesson 4 —the heart of the module— turns it into a gate by comparing it against a threshold, and connects that gate with the fitness function from architecture-decisions. Then the two applications that make the eval something alive: lesson 5 uses it to catch regressions when the prompt or the model changes, and lesson 6 puts it into CI so it blocks the deploy when the score drops. Lesson 7 opens the types of criterion (exact-match, contains, LLM-as-judge, statistical threshold) and warns that the judge is another AI component. Lesson 8 —the project— puts you to setting up the quality gate of a Mercado feature from end to end.

Two analogies: the standardized exam and quality control

Before dropping down to the code, two everyday images you'll recognize in every lesson of the module. Each captures one of the two central ideas.

The standardized exam with an answer key. Think of an admissions exam taken by thousands of students. Each one writes differently —different handwriting, different wording, different order—, and yet the exam can be graded objectively and automatically, because what's evaluated isn't "how nicely each one wrote" but how many answers match the key. The answer key is the success criterion; the whole exam —many questions, each with its key— is the eval-set; and the final grade —"85 out of 100"— is the score. Notice what this idea does: it turns a bunch of unrepeatable, mutually different answers into a single comparable number. That's exactly what you need for a probabilistic component: you can't demand it give the same wording every time, but you can measure what fraction of its responses meets the criterion. The score is the component's grade against its answer key. Keep it: the eval-set is the exam, the criterion is the key, the score is the grade.

The factory's quality control. Now imagine a production line making light bulbs. At the end of the line there's a quality-control station that takes a sample of each batch and counts how many bulbs are defective. There's a written rule: if the defect rate exceeds 2%, the whole batch is rejected —it doesn't go on sale—. It doesn't matter that most bulbs work; the threshold is the threshold, and a batch that crosses it doesn't pass. That station is the quality gate: it takes an aggregate measurement (the defect rate, the equivalent of the score) and compares it against a threshold to give a binary verdict —the batch passes or is rejected—. In AI, that station is the eval gate: it takes the component's score and compares it against the quality threshold; above the threshold, the component goes to production (deploy allowed); below, it's blocked. And the most powerful part: that station can be on the line all the time, checking each batch automatically. That's where the eval becomes a CI gate —lesson 6—: every change to the component goes through the station before reaching the customer.

Keep the two. The standardized exam is how you measure a component that responds differently each time —you grade it against a key and get a score—. The quality control is how that score becomes a deploy decision —you compare it against a threshold and it passes or is blocked—. The whole module is learning to set up those two things over a real feature.

The case: Mercado's support agent

Let's drop down to Mercado, the ecosystem's marketplace. Of its AI features, this module's protagonist is the customer support agent: a conversational assistant that answers frequent questions —"where's my order?", "how do I return a product?", "how long does shipping take?"—. It's a perfect candidate for the eval for a concrete reason: its responses are verifiable. For "how long does shipping take?" there's a correct answer —"3 to 5 business days"— that the agent must communicate, even if it words it differently each time. That gives us a clear success criterion: the response must contain the correct information.

The agent's eval-set is a set of those frequent questions, each with the key phrase the correct answer must contain. Running it against the agent tells us what fraction of the questions it answers well —the score—. And when someone changes the agent's prompt, or proposes dropping it to a cheaper model to save (module 2's cascade), we'll run the eval again: if the score dropped, the change regressed the quality and must not enter. The semantic search —the other protagonist feature of module 2— appears in the exercises, because its eval-set is built the same way (queries with the products that should come out) even though its criterion is different.

We're not going to build the agent nor decide which frequent questions its eval-set should cover —that's eval design, and it's AI Engineering—. We're going to take the eval-set as given and use it as a quality gate: run it, measure the score, compare it against the threshold, and let the gate decide whether a change gets deployed.

And to start, let's see the whole module condensed into an executed block. Look closely, because here's the thesis in action.

# Lesson 01 (intro) — the module in miniature: the eval as a deploy gate
# Everything SIMULATED. Zero network, zero API, zero keys. Deterministic output.

# The support agent's eval-set: each case is a question + the phrase
# the correct answer must contain (the success criterion).
# BOUNDARY NOTE: how these cases and this criterion are CHOSEN is AI Engineering.
# Here the eval-set ALREADY exists; we use it AS A GATE.
EVAL_SET = [
    {"id": "q1",  "question": "where is my order",         "must_contain": "tracking"},
    {"id": "q2",  "question": "how do i return a product",  "must_contain": "return"},
    {"id": "q3",  "question": "how long does shipping take", "must_contain": "3 to 5 days"},
    {"id": "q4",  "question": "can i pay in installments",  "must_contain": "installments"},
    {"id": "q5",  "question": "the product arrived broken", "must_contain": "refund"},
    {"id": "q6",  "question": "how do i change my address", "must_contain": "profile"},
    {"id": "q7",  "question": "i didn't get my invoice",    "must_contain": "email"},
    {"id": "q8",  "question": "i want to cancel my order",  "must_contain": "cancel"},
    {"id": "q9",  "question": "the coupon doesn't work",    "must_contain": "expiration"},
    {"id": "q10", "question": "how do i contact a seller",  "must_contain": "messages"},
]

# The correct answer ("gold") per case. Each contains the criterion phrase.
GOLD = {
    "q1":  "You can see the tracking of your order in your profile.",
    "q2":  "For a return, go to your order and click Return.",
    "q3":  "Standard shipping takes 3 to 5 days.",
    "q4":  "Yes, you can pay in installments with no interest by card.",
    "q5":  "We're sorry about that; you can request a refund from the order.",
    "q6":  "Change your address in the Profile, Addresses section.",
    "q7":  "We'll resend the invoice to your account email.",
    "q8":  "You can cancel the order if it hasn't shipped yet.",
    "q9":  "Check the coupon's expiration; it may have already expired.",
    "q10": "Message the seller from the Messages section.",
}
POOR_ANSWER = "Sorry, I don't have information about that."

def make_agent(competent_ids):
    # LLM STUB: simulates a version of the agent. Answers well the cases in
    # competent_ids; for the rest it gives a poor answer that fails the criterion.
    def agent(question, case_id):
        return GOLD[case_id] if case_id in competent_ids else POOR_ANSWER
    return agent

def run_eval(agent):
    passed = sum(
        case["must_contain"] in agent(case["question"], case["id"]).lower()
        for case in EVAL_SET
    )
    return passed / len(EVAL_SET)          # the score: fraction of cases that pass

def gate(score, threshold):
    return ("PASS", "deploy ALLOWED") if score >= threshold else ("FAIL", "deploy BLOCKED")

THRESHOLD = 0.80
ALL = {c["id"] for c in EVAL_SET}
version_a = make_agent(ALL - {"q9"})                          # good: 9/10
version_b = make_agent({"q1","q2","q3","q4","q6","q8"})       # regressed: 6/10

print(f"support agent eval gate — threshold = {THRESHOLD:.2f}\n")
print(f"{'version':<22}{'score':>7}   {'gate':<6}  result")
for name, agent in (("A (candidate)", version_a), ("B (prompt change)", version_b)):
    s = run_eval(agent)
    verdict, action = gate(s, THRESHOLD)
    print(f"{name:<22}{s:>7.2f}   [{verdict}]  {action}")

What to expect. When you run it:

support agent eval gate — threshold = 0.80

version                 score   gate    result
A (candidate)            0.90   [PASS]  deploy ALLOWED
B (prompt change)        0.60   [FAIL]  deploy BLOCKED

Stop at those two rows, because they're the whole module in miniature. Both are the same feature —Mercado's support agent— in two versions. Version A answers 9 of the eval-set's 10 questions well: its score is 0.90, above the quality threshold of 0.80, so the gate lets it through —deploy allowed—. Version B —maybe someone "improved" the prompt, or dropped it to a cheaper model— answers only 6 of 10 well: its score falls to 0.60, below the threshold, and the gate blocks it —deploy denied, in red—. Nobody had to read the responses one by one and opine on whether they were good. The gate decided it with a number against a threshold, just as the quality-control station rejects the batch that crosses 2% defects.

Don't understand yet how each piece works —that's what the lessons are for—. Keep the shape of the result: two versions of the same probabilistic component, one that passes the quality gate and another that fails it, decided by a score against a threshold, without a human having to judge by eye. That's exactly what "the eval as a fitness function" means: an automated test that governs whether a change to the component's quality can enter production.

The map of the six lessons

The six lessons that follow go in this order because each assembles the piece the next one needs.

Lesson	What it gives you	Why it goes here
2	The problem: why the exact `assert` breaks and what replaces it (score, not binary per case)	You can't build the gate without understanding why the classic test doesn't work
3	The anatomy: eval-set (cases + criterion), running it, the score	The base tool; everything else operates on the score this lesson produces
4	The gate: score against threshold = the component's fitness function	The heart of the module: where the score becomes a deploy decision
5	The regression: the eval detects when a prompt/model change lowers the quality	The gate comes to life: it protects against change, which is when quality breaks
6	The eval in CI: a score below the threshold blocks the deploy, like a red test	Turns the gate into something the pipeline enforces on its own, on every change
7	The types of criterion: exact-match, contains, LLM-as-judge, statistical threshold	Opens the box of the "success criterion" and warns that the judge is another AI component

The arc is: first you understand why the classic test doesn't apply (2), then you build the tool that replaces it —the eval-set and its score (3)—, then you turn it into a gate with a threshold (4), and with the gate ready you put it to work: it catches regressions (5) and lives in CI blocking deploys (6). Lesson 7 opens the types of criterion so you know what governs each one. Lesson 8 —the project— brings it all together over a Mercado feature you architect from scratch, so you confirm you learned the method and didn't memorize a table.

What this module does NOT touch

It's worth marking the boundary from now, because there are neighboring topics that look like they belong here and belong to another part of the ecosystem. This boundary is hard: cross it only to refer.

How to design good evals is AI Engineering, not here. This is the module's most important boundary, so read it carefully. Building a good eval-set is a deep craft: choosing representative cases that cover what matters (and not just the easy stuff), avoiding the bias of testing only what you already know works, measuring the relevance of a semantic search with serious metrics, deciding how many cases suffice, calibrating a judge so its grades correlate with human judgment, versioning the dataset as the feature evolves. All of that is eval design, it's a whole body of knowledge, and it's from the AI Engineering ecosystem. This module doesn't teach it: in these lessons the eval-set already exists —we give it to you ready-made— and your job is to use it as an architectural gate: run it, measure the score, compare it against the threshold, and govern the deploy with the result. The mental rule: if the question is "which cases and which criterion do I put in the eval-set?", it's AI Engineering; if the question is "how do I use the eval-set's score to decide whether this gets deployed?", it's from here.

The fitness function as a general concept is from architecture-decisions. This whole module says the eval is a fitness function —an automated test that verifies a system property (here, the quality of a probabilistic component) holds, and stops the change if it degrades—. But the general idea of a fitness function, applied to any architectural property (latency, coupling between layers, a module's size, forbidden dependencies), is taught in architecture-decisions-and-tradeoffs-guide (M6). Here we take that concept as known and specialize it to AI quality. If you want the complete framework of fitness functions as an architectural governance technique, that's the guide; here you see its version for probabilistic components.

The mechanics of CI/CD in depth are from another guide. In lesson 6 the eval lives in a pipeline and produces an exit code that blocks the deploy. But how a serious CI/CD pipeline is assembled —the runners, the stages, the environments, the rollback, the secrets— is from the delivery and infrastructure guides. Here the pipeline is the minimum needed to show the eval as the step that governs the deploy: a gate that passes or fails. We don't build the pipeline; we show where the gate fits.

Common mistakes

Testing the AI component with an exact assert (a mental-model mistake). What happens: the team writes tests as with any code —assert agent(question) == "expected answer"— and the test breaks on the first run because the LLM worded it differently. The wrong conclusion they draw is "AI can't be tested," and they abandon testing entirely. Why it happens: the exact-equality test is every programmer's reflex, and it works with deterministic code, so it's applied out of habit. How to spot it: if your test suite for an AI component uses == against a literal output, it's doomed to break or to be uselessly fragile. How to fix it: it's the whole module —change the shape of the test from a binary per case to a score over an eval-set (lessons 2 and 3)—.

Having no eval-set and "testing by eye" (an omission mistake). What happens: the component's quality is verified by looking at a few responses by hand before each release and saying "it looks good." It's not reproducible (everyone looks at different cases), it doesn't scale (nobody reviews hundreds of cases by hand), and it doesn't catch regressions (the change that broke the case nobody looked at passes quietly). Why it happens: setting up an eval-set costs effort up front, and "by eye" gives the illusion of verifying. How to spot it: if you can't say in a number how good your component is today, you're not measuring it —you're feeling it—. How to fix it: an eval-set with a criterion and a score (lesson 3), which is reproducible and runs on its own.

Changing the prompt or the model without running the eval (a process mistake). What happens: someone tweaks the agent's prompt to fix a case, or drops it to a cheaper model to save, and deploys it without running the eval. The change fixed what it was after but broke three things nobody reviewed —a silent regression—, and it's discovered weeks later through customer complaints. Why it happens: a prompt change feels harmless ("it's just text"), and without a gate that verifies it, nothing stops the deploy. How to spot it: if your prompt or model changes reach production without a score backing them, you're flying blind. How to fix it: the eval as a gate on every change (lesson 5) and in CI so it's mandatory (lesson 6).

Exercises

Exercise 1 — Translate the analogies to design. For each analogy of the module, say which piece of the eval it represents and give a concrete example in Mercado's support agent. (a) The standardized exam with its answer key. (b) The exam's final grade ("85 out of 100"). (c) The quality control that rejects the batch if the defect rate crosses 2%.

See solution

(a) The exam with its key → the eval-set with its success criterion. The exam is the set of cases (the agent's frequent questions), and the answer key is the criterion (the phrase each answer must contain). Example in Mercado: the question "how long does shipping take?" is a case, and its key is that the answer contain "3 to 5 days". The complete eval-set is the ten questions with their ten keys.
(b) The final grade → the score. It's the aggregate: what fraction of the cases met their criterion, condensed into a comparable number. Example in Mercado: if the agent answers 9 of the 10 questions well, its score is 0.90 —the component's grade against its key, no matter that it worded each answer differently—.
(c) The quality control with its threshold → the eval gate. It takes the aggregate measurement (the score) and compares it against a threshold to give a binary deploy verdict. Example in Mercado: with a threshold of 0.80, the 0.90 version passes (deploy allowed) and a 0.60 one is rejected (deploy blocked), just like the batch that crosses 2% defects.

The important thing: the exam is how you measure something that responds differently each time (score against key), and the quality control is how that score becomes a deploy decision (score against threshold).

Exercise 2 — The dangerous phrase. A colleague proposes testing the support agent like this: "Easy: I write a test that sends it 'how long does shipping take?' and verify it responds exactly 'Standard shipping takes 3 to 5 business days.'. If it matches, it passes." Identify the underlying mistake and explain what test they should write instead.

See solution

The underlying mistake is testing a probabilistic component with exact equality. The agent is an LLM: next time you send it the same question it may respond "Your order usually arrives in 3 to 5 days" —equally correct, worded differently—. The assert output == "..." would break, not because the response is wrong, but because it isn't identical letter by letter. The test would be fragile (it breaks with any valid rewording) or, worse, it would push you to force the model to repeat a fixed phrase, losing the naturalness that justifies using an LLM.

What they should write instead: a property test, not an equality one. The success criterion is that the response contain the correct information —the key phrase "3 to 5 days"—, no matter how it words it. That way both "Standard shipping takes 3 to 5 days" and "Your order usually arrives in 3 to 5 days" pass, because both communicate the correct fact. And even better: that case doesn't live alone, it lives inside an eval-set of many questions, and what matters isn't a case's verdict but the score of the set against the threshold. It's exactly lesson 2's change of shape.

Exercise 3 — Eval design or eval as a gate? For each activity, say whether it falls within this module (the eval as an architectural gate) or on the AI Engineering boundary (designing the eval), and why. (a) Running the eval-set and blocking the deploy if the score falls below 0.80. (b) Deciding which 200 customer questions must be in the eval-set for it to be representative. (c) Integrating the eval as a CI step that runs on every PR. (d) Calibrating a judge model so its grades correlate with those of a human evaluator.

See solution

(a) Block the deploy if the score falls → this module. It's using the score as a deploy gate: the essence of the eval as a fitness function. Lessons 4 and 6.
(b) Choose the 200 representative questions → boundary (AI Engineering). It's eval-set design: which cases compose it, how to ensure coverage and representativeness. It's a deep craft and it's not from here. This module takes the eval-set as given.
(c) Integrate the eval into CI → this module. It's putting the gate into the pipeline so it governs the deploy automatically. Lesson 6.
(d) Calibrate the judge model → boundary (AI Engineering). Making an LLM-as-judge's grades correlate with human judgment is design and calibration of the criterion, not use of the gate. This module uses the judge as a component (lesson 7), but calibrating it is AI Engineering.

The rule that separates: if the activity builds or tunes the eval-set or its criterion (b, d), it's AI Engineering; if it uses the eval-set's score to govern the deploy (a, c), it's from here.

Summary and next step

In this lesson you met the module's thesis: a probabilistic component isn't tested with an exact assert, but with an eval-set that produces a score, and that score against a threshold is the quality gate that governs its deploy —the probabilistic equivalent of the fitness function—. You saw why this is the third gate every AI feature needs, alongside module 2's latency and cost ones. The two analogies gave you the frame: the standardized exam with an answer key (how you measure something that responds differently each time: you grade it against a criterion and get a score) and the factory's quality control (how that score becomes a decision: you compare it against a threshold and the batch passes or is rejected). And in the executed block you saw the thesis in action: the same feature in two versions, one that passes the quality gate (score 0.90 → deploy allowed) and another that regressed and fails it (score 0.60 → deploy blocked), decided by a number against a threshold, without a human judging by eye.

Before moving on you should be able to: explain why an AI component's quality can't be left "to the eye"; name the three pieces of the eval (eval-set, criterion, score) and the fourth that turns them into a gate (the threshold); and separate the eval as a gate (this module) from eval design (AI Engineering), which is outside the boundary.

What follows is understanding in depth why the classic test doesn't work, because until you see the exact assert break you won't appreciate why you need a score. In lesson 2 you're going to execute the failed attempt —an assert output == expected that works with a normal function and breaks against a component that gives different wordings— and you're going to see the change of shape that replaces it: from a binary test per case to an aggregate score over an eval-set with a property criterion. It's the step from "I know AI isn't tested with ==" to "I know exactly what it's tested with, and why."

Resources

Anthropic — Claude docs, application evaluation (conceptual) — the conceptual guide to why and how to evaluate an LLM application: define cases with a success criterion and measure against them instead of against an exact output; the basis of the eval-set idea, without fixing a model version.
martinfowler.com — "Emerging Patterns in Building GenAI Applications", Bharani Subramaniam and Martin Fowler — the catalog of architecture patterns for LLM apps, with the evals pattern treated as a design piece that governs changes; the essayistic frame of the whole module.
Chip Huyen — AI Engineering (O'Reilly), evaluation chapters — the systematic treatment of AI-system evaluation; the go-to book for designing the evals this module takes as given (the boundary with AI Engineering).
architecture-decisions-and-tradeoffs-guide — Fitness functions (M6) — the general concept of a fitness function as an automated test that governs an architectural property; this module specializes it to the quality of a probabilistic component.

← Previous8. Project: design the budget for a Mercado AI feature Next →2. Why you can't assert a probabilistic component