Module 3: The Eval as a Fitness Function

7. Types of eval criterion

Next →

Overview

By the end of this lesson you'll open the box the module took for granted until now: the success criterion. In all the previous lessons the criterion was the same —contains, does the response contain the key phrase?—, because it's the simplest and let the gate's mechanics show without distractions. But contains is only one of several types of criterion, and the choice has architectural consequences, not just precision ones. You're going to execute four criteria over the same cases and see how each gives a different score for the same real quality: exact-match (too strict, fails correct paraphrases), contains (binary, catches the paraphrase but doesn't grade), LLM-as-judge (a judge that gives partial credit, more nuanced), and the statistical threshold (which operates on the aggregate score, not case by case). And you're going to run into the module's most important warning, the one that connects this module with all the others: the LLM-as-judge is another AI component, with its own latency, cost, and non-determinism, so judging quality with it recurses all the properties of this guide.

This matters because the criterion is what decides what counts as "a good response," and a gate is no better than its criterion. A badly chosen criterion produces a score that looks like a measurement but isn't: exact-match fails correct answers and makes the gate uselessly strict; a loose contains approves bad answers and makes the gate blind. Understanding the types of criterion at an architectural level —what each one governs, what fails in each one, what each one costs— is what lets you reason about a gate instead of blindly trusting its number. And the LLM-as-judge case is especially loaded, because it's tempting: "let's use a powerful model to judge whether the cheap model's response is good" sounds perfect, until you remember that the judge is an LLM —slow, expensive, non-deterministic, fallible— and that now you have two AI components instead of one, with the uncomfortable question of who evaluates the judge.

Connection with the module: this is the last topical lesson, and it closes the module by opening its most intimate piece. Lessons 2 to 6 built the complete gate —score (3), gate (4), regression (5), CI (6)— always with contains as the fixed criterion. Here you generalize that criterion to a family, and you see that the choice between its members is a design decision with precision and cost trade-offs. The judge-as-component warning reconnects you with the whole guide: latency and cost (module 2), non-determinism and probabilistic contract (module 1), trust boundary (module 4) —the judge has all those properties—. Here the hard boundary is also marked, for the last time: how to choose and calibrate the right criterion for a task is eval design, AI Engineering's; this lesson gives you the map of the types and their architectural consequences, not the recipe for which to use.

The analogy: four ways to grade the same answer

Go back to the exam, but now look at who grades and how. Imagine three students answer "how long does shipping take?" and you want to grade their answers. There are several ways, and each is a different criterion.

The first is the multiple-choice scanner: it only accepts the answer if it's identical, letter for letter, to the key. It's exact-match. Lightning-fast and objective, but it fails anyone who says the right thing in other words —useless for free responses—. The second is a grader with a list of keywords: it doesn't demand the exact phrase, it only verifies that a certain term appears ("did they mention '3 to 5 days'?"). It's contains. More flexible than the scanner, but binary and somewhat crude: it doesn't distinguish between a complete answer and one that only mentions the word in passing. The third is a human teacher with a rubric: they read the answer, understand what it meant, and give a partial grade —"this one mentions the topic but doesn't give the exact fact, I'll give it 0.4 out of 1"—. It's the LLM-as-judge: the most nuanced criterion, capable of partial credit and of understanding paraphrases, but also the slowest, the most expensive and —key— the most subjective, because the teacher can have a bad day and grade the same answer differently. And the fourth isn't a way of grading an answer, but of aggregating the grades of all of them: the exam's passing rule ("you pass with a 0.80 average"), which is the statistical threshold operating on the aggregate score.

Keep the four. The first three are ways of giving a verdict to one case —from the most rigid (exact-match) to the most nuanced (judge)—; the fourth aggregates the verdicts into the score the gate consumes. And keep the third above all, because it has a trap: the LLM-as-judge's "human teacher" is also an LLM. Hiring a model to judge another model doesn't get you out of the non-determinism problem —it puts a second probabilistic component into it that now also has to be governed—.

Worked example: four criteria, four scores

We're going to take three of the agent's responses to "how long does shipping take?" —a complete and correct one, a correct paraphrase, and a vague one— and grade them with the four criteria. The reference "gold" answer is "Shipping takes 3 to 5 days". The LLM-as-judge is simulated with a deterministic stub: a fixed rubric that gives partial credit (+0.4 if it mentions the shipping topic, +0.6 if it gives the correct range). It's not a real model —zero network, zero API— but it mimics what a judge would do: give a graded score, not a binary.

# Lesson 07 — types of eval criterion
# Everything SIMULATED (including the judge). Zero network, zero API, zero keys. Deterministic.

# Three agent outputs for the same question, with different quality.
CASES = [
    {"id": "c1", "output": "Shipping takes 3 to 5 days.",
     "exact": "Shipping takes 3 to 5 days.", "contains": "3 to 5 days"},
    {"id": "c2", "output": "It usually arrives in 3 to 5 days, depending on your area.",
     "exact": "Shipping takes 3 to 5 days.", "contains": "3 to 5 days"},
    {"id": "c3", "output": "It usually arrives fast, in a few days.",
     "exact": "Shipping takes 3 to 5 days.", "contains": "3 to 5 days"},
]

# --- Criterion 1: exact-match (literal equality against the gold) ---
def crit_exact(o, c):    return o == c["exact"]

# --- Criterion 2: contains (the output contains the key phrase) ---
def crit_contains(o, c): return c["contains"].lower() in o.lower()

# --- Criterion 3: LLM-as-judge, SIMULATED with a fixed rubric (stub, NOT a model) ---
# ARCHITECTURAL WARNING: a REAL judge would be another LLM -> non-deterministic,
# with its own latency, cost, and fallibility. Here we simulate it with a
# deterministic rubric that gives PARTIAL CREDIT (0..1), to see the shape of its output.
def llm_as_judge_stub(output):
    o = output.lower()
    score = 0.0
    if "shipping" in o or "arrives" in o:  # mentions the topic (shipping/arrival)
        score += 0.4
    if "3 to 5" in o:                      # gives the specific and correct fact
        score += 0.6
    return round(score, 2)

print(f"{'case':<6}{'exact_match':>13}{'contains':>11}{'llm_judge':>11}")
for c in CASES:
    e  = crit_exact(c["output"], c)
    co = crit_contains(c["output"], c)
    j  = llm_as_judge_stub(c["output"])
    print(f"{c['id']:<6}{str(e):>13}{str(co):>11}{j:>11.2f}")

# --- Criterion 4: the statistical threshold operates on the AGGREGATE score ---
n = len(CASES)
exact_score    = sum(crit_exact(c["output"], c)    for c in CASES) / n
contains_score = sum(crit_contains(c["output"], c) for c in CASES) / n
judge_score    = sum(llm_as_judge_stub(c["output"]) for c in CASES) / n
print(f"\naggregate score  exact={exact_score:.2f}  "
      f"contains={contains_score:.2f}  llm_judge={judge_score:.2f}")

What to expect. When you run it:

case    exact_match   contains  llm_judge
c1             True       True       1.00
c2            False       True       1.00
c3            False      False       0.40

aggregate score  exact=0.33  contains=0.67  llm_judge=0.80

Here are the four criteria over the same three cases, and here's why the choice of criterion changes the verdict. Read it by column.

Exact-match (aggregate score 0.33). Only c1 passes —it's identical to the gold—; c2 and c3 fail. Notice c2: "It usually arrives in 3 to 5 days, depending on your area" is a correct answer —it gives the exact range— and exact-match still fails it, because it isn't identical letter for letter. The aggregate score, 0.33, doesn't measure the agent's quality: it measures how identical its responses are to an arbitrary phrase. It's the multiple-choice scanner grading essays: too strict, useless for free text.

Contains (aggregate score 0.67). c1 and c2 pass —both contain "3 to 5 days"— and c3 fails —"in a few days" doesn't contain the exact range—. This is better: it recognizes that the correct paraphrase (c2) is valid, so the score rises to 0.67 and reflects the real quality better. But it's binary: c3 gets a flat zero, even though it isn't a terrible answer —it mentions it arrives in a few days, just without the precise fact—. Contains doesn't distinguish "vague but on the right track" from "completely wrong": both are FALSE. It's the keyword grader: flexible, but crude.

LLM-as-judge (aggregate score 0.80). Here the nuance appears. c1 and c2 get 1.00 —they mention shipping and give the range—, and c3 gets 0.40 —partial credit: it mentions it arrives in a few days (+0.4 for the topic) but doesn't give the range (no +0.6)—. That 0.40 is the key difference: where contains gave a binary zero to c3, the judge recognizes it's a partially good answer and gives it a graded score. The aggregate score, 0.80, captures reality better —the agent responds well, with one weak answer— than contains' binary 0.67. The judge is the teacher with a rubric: nuanced, capable of partial credit and of understanding intent.

The statistical threshold (the fourth column, implicit in the "aggregate score"). Notice that the gate's decision isn't made case by case, but on the aggregate. The three aggregate scores —0.33, 0.67, 0.80— are what a gate would compare against a threshold. And look at the consequence: the same agent, with the same real quality, passes or fails a gate with a 0.75 threshold depending on which criterion you choose —with exact-match (0.33) and contains (0.67) it fails, with the judge (0.80) it passes—. The criterion isn't an implementation detail: it's what defines what your gate measures.

Put the four columns together and you have the lesson: the success criterion is a design decision, not a datum. A too-strict criterion (exact-match) underestimates the quality; a binary one (contains) measures it coarsely; a graded one (judge) measures it finely but —as you'll see— brings its own cost. And the threshold operates on the aggregate, so the choice of criterion and the threshold value have to be thought out together.

Going deeper: the LLM-as-judge is another AI component

The recursion of properties. This is the most important idea of the lesson, and the one that connects module 3 with the whole guide. The LLM-as-judge is tempting: when contains or exact-match fall short —because the quality is too nuanced for a mechanical rule—, the natural answer is "let's use a powerful model to judge." And it works: an LLM judge can evaluate relevance, tone, factual correctness, things no string rule captures. But look at what you just did. To evaluate your ai_component (the agent), you added a second ai_component (the judge). And the judge, being an LLM, has all the properties this guide taught you to handle: it's slow and costs per call (module 2 —now you pay for two models: the one that responds and the one that judges—); it's non-deterministic (module 1 —the same judge can give different grades to the same response, so your score becomes noisy—); it's fallible (it can judge badly —approve a bad response or fail a good one—); and if you pass it user text to judge, it crosses a trust boundary (module 4). Evaluating with an LLM doesn't get you out of the non-determinism problem: it gives you a second probabilistic component that also has to be governed. The uncomfortable question that follows is "and who evaluates the judge?" —and the answer is that the judge needs its own calibration against human judgment, which is eval design (AI Engineering)—.

When to use each criterion, at an architectural level. There's no "best" criterion; there are trade-offs, and at the design level they're chosen like this. Exact-match works when the correct output is unique and structured —a code, an identifier, a JSON with a fixed value—; for free text, almost never. Contains works when there's an unambiguous key phrase the correct answer must include —like shipping's "3 to 5 days"—; it's cheap, deterministic and fast, which is why it's the workhorse of many evals. A step up, structural criteria: validating that the output meets a schema (it's a valid JSON with the required fields), which is deterministic and perfect when the form matters (you'll see it in depth in module 4, guardrails). And the LLM-as-judge works when the quality is genuinely nuanced —semantic relevance, tone, the correctness of a piece of reasoning— and no mechanical rule captures it; it's the most powerful and the most expensive, and it's only justified when the simpler ones fall short. The design rule: use the simplest criterion that captures what matters. Don't bring in an LLM judge —with its cost, latency, and noise— if a deterministic contains measures your property well. The judge is the last-resort tool, not the first.

The judge's cost multiplies. A detail that connects with module 2 and that people underestimate: if your eval-set has 500 cases and you use an LLM-as-judge, each run of the eval makes 500 calls to the judge model —plus the 500 to the model you're evaluating—. And the eval runs on every PR (lesson 6). Module 2's taximeter now runs twice over, and for the judge model (which is usually a powerful model, to judge well) the price per call is the premium. An eval with an LLM judge can cost more than the feature it evaluates. This isn't a reason not to use it —sometimes it's the only way to measure what matters— but it is a reason to use it with awareness of its cost: reserve it for the properties that truly need it, and use cheap criteria (contains, schema) for the rest.

The boundary, for the last time. All this lesson does is show you the map of the types of criterion and their architectural consequences —what each one governs, what it costs, what fails—. What it does NOT do —and is AI Engineering's— is teach you to choose and calibrate the right criterion for a concrete task: how to decide which key phrase matters in a contains, how to write a judge's rubric so its grades correlate with human judgment, how to validate that the judge has no biases, how to measure the relevance of a semantic search with serious metrics. That's the craft of designing evals, and it's deep. This module leaves you knowing that the criterion matters and that the judge is a component with its own properties; designing the criterion is the other guide.

Common mistakes

Using exact-match for free text (a too-strict-criterion mistake). What happens: the team evaluates a component that generates text —an agent, a summary, a description— with exact equality against a model answer. The score comes out artificially low (0.33 in the example) because it fails every correct paraphrase, and the team concludes the component is bad when in reality the criterion is inadequate. Why it happens: exact-match is the default criterion of whoever comes from classic tests. How to spot it: if your criterion fails responses a human would consider correct just for not being identical, it's too strict. How to fix it: for free text, use contains (if there's a key phrase) or a judge (if the quality is nuanced); reserve exact-match for unique, structured outputs.

Bringing in an LLM-as-judge without accounting for its cost and non-determinism (an over-engineering-the-criterion mistake). What happens: the team uses an LLM judge for the whole eval-set "because it's smarter," and discovers late that each run costs double (two models), that the eval in CI is now slow and expensive, and that the score varies between runs because the judge is non-deterministic —sometimes it blocks a good deploy over judge noise—. Why it happens: the judge sounds like the universal solution ("a smart model judging"), and its cost and its noise are invisible until they accumulate. How to spot it: if you use an LLM judge for properties a deterministic contains would measure just as well, you're overpaying and injecting noise. How to fix it: use the simplest criterion that captures the property; reserve the judge for the genuinely nuanced, and remember the judge is an ai_component with its own cost, latency, and non-determinism (it recurses the guide's properties).

Trusting the score without looking at the criterion (an opacity mistake). What happens: the team sees a high score (0.95) and trusts it without asking with which criterion it was computed. It turns out the criterion was a loose contains ("the response mentions 'days'"), which approves vague responses, so the 0.95 doesn't mean the component is good —it means the criterion is permissive—. Why it happens: the score is a clean number that invites trust, and the criterion that produced it stays hidden behind it. How to spot it: if you can't say with which criterion a score was computed, you don't know what that score measures. How to fix it: always look at the criterion alongside the score; a high number with a loose criterion is lesson 4's "useless gate" disguised as good news.

Exercises

Exercise 1 — Grade with each criterion. An agent response to "can I return a product?" is: "Yes, we accept returns within 30 days". The gold is "You can make the return within 30 days". Give the verdict under (a) exact-match against the gold, (b) contains with the key phrase "30 days", and (c) an LLM-as-judge that gives +0.5 if it mentions that you can return and +0.5 if it gives the window. Comment on which criterion best reflects the real quality.

See solution
  • (a) Exact-match: "Yes, we accept returns within 30 days" ≠ "You can make the return within 30 days" → FALSE. It fails a clearly correct answer just for being worded differently.
  • (b) Contains "30 days": the response contains "30 days" → TRUE (1). It recognizes the key information is there.
  • (c) LLM-as-judge: it mentions that you can return (+0.5) and gives the 30-day window (+0.5) → 1.00. It recognizes the response is complete.

Which best reflects the real quality: the response is correct and complete, so the judge (1.00) and the contains (TRUE) measure it well, and exact-match (FALSE) measures it badly —it's the inadequate criterion here, because the text is free—. Between contains and judge, for this response they give the same positive verdict; the judge would differ on a partial response (for example "yes, you can return" without the window would give contains=FALSE but judge=0.5, partial credit). The moral: exact-match is out of place for free text; contains and judge agree on the clear cases and separate on the nuanced ones.

Exercise 2 — The judge that has to be governed. A team proposes: "To evaluate the support agent, we'll use the most powerful model as a judge: we pass it the customer's question and the agent's response, and the judge gives a grade from 0 to 1. We run this over 800 cases on every PR." List at least three properties of this guide the judge, being an LLM, drags into the eval, and what problem each causes.

See solution

The judge is another ai_component, so it drags in its properties:

  1. Cost (module 2). Each run makes 800 calls to the judge model (plus the 800 to the agent), and the judge is the most powerful model, with the premium price. On every PR. The eval with a judge can cost more than the evaluated feature —the taximeter runs twice over—.
  2. Latency (module 2). 800 calls to the judge take time; the eval step in CI becomes slow, and a slow CI holds up the whole team. You may have to take it out of the per-PR pipeline and run it in a separate stage.
  3. Non-determinism (module 1). The judge is an LLM, so it can give different grades to the same response on different runs. The eval's score becomes noisy: it can block a good deploy over judge variation, not agent variation. The threshold has to have margin for the noise.

(A fourth, if the customer text is adversarial: trust boundary (module 4) —a malicious response or question could try to manipulate the judge—.)

The underlying problem: using an LLM judge doesn't eliminate the non-determinism, it doubles it. Now you have two probabilistic components —the agent and the judge— and the question "who evaluates the judge?" stays open (the judge needs its own calibration against humans, which is AI Engineering). The judge is powerful, but it isn't free or neutral: it's one more AI component, with everything that implies in this guide.

Exercise 3 — Choose the simplest criterion that works. For each property to evaluate, say which criterion you'd use (exact-match, contains, schema/structural, or LLM-as-judge) and why the simplest that works is the right one. (a) That the agent returns a JSON with the status and order_id fields. (b) That the response about shipping mentions the "3 to 5 days" window. (c) That the extracted order_id is exactly "MERCADO-8842". (d) That the agent's response to a complaint has an empathetic tone and solves the problem.

See solution
  • (a) JSON with status and order_id fields → schema/structural. The property is about form: validate that the output is a valid JSON with those fields. Deterministic, cheap, perfect for structure. You don't need a judge to verify a form.
  • (b) Mentions "3 to 5 days" → contains. There's an unambiguous key phrase; contains("3 to 5 days") verifies it, cheap and deterministic. Bringing in a judge here would be overpaying for what an in solves.
  • (c) order_id exactly "MERCADO-8842" → exact-match. The correct output is unique and structured; exact equality is the right criterion, there's no possible paraphrase of an identifier.
  • (d) Empathetic tone and solves the problem → LLM-as-judge. The property is genuinely nuanced —"empathetic" and "solves" aren't a key phrase or a form—; no mechanical rule captures it, so here the judge is justified, with awareness of its cost and its noise.

The rule that orders it: use the simplest criterion that captures the property. Structure → schema; key phrase → contains; unique output → exact-match; nuanced quality → judge (last resort). Escalating to the judge when a contains would suffice is paying extra cost, latency, and noise without gaining precision.

Summary and next step

In this lesson you opened the piece the module took for granted: the success criterion. With the analogy of the four ways to grade you saw that exact-match (the scanner) is too strict for free text, contains (the keyword grader) is flexible but binary, the LLM-as-judge (the teacher with a rubric) gives partial credit and nuance, and the statistical threshold operates on the aggregate score. You executed the four over the same cases and saw the same agent get different scores —exact 0.33, contains 0.67, judge 0.80— depending on the criterion: proof that the criterion is a design decision, not a datum. And you ran into the module's central warning: the LLM-as-judge is another AI component that drags in all the guide's properties —cost and latency (module 2), non-determinism (module 1), fallibility, trust boundary (module 4)—, so evaluating with it doubles the problem instead of solving it. The design rule you take away: use the simplest criterion that captures what matters, and reserve the judge for the genuinely nuanced.

Before moving on you should be able to: name the types of criterion and what each one governs; explain why the chosen criterion changes the score of the same real quality; articulate why the LLM-as-judge is an ai_component with its own properties and what that implies; and apply the rule of "the simplest criterion that works."

With this the module's topical journey closes. You now have the complete arc: why the exact assert doesn't work (2), how the score is built with an eval-set (3), how the score becomes a gate with a threshold (4), how the gate catches regressions (5), how it lives in CI blocking deploys (6), and which criteria feed it (7). What follows is the project: in lesson 8 you're going to set up the quality gate of Mercado's support agent from end to end —the eval-set, the gate with per-case detail, the version that passes and the one that regresses, the CI integration— and justify why the eval is the architectural quality gate that completes the trio with latency and cost. It's the step from "I understand each piece" to "I know how to set up the whole gate with my own hands."

Resources

← Previous6. The eval in CI: a gate on the deployNext →8. Project: build the eval gate for a Mercado AI feature