Module 3: The Eval as a Fitness Function

8. Project: build the eval gate for a Mercado AI feature

Next →

Overview

This is your graduation from the module. Over seven lessons you learned to test a probabilistic component without an exact assert: with an eval-set that produces a score, a threshold that turns it into a gate, the comparison against a baseline that catches regressions, and the CI step that blocks the deploy. You saw all of that applied to the support agent. Now it's your turn, from scratch, on a different Mercado feature: semantic search. The reason for changing feature is the usual one and it's hard: if I let you re-set-up the support agent's gate, I wouldn't know whether you learned the method or memorized the table. With a new feature —and, above all, with a different success criterion— the only way to solve it is to apply the method: define the criterion, run the eval-set, set up the gate, catch the regression, and put it in CI. That is, exactly, the proof that the module worked.

Your deliverable is four artifacts for semantic search: (1) the eval-set with its success criterion (given) and why the criterion is the right one for this feature; (2) the eval_gate executed with per-case detail, showing version A that passes and version B that regressed and is blocked; (3) the gate integrated into CI with its exit code; and (4) a decision brief (ADR style) that justifies why the eval is the architectural quality gate and how it composes with module 2's latency and cost gates. No part requires building the LLM or computing embeddings: the component is simulated with a deterministic stub. It's pure architecture work —define the gate, execute it, defend the decision— which is exactly what separates an AI feature with governed quality from one whose quality degrades without anyone noticing. Build it yourself first; reading the reference solution without having tried is like reading the score of a game you didn't play.

Connection with the module: this project closes the arc. Lesson 2 showed you why the exact assert doesn't work; lesson 3 manufactured the score; lesson 4 turned it into a gate; lesson 5 put it to catch regressions; lesson 6 put it into CI; lesson 7 opened the types of criterion. Here you produce the four artifacts with your own hands, from start to finish, over semantic search —and with a success criterion (relevant in the top-3) that is not the contains of the lessons, so you apply lesson 7—. And with this lesson the module closes: at the end is the summary of the eight lessons and the bridge to module 4 (guardrails and the trust boundary —validate the model's output before trusting it—) and to the AI Engineering ecosystem (where the evals we only used as a gate here are designed).

The project's case: Mercado's semantic search

The feature you have to govern —yours to solve— is this:

Mercado has a semantic search: when a customer types "something to listen to music while running," an AI component understands the intent and returns a list of products ordered by relevance. The team wants to put a quality gate on it: a change that worsens the relevance of the results should not be deployable. Set it up.

It's a sibling feature to the support agent's, but with a different quality profile, and that's why its success criterion is a different one. In the agent, the response was text and the criterion was contains (does the response contain the key phrase?). In search, the output is a product ranking, and what matters isn't a phrase but the position: does the relevant product come out on top, where the customer sees it? A customer doesn't check page 3 of results; if the relevant one isn't in the first few, for them the search failed. That's why the natural criterion is a top-k one: the search gets a case right if the relevant product appears in the top-3. It's a different criterion from the lessons' —you apply lesson 7— and it's the right one for this feature because it measures what the customer experiences: the relevant, on top.

The facts the team gives you (so you don't have to invent them): the eval-set has 10 representative queries, each with the relevant product that should come out on top (we give it to you; choosing those queries and their relevant product is eval design, AI Engineering's, not your job here). The success criterion is relevant in the top-3. The feature's quality threshold is 0.80 (the business decided that at least 8 of every 10 searches must put the relevant on top). There are two versions to evaluate: A, the production candidate, and B, one that dropped to a cheaper model to save (module 2's cascade) and regressed.

What you have to deliver

Follow the steps in order; each one rests on the previous.

Part 1 — The eval-set and its criterion

Write (or take as given) the eval-set: the 10 queries with their relevant product. And justify in one or two sentences why the top-3 criterion is the right one for semantic search —why the position matters more than a phrase, and why top-3 and not top-1 or top-10—. Don't invent the queries; the point is to understand which criterion measures the quality of THIS feature and why.

Part 2 — The eval_gate executed, with per-case detail

Set up the gate and run it over the two versions. For each, print: the per-case detail (which queries fail, that is, put the relevant one outside the top-3), the aggregate score, and the verdict (PASS/deploy allowed or FAIL/deploy blocked). Version A should pass; version B, the regressed one, should be blocked. Simulate the component with a deterministic stub.

Part 3 — The gate in CI

Turn the gate into a CI step that returns an exit code (0 if it passes, 1 if it fails) and show what the pipeline would do with each version (MERGE or BLOCK). It's lesson 6 applied to your feature.

Part 4 — The decision brief (ADR style)

Write a short decision document that answers: which gate did you put and why? Which criterion and which threshold, and where do they come from? How does this eval gate compose with module 2's latency and cost gates? Why is search's non-determinism governed by this gate? It's the architectural justification of your design.

Try the four parts before looking at the solution. What follows is a reference, not the only correct answer.

Reference solution

Part 1 — The eval-set and its criterion

The eval-set is 10 queries, each with the relevant product that should appear on top. The criterion is relevant in the top-3.

Why the top-3 criterion is the right one: in a search, the output isn't a phrase but a ranking, and what the customer experiences is the position —they check the first few results and rarely go down—. A contains (is the product in the list?) wouldn't work: the relevant product could be at position 50 and "count" as present, even though the customer never sees it. The criterion has to measure position, not mere presence. And top-3 (not top-1) because demanding the exact position 1 would be too strict —a good search puts the relevant among the first few, not necessarily first—; top-10 would be too loose —the relevant at position 9 is, for the customer, nearly invisible—. Top-3 captures "on top, where the customer sees it." (Which exact k, and how to build the queries, is eval design, AI Engineering; here the point is choosing the right type of criterion —position, not presence— for what the feature promises.)

Parts 2 and 3 — The gate executed and in CI

# Project M3 — the eval gate for Mercado's semantic search. SIMULATED.
# Zero network, zero API, zero keys. Deterministic.

# Mercado's product catalog (English ids).
CATALOG = ["earbuds", "running_shoes", "phone_case", "smartwatch",
           "water_bottle", "backpack", "sunglasses", "headphones",
           "yoga_mat", "power_bank", "laptop", "coffee_mug"]

# THE EVAL-SET: each case = query + the relevant product that should come out on top.
# BOUNDARY NOTE: choosing these queries and their relevant product is AI Engineering.
# Here the eval-set ALREADY exists; we use it AS A GATE.
EVAL_SET = [
    {"id": "s1",  "query": "something to listen to music while running", "relevant": "earbuds"},
    {"id": "s2",  "query": "running shoes",                    "relevant": "running_shoes"},
    {"id": "s3",  "query": "protect my phone",                 "relevant": "phone_case"},
    {"id": "s4",  "query": "watch that counts steps",          "relevant": "smartwatch"},
    {"id": "s5",  "query": "bottle for the gym",              "relevant": "water_bottle"},
    {"id": "s6",  "query": "backpack for the laptop",         "relevant": "backpack"},
    {"id": "s7",  "query": "glasses for the sun",             "relevant": "sunglasses"},
    {"id": "s8",  "query": "charge the phone without an outlet", "relevant": "power_bank"},
    {"id": "s9",  "query": "mat for doing yoga",              "relevant": "yoga_mat"},
    {"id": "s10", "query": "big headphones for home",         "relevant": "headphones"},
]

def make_search(competent_ids):
    # STUB of the search component: for the cases in competent_ids it puts the
    # relevant product ON TOP (top-3); for the rest it buries it (position 5).
    def search(query, case_id, relevant):
        others = [p for p in CATALOG if p != relevant]
        if case_id in competent_ids:
            return [relevant, others[0], others[1]]     # relevant in the top-3
        return others[:4] + [relevant]                  # relevant at position 5
    return search

def criterion_top_k(ranking, relevant, k=3):
    # The success criterion: the relevant product appears in the top-k of the ranking.
    return relevant in ranking[:k]

def run_eval(search, k=3):
    passed, detail = 0, []
    for case in EVAL_SET:
        ranking = search(case["query"], case["id"], case["relevant"])
        ok = criterion_top_k(ranking, case["relevant"], k)
        passed += ok
        detail.append((case["id"], ok))
    total = len(EVAL_SET)
    return passed, total, passed / total, detail

# Version A (candidate) and B (cheaper model that regressed).
ALL_IDS = {c["id"] for c in EVAL_SET}
A_IDS = ALL_IDS - {"s9"}                          # 9/10 = 0.90
B_IDS = {"s1", "s2", "s3", "s4", "s6", "s8"}      # 6/10 = 0.60

THRESHOLD = 0.80

def full_gate(name, ids):
    passed, total, score, detail = run_eval(make_search(ids))
    ok = score >= THRESHOLD
    fails = [cid for cid, o in detail if not o]
    print(f"--- {name} ---")
    print(f"  score = {passed}/{total} = {score:.2f}   threshold = {THRESHOLD:.2f}")
    print(f"  queries that fail (relevant outside top-3): {fails if fails else 'none'}")
    print(f"  GATE: {'PASS -> deploy allowed (green)' if ok else 'FAIL -> deploy blocked (red)'}")
    return 0 if ok else 1

print("=== EVAL GATE — Mercado semantic search (criterion: relevant in top-3) ===\n")
code_a = full_gate("version A (production candidate)", A_IDS)
print()
code_b = full_gate("version B (cheaper model, regression)", B_IDS)

# The gate in CI: the exit code decides MERGE or BLOCK.
print(f"\n=== In CI ===")
print(f"PR that deploys A: exit code = {code_a}  ->  {'MERGE' if code_a==0 else 'BLOCK'}")
print(f"PR that deploys B: exit code = {code_b}  ->  {'MERGE' if code_b==0 else 'BLOCK'}")

What to expect. When you run it:

=== EVAL GATE — Mercado semantic search (criterion: relevant in top-3) ===

--- version A (production candidate) ---
  score = 9/10 = 0.90   threshold = 0.80
  queries that fail (relevant outside top-3): ['s9']
  GATE: PASS -> deploy allowed (green)

--- version B (cheaper model, regression) ---
  score = 6/10 = 0.60   threshold = 0.80
  queries that fail (relevant outside top-3): ['s5', 's7', 's9', 's10']
  GATE: FAIL -> deploy blocked (red)

=== In CI ===
PR that deploys A: exit code = 0  ->  MERGE
PR that deploys B: exit code = 1  ->  BLOCK

Read the result by version. Version A puts the relevant product in the top-3 for 9 of the 10 queries —it only fails s9, the yoga mat—: score 0.90, above the 0.80 threshold, so the gate lets it through (green) and in CI the PR is merged (exit 0). Version B —the one that dropped to a cheaper model to save— gets only 6 of 10 right: it buries the relevant product outside the top-3 in four queries (s5, s7, s9, s10). Its score falls to 0.60, below the threshold, and the gate blocks it (red); in CI the PR is stopped (exit 1). The regression the cost saving introduced —searches that now hide the relevant— is caught before the deploy, without a human having to review rankings by hand. And notice the per-case detail: it tells you exactly which queries degraded, so if you wanted to fix version B you'd know where to start. The gate decides; the detail diagnoses.

Part 4 — The decision brief (ADR style)

Decision: put an eval gate as a quality gate for semantic search, integrated into CI, that blocks the deploy if the score falls below 0.80.

Context. Semantic search is a probabilistic component: it doesn't give the same ranking every time, and its quality —does the relevant come out on top?— can't be verified with an exact assert. Without a gate, a prompt or model change can degrade the relevance without anyone noticing until customers complain.

Criterion and threshold. The criterion is relevant in the top-3, because what the customer experiences is the position of the product, not its mere presence in the list. The threshold is 0.80: the business decided that at least 8 of every 10 searches must put the relevant on top. The threshold isn't technical —it comes from how much imperfection the search experience tolerates—.

Composition with the other gates. This eval gate is the third of the three gates the feature must pass, alongside the latency budget (≤ 800 ms) and the cost budget (≤ $5,000/month) from module 2. All three are orthogonal —fast, cheap, and good— and all three must pass. In particular, this gate protects against the cascade's side effect: dropping to a cheaper model saves money (passes the cost budget) but can degrade the relevance (fails the eval gate), like version B. The eval gate is what makes the cost optimization safe: you only go cheaper if the quality holds.

Why the non-determinism is governed. The search will keep being non-deterministic —it'll give rankings that vary—, but it can no longer worsen uncontrolled: any change that lowers the relevance below the threshold is impossible to deploy, because the pipeline blocks it. The non-determinism isn't eliminated; it's bounded with a gate that guarantees a quality floor. That's the eval's architectural role: the fitness function that governs that the probabilistic quality doesn't degrade.

Consequences. Every change to the component (prompt, model, data) must pass the eval in CI before the deploy. The eval-set must be kept representative over time (eval design, AI Engineering). The cost of running the eval is low as long as the criterion is a deterministic top-k; if in the future an LLM-as-judge is used to measure nuanced relevance, its cost and its noise will have to be accounted for (lesson 7).

That brief is the artifact that justifies the decision to the team: which gate, with which criterion and threshold, how it composes with the others, and why it contains the non-determinism. With the four parts —criterion, executed gate, CI, and brief— you have semantic search's quality gate set up from end to end.

Common mistakes

Using contains (presence) where top-k (position) goes (an inadequate-criterion mistake). What happens: the team evaluates the search with "is the relevant product in the list of results?" and the score comes out very high —because the product is almost always somewhere in the list, even if it's position 40—. The gate approves a search that in practice hides the relevant. Why it happens: contains is the criterion you were using in the agent, and it's applied out of habit. How to spot it: if your search criterion doesn't look at the position, it doesn't measure what the customer experiences. How to fix it: for a ranking, the criterion has to be a position one (top-k), not a presence one —apply lesson 7: the simplest criterion that captures what matters, and here what matters is that it come out on top—.

Approving the model change looking only at the cost (a scope mistake, cross with module 2). What happens: version B dropped a model and saved money, and the team approves it looking only at the cost budget —"it saves, and the searches look fine"— without running the eval. The relevance regression (0.60) reaches production. Why it happens: the saving is visible and immediate; the relevance drop is invisible without an eval. How to spot it: if you approved a model change without the eval's score, you're missing the quality gate. How to fix it: every cost optimization passes also through the eval gate; the three gates together, not separately.

Setting up the gate but leaving it as a notice in CI (a configuration mistake). What happens: the team puts the eval in CI but configured to notify without blocking, so version B is deployed anyway with a warning nobody reads. Why it happens: notice mode avoids friction. How to spot it: if your gate has never stopped a deploy, it's decorative. How to fix it: make the exit code block the pipeline (lesson 6); a gate that can't say "no" isn't a gate.

Exercises

Exercise 1 — Change the criterion from top-3 to top-1. The team proposes tightening the criterion: instead of "relevant in the top-3," require "relevant in position 1" (top-1). With the reference solution's stub, version A puts the relevant in position 1 for its competent cases ([relevant, ...]) and in position 5 for the rest. How does version A's score change with top-1 versus top-3? And in general, what effect does tightening the criterion have on the scores and on which versions pass the gate?

See solution

With this particular stub, version A's score doesn't change: in the competent cases the relevant is at position 1 (ranking[0]), which is in both the top-1 and the top-3; in the non-competent ones it's at position 5, outside both. So A gives 0.90 with top-1 and with top-3 —the stub is "all or nothing" on the position—.

But the general question is the one that matters: tightening the criterion can only lower or hold the score, never raise it. Top-1 is a subset of top-3 (everything that passes top-1 passes top-3, but not the reverse), so with a realistic component —where the relevant sometimes lands at position 2 or 3— the score with top-1 would be lower than with top-3, because the "relevant at position 2" cases now fail. Consequence for the gate: a stricter criterion makes more versions fail the threshold. This is a design decision with a trade-off: top-1 demands a nearly perfect search (the relevant first), which can be too strict and block versions the customer would consider good (the relevant in the top-3 serves them). The lesson: the criterion and the threshold are calibrated together, and tightening the criterion without lowering the threshold can turn a reasonable gate into an unreachable one. (Which k is right for this feature is eval design, AI Engineering.)

Exercise 2 — Search's three gates. Version B (the cheaper model) gives: latency 180 ms (budget 800 ms), cost $2,400/month (budget $5,000/month), quality score 0.60 (threshold 0.80). A colleague argues: "it's lightning-fast and dirt-cheap, we save a lot, let's deploy it." Answer with the three-gate analysis and explain why the saving doesn't save it.

See solution

Version B's three gates:

  • Latency budget: 180 ms ≤ 800 ms → PASS. It's very fast (faster than A, because the cheap model responds faster).
  • Cost budget: $2,400/month ≤ $5,000/month → PASS. It's cheap (saves against the budget).
  • Eval gate: 0.60 < 0.80 → FAIL. Its relevance regressed: it buries the relevant outside the top-3 in four of ten searches.

Overall verdict: it doesn't deploy. A single gate failing is enough, and the quality one fails. The colleague's argument looks only at two of the three gates —the operational ones (fast, cheap)— and ignores the quality one. And here's the point: version B is fast and cheap precisely because it's worse —it uses a weaker model that saves resources at the cost of relevance—. Deploying it would be serving a search that's speedy, cheap, and bad: the customer searches "running shoes" and doesn't see the shoes in the first results. The saving is real in dollars, but it buys a degradation of the experience the business didn't accept (it set the threshold at 0.80). The eval gate is exactly the gate that keeps "we save a lot" from becoming "we degraded the search without realizing." If the team wants the saving, the correct route is the cascade (module 2): use the cheap model only on the easy queries where it doesn't harm relevance, and the strong one on the hard ones —and verify with the eval that that mix keeps the score above the threshold—.

Exercise 3 — The criterion for another feature. Mercado wants to put a quality gate on the "describe your product" generator (you give it a product's name and specs, and the LLM writes a sales description). What type of success criterion would you use for its eval-set and why? Consider what makes a product description "good" and which criterion from lesson 7 captures it.

See solution

This is the hardest of the three cases, and the honest answer starts by recognizing it. A "good" product description is genuinely nuanced: it must be fluent, persuasive, mention the correct specs, have the brand tone, and not invent features the product doesn't have (not hallucinate). None of those properties is captured with a simple mechanical criterion.

A layered approach, applying the "simplest criterion that works" rule (lesson 7):

  • Cheap, deterministic criteria for the verifiable: a contains or a structural criterion for the objective properties —that the description mention the given key specs (the material, the size), that it not exceed a certain length, that it not include specs that were not in the input (a cheap defense against hallucinated data)—. These are fast, with no model cost, and catch objective errors.
  • An LLM-as-judge for the nuanced: fluency, persuasion, and brand tone aren't measured with string rules; here an LLM judge that grades those dimensions is justified. But —lesson 7— with awareness of its cost: the judge is another ai_component (slow, expensive, non-deterministic), so it's reserved for what truly needs it, and its rubric has to be calibrated against human judgment (that's AI Engineering).

The moral: there's no single criterion; they're combined —cheap criteria for the objective, a judge for the subjective—, always using the simplest that captures each property. And an important note: verifying that the description doesn't hallucinate specs is partly an eval problem (does the score drop when it invents?) and partly a guardrail problem —validate the output before publishing it—, which is module 4. The quality gate (eval) and the trust gate (guardrail) complement each other.

Module summary: the eight lessons

You closed the module of the eval as a fitness function. This is the complete arc you traveled:

LessonWhat you take away
1The thesis: a probabilistic component is tested with an eval-set that produces a score, and that score against a threshold is the quality gate —the probabilistic equivalent of the fitness function—. The two analogies (standardized exam, factory quality control).
2Why the exact assert breaks against a probabilistic component, and the change of shape that saves it: from equality to property, from boolean per case to aggregate score.
3The anatomy of the eval-set: cases + criterion + aggregation → score. The two levels of its output (per-case detail that diagnoses, score that decides). Why it's reproducible and "by eye" isn't.
4The heart: the score against a threshold is a gate (green/red, deploy allowed/blocked). It's a fitness function specialized to probabilistic quality. A loose threshold = useless gate.
5The gate catches regressions: the score of a change against a baseline —a new prompt improves, a cheap model regresses—. The eval verifies that going cheaper (module 2) didn't break the quality.
6The eval in CI: an exit code that blocks the deploy if the score drops, like a red test. The gate stops depending on human discipline and becomes inevitable.
7The types of criterion: exact-match, contains, LLM-as-judge, statistical threshold. The central warning: the judge is another AI component that recurses all the guide's properties.
8The project: semantic search's quality gate set up from end to end —top-3 criterion, executed gate, CI, and decision brief—.

The capability you gained: taking an AI component and putting a quality gate on it that governs its deploy —define the criterion, run the eval-set, compare against a threshold and a baseline, and put it in CI so it blocks regressions— without falling into the exact assert or "testing by eye." And with the boundary clear: this is the eval as an architectural gate; designing the eval —which cases, which criterion, how to calibrate a judge— is AI Engineering.

Where to go next

Module 4: guardrails and the trust boundary. The eval verifies that the component's quality is good on average, over an eval-set. But there's a different question the eval doesn't cover: on an individual production request, can you trust the model's output before using it? The answer is no —an LLM's output is untrusted until you validate it—, and module 4 teaches you to put guardrails: validate the model's input and output at the boundary (a schema, a rule), and treat prompt injection as a trust boundary problem —the model reading user data crosses a security boundary—. If the eval is the aggregate quality gate (is it good in general?), the guardrail is the per-request trust gate (can I use this output?). Both are necessary.

The AI Engineering ecosystem: designing the evals in depth. Throughout this module the eval-set came given: we gave you the queries, the criterion, and the threshold. Building good evals is a deep craft —choosing representative cases without bias, measuring relevance with serious metrics, calibrating an LLM-as-judge against human judgment, versioning the dataset—, and it's from the AI Engineering ecosystem. If you're going to put AI features into production for real, that's the next body of knowledge: this module taught you to use the eval as a gate; AI Engineering teaches you to design it well.

architecture-decisions-and-tradeoffs-guide: the fitness function as a general concept. This whole module specialized an idea taught there in general: the fitness function as an automated test that governs an architectural property. If you want the complete framework —fitness functions for latency, coupling, dependencies, module size, and not just for AI quality—, that guide is the source. The eval is a fitness function; that's the root concept.

With this you have the third gate of an AI feature: latency (module 2), cost (module 2), and quality (this module). A production-ready AI feature passes all three, and you now know how to set up the quality one with your own hands.

Resources

← Previous7. Types of eval criterionNext →Module introduction: guardrails and the trust boundary