Module 3: The Eval as a Fitness Function

5. Catching regressions when the prompt or model changes

Next →

Overview

By the end of this lesson you'll see lesson 4's gate doing the work it really exists for: protecting quality at the moment of change. An AI component doesn't degrade on its own, sitting in production; it degrades when someone touches it —tweaks the prompt to fix a case, or drops it to a cheaper model to save (module 2's cascade)—. That change can improve the quality, leave it the same, or worsen it —a regression—, and the problem is that at a glance the three look identical: the component keeps responding, in a similar tone, and nothing warns that it got worse. The eval is what makes the difference visible. You're going to run it against a baseline —the component's score in production today, 0.80— and against two changes: a new prompt that raises the score to 1.00 (improvement → accept) and a cheaper model that lowers it to 0.60 (regression → reject). The same component, the same eval-set, and the score tells you at a glance which change helped and which broke it.

This matters because the silent regression is one of the most expensive ways to fail with AI. A prompt change feels harmless —"it's just text, what could go wrong?"— and a model change justifies itself —"we save half the cost"—. Without a gate, both reach production without anyone measuring whether the quality survived, and the drop is discovered weeks later, through customer complaints, when nobody remembers which change caused it. The eval turns that invisible risk into a visible number before the deploy: you run the eval on the change, compare against the baseline, and if the score dropped, you know it today, not in a month. And there's a direct connection with module 2 that closes here: the cascade made things cheaper by sending queries to the cheap model, and the question it left open —"what if the cheap one answers worse?"— is exactly a regression question the eval answers. The cost gate (module 2) and the quality gate (this module) work together: one lets you go cheaper, the other verifies that going cheaper didn't break the quality.

Connection with the module: this lesson puts the gate in motion. Lesson 4 set it up statically —one version, one verdict—; here you run it over successive versions to detect the change between them. The new idea is the comparison against a baseline: it's not enough to ask "is this score good?", you have to ask "is this score worse than the one before?". That comparison is the core of regression detection, and it's what lesson 6 is going to automate in CI —so no change reaches production without the eval comparing it against the baseline first—. In one sentence: here the gate stops being a snapshot and becomes a guardian of change.

The analogy: the blood test against your history

Think of an annual medical checkup. The doctor asks for a blood test and looks at, say, your cholesterol. But notice what they don't do: they don't look at your number in isolation and say "180, it's a number, it's fine." What they do is pull up your history —your cholesterol last year was 160— and compare: it went up 20 points. That change relative to your baseline is the signal, more than the absolute number. A value that in itself seems acceptable can be an alarm if it worsened relative to your history, because it reveals a trend. And the reverse: to know whether a new treatment worked, the doctor doesn't guess —they measure you again and compare against the previous value—. The measurement against the baseline is how you distinguish "improved," "same," and "worsened," which by eye are indistinguishable.

An AI component is the same. The eval's score is your "blood test" of quality, and the baseline is your history: the score it had in production before the change. When someone tweaks the prompt or changes the model, you don't just ask "is the new score good?" —you ask "did the new score worsen relative to the baseline?"—. If it dropped, the change is a regression, even if the score is still above some minimum. If it rose, the change is an improvement, and you accept it with evidence, not with faith. Without the baseline, you're looking at a loose number without knowing whether your component is getting better or worse with each change —like a patient who only knows their cholesterol today and has no idea whether it's rising—. The eval run against a baseline is the checkup that turns each change into a measured "better, same, or worse."

Worked example: improvement, no change, and regression

We're going to take Mercado's support agent with a production baseline of 0.80 and pass two changes through the eval, comparing each against that baseline. The first change is a new prompt that, in addition to what it already answered, now gets the coupon and seller questions right —it rises to 1.00, an improvement—. The second is dropping to a cheaper model (module 2's cascade) that answers fewer cases well —it falls to 0.60, a regression—. The verdict rule compares the change's score against the baseline: if it falls below the threshold it's a regression, if it rises it's an improvement, if it stays the same it's no change.

# Lesson 05 — catching regressions when the prompt or model changes
# Everything SIMULATED. Zero network, zero API, zero keys. Deterministic output.

# (Reuses EVAL_SET, GOLD, POOR_ANSWER, make_agent, run_eval from lessons 3-4.)
EVAL_SET = [
    {"id": "q1",  "question": "where is my order",         "must_contain": "tracking"},
    {"id": "q2",  "question": "how do i return a product",  "must_contain": "return"},
    {"id": "q3",  "question": "how long does shipping take", "must_contain": "3 to 5 days"},
    {"id": "q4",  "question": "can i pay in installments",  "must_contain": "installments"},
    {"id": "q5",  "question": "the product arrived broken", "must_contain": "refund"},
    {"id": "q6",  "question": "how do i change my address", "must_contain": "profile"},
    {"id": "q7",  "question": "i didn't get my invoice",    "must_contain": "email"},
    {"id": "q8",  "question": "i want to cancel my order",  "must_contain": "cancel"},
    {"id": "q9",  "question": "the coupon doesn't work",    "must_contain": "expiration"},
    {"id": "q10", "question": "how do i contact a seller",  "must_contain": "messages"},
]
GOLD = {
    "q1": "You can see the tracking of your order in your profile.",
    "q2": "For a return, go to your order and click Return.",
    "q3": "Standard shipping takes 3 to 5 days.",
    "q4": "Yes, you can pay in installments with no interest by card.",
    "q5": "We're sorry about that; you can request a refund from the order.",
    "q6": "Change your address in the Profile, Addresses section.",
    "q7": "We'll resend the invoice to your account email.",
    "q8": "You can cancel the order if it hasn't shipped yet.",
    "q9": "Check the coupon's expiration; it may have already expired.",
    "q10": "Message the seller from the Messages section.",
}
POOR_ANSWER = "Sorry, I don't have information about that."

def make_agent(competent_ids):
    def agent(question, case_id):
        return GOLD[case_id] if case_id in competent_ids else POOR_ANSWER
    return agent

def run_eval(agent):
    passed = sum(
        case["must_contain"] in agent(case["question"], case["id"]).lower()
        for case in EVAL_SET
    )
    return passed / len(EVAL_SET)

ALL_IDS = {c["id"] for c in EVAL_SET}
BASELINE = 0.80          # the agent's score IN PRODUCTION today (the "history")
THRESHOLD = 0.80         # the gate's threshold

# The deploy candidates: each is a version of the agent after a change.
BASE_IDS    = {"q1","q2","q3","q4","q5","q6","q7","q8"}   # production today -> 0.80
PROMPT2_IDS = ALL_IDS                                     # new prompt   -> 1.00
CHEAP_IDS   = {"q1","q2","q3","q4","q6","q8"}             # cheap model  -> 0.60

candidates = [
    ("baseline (production today)", BASE_IDS),
    ("change 1: new prompt",        PROMPT2_IDS),
    ("change 2: cheap model",       CHEAP_IDS),
]

print(f"baseline in production = {BASELINE:.2f}   threshold = {THRESHOLD:.2f}\n")
print(f"{'candidate':<28}{'score':>7}{'vs baseline':>13}   verdict")
for name, ids in candidates:
    score = run_eval(make_agent(ids))
    delta = score - BASELINE
    if score < THRESHOLD:
        verdict = "REGRESSION -> reject"
    elif delta > 0:
        verdict = "improvement -> accept"
    else:
        verdict = "no change -> ok"
    print(f"{name:<28}{score:>7.2f}{delta:>+13.2f}   {verdict}")

What to expect. When you run it:

baseline in production = 0.80   threshold = 0.80

candidate                     score  vs baseline   verdict
baseline (production today)    0.80        +0.00   no change -> ok
change 1: new prompt           1.00        +0.20   improvement -> accept
change 2: cheap model          0.60        -0.20   REGRESSION -> reject

Here's the medical checkup of quality, and here's why the baseline is what gives each number meaning. Read it row by row.

The baseline (0.80). It's your history: the agent's score as it stands in production today. It serves as reference —the vs baseline is +0.00 because it's compared with itself—. Everything that follows is measured against this line.

Change 1: the new prompt (1.00, +0.20). Someone rewrote the agent's prompt and now it also gets the coupon and seller questions right. The score rose to 1.00, +0.20 over the baseline: it's an improvement, and the verdict is accept. And notice the value of this: the improvement isn't an impression ("the new prompt looks better") but a measured fact —two more cases that used to fail now pass—. You accept the change with evidence, and along the way the new baseline rises to 1.00 for the next checkup.

Change 2: the cheap model (0.60, −0.20). Someone, looking to save, dropped the agent to a cheaper model —exactly module 2's cascade—. The component keeps responding, in a similar tone, and by eye nobody would notice anything odd. But the eval gives it away: the score fell to 0.60, −0.20 below the baseline and below the threshold. It's a regression, and the verdict is reject. Without the eval, this change would have reached production —"we save half, and the responses look fine"— and the quality drop would have been discovered weeks later through complaints. With the eval, it's detected before the deploy, with a number nobody can argue with.

Put the three together and you have the lesson: the same component, three states —same, better, worse— that are indistinguishable by eye and that the eval separates with a number. The comparison against the baseline is what turns "here's a score" into "this change improved or worsened the quality." And note the close with module 2: change 2 is exactly the cascade's cost optimization, and the eval is what verifies that optimization didn't come out expensive in quality. The two gates, cost and quality, need each other: going cheaper without measuring quality is gambling, and the eval turns the gamble into an informed decision.

Going deeper: the regression, the baseline, and the interaction with cost

What exactly a regression is. A regression is a change that worsens a property that was previously fine. In classic software it's a test that was green and goes red because of an unrelated change —you fixed A and broke B—. In an AI component it's a change (of prompt, of model, of context data) that lowers the eval's score. The key word is lowers: it's not that the component is bad in absolute terms, it's that it's worse than before. That's why a regression is only seen with a baseline —you need the "before" to know there was a "worse"—. A component with score 0.60 could be perfectly fine if its baseline was always 0.60; the problem is when its baseline was 0.80 and a change lowered it to 0.60. The eval without a baseline measures absolute quality; the eval with a baseline also measures the direction of the change, which is what matters when someone touches the component.

Why a prompt change isn't "just text." The instinct to treat a prompt change as harmless comes from it looking like text editing, not code. But in an AI component the prompt is part of the program: changing it changes the component's behavior in all cases, not just the one you wanted to fix. You tweak the prompt so it responds better about coupons and, unintentionally, change how it responds about shipping —because the model processes the whole prompt together—. That's why a prompt change needs the full eval, not a review of the case you touched: the effect is spread across the whole eval-set, and only by running it entire do you see whether you gained on one side and lost on another. "It's just text" is precisely the mental model that produces silent regressions.

The interaction with module 2's cost budget. This is the most important connection of the lesson. Module 2 taught you to lower the cost with the cascade: send the easy queries to a cheap model. But it left open, on purpose, the quality question: "what if the cheap one answers worse?". Now you have the tool to answer it. When you propose dropping a model to save, you run the eval: if the score holds, going cheaper came free in quality —go ahead—; if the score falls (like change 2, to 0.60), going cheaper had a hidden cost in quality, and you have to decide whether it's worth it or whether the cascade should send those queries to the strong model. The design rule: no cost optimization is accepted without running the eval, because the dollar saving can hide a quality loss that only the eval makes visible. Module 2's two gates and this module's form a system: budget and quality are verified together, not separately.

The baseline moves (and that's good). A nuance: the baseline isn't fixed forever. Every time you accept an improvement, the new score becomes the baseline for the next change. You accepted the new prompt (1.00); now the next change is compared against 1.00, not against 0.80. This is desirable: the baseline rises with the improvements, so the "don't get worse" bar rises with you. A change that gives 0.90 would be an improvement over the old baseline of 0.80 but a regression over the new one of 1.00 —and so it should be: once you reached 1.00, going back to 0.90 is losing ground—. The living baseline is what turns the eval into a quality ratchet: every time you go up, you're not allowed to go below that without noticing.

Common mistakes

Changing the prompt or the model without running the eval (a process mistake). What happens: someone tweaks the prompt or drops a model and deploys directly, trusting that "it looks fine." The change fixes what it was after but breaks cases nobody looked at again —a silent regression— and it's discovered weeks later through customer complaints, when it's already expensive to trace what caused it. Why it happens: a prompt change feels like editing text, and a model change justifies itself by the saving; without a mandatory gate, nothing stops the deploy. How to spot it: if your prompt or model changes reach production without a score comparing them against the baseline, you're flying blind. How to fix it: run the eval on every change to the component and compare against the baseline; lesson 6 makes it mandatory by putting it into CI.

Looking at the score with no baseline (a reference mistake). What happens: the team runs the eval on a change, sees 0.75 and says "it's above half, acceptable" —without noticing the baseline was 0.90 and the change was a 0.15 drop—. The absolute score looked tolerable, but the direction was downward. Why it happens: without pulling up the history, a loose number always looks "more or less fine." How to spot it: if you evaluate a change without comparing against the previous score, you're not detecting regressions, only measuring an absolute value. How to fix it: store the baseline and always report the vs baseline (the delta); a regression is a drop relative to the history, not a low value in the abstract.

Accepting the cost optimization without verifying quality (a scope mistake, cross with module 2). What happens: the team drops to a cheaper model because "it saves half" and deploys looking only at the cost budget, without running the eval. The cost dropped, yes, but so did the quality —a regression the cost gate can't see, because it only measures dollars—. Why it happens: the saving is immediate and visible; the quality loss is deferred and invisible without an eval. How to spot it: if you approved a model change looking only at the cost, you're missing half the analysis. How to fix it: every cost optimization passes also through the eval —the cost budget and the eval gate together—; going cheaper is only accepted if the score holds.

Exercises

Exercise 1 — Classify each change. The production baseline is 0.85 and the threshold is 0.80. For each change, say whether it's an improvement, no change, or a regression, and whether it's accepted or rejected. (a) Change X: score 0.90. (b) Change Y: score 0.82. (c) Change Z: score 0.78.

See solution

Comparing each score against the baseline (0.85) and the threshold (0.80):

  • (a) X: 0.90. +0.05 over the baseline and above the threshold → improvement, accept. Quality rises; the new baseline becomes 0.90.
  • (b) Y: 0.82. −0.03 below the baseline but above the threshold (0.82 ≥ 0.80). Here there's a nuance: it doesn't cross the threshold, so it isn't a regression that blocks the deploy, but it did drop relative to the baseline. It's a slight degradation: technically deployable, but you should ask why the change cost 0.03 of quality and whether it was worth it. A strict team flags it for review; a loose one lets it through for being above the threshold. What matters: the negative delta is a signal even if it doesn't cross the threshold.
  • (c) Z: 0.78. −0.07 below the baseline and below the threshold (0.78 < 0.80) → regression, reject. It worsened and also falls below the acceptable minimum. It's blocked.

The moral: there are two references, the baseline (did it worsen?) and the threshold (is it acceptable?). Z fails both and is rejected without doubt; Y is in the gray zone —it dropped but is still above the minimum— and deserves a look; X improves and raises the baseline.

Exercise 2 — The regression disguised as savings. A colleague proposes: "I dropped the support agent to the cheap model. The monthly cost fell from $3,000 to $1,400 —we save 53%—. I tested five questions by hand and the responses look fine. Ready to deploy." Identify what this analysis is missing and why the saving isn't enough to approve the change.

See solution

What it's missing is running the full eval and comparing it against the quality baseline. The analysis measures the cost well (it fell 53%, module 2's cost gate passes), but it verifies the quality in the worst possible way: "I tested five questions by hand and they look fine." That's exactly the "testing by eye" mistake: five cases aren't the eval-set, "they look fine" isn't a score, and there's no comparison against the baseline. The cheap model could be answering well those five easy questions it chose and failing the fifteen hard ones it didn't test —exactly change 2's regression (0.60 against a baseline of 0.80)—.

Why the saving isn't enough: cost and quality are orthogonal gates (lesson 4, exercise 3). A change can improve one and worsen the other, and this change does exactly that: it lowers the cost and —probably— lowers the quality. Approving it by looking only at the cost is seeing half the picture. The right thing: run the full eval-set on the cheap model, compare the score against the baseline, and only then decide. If the score holds, the saving is real and free; if the score falls, the "saving" has a hidden cost in quality that the business has to decide whether it accepts —or send those queries to the strong model via the cascade—.

Exercise 3 — The baseline that moves. The agent starts with a baseline of 0.80. A change is accepted that raises it to 0.95 (new baseline). Then a change arrives that gives 0.88. Under a fixed threshold of 0.80, does it pass the gate? Is it an improvement or a regression? Explain why the verdict depends on what you compare against.

See solution

The change that gives 0.88 produces two verdicts depending on the reference:

  • Against the fixed threshold (0.80): 0.88 ≥ 0.80 → it passes the gate. It's good enough in absolute terms: deploying it wouldn't serve unacceptable quality.
  • Against the living baseline (0.95): 0.88 < 0.95, a drop of −0.07 → it's a regression. It worsened relative to where you already were.

How they reconcile: the threshold gate says "this is acceptable for production" (yes, 0.88 is), but the regression analysis says "this is worse than what you already had" (yes, you dropped from 0.95 to 0.88). Both are true at once. A mature team does not deploy this change just because it passes the threshold: it asks why it lost 0.07 relative to the baseline and whether that price buys something (cost saving? another improvement?). Giving up 0.95 in exchange for 0.88 with no good reason is losing ground gained.

The moral: the threshold is an absolute floor (is it acceptable?), the baseline is a relative reference (did it improve or worsen?), and you need both. Once you go up to 0.95, the living baseline protects that achievement: it doesn't let you drop to 0.88 without the comparison flagging it as a regression, even though the fixed threshold would let it through.

Summary and next step

In this lesson you put the gate to work where it really matters: at the moment of change. You saw that an AI component doesn't degrade on its own —it degrades when someone touches the prompt or the model— and that the three directions of change (better, same, worse) are indistinguishable by eye but are separated with a number. With the analogy of the blood test against your history you understood the role of the baseline: it's not enough to ask "is this score good?", you have to ask "did it worsen relative to the one before?" —that's the definition of a regression—. You executed the eval over a baseline of 0.80 and two changes: a new prompt that rose to 1.00 (improvement → accept) and a cheaper model that fell to 0.60 (regression → reject), detected before the deploy. And you closed the loop with module 2: the cheap-model change is the cascade's cost optimization, and the eval is what verifies that going cheaper didn't come out expensive in quality —the cost and quality gates work together—.

Before moving on you should be able to: define a regression as a change that lowers the score relative to a baseline; explain why a prompt change isn't "just text" and needs the full eval; articulate why every cost optimization must pass through the eval; and distinguish the role of the threshold (absolute floor) from the role of the baseline (relative reference).

What follows is making this protection mandatory and automatic. So far you run the eval by hand and decide; but a process that depends on someone remembering to run the eval will fail the day someone is in a hurry. In lesson 6 you're going to put the eval into CI: a pipeline step that runs the eval on every change, produces an exit code, and blocks the deploy if the score falls below the threshold —just as a red test blocks the merge—. You're going to see two PRs pass through the pipeline: one improves the prompt (exit 0 → MERGE) and another drops the model and regresses (exit 1 → BLOCK). It's the step from "I run the eval when I remember" to "the pipeline won't let me deploy a regression even if I want to."

Resources

← Previous4. The eval as a quality gate (fitness function)Next →6. The eval in CI: a gate on the deploy