Module 7: The Data and Feedback Loop

5. Closing the loop: the feedback becomes the eval-set

Next →

Overview

By the end of this lesson you'll know how to do what gives the module its name: close the loop —take the feedback you captured and feed it back to something that improves the system in a verifiable way—. It's the module's heart because it's where everything before it makes sense: the flywheel (lesson 2) only spins if the loop closes, the observability (lesson 3) is only useful if you act on what you see, and the captured feedback (lesson 4) is dead raw material until you feed it back. The thesis is concrete: the cases a user reported as bad, with their correction, become new cases of the eval-set —module 3's— and that augmented eval-set catches regressions the old one let pass. Production feedback stops being a table that piles up and becomes the quality gate that gets more honest with every real failure it discovers.

This matters because it's the difference between an open loop and a closed one, and it's the module's most expensive error. An open loop captures feedback and lets it die on a dashboard: you spend the effort of collecting and get no improvement —lesson 1's suggestion box no one reads—. A closed loop turns every thumbs_down into a concrete system improvement. And of the three possible destinations of the feedback —the eval-set, the prompt, and the retrieval—, this lesson focuses on the most fundamental: the eval-set, because it's the safety net. A case you add to the eval-set is watched forever: any future change that breaks it again is detected before the deploy. The other two destinations (prompt and retrieval) fix the failure; the eval-set guarantees it doesn't come back. That's why, no matter what, every reported failure goes to the eval-set first.

Connection with the module: this lesson is where the loop closes. Lesson 2 showed you the wheel only spins if you close it; lesson 3 gave you the eyes (observability); lesson 4 the raw material (the captured feedback); here you turn that raw material into a verifiable improvement. And it's the reunion with module 3: the eval-set we built there as a static gate becomes alive here —it grows with reality—. Lesson 6 will open the other improvement levers (prompt, RAG, fine-tune) and lesson 7 will route each failure to its lever, but they all share this safety net: the failure also goes to the eval-set. Closing the loop toward the eval is the gesture that makes it so none of the other improvements can regress without the gate noticing.

Analogy: the pilot's "never again" list

In aviation there's a practice that saved countless lives: every incident —a near-miss, a procedural error, a confusing instrument reading— is investigated, and out of that investigation comes a permanent change to the checklists. If a pilot almost took off with the flaps misconfigured because an alarm wasn't clear, it's not enough for that pilot to be more careful: an item is added to the checklist that all pilots review before every takeoff, forever. The individual incident becomes a permanent verification. That's why flying gets safer every decade: the industry has a closed loop that turns every failure into a barrier that prevents that failure from repeating.

That's exactly the idea of closing the loop toward the eval-set. A thumbs_down —a case where the support agent answered badly— is the "near-miss". It's not enough for someone to fix that particular response (the equivalent of "that pilot should be more careful"): that fixes today's case but doesn't prevent a future change from breaking it again. Closing the loop toward the eval is adding the item to the permanent checklist: the reported case becomes an eval-set case that every future version of the component has to pass before deploying. The individual failure becomes a permanent barrier. And that's why a system with a closed loop becomes more and more robust: its eval-set accumulates all the failures that ever happened to it, and none can come back without the gate stopping it.

The other side of the analogy is the warning. An airline that investigates incidents but doesn't update the checklists learns nothing: the same accidents repeat, because nothing changed in the procedure. That's the open loop —capturing the failure and not turning it into a barrier—, and in aviation it costs lives; in an AI system it costs the quality degrading silently while the team believes it "already has feedback".

Worked example: the feedback becomes cases, and the gate gains eyes

We're not going to say that closing the loop improves the gate: we're going to execute it. We model the complete closing: we take the support agent's feedback log, convert the thumbs_down (with their correction) into new eval-set cases, and then run two candidate versions of the agent against the old eval-set and against the augmented one. You'll see the key moment: a version the old eval approved is blocked by the augmented eval, because now the gate tests the cases the users reported.

# Lesson 05 (M7) — CLOSE the loop: feedback becomes the eval-set. SIMULATED.
# Zero network, zero API, zero keys. Deterministic.
#
# The loop "closes" when the feedback RETURNS to something that improves the system.
# Here we feed it back to the EVAL-SET (M3's gate): the cases the user reported
# as bad become new cases, and the gate stops being blind to them.

# --- The ORIGINAL eval-set (M3): 6 "easy" cases with must_contain criteria. ---
EVAL_SET_V1 = [
    {"id": "q1", "question": "where is my order",        "must_contain": "tracking"},
    {"id": "q2", "question": "how do I return a product", "must_contain": "return"},
    {"id": "q3", "question": "how long does shipping take", "must_contain": "3 to 5 business days"},
    {"id": "q4", "question": "can I pay in installments", "must_contain": "installments"},
    {"id": "q5", "question": "I want to cancel my order", "must_contain": "cancel"},
    {"id": "q6", "question": "how do I contact a seller", "must_contain": "messages"},
]

# --- The production feedback log: the thumbs_down carry the human's CORRECTION
#     (the correct datum that should have been given). That correction is the new case's criterion. ---
FEEDBACK_LOG = [
    {"input": "the product arrived broken", "thumbs": "down", "correction": "refund"},
    {"input": "my coupon doesn't work",   "thumbs": "down", "correction": "expiration"},
    {"input": "I didn't receive my invoice", "thumbs": "down", "correction": "email"},
    {"input": "how do I change my address", "thumbs": "down", "correction": "profile"},
    {"input": "where is my order",   "thumbs": "up",   "correction": None},   # up: generates no case
]

def feedback_to_eval_cases(log, start=1):
    # The conversion: each thumbs_down with a correction becomes an eval-set case.
    # input -> question ; correction -> must_contain (the success criterion).
    cases = []
    for i, r in enumerate(log, start=start):
        if r["thumbs"] == "down" and r["correction"]:
            cases.append({"id": f"fb{i}", "question": r["input"], "must_contain": r["correction"]})
    return cases

new_cases = feedback_to_eval_cases(FEEDBACK_LOG)
EVAL_SET_V2 = EVAL_SET_V1 + new_cases

print("=== Close the loop: feedback -> eval-set cases ===")
print(f"  eval-set v1          : {len(EVAL_SET_V1)} cases")
print(f"  thumbs_down -> cases : {len(new_cases)}  {[c['id'] for c in new_cases]}")
print(f"  eval-set v2          : {len(EVAL_SET_V2)} cases  (coverage +{len(new_cases)})")
print()

# --- Two candidate agent versions, simulated with deterministic stubs. ---
# "known" = the set of inputs the version answers well.
EASY = {"where is my order", "how do I return a product", "how long does shipping take",
        "can I pay in installments", "I want to cancel my order", "how do I contact a seller"}
REPORTED = {"the product arrived broken", "my coupon doesn't work",
            "I didn't receive my invoice", "how do I change my address"}

GOLD = {
    "where is my order": "You can see the tracking in your profile.",
    "how do I return a product": "For a return, go to your order.",
    "how long does shipping take": "Shipping takes 3 to 5 business days.",
    "can I pay in installments": "You can pay in installments with no interest.",
    "I want to cancel my order": "You can cancel the order if it wasn't shipped.",
    "how do I contact a seller": "Message the seller from Messages.",
    "the product arrived broken": "You can request a refund from the order.",
    "my coupon doesn't work": "Check the coupon's expiration; it may have expired.",
    "I didn't receive my invoice": "We resend the invoice to your account's email.",
    "how do I change my address": "Change your address in the Profile section.",
}
POOR = "Sorry, I don't have information about that."

def make_agent(known):
    def agent(question):
        return GOLD[question] if question in known else POOR
    return agent

# IMPROVED version: learned the reported cases (the team fixed the prompt).
improved = make_agent(EASY | REPORTED)
# CHEAPER version: a more economical model (M2 cascade). Answers the common
# cases well (grazes the threshold on the old eval) but NEVER learned the reported ones.
cheaper = make_agent(EASY - {"how do I contact a seller"})

def run_eval(agent, eval_set):
    passed = sum(c["must_contain"] in agent(c["question"]).lower() for c in eval_set)
    return passed, len(eval_set), passed / len(eval_set)

THRESHOLD = 0.80
def show(name, agent):
    p1, n1, s1 = run_eval(agent, EVAL_SET_V1)
    p2, n2, s2 = run_eval(agent, EVAL_SET_V2)
    g1 = "PASS" if s1 >= THRESHOLD else "FAIL"
    g2 = "PASS" if s2 >= THRESHOLD else "FAIL"
    print(f"  {name}")
    print(f"     vs eval-set v1 (old)      : {p1}/{n1} = {s1:.2f}  [{g1}]")
    print(f"     vs eval-set v2 (augmented): {p2}/{n2} = {s2:.2f}  [{g2}]")

print(f"=== The gate with the old eval-set vs the augmented (threshold {THRESHOLD:.2f}) ===")
show("IMPROVED version (learned from the feedback)", improved)
show("CHEAPER version (economical model, didn't learn the reported)", cheaper)
print()
print("The lesson in one line:")
print("  the CHEAPER version PASSES the old eval-set (0.83) but FAILS the augmented (0.50).")
print("  Without closing the loop, that version blind to the reported cases would have")
print("  been deployed. The feedback turned into cases gave the gate EYES.")

What to expect. When you run it, the output is exactly this:

=== Close the loop: feedback -> eval-set cases ===
  eval-set v1          : 6 cases
  thumbs_down -> cases : 4  ['fb1', 'fb2', 'fb3', 'fb4']
  eval-set v2          : 10 cases  (coverage +4)

=== The gate with the old eval-set vs the augmented (threshold 0.80) ===
  IMPROVED version (learned from the feedback)
     vs eval-set v1 (old)      : 6/6 = 1.00  [PASS]
     vs eval-set v2 (augmented): 10/10 = 1.00  [PASS]
  CHEAPER version (economical model, didn't learn the reported)
     vs eval-set v1 (old)      : 5/6 = 0.83  [PASS]
     vs eval-set v2 (augmented): 5/10 = 0.50  [FAIL]

The lesson in one line:
  the CHEAPER version PASSES the old eval-set (0.83) but FAILS the augmented (0.50).
  Without closing the loop, that version blind to the reported cases would have
  been deployed. The feedback turned into cases gave the gate EYES.

Read the result in two parts, because each is half the lesson.

Part 1: the mechanical conversion of feedback to cases. The feedback log had five entries: four thumbs_down (with their correction) and one thumbs_up. The feedback_to_eval_cases function converts only the thumbs_down into cases —the thumbs_up generates no case, because a success isn't a failure to watch—, and the conversion is direct: the user's input becomes the question, and the correction becomes the must_contain (the success criterion). So "the product arrived broken" with correction "refund" becomes a case that tests: does the agent's response contain "refund"? The eval-set went from 6 cases to 10. Notice how elegant the mechanism is: the correction the human agent made in production —the correct datum the model should have given— is exactly the new case's success criterion. The human, by correcting, already gave you the answer key.

Part 2: the gate gains eyes. Here's the punch. Two candidate versions are tested against the two eval-sets. The improved version —which learned the reported cases— passes both (1.00 and 1.00): correct, it's a good version and both gates confirm it. But look at the cheaper version —a more economical model that answers the common cases well but never learned the reported ones—. Against the old eval-set, it scores 0.83: PASSES (grazes the threshold, but passes). Against the augmented eval-set, it scores 0.50: FAILS. It's the same version, with two opposite verdicts, and the one that matters is the augmented one's, because it tests the cases the users actually had. Without closing the loop, this cheaper version —blind to the four failures the customers reported— would have been deployed with the old eval's blessing. The feedback turned into cases is what gave the gate the eyes to see it.

Stop at the asymmetry, because it's the reason for the lesson: the old eval had a blind spot exactly in the cases where the users complained, and that's why it gave a false reassurance. Closing the loop doesn't make the gate stricter on a whim; it makes it honest, because it adds exactly the cases reality proved to fail. An eval-set that only tests what already works is an exam that only asks what you already know: you always pass, and it means nothing.

The three destinations of the feedback (and why the eval goes first)

The example closed the loop toward the eval-set, but the feedback has three possible destinations, and a good loop uses all three. It's worth seeing them, and understanding why the eval always goes first.

Destination 1: the eval-set (the safety net). Every reported failure becomes an eval-set case. This doesn't fix the failure —the new case, at first, the current version will fail— but it guarantees that, once fixed, it can't come back without the gate noticing. It's the pilot's "never again" list: the case is watched forever. That's why the eval goes first: it's the permanent barrier, independent of how you fix the failure. A failure you fix only in the prompt but don't add to the eval can come back the day someone changes the prompt again.

Destination 2: the prompt (the quick fix). Many failures are fixed by adjusting the prompt: adding an instruction ("if the customer says the product arrived broken, offer a refund"), adding an example (few-shot) of the failing case, or clarifying the expected format. It's the cheapest and most reversible lever (lesson 6), and it's usually the first attempt. The correction you captured in the feedback can be turned into a few-shot example almost directly. Boundary: how a good prompt is designed is AI Engineering; here, that the feedback feeds the prompt.

Destination 3: the retrieval (when information is missing). If the failure is that the model didn't have the information —it didn't know the warranty policy because that document wasn't in its context—, the correct destination is the retrieval (RAG): add the missing document to the index so it's retrieved next time. The feedback's correction tells you what information was missing. Boundary: how the RAG index is built is AI Engineering; here, that the feedback signals what to add.

Lesson 7 is precisely about routing each failure to the correct destination (do I fix it in the prompt, in the retrieval, or consider fine-tune?). But the rule that cuts across all three, and that's installed here, is: whatever the destination of the fix, the case also goes to the eval-set. The eval is the safety net that makes any fix verifiable —after improving the prompt or the retrieval, you run the augmented eval and confirm that the case now passes and that you didn't break any other—. Without that net, you don't know if your fix worked nor if it caused a regression elsewhere.

Common mistakes

Not closing the loop: capturing feedback and never using it (of process). What happens: it's the module's most expensive error. The team captures thumbs_down —it has the observability, it has the feedback loop— but the data piles up in a table no one turns into improvements. Not one thumbs_down became an eval case, nor a prompt adjustment, nor a retrieval document. It's the airline that investigates incidents but doesn't update the checklists: the same failures repeat. Why it happens: capturing is visible (a button appears); closing is invisible architecture work that competes for priority and always loses. How to detect it: if you have a month of feedback and your eval-set didn't grow, your loop is open. How to fix it: make the feedback→eval conversion a routine (ideally automated): every thumbs_down with a correction becomes a candidate eval-set case. Lesson 7 assembles the complete flow.

Fixing the particular case without adding it to the eval-set (of omission). What happens: the team sees a thumbs_down, fixes the response in the prompt, and considers the case closed —without adding it to the eval-set—. The failure was fixed today, but since it didn't stay in the gate, a future change (another prompt adjustment, a cheaper model) breaks it again, and no one finds out until a customer complains about the same problem again. It's "that pilot should be more careful" instead of adding the item to everyone's checklist. Why it happens: fixing the prompt feels like solving the problem; adding the case to the eval is an extra step that seems redundant ("I already fixed it"). How to detect it: if your fixes of reported failures don't leave a new case in the eval-set, you have no protection against the regression. How to fix it: the hard rule —every reported failure goes to the eval-set first, wherever it's fixed—; the eval is what prevents the failure from coming back.

Adding the thumbs_up to the eval as a "passing case" and inflating the score (of method). What happens: in the eagerness to "use all the feedback", the team also converts the thumbs_up into eval-set cases. Since those cases already pass (that's why they got up), they inflate the score artificially: the eval-set fills with easy cases the component already resolves, and the score rises without the real quality improving —the exam gets easier, not the student smarter—. Why it happens: it seems logical to "make use of all the feedback", and a higher score feels like progress. How to detect it: if your eval-set grows but fills with cases that always pass, your score rises without meaning anything. How to fix it: the cases that go to the eval-set are the ones that failed (the thumbs_down with correction), because they're the ones that reveal blind spots; the successes add no quality information. (Choosing how many and which cases to keep the eval-set representative is eval design, AI Engineering; the simple rule here is: the failures go in, the successes don't inflate.)

Exercises

Exercise 1 — The "never again" list. Explain, with the aviation analogy, why fixing a thumbs_down only in the prompt (without adding it to the eval-set) is like telling a pilot "be more careful" instead of updating the checklist. Then say what the eval-set guarantees that the prompt fix alone doesn't.

See solution

Fixing a thumbs_down only in the prompt is like "that pilot should be more careful" because it fixes today's case but doesn't create a permanent barrier. In aviation, asking a pilot to be careful doesn't prevent another (or the same one, on a bad day) from making the same error tomorrow; only updating the checklist everyone reviews turns the incident into a lasting protection. With the prompt it's the same: you adjusted the prompt so it answers "the product arrived broken" well today, but nothing prevents a future change —another adjustment, a cheaper model— from breaking that case again. The fix is real, but it's fragile and temporary.

What the eval-set guarantees and the prompt alone doesn't: that the failure can't come back without being detected. By adding the case to the eval-set, every future version of the component has to pass that case before deploying; if a change breaks it, the gate blocks it (as you saw with the cheaper version: 0.50, FAIL). The prompt fixes; the eval-set protects. That's why the rule is: the failure goes to the eval-set first, and then you fix it where it belongs —the eval is the permanent checklist, the prompt is the fix of today's flight—.

Exercise 2 — The two verdicts of the cheaper version. In the example, the cheaper version scored 0.83 (PASS) against the old eval and 0.50 (FAIL) against the augmented one. Explain why the augmented eval's verdict is the correct one, and what would have happened in production if the team had deployed this version trusting the old eval.

See solution

The augmented eval's verdict (0.50, FAIL) is the correct one because it tests the cases that the users actually had and reported as bad —"the product arrived broken", "the coupon doesn't work", "I didn't receive the invoice", "how do I change my address"—. The cheaper version fails exactly those cases (it never learned them), and those cases are a real part of the traffic. The old eval gave 0.83 (PASS) only because it didn't test those cases: its high score was a false reassurance produced by a blind spot, not by quality. An exam that doesn't include the questions you'd fail gives you a high grade that means nothing.

If the team had deployed the cheaper version trusting the old eval, in production this would have happened: the common cases (tracking, shipping, installments) would keep working —that's why the old eval approved it—, but every customer who asked about a broken product, an expired coupon, an invoice, or an address change would get a bad response ("Sorry, I don't have information about that"). The live approval_rate would drop, the thumbs_down would pile up, and the team would discover the problem from the complaints —weeks later—, when the augmented eval would have blocked it before the deploy. Closing the loop turned an invisible-until-the-complaints regression into a caught-before-the-deploy regression. That's the whole value.

Exercise 3 — Which destination does each failure go to? For each of these reported failures of the support agent, say which feedback destination you'd route it to primarily (prompt, retrieval, or considering fine-tune) and why, and confirm which destination all of them receive without exception. (a) The agent answers correctly but always in a tone that's too dry, without greeting. (b) The agent doesn't know the warranty policy for refurbished products because that document isn't in its context. (c) The agent makes up shipping tracking numbers that don't exist.

See solution
  • (a) Dry tone, no greeting → the prompt. It's an instruction/format problem, systematic and cross-cutting across many responses. It's fixed by adding a tone instruction to the prompt ("greet and use a warm tone"). It's the cheapest and most reversible lever. It needs neither retrieval (no information is missing) nor fine-tune (it's a simple instruction adjustment).
  • (b) Doesn't know the warranty policy because the document is missing → the retrieval (RAG). The problem is that information is missing from the model's context, not that the model reasons badly. The correct destination is to add the warranty document to the retrieval index so it's retrieved when it applies. Putting the entire warranty policy in the fixed prompt doesn't scale if there are many policies; retrieving it when needed does.
  • (c) Makes up tracking numbers (data hallucination) → this is guardrail/verification (module 4/5) more than a loop improvement lever. The underlying fix is not letting the model assert a tracking number without verifying it against the source of truth (verification, module 5) and validating the output (guardrail, module 4). The loop's feedback here serves to detect the problem, but the solution is containment, not prompt/retrieval/fine-tune. (If the pattern were "the model doesn't distinguish when it has the datum", a prompt adjustment so it says "let me verify" helps —so there's a prompt component—.)

And the destination they all receive without exception: the eval-set. Each of the three failures becomes an eval-set case (with the appropriate criterion: tone for a, presence of the correct policy for b, absence of a made-up number for c), so none can come back without being detected. The fix lever changes with the failure; the eval's safety net is universal.

Summary and next step

In this lesson you learned to close the loop, the module's heart: the cases a user reported as bad, with their correction, become new cases of the eval-set —module 3's—, and that augmented eval-set catches regressions the old one let pass. You saw it with the analogy of aviation's "never again" list —every incident becomes a permanent verification that prevents that failure from repeating— and you measured it: the feedback turned 4 thumbs_down into 4 new cases (the eval-set grew from 6 to 10), and a cheaper version the old eval approved (0.83) the augmented eval blocked (0.50), because now the gate tests the cases the users reported. You understood the three destinations of the feedback —eval-set, prompt, retrieval— and the rule that cuts across them: whatever the fix, the case also goes to the eval-set, which is the safety net that makes any improvement verifiable.

Before moving on you should be able to: explain the mechanical conversion of feedback to eval cases (input → question, correction → criterion); explain why the augmented eval is more honest (not stricter on a whim); name the three destinations of the feedback and why the eval goes first; and recognize the most expensive error (capturing and not closing) and its two variants (fixing without adding to the eval; inflating the eval with successes).

What follows is opening the improvement levers that destinations 2 and 3 anticipated. When the feedback tells you the component is failing, do you adjust the prompt, set up a RAG, or train a fine-tune? In lesson 6 you'll see those three options as an architectural decision: each has a different profile of latency, cost, data freshness, and maintenance, and choosing wrong —jumping to fine-tune when the prompt was enough— is an expensive error. You'll execute the comparison and a recommender that chooses per feature. It's the step from "I closed the loop toward the eval" to "I know which lever to move to fix what the eval revealed, and what each one implies".

Resources

  • Chip Huyen — AI Engineering (O'Reilly) — the treatment of how production feedback becomes evaluation and improvement data; the conceptual backing for turning reported failures into eval cases (the mechanics of designing the representative eval-set is its boundary).
  • Chip Huyen — Designing Machine Learning Systems (O'Reilly) — the chapter on feedback loops and continuous evaluation treats closing the loop as the mechanism that keeps an ML system healthy in production; the frame of this lesson.
  • Anthropic — Claude docs, evaluating applications — the conceptual guide to building and growing a set of evaluation cases from real failures; without pinning a model version. In English.
  • architecture-decisions-and-tradeoffs-guide — Fitness functions — the eval as a fitness function that governs quality; this lesson makes it alive (it grows with the feedback), but the root concept of the fitness function that's maintained over time lives there.
  • AI Engineering ecosystem (referral) — for the design of the eval-set (which cases, how many, how to keep it representative as the feedback grows it); this module closes the loop toward the eval, AI Engineering designs the set.
← Previous4. The feedback loop as a design decisionNext →6. Prompt vs RAG vs fine-tune as an architectural choice