Module 7: The Data and Feedback Loop

7. Routing the feedback signal to the right lever

Overview

By the end of this lesson you'll know how to assemble the complete data loop, uniting everything from the module into a flow that works: the feedback is grouped by failure type, each group is routed to the lever that corresponds to it —prompt, retrieval, or considering fine-tune—, every failure also goes to the eval-set as a safety net, and the eval verifies that the fix improved without regressing. It's the module's synthesis because it brings the pieces together: the capture (lesson 4) produces the feedback, closing the loop toward the eval (lesson 5) is the universal safety net, and the levers (lesson 6) are the fixing tools. What was missing was the routing: given a concrete failure, which lever fixes it? The thesis is that the failure type determines the lever —not all failures are fixed the same way—, and that the operational rule is to start with the cheapest lever that works.

This matters because a loop without routing is a loop that applies the same tool to all problems —the hammer for everything—, and that wastes effort and sometimes fixes nothing. A failure where the correct datum existed but wasn't retrieved isn't fixed by touching the prompt; it's fixed by improving the retrieval. A systematic tone or format failure isn't fixed by adding documents to the index; it's fixed in the prompt. A failure that is a whole class of behavior that neither the prompt nor the retrieval achieves, and that is stable and high-volume, is the rare case where considering fine-tune makes sense. Routing well is diagnosing the failure type and sending it to its lever; routing badly is spending weeks on a fine-tune for a problem a prompt adjustment solved in an afternoon.

Connection with the module: this is the lesson that closes the arc. You went through the flywheel (2), the observability (3), the capture (4), the closing toward the eval (5), and the levers (6); here you assemble them into the complete and executable data loop. And it prepares the project: in lesson 8 you'll set up this entire loop over the semantic search. The boundary stays firm: how each lever is implemented (indexing the retrieval, training the fine-tune) is AI Engineering; routing the feedback to the correct lever and verifying with the eval is from here.

Analogy: the emergency room triage

In a hospital's emergency room there's a process that decides everything else: the triage. When patients arrive, an experienced nurse doesn't treat them in order of arrival nor send them all to the same specialist. They classify them by type and severity, and route each one to where it belongs: the broken bone goes to trauma, the chest pain goes to cardiology with top priority, the cold goes to general consultation and can wait. The triage cures no one —the specialists do that—; its job is to diagnose the type and send each case to the correct lever, in the correct order. A hospital without triage, treating everyone the same and in order of arrival, would collapse: it would send the heart attack to wait behind the cold and the cold to occupy a cardiologist.

Closing the data loop is exactly a triage of the feedback. The thumbs_down that arrive aren't all the same problem: some are "the datum existed but wasn't retrieved" (they go to retrieval), others are "the tone is wrong in all the responses" (they go to the prompt), others are "there's a whole class of task the system doesn't master" (they go to considering fine-tune). Your job when closing the loop isn't to apply the same fix to all —that's the hospital without triage—; it's to classify each failure by its type and route it to the lever that cures it. And like in the hospital, there's a priority order: you attend first to the cheapest and what hurts the most people, and reserve the most expensive lever (fine-tune, the surgery) for the cases that really need it.

The analogy also illuminates the safety net. In a good hospital, besides curing each patient, each case is recorded in the history —to detect patterns, so the same error doesn't repeat, to learn—. That record is the eval-set: no matter what happens with the triage, each failure is noted permanently, so that if it appears again, the system already recognizes it. Curing (the lever) and recording (the eval) are distinct things, and both are necessary.

Worked example: the feedback triage, executed end to end

We're not going to describe the complete loop: we're going to execute it. We model the closing of the support agent's loop end to end: we take the feedback labeled by failure type, group it, route each group to its lever, put all the failures into the eval-set, apply the cheap and reversible levers, and re-run the eval to prove the system improved without regressing. Notice the score rising at the end: that's the proof the loop really closed.

# Lesson 07 (M7) — route the feedback SIGNAL to the correct lever. SIMULATED.
# Zero network, zero API, zero keys. Deterministic.
#
# Feedback isn't fixed in a single way. It's GROUPED by failure type and each
# group is routed to its lever: prompt / retrieval(RAG) / eval-only / consider-finetune.
# And ALWAYS, no matter what, the failed cases enter the eval-set (the safety net).

# The feedback_loop, already labeled by FAILURE TYPE (the diagnosis):
#   retrieval_miss : the correct datum EXISTS but wasn't retrieved/shown
#   prompt_gap     : systematic tone/format/instruction problem
#   knowledge_gap  : a whole stable behavior class that prompt+RAG don't cover
FEEDBACK_LOG = [
    {"id": "f1",  "input": "return policy for electronics",           "failure": "retrieval_miss"},
    {"id": "f2",  "input": "warranty for a refurbished product",      "failure": "retrieval_miss"},
    {"id": "f3",  "input": "hours of operation on holidays",          "failure": "retrieval_miss"},
    {"id": "f4",  "input": "answers very curtly and without a greeting", "failure": "prompt_gap"},
    {"id": "f5",  "input": "doesn't use the list format we asked for", "failure": "prompt_gap"},
    {"id": "f6",  "input": "translate legal jargon of B2B contracts",  "failure": "knowledge_gap"},
    {"id": "f7",  "input": "classify 40 rare complaint subtypes",      "failure": "knowledge_gap"},
    {"id": "f8",  "input": "didn't find the manual for product X",     "failure": "retrieval_miss"},
]

from collections import Counter
clusters = Counter(r["failure"] for r in FEEDBACK_LOG)

# The router: maps a failure type to the appropriate architectural lever.
# BOUNDARY NOTE: how each lever is IMPLEMENTED (indexing, training) is AI Eng.
def route(failure_type):
    return {
        # The datum exists but didn't arrive: improve the RETRIEVAL (RAG).
        "retrieval_miss": ("retrieval (RAG)", "the datum exists; it just wasn't retrieved/shown"),
        # Systematic tone/format/instruction: fixed in the PROMPT.
        "prompt_gap":     ("prompt",          "instruction/format: cheap and reversible, start here"),
        # A whole stable class that prompt+RAG don't cover: CONSIDER fine-tune (with its costs).
        "knowledge_gap":  ("consider fine-tune", "stable + volume; weigh re-training cost (L6)"),
    }[failure_type]

print("=== Feedback grouped by failure type, and its lever ===")
print(f"{'failure type':<16}{'cases':>6}   lever -> reason")
for ftype, count in clusters.most_common():
    lever, reason = route(ftype)
    print(f"{ftype:<16}{count:>6}   {lever}  ->  {reason}")
print()

# The safety net: ALL failed cases enter the eval-set, no matter how they're fixed.
new_eval_cases = [r["id"] for r in FEEDBACK_LOG]
print(f"Safety net: the {len(new_eval_cases)} failed cases enter the eval-set "
      f"(M3), no matter the lever.")
print(f"  {new_eval_cases}")
print()

# --- The loop, closed and VERIFIED: apply the levers and re-run the eval. ---
# Model: the system "resolves" a case if its lever was applied. The eval measures
# the fraction of cases (old + new) resolved, before and after closing the loop.
BASE_SOLVED = 6      # the system already resolved 6 base cases of the original eval-set
BASE_TOTAL = 6

# Before closing the loop: the augmented eval-set includes the 8 failures, unresolved.
before_solved = BASE_SOLVED
before_total = BASE_TOTAL + len(FEEDBACK_LOG)

# We apply the cheap and reversible levers NOW (prompt + retrieval); fine-tune
# stays as a proposal to evaluate, not applied in this iteration.
applied_now = {"retrieval_miss", "prompt_gap"}
fixed = sum(1 for r in FEEDBACK_LOG if r["failure"] in applied_now)
deferred = sum(1 for r in FEEDBACK_LOG if r["failure"] not in applied_now)

after_solved = BASE_SOLVED + fixed
after_total = BASE_TOTAL + len(FEEDBACK_LOG)

THRESHOLD = 0.80
def gate(s):
    return "PASS" if s >= THRESHOLD else "FAIL"

s_before = before_solved / before_total
s_after = after_solved / after_total
print("=== The loop closed and verified by the eval (threshold 0.80) ===")
print(f"  before closing the loop  : {before_solved}/{before_total} = {s_before:.2f}  [{gate(s_before)}]")
print(f"  after applying prompt+RAG: {after_solved}/{after_total} = {s_after:.2f}  [{gate(s_after)}]")
print(f"  cases still unresolved   : {deferred}  (the knowledge_gap: fine-tune decision pending)")
print()
print("The complete loop: observe -> capture -> group -> route to the cheapest")
print("lever that works -> re-run the eval to PROVE it improved without regressing.")

What to expect. When you run it, the output is exactly this:

=== Feedback grouped by failure type, and its lever ===
failure type     cases   lever -> reason
retrieval_miss       4   retrieval (RAG)  ->  the datum exists; it just wasn't retrieved/shown
prompt_gap           2   prompt  ->  instruction/format: cheap and reversible, start here
knowledge_gap        2   consider fine-tune  ->  stable + volume; weigh re-training cost (L6)

Safety net: the 8 failed cases enter the eval-set (M3), no matter the lever.
  ['f1', 'f2', 'f3', 'f4', 'f5', 'f6', 'f7', 'f8']

=== The loop closed and verified by the eval (threshold 0.80) ===
  before closing the loop  : 6/14 = 0.43  [FAIL]
  after applying prompt+RAG: 12/14 = 0.86  [PASS]
  cases still unresolved   : 2  (the knowledge_gap: fine-tune decision pending)

The complete loop: observe -> capture -> group -> route to the cheapest
lever that works -> re-run the eval to PROVE it improved without regressing.

Read the output in three parts, because together they're the complete loop.

The triage: group and route. The first table is the triage in action. The eight captured failures were grouped by type, and each group was routed to its lever. The four retrieval_miss —cases where the datum existed but wasn't retrieved (the return policy, the warranty, the holiday hours, the product manual)— go to retrieval (RAG): the fix is to make sure those documents are in the index and get retrieved. The two prompt_gap —curt tone, incorrect format— go to the prompt: they're instruction problems, and they're fixed there, which is the cheapest. The two knowledge_gap —translate legal jargon, classify 40 rare subtypes— go to consider fine-tune, with lesson 6's warning: only if they're stable and of sufficient volume to justify the cost. Notice that the routing diagnoses each failure instead of applying the same tool to all: it's the triage that sends the broken bone to trauma and the chest pain to cardiology.

The safety net: everyone to the eval-set. The middle line is lesson 5's universal rule, applied without exception: the eight failed cases enter the eval-set, whatever lever they're fixed with. The tone failure goes to the eval (with a tone criterion), the missing datum goes to the eval (with a policy-presence criterion), the specialized task goes to the eval (with its criterion). Curing is the lever; recording is the eval. Both are necessary, and the eval is universal: no failure is closed without leaving its case in the gate, so it can't come back.

The verification: the score rises, and it's proven. The last part is what distinguishes a closed loop from a good intention. Before fixing anything, the augmented eval-set —which now includes the eight failures— gives 6/14 = 0.43: FAILS. That's honest: the system resolves the six base cases, but fails the eight the feedback revealed. Then we apply the cheap and reversible levers (prompt + retrieval, which fix six of the eight) and leave pending the fine-tune decision for the two knowledge_gap (which requires more cost analysis). We re-run the eval: 12/14 = 0.86: PASSES. The score rose from 0.43 to 0.86, and —this is crucial— we didn't assert it, we measured it with the augmented eval. The loop closed and it was proven to close. The two cases left unresolved are the fine-tune ones, correctly deferred: we didn't jump to the expensive lever in a hurry, we applied all the cheap ones first and left the costly decision for a separate analysis.

The lesson in one sentence: the loop closes with a triage —group the feedback by type, route each group to the cheapest lever that works, record everything in the eval— and it's verified by re-running the eval, which proves you improved without regressing.

The flow of the complete loop, and the rules that govern it

The example executed the loop; it's worth seeing it as the complete flow of the entire module, with the rules that make it work.

flowchart TD
    U["Usage (production)"] --> O["Observability<br/>(quality signal, L3)"]
    O --> C["Feedback capture<br/>(thumbs / correction / action, L4)"]
    C --> G["Group by failure type<br/>(the triage)"]
    G --> R{"Route<br/>by type"}
    R -->|"missing datum"| RAG["retrieval / RAG"]
    R -->|"tone / format"| PR["prompt"]
    R -->|"stable class +<br/>high volume"| FT["consider fine-tune (L6)"]
    G --> EV["ALL to the eval-set<br/>(safety net, L5)"]
    RAG --> RE["Re-run the eval<br/>(verify: improved without regressing)"]
    PR --> RE
    FT --> RE
    EV --> RE
    RE --> U

Rule 1: diagnose the failure type before choosing the lever. The correct routing depends on classifying well. A failure of "the model didn't have the datum" and one of "the model had the datum but said it with a bad tone" look similar in the customer's complaint, but they go to different levers (retrieval vs prompt). The triage starts with the diagnosis: what type of failure is it? This is usually done by grouping the thumbs_down by pattern —many similar cases reveal a systematic failure type—.

Rule 2: start with the cheapest lever that works. It's lesson 6's ladder, applied to the routing. Facing a failure, the preference order is prompt (cheap, instant) → retrieval (freshness/volume) → fine-tune (expensive, rigid). In the example, we applied prompt and retrieval immediately (cheap and reversible) and deferred fine-tune. It's not that fine-tune is forbidden; it's that it's reserved for when the cheap levers fall short, and its decision deserves a separate cost analysis (is the behavior stable?, does the volume justify re-training?).

Rule 3: every failure goes to the eval-set, no matter what. The universal safety net. Curing the failure (with the lever) and protecting yourself from it coming back (with the eval) are distinct things and both are mandatory. A failure fixed in the prompt but not recorded in the eval can come back the day someone touches the prompt; a failure in the eval is protected forever.

Rule 4: verify by re-running the eval. An unverified fix is a hope, not a fact. After moving the lever, you re-run the augmented eval and confirm two things: that the fixed cases now pass, and that you didn't break any other case (a collateral regression). The score that rises (0.43 → 0.86) is the executed proof that the loop really closed. Without this verification, you wouldn't know if your fix worked nor if it caused damage elsewhere.

These four rules are the complete data loop, and they're the capability you take from the module: observe, capture, group, route to the cheapest lever that works, record everything in the eval, and verify with the eval that you improved without regressing. That cycle, repeated, is lesson 2's flywheel really spinning.

Common mistakes

Applying the same lever to all failures (of lack of triage). What happens: the team has a favorite lever —usually the prompt, sometimes fine-tune— and applies it to all the feedback without diagnosing the failure type. It puts into the prompt information that should be in the retrieval (and the prompt grows until it bursts the context window), or trains a fine-tune for a tone problem a prompt adjustment solved. It fixes some failures by luck and others not, because the tool doesn't match the problem. Why it happens: it's easier to apply the tool you master than to diagnose each case; it's the hospital without triage that treats everyone the same. How to detect it: if all your fixes use the same lever regardless of the failure, you're not routing. How to fix it: classify each failure by type before choosing the lever —missing datum → retrieval, tone/format → prompt, stable class → consider fine-tune—.

Jumping to the expensive lever without exhausting the cheap ones (of inverted order). What happens: facing a failure, the team goes straight to fine-tune (or a complex RAG) without trying the prompt first. It invests weeks in the expensive lever for a problem the cheap one solved in hours. It's the same as lesson 6's over-engineering, but seen from the routing: the triage sent the cold to surgery. Why it happens: the expensive lever has prestige and gives a feeling of "really solving"; the effort order is inverted relative to the prestige one. How to detect it: if you're applying an expensive lever and didn't try the cheap one for that same failure, you inverted the order. How to fix it: go up the ladder —prompt first, retrieval if you need freshness/volume, fine-tune only if the two fall short—; apply the cheap and reversible immediately, and defer the expensive decision to a cost analysis (like the example's knowledge_gap).

Fixing without verifying and believing the loop closed (of process). What happens: the team moves the lever —adjusts the prompt, adds documents to the retrieval— and considers the failure closed without re-running the eval. It doesn't know if the fix really resolved the case, nor if it caused a regression elsewhere. Sometimes the fix of case A broke case B, and no one finds out until a customer complains about B. Why it happens: moving the lever feels like finishing; re-running the eval is an extra step that seems optional. How to detect it: if your fixes don't end with an eval that rises (and no cases that drop), you didn't close the loop, you only tried. How to fix it: always verify by re-running the augmented eval —the score that rises without other cases dropping is the proof that the fix worked and damaged nothing—.

Exercises

Exercise 1 — The triage. Explain, with the emergency-room analogy, why routing the feedback by failure type is better than applying the same lever to all. Then, for these three thumbs_down of the support agent, say which "specialist" (lever) you'd route them to and why: (a) "asked about the refund policy for a digital product and the agent said it didn't know it, even though that policy exists in our documentation"; (b) "the agent answered well but was too informal with the customer when our brand uses a formal tone"; (c) "the customer asked the agent to write the response in the legal dialect of a B2B contract, something we do thousands of times a month with a very specific format".

See solution

Routing by failure type is better than applying the same lever to all for the same reason triage is better than treating everyone the same: each type of problem has a different treatment, and applying the wrong treatment doesn't cure and wastes resources. Sending a missing datum to the prompt (when it should go to retrieval) inflates the prompt without solving the root; training a fine-tune for a tone problem (when the prompt was enough) spends weeks on something that was fixed in hours. The triage sends each case to the lever that cures it.

(a) Policy that exists but the agent didn't know → retrieval (RAG). The datum exists in the documentation but didn't reach the model: it's a retrieval_miss. The fix is to make sure that document is in the index and gets retrieved. It's not a prompt problem (the model reasons well, it's missing the datum) nor fine-tune (no need to bake anything, just retrieve the correct document).
(b) Was informal when the brand is formal → prompt. It's a systematic tone/format problem: a prompt_gap. It's fixed by adding a tone instruction to the prompt ("always use a formal tone"). Cheap, instant, reversible. It needs neither retrieval (no information is missing) nor fine-tune (it's a simple instruction adjustment).
(c) Write in B2B legal dialect, thousands of times a month, very specific format → consider fine-tune. It's a knowledge_gap: a whole class of behavior (a very specific legal style) that the prompt can approximate but perhaps not with the necessary consistency, and it's stable (the legal format doesn't change) and high-volume (thousands a month). It meets lesson 6's criteria. Even so, "consider" isn't "do": first you try the prompt with examples and measure; only if it falls short do you weigh the cost of fine-tune.

Exercise 2 — The verification that was missing. A team grouped its feedback, routed each group to its lever, applied the fixes, and announced "we closed the loop, the quality improved". But it didn't re-run the eval. Explain what they can't know for having skipped the verification, and describe what would have happened if a failure's fix had broken another case that used to pass.

See solution

For having skipped the verification, the team can't know two things: first, whether their fixes really resolved the failures (moving the lever is a hypothesis; that it worked is a fact only the eval confirms); and second, whether some fix caused a regression —broke a case that used to pass—. They announced "the quality improved" as a belief, not as a datum. It could be true, it could be false, and they have no way to distinguish it.

If a failure's fix had broken another case, here's what would have happened without verification: the team believes it raised the quality (fixed the failures it saw), but in reality it moved it around —a prompt adjustment to fix the tone, for example, could have changed how the model answers another category of questions and broken it—. Since it didn't re-run the augmented eval, the broken case goes unnoticed until a customer complains about that new problem, weeks later. The damage is double: they introduced a regression and believed it was an improvement.

With verification, this doesn't happen: on re-running the augmented eval (which includes both the old cases and the new feedback ones), a broken case would make the score drop or would appear as a case that now fails, and the team would see it before deploying. The rule is that a fix isn't closed until the eval rises and no previous case drops. The score that rises without regressions (0.43 → 0.86 in the example, without breaking the 6 base) is the only proof that the loop really closed. Announcing the improvement without that number is closing the loop blindfolded.

Exercise 3 — The complete loop in a new feature. Mercado launches a new assistant that recommends complementary products ("you bought a camera, want a case?"). After a month, the feedback shows three failure patterns: (a) it recommends accessories that aren't compatible with the purchased product (compatibility data that exists in the product's sheet but the assistant didn't consult); (b) its messages are too pushy and users complain about the tone; (c) for a whole category of technical products (professional photography gear), its recommendations are consistently poor because it doesn't understand that specialized domain, and that category represents a small fraction of the traffic. Design the closing of the loop: route each pattern, say what goes to the eval-set, and give the priority order.

See solution

Routing of each pattern:

(a) Recommends incompatible accessories, with the compatibility datum existing in the sheet → retrieval (RAG). It's a retrieval_miss: the information exists (the product's sheet has the compatibility) but the assistant didn't consult it. The fix is to make sure the compatibility is retrieved and passed to the model before recommending. It's not prompt (the model reasons well, it's missing the datum) nor fine-tune (no need to bake anything).
(b) Too pushy tone → prompt. It's a prompt_gap: a systematic tone problem. It's fixed with a tone instruction in the prompt ("recommend subtly, without insisting"). Cheap, reversible.
(c) Poor recommendations in professional photography, small-traffic category → NOT fine-tune (for now). Although it's a knowledge_gap (a specialized domain the model doesn't master), it doesn't meet lesson 6's volume criterion: the category is a small fraction of the traffic, so the cost of training a fine-tune isn't amortized. The best option is probably RAG (add professional-photography documentation to the index) or improving the prompt with domain context; if even that falls short and the category grew in volume, then fine-tune would be reconsidered. Diagnosing the knowledge_gap well doesn't mean jumping to fine-tune: it means weighing its criteria (here, the volume doesn't justify it).

What goes to the eval-set: the three patterns, with their criterion (correct compatibility for a, subtle tone for b, recommendation quality in photography for c), without exception. The safety net is universal.

Priority order: first (b) —the prompt is the cheapest and the pushy tone affects all users—; then (a) —the compatibility retrieval, which affects many recommendations and is a medium-effort fix—; and (c) last —it affects little traffic and its best lever (domain RAG) requires more work, so it's attended after the cheap and high-impact—. The order follows the rules: the cheapest lever and what hurts the most people, first; the expensive and low-impact, later. And all verified by re-running the eval after each fix.

Summary and next step

In this lesson you assembled the complete data loop, the module's synthesis: the feedback is grouped by failure type, each group is routed to its lever —prompt, retrieval, or considering fine-tune—, every failure goes to the eval-set as a safety net, and the eval verifies that the fix improved without regressing. You saw it with the analogy of the emergency-room triage —classify each case and route it to the correct specialist, in the correct order, recording everything in the history— and you executed it end to end: the eight failures were grouped and routed (retrieval, prompt, consider fine-tune), the eight entered the eval-set, and on applying the cheap levers (prompt+RAG) and deferring fine-tune, the score rose from 0.43 to 0.86, measured and proven by the augmented eval. You understood the four rules of the loop: diagnose the type before choosing the lever, start with the cheapest that works, everything goes to the eval, and verify by re-running the eval.

Before moving on you should be able to: do the triage of a set of failures (classify by type and route to the lever); explain why the order is prompt → retrieval → fine-tune; justify why every failure goes to the eval-set even if it's fixed on another lever; and explain why the verification (re-running the eval) is what distinguishes a closed loop from a good intention.

What follows is the project, where you set up this complete loop with your own hands over a new feature: the semantic search, which learns from a different feedback —the users' clicks—. You'll build the observability with its quality signal, capture the clicks, close the loop by turning them into eval-set cases, and write the ADR of the prompt vs RAG vs fine-tune decision for a live catalog. It's the step from "I understand the data loop" to "I set up a feature's data loop end to end, and proved it by executing it".

Resources

Chip Huyen — AI Engineering (O'Reilly) — the treatment of how production feedback is analyzed, classified by failure type, and feeds the different improvement levers (prompt, RAG, fine-tune); the backing of this lesson, with the mechanics of each lever as its boundary.
Chip Huyen — Designing Machine Learning Systems (O'Reilly) — the iterative improvement cycle of an ML system (observe, diagnose, act, verify) that this lesson applies to the data loop of an AI feature.
martinfowler.com — "Emerging Patterns in Building GenAI Applications", Bharani Subramaniam and Martin Fowler — the pattern catalog places the data loop, the continuous evaluation, and the choice of levers in the complete architectural map; the module's synthesis frame. In English.
AI Engineering ecosystem (referral) — for implementing each lever you route the feedback to: building the retrieval, doing serious prompt engineering, training the fine-tune. This module teaches how to route and verify; AI Engineering teaches how to build the lever.

← Previous6. Prompt vs RAG vs fine-tune as an architectural choice Next →8. Project: build the data loop for a Mercado AI feature