8. Project: build the data loop for a Mercado AI feature

Overview

This is your graduation from the module. Over seven lessons you learned to design the data loop of an AI feature: the flywheel that compounds, the observability that sees the quality, the capture of the three feedback signals, the closing of the loop toward the eval-set, the three improvement levers, and the routing of each failure to its lever. You saw all that applied to the support agent. Now it's your turn, from scratch, over a different Mercado feature: the semantic search. The reason to change features is the usual one and it's hard: if I let you re-set-up the support agent's loop, I wouldn't know whether you learned the method or memorized the table. And the semantic search has a twist that forces applying the method and not the memory: its feedback isn't an explicit thumbs, it's implicit —the users' clicks—. The only way to solve it is to apply what you learned: set up the observability with the correct quality signal, capture the clicks, turn them into eval cases, and decide the lever. That is, exactly, the proof that the module worked.

Your deliverable is four artifacts for the semantic search: (1) the observability with its quality signal (the relevant_click_rate); (2) the feedback_loop that captures the clicks (the implicit action signal); (3) the closing of the loop —the clicks to low positions become eval-set cases and catch a regression the old eval approved—; and (4) a decision brief (ADR-style) that answers the prompt vs RAG vs fine-tune choice for this feature and how the loop composes with the gates of the previous modules. No part requires building the LLM nor computing embeddings: the component is simulated with a deterministic stub. It's pure architecture work —set up the loop, execute it, defend the decision— which is exactly what separates an AI feature that improves over time from one that freezes where it was born. Build it yourself first; reading the reference solution without having tried it is like reading the scoreboard of a game you didn't play.

Connection with the module: this project closes the arc. Lesson 2 showed you the flywheel; lesson 3 the observability; lesson 4 the capture; lesson 5 the closing toward the eval; lesson 6 the levers; lesson 7 the routing. Here you produce the four artifacts with your own hands, from start to finish, over the semantic search —and with a feedback signal (the implicit click) that is not the thumbs of the lessons, so you apply the method—. And with this lesson the module closes: at the end are the summary of the eight lessons and the bridge to module 8 (the guide's capstone, which brings everything together) and to the AI Engineering ecosystem (where the pieces —RAG, fine-tune, embeddings— that here we only decide and feed back are built).

The project's case: Mercado's semantic search

The feature you get to govern —yours to solve— is this:

Mercado has a semantic search: when a customer types "something to listen to music while running", an AI component understands the intent and returns a list of products ordered by relevance. The team wants the search to improve with use: to learn from what the customers do. But the search doesn't have an obvious thumbs up/down —no one is going to rate every search—. Design its data loop.

It's a sibling feature of the support agent's, but with a different feedback profile, and that's why its loop is another. In the agent, the feedback was explicit (thumbs) or a human correction; in the search, the feedback is the user's action: at which position of the ranking they clicked. A click in the top-3 means the search put the relevant thing where the customer sees it; a click at a low position (or a query reformulation with no click) means the search buried the relevant thing —and, revealingly, the product the user ended up clicking is exactly the one that should have come up top—. The click is high-volume implicit feedback: the user produces it anyway, with no extra work, and every search generates it. It's lesson 4's action signal in its purest form.

The facts the team gives you (so you don't have to invent them): you have a production click log with seven searches, each with the ranking shown, the click position, and the clicked product. The quality criterion is the same as the module 3 project: relevant in the top-3. The gate threshold is 0.80. The search's original eval-set has 3 cases (the module 3 ones). And there's a candidate version: a cheaper model (the module 2 cascade) that gets the original cases right but whose real relevance —revealed by the clicks— degraded.

What you have to deliver

Follow the steps in order; each one builds on the previous.

Part 1 — The observability with the quality signal

From the click log, compute the search's quality signal: the relevant_click_rate (fraction of searches where the click landed in the top-3) and the average click position. Explain why the click is the appropriate quality signal for this feature (implicit feedback, free, high-volume) and what it reveals that a classic server log wouldn't see.

Part 2 — The feedback_loop that captures the clicks

Design what the capture point saves for each search so as not to lose the signal (the query, the ranking shown, and the position/product of the click). Remember from lesson 4: what you don't capture isn't recovered —if you only save the query and the result, you lose the click position, which is the signal—.

Part 3 — Close the loop: the clicks become eval-set cases

Convert the clicks to low positions (outside the top-3) into new eval-set cases: the query becomes the case, and the clicked product becomes the "relevant" one that should have come up top. Run the gate with the candidate version against the old eval-set and the augmented one, and show how the augmented one catches the regression the old one approved. Simulate the component with a deterministic stub.

Part 4 — The decision brief (ADR-style)

Write a brief decision document that answers: which lever (prompt, RAG, or fine-tune) would you use for the semantic search and why? How does this feature close the loop (from the click signal to the improvement)? How does the data loop compose with the gates of the previous modules (M2 budgets, M3 eval, M4 guardrails, M5 resilience, M6 shell)? It's the architectural justification of your design.

Try the four parts before looking at the solution. What follows is a reference, not the only correct answer.

Reference solution

Parts 1, 2, and 3 — The loop executed

# Project M7 — the DATA LOOP of Mercado's semantic search. SIMULATED.
# Zero network, zero API, zero keys. Deterministic.
#
# A feature different from the lessons (support agent) on purpose: the search
# learns from an IMPLICIT feedback —the CLICKS—. Which position the user clicked
# IS the relevance signal: click up = good; click down = the ranking buried the
# relevant one, and the product they clicked reveals what should have been up top.

CATALOG = ["earbuds", "running_shoes", "phone_case", "smartwatch", "water_bottle",
           "backpack", "sunglasses", "headphones", "yoga_mat", "power_bank"]

# --- The production CLICK LOG: for each search, the ranking shown and at which
#     position (1-based) the user clicked. If they clicked down, the clicked
#     product is the implicit "correction" (the relevant one the ranking sank). ---
CLICK_LOG = [
    # (query, ranking_shown, click_position, clicked_product)
    ("something to listen to music while running", ["earbuds", "phone_case", "backpack"], 1, "earbuds"),
    ("running shoes",                        ["running_shoes", "yoga_mat", "backpack"], 1, "running_shoes"),
    ("protect my phone",                     ["phone_case", "earbuds", "backpack"], 1, "phone_case"),
    # --- clicks DOWN: the ranking buried the relevant one (implicit failure signal) ---
    ("bottle for the gym",  ["backpack", "yoga_mat", "sunglasses", "smartwatch", "water_bottle"], 5, "water_bottle"),
    ("glasses for the sun", ["backpack", "phone_case", "earbuds", "yoga_mat", "sunglasses"], 5, "sunglasses"),
    ("charge my phone without an outlet", ["backpack", "earbuds", "phone_case", "yoga_mat", "power_bank"], 5, "power_bank"),
    ("watch that counts steps", ["earbuds", "phone_case", "backpack", "sunglasses", "smartwatch"], 5, "smartwatch"),
]

# ===================== PART 1: OBSERVABILITY (quality signal) =====================
TOP_K = 3
clicks_in_top3 = sum(1 for *_r, pos, _p in CLICK_LOG if pos <= TOP_K)
relevant_click_rate = clicks_in_top3 / len(CLICK_LOG)
avg_click_pos = sum(pos for *_r, pos, _p in CLICK_LOG) / len(CLICK_LOG)

print("=== PART 1 — Observability of the semantic search ===")
print(f"  searches logged              : {len(CLICK_LOG)}")
print(f"  clicks in the top-3          : {clicks_in_top3}/{len(CLICK_LOG)}")
print(f"  relevant_click_rate (quality): {relevant_click_rate:.0%}")
print(f"  average click position       : {avg_click_pos:.1f}  (1.0 = ideal)")
print(f"  signal: {'OK' if relevant_click_rate >= 0.80 else 'ALERT: the ranking buries the relevant one'}")
print()

# ===================== PART 2: CLOSE THE LOOP (click -> eval-set) =====================
# The ORIGINAL eval-set of the search (M3): queries with their relevant product, top-3 criterion.
EVAL_SET_V1 = [
    {"id": "s1", "query": "something to listen to music while running", "relevant": "earbuds"},
    {"id": "s2", "query": "running shoes",                   "relevant": "running_shoes"},
    {"id": "s3", "query": "protect my phone",               "relevant": "phone_case"},
]

# Conversion: a click OUTSIDE the top-3 reveals a new case (query -> clicked product).
def clicks_to_eval_cases(log, k=TOP_K, start=1):
    cases = []
    for i, (query, _rank, pos, product) in enumerate(log, start=start):
        if pos > k:   # the user had to scroll down: the relevant one was sunk
            cases.append({"id": f"c{i}", "query": query, "relevant": product})
    return cases

new_cases = clicks_to_eval_cases(CLICK_LOG)
EVAL_SET_V2 = EVAL_SET_V1 + new_cases

print("=== PART 2 — Close the loop: clicks below -> eval-set cases ===")
print(f"  eval-set v1             : {len(EVAL_SET_V1)} cases")
print(f"  clicks outside the top-3: {len(new_cases)} -> {[c['id'] for c in new_cases]}")
print(f"  eval-set v2 (augmented) : {len(EVAL_SET_V2)} cases")
print()

# ===================== PART 3: THE GATE with the augmented eval =====================
def make_search(competent_ids):
    # STUB: for the competent cases it puts the relevant one in the top-3; else at pos 5.
    def search(query, case_id, relevant):
        others = [p for p in CATALOG if p != relevant]
        if case_id in competent_ids:
            return [relevant, others[0], others[1]]
        return others[:4] + [relevant]
    return search

def run_eval(search, eval_set, k=TOP_K):
    passed = 0
    for c in eval_set:
        ranking = search(c["query"], c["id"], c["relevant"])
        passed += c["relevant"] in ranking[:k]
    return passed, len(eval_set), passed / len(eval_set)

ALL_V1 = {c["id"] for c in EVAL_SET_V1}
ALL_V2 = {c["id"] for c in EVAL_SET_V2}
THRESHOLD = 0.80

# Candidate version: a cheaper model. Gets the 3 easy ones (v1) right but sinks
# the relevant one in the cases the CLICKS revealed -> the v2 eval catches it.
cheaper = make_search(ALL_V1)     # only competent in the 3 original cases

def gate(s):
    return "PASS -> deploy allowed" if s >= THRESHOLD else "FAIL -> deploy blocked"

p1, n1, s1 = run_eval(cheaper, EVAL_SET_V1)
p2, n2, s2 = run_eval(cheaper, EVAL_SET_V2)
print(f"=== PART 3 — The gate: cheaper version vs old and augmented eval (threshold {THRESHOLD:.2f}) ===")
print(f"  vs eval-set v1 (old)      : {p1}/{n1} = {s1:.2f}  [{gate(s1)}]")
print(f"  vs eval-set v2 (augmented): {p2}/{n2} = {s2:.2f}  [{gate(s2)}]")
print()
print("The production clicks became eval cases. The cheaper version that")
print("the old eval approved is now BLOCKED: the loop gave the gate eyes.")

What to expect. When you run it, the output is exactly this:

=== PART 1 — Observability of the semantic search ===
  searches logged              : 7
  clicks in the top-3          : 3/7
  relevant_click_rate (quality): 43%
  average click position       : 3.3  (1.0 = ideal)
  signal: ALERT: the ranking buries the relevant one

=== PART 2 — Close the loop: clicks below -> eval-set cases ===
  eval-set v1             : 3 cases
  clicks outside the top-3: 4 -> ['c4', 'c5', 'c6', 'c7']
  eval-set v2 (augmented) : 7 cases

=== PART 3 — The gate: cheaper version vs old and augmented eval (threshold 0.80) ===
  vs eval-set v1 (old)      : 3/3 = 1.00  [PASS -> deploy allowed]
  vs eval-set v2 (augmented): 3/7 = 0.43  [FAIL -> deploy blocked]

The production clicks became eval cases. The cheaper version that
the old eval approved is now BLOCKED: the loop gave the gate eyes.

Read the result in parts, because it's the whole loop over a new feature.

Part 1 — the quality signal reveals the problem. Of the 7 searches, only 3 had the click in the top-3: a relevant_click_rate of 43%, with an average click position of 3.3 (against the ideal of 1.0). The dashboard marks ALERT. Notice what the click did: without any user rating anything, their behavior —where they clicked— revealed that in 4 of 7 searches the ranking buried the relevant thing. A classic server log would have seen 7 searches with status 200 and low latency: all green. The action signal (the click) is the only one that sees the search is failing more than half its users. And it's the ideal signal for this feature precisely because it's implicit (the user already clicks, you ask nothing extra), free, and abundant (every search produces it).

Part 2 — the clicks become eval cases. The 4 clicks to a low position (c4 to c7) become new eval-set cases: the query becomes the case, and the product the user ended up clicking becomes the "relevant" one that should have come up top. Notice how elegant: the user, by clicking "water_bottle" after searching "bottle for the gym", told you which was the correct answer —without writing it, just with their click—. The eval-set grew from 3 to 7 cases, and those 4 new cases are exactly the ones the search is failing in production. It's the same conversion as lesson 5, but with implicit feedback instead of an explicit correction.

Part 3 — the augmented eval catches the regression. The cheaper version —the economical model that gets the 3 original cases right but buried the relevant thing in the ones the clicks revealed— is tested against the two eval-sets. Against the old one (the 3 module 3 cases), it scores 3/3 = 1.00: PASSES, deploy allowed. Against the augmented one (with the 4 cases from the clicks), it scores 3/7 = 0.43: FAILS, deploy blocked. The same version, two opposite verdicts, and the augmented one is the correct one because it includes what the users really experienced. Without closing the loop, this cheaper version —which in production had a relevant_click_rate of 43%— would have been deployed with the old eval's blessing. The clicks turned into cases gave the gate the eyes to block it. The loop closed: from the action signal (click) to the verifiable improvement (regression caught).

Part 4 — The decision brief (ADR-style)

Decision: set up a data loop for the semantic search that captures the clicks as a quality signal, feeds them back to the eval-set, and uses RAG as the knowledge lever; NOT fine-tune.

Context. The semantic search operates over Mercado's catalog, which is large (millions of products) and changes constantly (products that come in, go out, change price and stock daily). Its quality —does it put the relevant thing up top?— degrades invisibly to classic monitoring: it responds 200 in all searches even when it buries the relevant thing. It needs a data loop to improve with use.

Quality signal and capture. The signal is the relevant_click_rate: the fraction of searches where the click lands in the top-3. It's implicit feedback (the user already clicks, with no extra work), free and high-volume —ideal for a high-traffic feature with no natural moment to ask for a thumbs—. The capture point saves, per search: the query, the ranking shown, and the position/product of the click. Saving only the query and the result would lose the click position, which is the signal; and what isn't captured isn't recovered.

Closing the loop. The clicks to low positions become eval-set cases (query → case, clicked product → relevant), which catch regressions the old eval approved (the cheaper version: 1.00 against the old eval, 0.43 against the augmented). Those cases also signal what to improve in the retrieval (the products that got buried).

Lever: RAG, not fine-tune. The choice is decided by the data freshness (lesson 6): the catalog changes daily, so fine-tune is ruled out —it would freeze with the training-day catalog and you'd have to re-train constantly—. And the catalog is enormous, more than fits in a prompt. RAG is the only lever that gives freshness over a lot of knowledge: you update the index and the change is reflected in the next search. (How the RAG is built —embeddings, indexing— is AI Engineering; that RAG is the correct lever here is the architecture decision.)

Composition with the previous gates. The data loop doesn't replace the gates of the previous modules; it complements them and closes them in a cycle:

• M2 (budgets): the loop's observability watches the cost/latency budget live (tokens, cost, latency per search).
• M3 (eval): the eval-set is the safety net the loop feeds the failures back to; the eval grows with the clicks.
• M4 (guardrails): the search's output keeps being validated at the boundary before being shown.
• M5 (resilience): when the model goes down, the search degrades to keywords; the loop measures whether that degradation affects the relevant_click_rate.
• M6 (deterministic shell): the loop respects the shell —the feedback improves the model, but the model keeps proposing, not disposing—.

Why the system improves over time. With this loop, the search stops being a fixed snapshot: each click reveals a failure, each failure becomes an eval case and a signal of what to improve in the retrieval, and each improvement is verified with the augmented eval. It's lesson 2's flywheel spinning: more use → more clicks → more failure cases discovered and corrected → better search → more use. The search's quality isn't decided by the model; it's decided by this loop.

Consequences. You have to instrument the click capture from the start (what isn't captured isn't recovered). The eval-set grows and has to be kept representative (eval design, AI Engineering). Each change of the component (model, retrieval) passes through the augmented eval before the deploy. The RAG lever requires keeping the index fresh, which is its maintenance cost.

That brief is the artifact that justifies the decision to the team: which quality signal, how it's captured, how the loop closes, which lever and why, and how the loop composes with everything before it. With the four artifacts —observability, capture, closing of the loop, and brief— you have the semantic search's data loop set up end to end.

Common mistakes

Using an explicit thumbs where the implicit click is the natural signal (of inadequate capture). What happens: the team, out of the support agent's habit, adds a "was this result helpful?" button to each search. Almost no one uses it —no one rates searches—, so the signal is scarce and biased (only the very frustrated respond). Meanwhile, the action signal that does abound —the click— isn't being captured. Why it happens: the thumbs is the most obvious signal and the one used in the previous feature, and it's applied by inertia. How to detect it: if your quality signal depends on the user doing extra work in a high-traffic feature, you'll have little data. How to fix it: for the search, the natural signal is the click (implicit feedback, free, abundant); capture it instead of asking for a thumbs no one gives.

Saving only the search result and losing the click position (of irreversible omission). What happens: the system records the query and the products shown, but not which one the user clicked nor at which position. When the team wants to compute the relevant_click_rate, it discovers it doesn't have the datum —it knows what was shown, not what the user chose—. All that action signal was lost. Why it happens: saving the result seems enough, and the click position happens in the front-end, at a point that wasn't being instrumented. How to detect it: if you can't say at which position your users clicked last week, you didn't capture the signal. How to fix it: save the triple (query, ranking shown, click with its position) at the capture point; and remember it's irreversible —the clicks you didn't instrument aren't recovered—.

Putting the search into fine-tune "so it learns the catalog" (of wrong lever). What happens: the team decides to train a model with the catalog so the search "knows the products". It works for a while, but the catalog changes daily —new products, prices, stock— and the trained model stays with the old catalog, recommending products that no longer exist. Now you have to re-train constantly. Why it happens: fine-tune sounds like "really teaching it the catalog", and freshness wasn't considered. How to detect it: if your lever for knowledge that changes daily is fine-tune, you'll always be out of date. How to fix it: for a live catalog, the lever is RAG —the index updates and the change is reflected immediately—; fine-tune would freeze. Data freshness rules out fine-tune (lesson 6).

Exercises

Exercise 1 — The click as a correction. In the project, a click to position 5 on "water_bottle" after the search "bottle for the gym" became an eval case. Explain why the click is at once a failure signal and a correction, and compare it with the explicit correction of the support agent (lesson 5). What advantage does the click have over asking the user to correct?

See solution

The click to position 5 is a failure signal because it reveals that the ranking didn't put the relevant thing up top: the user had to scroll down to position 5 to find what they were looking for, which means positions 1-4 weren't what they wanted. If the search had gotten it right, the click would have landed in the top-3. And it's at once a correction because the product the user ended up clicking ("water_bottle") is exactly the correct answer —what should have come up top—. The user, with their click, not only said "this is wrong" but "this was the right one", just like the explicit correction of the support agent, where the human rewrote the response and that rewrite became the case's criterion.

The difference with the agent's correction: in the agent, the correction was explicit —the human edited the text on purpose—; in the search, the correction is implicit —it arises from the user's natural behavior, without them knowing they're correcting anything—. The user just wanted their bottle; by clicking it, unintentionally, they handed the system the correct answer for that query.

The advantage of the click over asking for an explicit correction: it's free and abundant. Asking the user to correct (to write which was the correct result) requires work on their part, so almost no one would do it, and you'd have little and biased data. The click, on the other hand, the user does anyway —it's part of using the search—, so the signal is high-volume and doesn't depend on the user's goodwill. For a high-traffic feature like the search, this abundance is decisive: implicit feedback scales where explicit doesn't.

Exercise 2 — The two verdicts, again. The cheaper version scored 1.00 against the old eval (3 cases) and 0.43 against the augmented one (7 cases). A colleague says: "the old eval with 3 cases already approved it; adding cases just to fail it is cheating". Explain why they're wrong, connecting it with the 43% relevant_click_rate measured in production.

See solution

They're wrong because they confuse making the exam harder arbitrarily with making the exam representative of reality. The 4 cases that were added didn't come from nowhere nor were they chosen to fail the cheaper version: they're the real searches the users made and where the search failed —we know because the user had to scroll down to position 5 to find the relevant thing—. Adding those cases isn't cheating; it's making the eval-set include what the users really experience. The old eval of 3 cases gave 1.00 only because it didn't test the failing searches: its perfect score was a blind spot, not quality.

And there's independent evidence confirming it: the relevant_click_rate measured in production was 43% —exactly the augmented eval's score (3/7 = 0.43)—. It's no coincidence: the augmented eval, by including the cases the clicks revealed, converges with the production reality, while the old eval (1.00) was completely disconnected from it. The old eval said "perfect", production said "43% right", and the augmented eval (0.43) agrees with production. Adding the click cases didn't unfairly fail the cheaper version; it revealed that it was already failing in production, where it matters. Closing the loop made the gate honest, not cheating.

Exercise 3 — The loop of another feature. Apply the method to a new feature: the "describe your product" generator, where a seller gives the name and specs of a product and the LLM writes a sales description. Design its data loop: (a) which quality signal would you capture (and is it implicit or explicit)?, (b) how do you close the loop with that signal?, and (c) which lever (prompt, RAG, or fine-tune) would you choose, considering that Mercado has a fixed brand tone and millions of products?

See solution

• (a) The quality signal → the rate of descriptions the seller published without editing (action signal, implicit), and how much they edited when they did. If the seller publishes the generated description as is, it helped; if they rewrite it entirely, it failed. It's lesson 4's implicit feedback (the action): it arises from the normal flow (the seller is going to publish anyway) and doesn't require asking them for a thumbs. An explicit complement (an optional thumbs) can be added, but the main and abundant signal is the action of publishing/editing.
• (b) Closing the loop: the descriptions the seller rewrote are the corrections —the edited version is what should have been generated—. They become eval-set cases (input: name + specs; criterion: properties of the edited description, like mentioning the key specs and respecting the tone). Those cases catch regressions and feed the improvement lever. The conversion is the same as lesson 5, with the seller's edit as the correction.
• (c) The lever → here yes, fine-tune is a serious candidate (unlike the search). The brand tone is fixed (it doesn't change) and the volume is enormous (millions of products): it meets lesson 6's criteria —stable behavior + high volume—. Fine-tune would bake the brand style, with short prompts and cheap inference at massive scale, and since the tone doesn't change, fine-tune's rigidity doesn't hurt. But —following the ladder— first you try the prompt with examples of the brand tone and measure whether it suffices; often a good prompt with few-shot achieves the tone without the cost of training. Only if the prompt doesn't give the necessary consistency at that volume is fine-tune justified. Important note: the product's specs (which do change per product) don't go in the fine-tune —they go in the prompt/input—; what's baked is the style, not the variable data. That distinction (stable style → fine-tune; variable data → prompt/RAG) is the key to the decision.

The contrast with the search is the lesson: the search operates over a live catalog (freshness rules → RAG, never fine-tune); the generator applies a fixed tone at high scale (stability + volume → fine-tune is a candidate). The same decision family, opposite answers, because the feature profiles are opposite.

Module summary: the eight lessons

You closed the data and feedback loop module. This is the complete arc you traveled:

Lesson	What you take away
1	The thesis: an AI-native system improves because it listens to its use and feeds it back. The three analogies (the restaurant, the maps app, the suggestion box). The feedback becomes eval cases, and the augmented eval catches a blind spot.
2	The flywheel: use → data → better system → more use, measured —the system with a loop climbs from 0.60 to 1.00, the one without freezes at 0.60—. The snowball that compounds vs the rock that rolls without changing.
3	The observability for AI: tokens, cost, latency and —the new thing— the quality signal. A classic log sees 100% of 2xx while a feature has 65% approval: the engine runs, the trip doesn't.
4	The feedback loop as a design decision: three signals (thumbs, correction, action) that don't coincide (62% / 50% / 38%). The action is the most honest. The capture point is irreversible: what isn't captured isn't recovered.
5	Closing the loop: the feedback becomes eval-set cases (M3). A version that passed the old eval (0.83) fails the augmented one (0.50). Aviation's "never again" list. The most expensive error: capturing and not closing.
6	The decision prompt vs RAG vs fine-tune: their profile of latency, cost, freshness, and maintenance. The ladder (prompt → RAG → fine-tune). Freshness decides first; don't jump to fine-tune for prestige.
7	The routing: the feedback triage —group by type, route to the cheapest lever that works, everything to the eval, verify by re-running the eval (0.43 → 0.86)—. The four rules of the loop.
8	The project: the semantic search's data loop set up end to end —observability with the relevant_click_rate, click capture, closing toward the eval, and ADR with the RAG decision—.

The capability you gained: take an AI feature and design its complete data loop —the observability with its quality signal, the feedback capture, the closing toward the eval-set, and the decision of which lever to move to improve it— so the system improves with use instead of freezing. And with the boundary clear: this is the decision and the loop; building the pieces (RAG, fine-tune, embeddings, the eval-set design) is AI Engineering.

Where to go next

Module 8: the guide's capstone. This module closed the last piece of the shell that surrounds the AI component. In module 8 you'll bring everything together: architect an AI feature in Mercado end to end —where the component lives, its latency/cost budget with model cascade (M2), its eval gate (M3), its guardrails and trust boundary (M4), its fallback when the model goes down (M5), the deterministic shell that validates its actions (M6), and the data loop that improves it (this module)—. The loop you set up here is one of the seven pieces the capstone integrates.

The AI Engineering ecosystem: building the pieces. Throughout this module you treated RAG, fine-tune, and embeddings as boxes with properties, and the eval-set as something that grows but you didn't design. Building those pieces —how RAG is indexed and retrieved, how the dataset is assembled and a fine-tune is run, how a representative eval-set is designed— is the body of knowledge of the AI Engineering ecosystem. If you're going to take AI features to production for real, that's the next step: this module taught you to decide which lever and to close the loop; AI Engineering teaches you to build the levers.

architecture-decisions-and-tradeoffs-guide: the fitness function as a general concept. The eval-set this module grows with feedback is a fitness function —an automated test that governs a property of the system and stops the change if it degrades—. The general concept, applied to any architectural property, is in that guide. The data loop is what keeps that fitness function representative over time.

With this you have the last architectural property of an AI feature: not only contained (M1-M6), but capable of improving over time (this module). An AI feature ready for production not only survives its non-determinism; it turns it into a trajectory of improvement, and now you know how to design the loop that makes it possible.

Resources

• Chip Huyen — AI Engineering (O'Reilly) — the treatment of the data flywheel, production feedback, and the choice between prompt, RAG, and fine-tune; the reference for building the pieces this project only decides and feeds back (the boundary with AI Engineering).
• Chip Huyen — Designing Machine Learning Systems (O'Reilly) — the chapters on feedback loops and monitoring treat the data loop as an architecture property of the complete system; the project's frame.
• martinfowler.com — "Emerging Patterns in Building GenAI Applications", Bharani Subramaniam and Martin Fowler — the pattern catalog with observability, feedback, and the choice of RAG/fine-tune as design pieces; the frame that surrounds this project and the module 8 capstone. In English.
• Anthropic — Claude documentation — the conceptual reference for instrumenting an application with an LLM (measure usage, capture signals) and for choosing between putting knowledge in the prompt or retrieving it; without pinning a model version. In English.
• AI Engineering ecosystem (referral) — for building the catalog RAG, designing the representative eval-set, and evaluating whether a style fine-tune is justified; this module set up the loop, AI Engineering builds what the loop feeds.