1. Module introduction: the data and feedback loop

Overview

By the end of this lesson you'll understand the idea that sustains the whole module, and that marks a turn from the previous six: a well-designed AI-native system doesn't just contain the AI component's non-determinism, it improves it over time, because it listens to its own usage and feeds back what it hears. In modules 1 through 6 you built the deterministic shell that protects the system from the AI component: you placed it behind a boundary, gave it budgets, an eval gate, guardrails, resilience, and a layer that validates its actions. This module does the complementary thing: it uses what the system learns from its users so the AI component gets better and better. It's the difference between a frozen system —as good the day of its launch as the day of its death— and one that compounds improvements and pulls away from its competitors over time.

This matters because the improvement of an AI system rarely comes from a bigger model; it comes from a well-designed data loop. Usage generates data (what did the user ask?, did the answer help?, what did they do next?), that data reveals where the system fails, and those failures —once you capture them and feed them back— become the system's improvements. That self-reinforcing cycle —usage → data → better system → more usage— is called a flywheel, an inertia wheel that, once it starts spinning, costs less and less to keep in motion. But the flywheel doesn't appear on its own: you have to design it, with three pieces that are the module's content —the observability that lets you see quality, the feedback loop that captures the user's signal, and the act of closing the loop by feeding that signal back into something that improves the system—.

Connection with the module: this lesson is the map, not the territory. Here you don't build the loop yet; you understand why the six lessons that follow go in the order they go. First the motivation: lesson 2 measures the flywheel —two identical systems, one with a loop and one without, diverging over time—. Then the precondition: lesson 3 sets up observability for AI, because you can't improve what you can't see, and a classic server log doesn't see quality. Then the capture: lesson 4 designs the feedback loop —the three feedback signals and the point where they're captured—. After that the heart: lesson 5 closes the loop by turning feedback into cases of module 3's eval-set. Lesson 6 opens the architectural decision prompt vs RAG vs fine-tune —which lever to move, with what latency, cost, and freshness profile—. And lesson 7 is the synthesis: routing each type of feedback to its lever. Lesson 8 —the project— has you assemble the data loop of a Mercado feature end to end.

Three analogies: the restaurant, the maps app, and the suggestion box

Before dropping to the code, three everyday images you'll recognize in every lesson of the module. Each captures a part of the data loop.

The restaurant that notes which dishes get sent back and adjusts the menu. Think of a well-run restaurant. It doesn't just cook and serve; it observes what happens with each dish. Which ones get sent back half-eaten? Which ones do people ask for the recipe of? Which one always stays on the menu without anyone ordering it? The chef who pays attention turns that observation into changes: removes the dish no one finishes, moves the one everyone praises to the front, adjusts the seasoning of the one that comes back cold. The restaurant improves because it listens to the diner, not because the chef is an isolated genius. And here's the important thing: that listening is deliberate —someone decided to note what gets sent back, someone decided to ask—. A restaurant that serves and doesn't look will never know why its clientele leaves. That act of observing usage and adjusting accordingly is the data loop: the system improves because it has a mechanism to listen to whoever uses it and act on what it hears.

The maps app that learns from millions of trips. When you open a maps app and it tells you "this route is congested, take this other one", it doesn't know by magic: it knows because millions of people who already traveled those streets gave it data —how long they took, where they braked, what detour they took—. Each trip of yours, in turn, feeds the app for the next driver. That's the purest form of the flywheel: more users produce more traffic data, more data makes better routes, better routes attract more users. The app isn't useful despite having many users; it's useful because it has them —usage is literally the raw material that improves it—. A just-launched maps app, with no trips, is barely a drawing of streets; with millions of trips, it's a traffic oracle. The difference between the two isn't the map; it's the data loop that turns usage into improvement.

The suggestion box that's actually read and acted upon. Almost every office has a suggestion box. In most, it's decoration: people drop little papers, no one reads them, nothing changes, and over time they stop dropping papers because they learned it's useless. In a few, someone opens the box every week, reads the suggestions, and acts —fixes the printer everyone complained about, changes the coffee, adjusts the schedule—. The difference between the two boxes isn't the box; it's whether the loop closes. Capturing feedback and not using it is worse than not capturing it: you spend the effort of collecting and don't get the improvement, and on top of that you train your users not to bother giving feedback. This is the module's most expensive error, and this analogy is the one that names it: a suggestion box no one reads is exactly an AI system that captures thumbs up/down and never feeds them back. Feedback is only worth it if it closes the loop.

Put the three together and you have the whole module. The restaurant is the data loop as a decision (listening deliberately and adjusting). The maps app is the flywheel (usage compounds the improvement). The suggestion box is the warning (the loop is only worth it if it closes). Everything else is details of how to assemble those three things over a real feature.

The case: Mercado's support agent

Let's drop to Mercado, the ecosystem's marketplace. Of its AI features, the protagonist of this module is the customer support agent: it answers frequent questions and proposes answers that a human agent reviews before sending. It's the ideal case for the data loop for a concrete reason: it generates rich and natural feedback. Every time the agent proposes an answer, measurable things happen —the customer marks it with thumbs up/down, the human agent sends it as is or corrects it, the ticket gets resolved or escalated—. That torrent of signals is exactly the raw material of the flywheel: it tells us, without our having to guess, where the agent answers well and where it fails.

And there's a direct connection with module 3 that will be the backbone of this module: the support agent already has an eval-set —the set of cases with criteria we assembled as a quality gate—. The central idea here is that that eval-set shouldn't be static. When a customer marks thumbs_down and the human agent corrects the answer, we've just discovered a case where the system fails that wasn't in the eval-set. Adding it to the eval-set is closing the loop: the failure reported by a real user becomes a case the gate will watch forever. The semantic search —which improves with users' clicks— appears in the project, because its feedback is implicit (which result you clicked) and helps to see the loop from another angle.

We're not going to build the agent nor decide how it's trained with this data —that's AI Engineering—. We're going to design the loop: observe the usage, capture the feedback, and feed it back to the eval-set (and to the prompt, and to the retrieval) so the system improves. And to start, let's see the whole module condensed into an executed block. Look closely, because here's the thesis in action.

Worked example: production feedback becomes the eval-set

We're not going to say that closing the loop improves the system: we're going to execute it. We model a fragment of the support agent's loop —the system logs its interactions with the user's feedback, the thumbs_down cases become new eval-set cases, and we re-run the gate— and observe something revealing: the same version of the agent that passes the old eval-set fails the augmented one. Production feedback caught a blind spot the original eval-set didn't see.

# Lesson 01 (intro M7) — the module in miniature: the closed data loop.
# All SIMULATED. Zero network, zero API, zero keys. Deterministic output.
#
# 1) The system LOGS its interactions with the user's feedback (thumbs up/down).
#    That is observability: a QUALITY signal, not just logs.
# 2) The thumbs_down cases are CONVERTED into new eval-set cases.
# 3) A candidate version PASSES the old eval-set (which didn't cover those cases) but
#    FAILS the augmented eval-set -> the loop caught a blind spot invisible before.

# --- The ORIGINAL eval-set of the support agent (from M3): 6 "easy" cases. ---
EVAL_SET_V1 = [
    {"id": "q1", "question": "where is my order",           "must_contain": "tracking"},
    {"id": "q2", "question": "how do I return a product",   "must_contain": "return"},
    {"id": "q3", "question": "how long does shipping take", "must_contain": "3 to 5 business days"},
    {"id": "q4", "question": "can I pay in installments",   "must_contain": "installments"},
    {"id": "q5", "question": "I want to cancel my order",   "must_contain": "cancel"},
    {"id": "q6", "question": "how do I contact a seller",   "must_contain": "messages"},
]

# --- The PRODUCTION LOG: real interactions with the user's feedback. ---
# Each row: user input, model proposal, feedback (up/down) and, if the user
# corrected, the correction (the correct datum the agent should have given).
# BOUNDARY NOTE: how the feedback capture is designed and how it's trained on is
# AI Engineering; here the LOOP as an architecture decision.
PROD_LOG = [
    # (input, model_proposal, feedback, correction_or_None)
    ("where is my order",           "You can see the tracking in your profile.",     "up",   None),
    ("how long does shipping take", "Shipping takes 3 to 5 business days.",           "up",   None),
    # --- cases the user marked BAD (thumbs_down) with their correction: ---
    ("the product arrived broken",  "Sorry, I don't have information about that.",    "down", "refund"),
    ("my coupon doesn't work",      "Coupons always work.",                          "down", "expiration"),
    ("I didn't receive my invoice", "We don't handle invoices.",                     "down", "email"),
    ("can I pay in installments",   "Yes, you can pay in installments with no interest.", "up", None),
]

# --- Observability: the QUALITY signal that a classic server log doesn't have. ---
ups = sum(1 for *_r, fb, _c in PROD_LOG if fb == "up")
downs = sum(1 for *_r, fb, _c in PROD_LOG if fb == "down")
approval_rate = ups / (ups + downs)

print("=== Observability: the quality signal (the server log isn't enough) ===")
print(f"  interactions logged      : {len(PROD_LOG)}")
print(f"  thumbs_up / thumbs_down   : {ups} / {downs}")
print(f"  approval_rate             : {approval_rate:.0%}")
print()

# --- Close the loop: the thumbs_down become NEW eval-set cases. ---
new_cases = []
for i, (q, _proposal, fb, correction) in enumerate(PROD_LOG, start=1):
    if fb == "down":
        new_cases.append({"id": f"fb{i}", "question": q, "must_contain": correction})

EVAL_SET_V2 = EVAL_SET_V1 + new_cases

print("=== Close the loop: the thumbs_down become eval-set cases ===")
print(f"  eval-set v1 (original)    : {len(EVAL_SET_V1)} cases")
print(f"  new cases from feedback   : {len(new_cases)}  -> {[c['id'] for c in new_cases]}")
print(f"  eval-set v2 (augmented)   : {len(EVAL_SET_V2)} cases")
print()

# --- The agent STUB: answers well ONLY the "easy" cases from v1. ---
# It's the version currently in production: it never learned the cases the
# user reported as bad (arrived broken, expired coupon, invoice by email).
GOLD_EASY = {
    "where is my order":           "You can see the tracking in your profile.",
    "how do I return a product":   "For a return, go to your order and press Return.",
    "how long does shipping take": "Standard shipping takes 3 to 5 business days.",
    "can I pay in installments":   "Yes, you can pay in installments with no interest by card.",
    "I want to cancel my order":   "You can cancel the order if it hasn't shipped yet.",
    "how do I contact a seller":   "Message the seller from the Messages section.",
}

def agent(question):
    # Answers well what it already knew; for the rest, a poor answer that fails the criterion.
    return GOLD_EASY.get(question, "Sorry, I don't have information about that.")

def run_eval(eval_set):
    passed = sum(case["must_contain"] in agent(case["question"]).lower() for case in eval_set)
    return passed, len(eval_set), passed / len(eval_set)

THRESHOLD = 0.80
p1, n1, s1 = run_eval(EVAL_SET_V1)
p2, n2, s2 = run_eval(EVAL_SET_V2)

def verdict(score):
    return "PASS -> deploy allowed" if score >= THRESHOLD else "FAIL -> deploy blocked"

print(f"=== The SAME agent version against both eval-sets (threshold {THRESHOLD:.2f}) ===")
print(f"  against eval-set v1 (old)       : {p1}/{n1} = {s1:.2f}   [{verdict(s1)}]")
print(f"  against eval-set v2 (augmented) : {p2}/{n2} = {s2:.2f}   [{verdict(s2)}]")

What to expect. When you run it, the output is exactly this:

=== Observability: the quality signal (the server log isn't enough) ===
  interactions logged      : 6
  thumbs_up / thumbs_down   : 3 / 3
  approval_rate             : 50%

=== Close the loop: the thumbs_down become eval-set cases ===
  eval-set v1 (original)    : 6 cases
  new cases from feedback   : 3  -> ['fb3', 'fb4', 'fb5']
  eval-set v2 (augmented)   : 9 cases

=== The SAME agent version against both eval-sets (threshold 0.80) ===
  against eval-set v1 (old)       : 6/6 = 1.00   [PASS -> deploy allowed]
  against eval-set v2 (augmented) : 6/9 = 0.67   [FAIL -> deploy blocked]

Read the output calmly, because here's the whole module in miniature, in three acts.

Act 1: observability reveals what the server log hides. The system logged 6 interactions, and of them 3 received thumbs_up and 3 thumbs_down: an approval_rate of 50%. Stop at that number. If this agent lived with only classic monitoring, you'd see HTTP status 200 in all 6 interactions —the server responded, everything "green"—. But half the answers were bad, and only the quality signal (the approval_rate) reveals it. Observability for AI is what turns "the service responded" into "the service responded well".

Act 2: the feedback becomes eval-set cases. The 3 thumbs_down cases —"the product arrived broken", "my coupon doesn't work", "I didn't receive my invoice"— carried the user's correction (the correct datum the agent should have given). The loop turns them into 3 new eval-set cases (fb3, fb4, fb5): the input becomes the question, and the correction becomes the criterion. The agent's eval-set went from 6 cases to 9. Notice what just happened: the failures real users reported are now a permanent part of the quality gate. The eval-set stopped being a fixed snapshot and started to grow with reality.

Act 3: the augmented eval-set catches a blind spot. And here's the punch. The same version of the agent —the one currently in production, which only knows how to answer the 6 easy cases— is measured against both eval-sets. Against the old one, it scores 6/6 = 1.00: perfect, deploy allowed, all green. Against the augmented one, it scores 6/9 = 0.67: below the 0.80 threshold, deploy blocked. It's the same agent, with two opposite verdicts. Which is true? The one from the augmented eval-set, because it includes the cases real users actually had. The old eval-set gave a false reassurance —"1.00, we're perfect"— precisely because it didn't test what was failing. Closing the loop gave the gate eyes: now it sees the blind spot that used to make it approve an agent that in production had an approval_rate of 50%.

Don't yet understand how each piece works —that's what the lessons are for—. Keep the shape of the result: usage generated feedback, the feedback became eval cases, and the augmented eval-set revealed that the system was worse than the old eval-set said. That's exactly what "the data loop as architecture" means: a mechanism that turns usage into an increasingly honest gate, and in the long run, into an increasingly better system.

The map of the six lessons

The six lessons that follow go in this order because each one assembles the piece the next one needs.

Lesson	What it gives you	Why it goes here
2	The flywheel: why the data loop compounds (usage → data → better system → more usage), measured	Installs the motivation: without a loop, the system freezes; with a loop, it pulls away from the competition
3	Observability for AI: tokens, cost, latency and —the new one— the quality signal	You can't close the loop if you don't see the quality; it's the precondition of everything else
4	The feedback loop: the three signals (thumbs, correction, action) and their capture	The loop's raw material; you have to design the point where the signals are captured
5	Closing the loop: the feedback becomes cases of module 3's eval-set	The heart: where the captured signal becomes a verifiable improvement
6	The decision prompt vs RAG vs fine-tune: their latency, cost, freshness profile	When the feedback asks for a change, you have to know which lever to move and what it implies
7	The routing: each type of failure to its lever, with the eval as a safety net	The synthesis: brings the capture (4), the eval (5), and the levers (6) together into a complete loop

The arc is: first you see why the loop matters (2), then you set up the observability that makes it possible (3), then you capture the feedback (4), then you close it by feeding it back to the eval (5). With the basic loop working, lesson 6 opens the levers to improve (prompt, RAG, fine-tune) as an architectural decision, and lesson 7 routes each signal to its lever. Lesson 8 —the project— brings everything together over the semantic search, a feature you architect from scratch, so you confirm you learned the method and didn't memorize a table.

What this module does NOT touch

It's worth marking the boundary from now, because there are neighboring topics that seem to be from here and are from another part of the ecosystem. This boundary is hard: cross it only to defer.

The mechanics of RAG, fine-tuning, and embeddings is AI Engineering, not here. This is the module's most important boundary. In lesson 6 you'll decide between prompt, RAG, and fine-tune as architectural options, and in lesson 7 you'll route feedback toward them. But how those pieces are built —how a vector store is indexed, how embeddings are chosen and computed, how a training dataset is assembled, how a fine-tune is run and evaluated, what retrieval metric to use— is a deep body of knowledge, and it belongs to the AI Engineering ecosystem. This module doesn't teach it: it treats those pieces as boxes with properties (latency they add, cost, data freshness, maintenance) and teaches you to choose between them and to close the loop with them. The mental rule: if the question is "how do I build the RAG or how do I train the fine-tune?", it's AI Engineering; if the question is "should I use prompt, RAG, or fine-tune for this feature, and how do I feed the feedback back?", it's from here.

The design of the eval-set is AI Engineering; using it and growing it with feedback is from here. Module 3 already marked this boundary: choosing the cases and the criterion of an eval-set (coverage, representativeness, calibrating a judge) is AI Engineering. This module adds a turn: the cases that arise from production feedback are added to the eval-set. Turning a thumbs_down into an eval case is the data loop (from here); deciding whether that case is representative of the real universe of failures, or how to balance the dataset, is eval design (AI Engineering). We close the loop; they design the set.

Data engineering and ML pipelines in depth are from other guides. A serious production data loop involves capture, storage, labeling, and retraining pipelines that are a whole discipline. Here the loop is the minimum necessary to show the architectural decision: where the feedback is captured, what it's fed back to, and how the eval verifies that the improvement was real. We don't build the pipeline; we show the shape of the loop and the design decisions that define it.

Common mistakes

Not capturing feedback and flying blind (of omission). What happens: the team launches the AI feature and measures only what a server measures —did it respond?, how fast?—, with no signal of whether the answers were good. The feature can be degrading for weeks and no one knows, because the dashboard is all green. The first warning is customer complaints or the drop of a business metric, when the damage is already done. Why it happens: capturing the quality signal costs design work (where do I put the thumbs?, how do I record the correction?), and server monitoring comes free with the infrastructure. How to detect it: if you can't state in a number what your AI feature's approval_rate is today, you're flying blind. How to fix it: set up observability for AI (lesson 3) before you need it —the quality signal is the precondition of the whole loop—.

Logging like a classic server, with no quality signal (of mental model). What happens: a variant of the previous one. The team does have observability —logs, latencies, error rate— but it's that of a classic service: it measures availability, not quality. An AI component that responds HTTP 200 with a hallucination counts as "success" on that dashboard. The feature looks perfectly healthy —100% 2xx, low latency— while delivering bad answers. Why it happens: the instinct is to reuse the monitoring you already have, and that monitoring never had to measure "was the answer right?" because classic components don't fail that way. How to detect it: if your dashboard doesn't have a quality column (approval_rate, or similar), you're not observing the part that matters of an AI feature. How to fix it: add the quality signal to the dashboard (lesson 3); the HTTP status says whether the service responded, the quality signal says whether it responded well.

Closing the loop badly: capturing feedback and never using it (of process). What happens: the team puts the thumbs up/down, people use it, and the data piles up in a table no one looks at. It's the suggestion box no one opens: you spent the effort of collecting and get no improvement, and worse, the users learn that their feedback changes nothing and stop giving it. Why it happens: capturing is a visible product change (a button appears); feeding back is invisible architecture work (turning feedback into eval cases, into prompt examples, into retrieval documents) that no one prioritizes. How to detect it: if you have a month of saved thumbs_down and not one became an eval case, a prompt adjustment, or a retrieval document, your loop is open. How to fix it: close the loop (lessons 5 and 7) —each thumbs_down must have a destination: to the eval-set, to the prompt, or to the retrieval—.

Exercises

Exercise 1 — Translate the analogies to design. For each analogy of the module, say which piece of the data loop it represents and give a concrete example in Mercado's support agent. (a) The restaurant that notes which dishes get sent back and adjusts the menu. (b) The maps app that learns from millions of trips. (c) The suggestion box that's actually read and acted upon.

See solution

• (a) The restaurant that notes and adjusts → the data loop as a deliberate decision. Observe the usage (which dishes get sent back) and act on it (adjust the menu). Example in Mercado: log each thumbs_down of the support agent and use it to improve —it's not enough to cook (answer), you have to note what gets sent back (which answers failed) and adjust—. The key: the listening is deliberate; someone decided to measure.
• (b) The maps app that learns from trips → the flywheel. More usage produces more data, more data improves the system, a better system attracts more usage. Example in Mercado: the more customers use the agent and the search, the more feedback is generated, the more failure cases are discovered and corrected, the better the feature becomes, and the more people use it. Usage isn't a cost; it's the raw material of the improvement.
• (c) The suggestion box that's read and acted upon → closing the loop. Feedback is only worth it if it returns to something that improves the system. Example in Mercado: the thumbs_down don't pile up in a dead table; they become eval-set cases, prompt examples, or retrieval documents. The warning: a box no one opens (feedback no one feeds back) is worse than having no box.

The important thing: the restaurant is how you decide to listen, the maps app is why listening compounds (the flywheel), and the suggestion box is the warning that capturing without feeding back is useless.

Exercise 2 — The two verdicts of the same agent. In the worked example, the same version of the agent scored 1.00 against the old eval-set and 0.67 against the augmented one. A colleague says: "so the augmented eval-set is more 'unfair', because it lowers the score of an agent that was at 1.00". Explain why that reading is backwards, and which number really reflects the agent's quality in production.

See solution

The reading is backwards because it confuses the score with reality. The old eval-set isn't more "fair"; it's more blind. It gave 1.00 not because the agent was perfect, but because it only tested the 6 cases the agent already knew how to answer —it never tested "the product arrived broken", "the coupon doesn't work", nor "I didn't receive my invoice", which are cases where the agent fails in production—. An exam that only asks you what you already know always gives you 100; that doesn't make you wise, it makes the exam useless.

The number that reflects the real quality is the one from the augmented eval-set (0.67), and there's independent evidence confirming it: the production approval_rate was 50%. The agent the old eval declared "perfect" had half of its real users marking thumbs_down. The augmented eval (0.67) is much closer to that reality (50%) than the old one (1.00). Closing the loop didn't "lower the score" of a good agent; it discovered that the agent wasn't as good as the old eval pretended. The gate became more honest, not more unfair. And that honesty is exactly what prevents deploying a bad agent thinking it's perfect.

Exercise 3 — Data loop or AI Engineering mechanics? For each activity, say whether it falls within this module (the architectural decision and the loop) or on the AI Engineering boundary (the mechanics), and why. (a) Turning the week's thumbs_down into new eval-set cases. (b) Choosing the embeddings algorithm and the vector dimension for the RAG index. (c) Deciding that the semantic search should use RAG and not fine-tune because the catalog changes daily. (d) Assembling the training dataset and running the model's fine-tune.

See solution

• (a) Turning thumbs_down into eval cases → this module. It's closing the loop: feeding production feedback back to module 3's eval-set. The essence of lesson 5.
• (b) Choosing the embeddings algorithm and the vector dimension → boundary (AI Engineering). It's the mechanics of how the RAG is built. This module treats RAG as a box with properties (freshness, latency); how it's implemented on the inside isn't from here.
• (c) Deciding RAG and not fine-tune because the catalog changes daily → this module. It's the architectural decision of lesson 6: choose the lever according to the feature's profile (here, data freshness rules). We're not building the RAG; we're choosing it.
• (d) Assembling the dataset and running the fine-tune → boundary (AI Engineering). It's the mechanics of the training. This module decides whether fine-tune is worth it and what it implies for the system; how it's trained is AI Engineering.

The rule that separates: if the activity decides which lever or feeds the feedback back (a, c), it's from here; if it builds or trains the piece on the inside (b, d), it's AI Engineering.

Summary and next step

In this lesson you met the module's thesis: a well-designed AI-native system improves over time because it has a data loop —it listens to its own usage and feeds back what it hears—. You saw the three pieces that make the flywheel spin: the observability that lets you see quality, the feedback loop that captures the user's signal, and the act of closing the loop by feeding that signal back to something that improves the system. The three analogies gave you the frame: the restaurant that notes which dishes get sent back (the loop as a decision), the maps app that learns from millions of trips (the flywheel that compounds), and the suggestion box that's actually read and acted upon (the warning: capturing without feeding back is useless). And in the executed block you saw the thesis in action: production feedback became eval-set cases, and the same version of the agent the old eval declared perfect (1.00) the augmented eval revealed as insufficient (0.67) —the loop gave the gate eyes, and that honesty matched the real 50% approval_rate—.

Before moving on you should be able to: explain the flywheel (usage → data → better system → more usage) and why it compounds; name the three pieces of the loop (observability, feedback capture, closing the loop); recognize the most expensive error (capturing feedback and never using it); and separate the decision and the loop (this module) from the mechanics of RAG/fine-tune/embeddings (AI Engineering), which is outside the boundary.

What follows is understanding why the loop matters so much, because until you see the flywheel compound you won't appreciate why it's worth designing. In lesson 2 you'll execute two systems identical at the start —one with a data loop, one without— and you'll measure how they diverge round after round: the one with the loop climbs from 0.60 to 1.00 by learning from what users report, while the one without stays frozen at 0.60. It's the step from "the data loop sounds good" to "the data loop is the difference between a system that improves and one that dies where it was born".

Resources

• Chip Huyen — AI Engineering (O'Reilly) — the systematic treatment of the data flywheel and of monitoring applications with foundation models; the go-to book for the pieces (RAG, fine-tune, evals) this module takes as boxes with properties. The boundary with AI Engineering.
• Chip Huyen — Designing Machine Learning Systems (O'Reilly) — the chapter on feedback loops and monitoring treats the data loop as a system architecture decision, not as an ML detail; the conceptual frame of the whole module.
• martinfowler.com — "Emerging Patterns in Building GenAI Applications", Bharani Subramaniam and Martin Fowler — the catalog of architecture patterns for LLM apps, with observability, feedback, and the choice of RAG/fine-tune treated as design pieces; the essayistic frame of the whole module.
• Anthropic — Claude documentation — the conceptual reference for the observability of an application with an LLM (what to measure of a model call: tokens, usage, latency) and for the choice between putting the knowledge in the prompt or retrieving it; without pinning a model version. In English.
• AI Engineering ecosystem (referral) — for the mechanics of RAG, fine-tuning, embeddings, and evaluation dataset design: how the pieces this module decides and feeds back are built. This module teaches the decision and the loop; AI Engineering teaches how to build.