Module 7: The Data and Feedback Loop

2. The flywheel: usage becomes a better system

Next →

Overview

By the end of this lesson you'll understand —and have measured— the idea that motivates the whole module: a system with a data loop compounds improvements over time, and one without a loop stays frozen where it was born. Module 1 gave you the lesson that an LLM isn't a normal function; this one gives you an equally important lesson about the system that surrounds it: its quality isn't a fixed value decided on launch day, but a trajectory that depends on whether the architecture has a mechanism to learn from its usage. Two teams can launch the same feature, with the same model, on the same day; six months later, the one that designed a data loop has a notably better feature, and the one that didn't has exactly the same one from day one. The difference isn't in the model; it's in the loop.

The technical name for that phenomenon is flywheel —an inertia wheel—, and the metaphor is precise: an inertia wheel costs a lot to start spinning, but once in motion, each push accumulates on the previous momentum and the wheel spins faster and faster with less and less effort. The data flywheel spins like this: usage generates data (feedback on what works and what doesn't), the data lets you improve the system, the better system attracts more usage, and more usage generates more data. Each turn makes the next one easier. And the property that makes it powerful is that it compounds: this week's improvement stacks on last week's, so the advantage doesn't grow linearly but accumulates —that's why systems with a flywheel pull away from their competitors in a way a competitor without a flywheel can't reach with just a bigger model—.

Connection with the module: this lesson installs the motivation of the whole module. In lesson 1 you saw the loop in miniature —the feedback became eval cases—; here you see why it's worth it to set up everything that comes: the observability (lesson 3), the feedback capture (lesson 4), and the closing of the loop (lesson 5) aren't decorations, they're the pieces that make this flywheel spin. Everything you do in the coming lessons is, at bottom, building and accelerating the wheel you'll measure here. And the boundary is the usual one: here the flywheel is the architecture decision (setting up the loop); how to train a model with the data the flywheel accumulates is AI Engineering.

Analogy: the snowball rolling down the slope

You already saw the maps app in lesson 1; add now this one, which captures the compounding part. Imagine a snowball at the top of a slope. At first it's small and you have to push it with effort. But as soon as it starts rolling, something happens that doesn't happen with a rock: as it turns, it picks up more snow, and being bigger, it picks up even more on each turn. Its growth isn't constant; it accelerates, because each turn makes it bigger and a bigger ball sweeps more surface on the next turn. A rock rolling down the same slope reaches the bottom the same size it started: rolling doesn't change it. The snowball reaches the bottom turned into an avalanche.

That's exactly the difference between a system with a data loop and one without it. The system without a loop is the rock: it rolls (it gets used) but doesn't change; the usage passes through it without leaving a trace, and it reaches the end of the year the same as it started. The system with a loop is the snowball: each use picks up data, each datum improves it, and a better system attracts more usage that picks up more data. The growth compounds. And there's a hard competitive consequence: if you have the snowball and your competitor has the rock, the gap between you doesn't stay constant —it widens each turn—. That's why the data loop isn't a minor optimization; it's, often, the architectural advantage that defines who wins an AI market.

An honest detail of the analogy: the snowball only compounds if it actually picks up snow on each turn. A ball on bare ice rolls without growing. In our terms: the flywheel only spins if the loop is closed —if the collected feedback is actually fed back to the system—. A system that captures feedback but never uses it is a ball on ice: it moves, it seems like it should grow, but it doesn't grow. That's lesson 1's suggestion-box warning, seen from the flywheel.

Worked example: two identical systems, one with a loop and one without

We're not going to assert that the flywheel compounds: we're going to execute it and measure it. We model two systems identical on launch day —the same support agent, which starts knowing how to answer only the 4 most common question types— and give them rounds of usage. System A has a data loop: each round, the users' feedback reveals the question types that fail the most, and the team fixes them (and adds them to the eval-set). System B has no loop: the usage passes through it without leaving a trace, and its coverage never changes. We measure the quality of each round after round, where quality is the fraction of real traffic the system resolves well.

# Lesson 02 (M7) — the FLYWHEEL: usage becomes a better system. SIMULATED.
# Zero network, zero API, zero keys. Deterministic.
#
# The thesis: usage -> data (feedback) -> better system -> more usage. We model TWO
# systems identical at the start; one has the data loop and the other doesn't, and
# we measure how the quality diverges round after round.

# The REAL distribution of questions reaching Mercado's support agent:
# 12 question types, each with its frequency (how often it's asked).
# They sum to 1.0. NOTE: choosing this distribution is data design (AI Eng); here the LOOP.
TRUE_DISTRIBUTION = {
    "tracking":        0.20,
    "shipping_time":   0.16,
    "return":          0.13,
    "installments":    0.11,
    "cancel":          0.09,
    "broken_product":  0.08,
    "invoice":         0.07,
    "coupon":          0.06,
    "address_change":  0.05,
    "seller_contact":  0.03,
    "size_change":     0.01,
    "missing_product": 0.01,
}

# The system starts knowing how to answer only the 4 most common types (the "easy" ones).
INITIAL_KNOWN = {"tracking", "shipping_time", "return", "installments"}

def quality(known):
    # The score = fraction of the REAL traffic the system resolves well
    # (sum of frequencies of the types it knows how to answer).
    return sum(freq for t, freq in TRUE_DISTRIBUTION.items() if t in known)

def rank_unknown_by_frequency(known):
    # The feedback surfaces first in what MOST people ask and fails: the unknown
    # types ordered by frequency (the most voluminous thumbs_down).
    unknown = [(t, f) for t, f in TRUE_DISTRIBUTION.items() if t not in known]
    return sorted(unknown, key=lambda tf: tf[1], reverse=True)

ROUNDS = 6
LEARN_PER_ROUND = 2   # each round, the loop learns the 2 most frequent types that fail

# --- System A: WITH data loop. Each round learns from what the user reported. ---
known_loop = set(INITIAL_KNOWN)
# --- System B: NO loop. Never captures feedback; its coverage doesn't change. ---
known_noloop = set(INITIAL_KNOWN)

print(f"{'round':<7}{'WITH loop':>12}{'NO loop':>12}   types learned this round (from feedback)")
print("-" * 78)
for r in range(ROUNDS):
    s_loop = quality(known_loop)
    s_noloop = quality(known_noloop)
    to_learn = [t for t, _f in rank_unknown_by_frequency(known_loop)[:LEARN_PER_ROUND]]
    print(f"{r:<7}{s_loop:>12.2f}{s_noloop:>12.2f}   {to_learn if to_learn else '(covers everything)'}")
    # The loop closes: the types the user reported get fixed (and enter the eval-set).
    known_loop.update(to_learn)

print("-" * 78)
print(f"{'final':<7}{quality(known_loop):>12.2f}{quality(known_noloop):>12.2f}")
print()
print("The flywheel, in numbers:")
print(f"  start (both)           : score {quality(INITIAL_KNOWN):.2f}")
print(f"  WITH loop after {ROUNDS} rounds: score {quality(known_loop):.2f}"
      f"  (+{(quality(known_loop)-quality(INITIAL_KNOWN))*100:.0f} points)")
print(f"  NO loop after {ROUNDS} rounds  : score {quality(known_noloop):.2f}"
      f"  (+{(quality(known_noloop)-quality(INITIAL_KNOWN))*100:.0f} points)")
print(f"  final gap WITH vs NO loop: {(quality(known_loop)-quality(known_noloop))*100:.0f} points")

What to expect. When you run it, the output is exactly this:

round     WITH loop     NO loop   types learned this round (from feedback)
------------------------------------------------------------------------------
0              0.60        0.60   ['cancel', 'broken_product']
1              0.77        0.60   ['invoice', 'coupon']
2              0.90        0.60   ['address_change', 'seller_contact']
3              0.98        0.60   ['size_change', 'missing_product']
4              1.00        0.60   (covers everything)
5              1.00        0.60   (covers everything)
------------------------------------------------------------------------------
final          1.00        0.60

The flywheel, in numbers:
  start (both)           : score 0.60
  WITH loop after 6 rounds: score 1.00  (+40 points)
  NO loop after 6 rounds  : score 0.60  (+0 points)
  final gap WITH vs NO loop: 40 points

Read the table calmly, because there's the flywheel turned into a number.

The two systems start identical. In round 0, both have a score of 0.60: both know how to answer the four most common question types (tracking, shipping time, return, installments), which together are 60% of the traffic. This is the honest starting point: the same day, the same model, the same coverage. If you kept only this snapshot, you'd say the two features are equal. And they are —that day—.

The system with a loop climbs; the system without a loop stays flat. From round 1 on, the trajectories diverge. The system with a loop listens to its usage: the feedback shouts that "cancel" and "broken_product" are the most frequent types it's failing (they're 9% and 8% of the traffic), so the team fixes them and adds them to the eval-set. Its score rises to 0.77. The following round it learns "invoice" and "coupon" (0.90); then "address_change" and "seller_contact" (0.98); then the two rare ones that were left (1.00). In four rounds it covers all the traffic. The system without a loop, meanwhile, stays stuck at 0.60 every round: without feedback capture, it never finds out what it's failing, so it never fixes it. The usage passes through it and leaves no trace. It's the rock that reaches the bottom the same size.

The final gap is the competitive advantage. After six rounds, the system with a loop resolves 100% of the traffic and the one without a loop 60%: a gap of 40 points between two features that on launch day were identical. Burn that figure in, because it's the whole module's argument: the quality of an AI feature isn't decided by the model you chose, it's decided by whether you designed the loop that improves it. And notice a detail of the order in which the loop learned: it always attacked the most frequent thing that was failing first —cancel (9%) before size_change (1%)—. That's not chance; it's the most useful property of real feedback: the volume of thumbs_down orders the failures by impact, so you fix first what hurts the most people. The loop doesn't just tell you what is broken; it tells you what to fix first.

Why the flywheel compounds (and what can slow it down)

The example touched, without fully developing them, the properties that make the flywheel an architectural decision and not a decoration. It's worth seeing them explicit.

Usage is the raw material, not a byproduct. In a classic system, usage is something you endure —more traffic is more load, more cost, more things that can fail—. In a system with a data loop, usage is also the input that improves you: each interaction is a potential datum about what works and what doesn't. This change of perspective is the module's heart: you stop seeing usage as a cost to endure and start seeing it as the source of your improvement. That's why an AI feature with few users is at a double disadvantage —it not only has less revenue, it has less data to improve—.

The improvement accumulates, it doesn't reset. The system with a loop doesn't go back to zero each round; it starts from where it was. Round 3's coverage (0.98) includes everything learned in rounds 1 and 2. That's why the growth is the snowball and not the rock: each turn stacks on the previous ones. And this has a consequence on the gap with a competitor: it doesn't stay, it widens. If you compound and your competitor doesn't, each round your advantage grows, and there comes a point where the competitor can't catch you with a one-off effort —they'd have to set up their own flywheel and wait for it to spin—.

The flywheel slows down if the loop doesn't close. Here's the hook with the lessons to come. In the model, the system with a loop improved because the feedback was fed back —the failing types were learned—. If the system had captured the feedback but never used it (the closed suggestion box), its curve would be identical to the loopless system's: flat at 0.60. That is, capturing isn't enough; you have to close. That's why the next lessons matter so much: observability (3) lets you see what fails, capture (4) lets you collect the feedback, and closing the loop (5) is what really makes the wheel spin. A flywheel with any of those pieces broken doesn't spin.

The data loop's flywheel:

        ┌──────────────┐
        │     USE      │
        │(more traffic)│
        └──────┬───────┘
               │ generates
               ▼
        ┌──────────────┐         The CLOSED loop spins the wheel.
        │     DATA      │         If any arrow breaks —data isn't
        │  (feedback)   │         captured, or captured and not fed
        └──────┬───────┘         back— the wheel stops and the
               │ improve         system freezes (the stone, not the
               ▼                 snowball).
        ┌──────────────┐
        │ BETTER SYSTEM│
        │(more coverage)│
        └──────┬───────┘
               │ attracts
               ▼
           (more USE) ──► the wheel turns again, faster

Common mistakes

Treating usage as cost and not as raw material (of mental model). What happens: the team designs the AI feature as they'd design a classic service —minimize the cost per request, withstand the traffic spike— and never asks what data that traffic produces nor how to use it. They optimize the feature to endure the usage, not to learn from it. Result: an efficient feature that never improves, because the usage passes through it without leaving a trace. Why it happens: the classic-systems instinct sees traffic as load; the idea that traffic is your improvement input is new and has to be adopted deliberately. How to detect it: if no one on the team can say what's done with the feedback each interaction generates, you're treating usage as cost. How to fix it: design the feedback capture point (lesson 4) as part of the architecture, not as an extra —usage is the snow the ball needs to grow—.

Believing a bigger model replaces the loop (of strategy). What happens: when the feature doesn't improve, the reflex is "let's switch it to a more powerful model". Sometimes it helps, but it doesn't set up the flywheel: a bigger model gives you a one-off jump in quality, not a trajectory of improvement. The feature with a big model and no loop is still the rock —a better rock, but a rock—: it stays where the new model left it. Why it happens: switching models is a concrete and visible action; setting up a data loop is diffuse architecture work. How to detect it: if your plan to improve the feature is a list of models to try and no mechanism to learn from the usage, you're missing the flywheel. How to fix it: the model gives you the starting point; the loop gives you the slope. You need both, but the slope is the one that compounds —and the one a competitor can't copy by buying the same model—.

Capturing feedback and not closing it, believing the flywheel spins on its own (of process). What happens: the team puts the thumbs up/down, sees the users use it, and assumes "we already have the flywheel". But the data piles up without being fed back: the ball is on ice, it moves but doesn't grow. The quality curve is flat like the loopless system's, and the team doesn't understand why, "having feedback", the feature doesn't improve. Why it happens: capturing is visible (a button appears) and gets confused with closing (which is invisible). How to detect it: if you have weeks of saved feedback and your quality curve is flat, your loop is open. How to fix it: close the loop (lessons 5 and 7) —the feedback must reach the eval-set, the prompt, or the retrieval, or the wheel doesn't spin—.

Exercises

Exercise 1 — The snowball and the rock. Explain, with the snowball analogy, why the gap between the system with a loop and the system without a loop widens instead of staying constant. Then answer: in the worked example, in which round did the gap start to widen, and why did the system with a loop attack "cancel" and "broken_product" before "size_change"?

See solution

The gap widens because the system with a loop compounds and the one without doesn't. The snowball (with a loop) picks up more snow each turn and, being bigger, picks up even more on the next —its growth accumulates—. The rock (without a loop) rolls but doesn't change size. Since one grows and the other stays the same, the distance between them isn't constant: it increases each round. In system terms: each improvement of the system with a loop stacks on the previous ones (round 3's coverage includes everything from rounds 1 and 2), while the loopless one stays at its starting point forever.

In the example, the gap started to widen in round 1: there the system with a loop rose to 0.77 and the loopless one stayed at 0.60, opening 17 points; in the following rounds the gap grew to 40 points. The system with a loop attacked "cancel" (9% of the traffic) and "broken_product" (8%) before "size_change" (1%) because the feedback volume orders the failures by impact: the most frequent types generate more thumbs_down, so they appear first and strongest in the feedback. Fixing the most frequent first maximizes each round's improvement —you go up 17 points attacking cancel+broken, you'd go up barely 2 attacking the two rare types—. The loop doesn't just say what's broken; the volume says what to fix first.

Exercise 2 — The slowed flywheel. A team captured thumbs up/down for three months but never converted any into an eval case, nor adjusted the prompt, nor touched the retrieval. Draw (in words) how their quality curve would look compared to the example's two, and explain with the snowball analogy why. Then say what minimum they'd have to do for the wheel to start spinning.

See solution

Their quality curve would look flat, identical to the example's loopless system (0.60 in every round), not the one with a loop. With the analogy: it's a snowball on bare ice —it's rolling (capturing feedback, generating data) but doesn't pick up snow (doesn't feed anything back), so it doesn't grow—. The motion gives the illusion that it should improve ("we have feedback, don't we?"), but without closing the loop, capturing is indistinguishable from not capturing in quality terms: the curve is the same.

The minimum for the wheel to start spinning is to close the loop: take the accumulated thumbs_down and feed them back to something that improves the system —most directly, turn them into eval-set cases (lesson 5), and from there route each failure type to its lever: prompt, retrieval, or considering fine-tune (lesson 7)—. As soon as a thumbs_down becomes an eval case that a fix makes pass, the ball picked up its first handful of snow and the curve starts to rise. The hard lesson: capturing without closing doesn't move the needle; the flywheel spins only when the loop closes.

Exercise 3 — The model or the loop? Two Mercado features have the same problem: their quality stalled. For feature X, the team proposes "upgrade to a more powerful model"; for feature Y, "set up a data loop that learns from feedback". Explain what each option gives (a one-off jump vs a trajectory) and in which situation each is the right answer. Can a feature need both?

See solution

Upgrading to a more powerful model (feature X) gives a one-off jump in quality: the feature improves all at once up to where the new model takes it, and stays there. It's the right answer when the current model's quality ceiling is the problem —the feature already learned all its model can give and needs more raw capacity—. But it doesn't set up the flywheel: it's a better rock, not a snowball. If the big loopless model stalls again, there's no mechanism to keep improving.

Setting up a data loop (feature Y) gives a trajectory of improvement: the feature rises round after round as it learns from its usage, and keeps rising while the loop is closed. It's the right answer when the problem is coverage or unseen cases —the feature fails at things the feedback can reveal and that a prompt/retrieval fix can resolve—, which is most real cases. The loop compounds; the big model doesn't.

Yes, a feature usually needs both, and in an order: the model gives you the starting point (the height from which you start), the loop gives you the slope (how fast you improve). An excellent model without a loop starts high and stays there; a modest model with a loop starts lower but climbs. The ideal is a good model with a loop: it starts high and keeps rising. And the strategic part: the model can be bought by any competitor (they all have access to the same models), but the loop —the accumulated data from your usage— is yours and isn't bought. That's why, in the long run, the slope beats the height.

Summary and next step

In this lesson you measured the whole module's motivation: a system with a data loop compounds improvements and pulls away from the competition; one without a loop freezes where it was born. You saw it with the snowball analogy (which picks up snow and accelerates) versus the rock (which rolls without changing), and you quantified it: two systems identical on launch day (0.60 both) ended, six rounds later, at 1.00 the one with a loop and 0.60 the one without —a 40-point gap that on day one didn't exist—. You saw the three properties that make the flywheel an architecture decision: usage is raw material (not cost), the improvement accumulates (doesn't reset), and the wheel only spins if the loop is closed (capturing isn't enough). And you saw a practical property of real feedback: its volume orders the failures by impact, so you fix first what hurts the most people.

Before moving on you should be able to: explain the flywheel and why it compounds; distinguish the one-off jump of a better model from the trajectory a loop gives; recognize a snowball on ice (feedback captured and not closed); and place the hard boundary (here the flywheel as a decision; training with the data is AI Engineering).

What follows is the first piece that makes the wheel spin: observability. You can't improve what you can't see, and —this is the important thing— an AI component needs an observability that a classic server log doesn't give. In lesson 3 you'll execute a two-view dashboard: the classic one, which says "all green, 100% successful responses", and the AI one, which adds the quality signal and reveals a degraded feature that the server monitoring left completely invisible. It's the step from "the data loop compounds" to "for it to compound, first I have to see what's failing".

Resources

← Previous1. Module introduction: the data and feedback loopNext →3. Observability for AI: beyond the server log