Module 7: The Data and Feedback Loop
6. Prompt vs RAG vs fine-tune as an architectural choice
Overview
By the end of this lesson you'll be able to make the decision that appears every time the feedback tells you the component fails: do I improve the system by adjusting the prompt, setting up a RAG, or training a fine-tune? They're the three classic levers for improving an AI component, and the lesson's thesis is that choosing between them is an architectural decision —not an implementation decision— because each one has a different profile of latency, cost, data freshness, and maintenance, and that profile, not the fashion, is what should decide. Changing the prompt is cheap, instantaneous, and reversible; RAG brings fresh data on every query in exchange for a retrieval hop; fine-tune bakes the knowledge into the model —short prompts, cheap inference— but it freezes when trained. The correct decision depends on what your feature needs, and the module's most expensive error in this dimension is jumping to fine-tune when prompt or RAG were enough.
This matters because the three levers are frequently confused, and "fine-tune" has an aura of a serious solution that pulls teams toward the most expensive and most rigid option for the wrong reasons. Fine-tune sounds like "really teaching the model", while adjusting a prompt sounds like a patch. But in most real cases, the correct hierarchy is the reverse: start with the prompt, go up to RAG if you need fresh or large knowledge, and consider fine-tune only when prompt and RAG fall short and the behavior is stable and high-volume. Choosing the lever by its architectural profile —and not by its prestige— is what separates a system that improves fast and cheap from one that spends months training a model that goes obsolete as soon as a policy changes.
Connection with the module: this lesson opens the levers that lessons 5 and 7 connect with the feedback. In lesson 5 you closed the loop toward the eval-set and saw that the feedback has three destinations (eval, prompt, retrieval); here you examine in depth the profile of each improvement lever so you can choose well. Lesson 7 will use this decision to route each type of failure to its lever. And this module's boundary is here harder than ever: you will NOT learn to build a RAG, to compute embeddings, nor to run a fine-tune —that's the mechanics, and it's from the AI Engineering ecosystem—. You'll learn to decide between the three by treating them as boxes with architectural properties. The mental rule: "how do I index the vector store or how do I train the model?" is AI Engineering; "should I use prompt, RAG, or fine-tune for this feature?" is from here.
Analogy: how you give information to a new employee
Imagine you hire a competent employee who doesn't know your company, and you have to give them the information they need to serve customers. You have three ways to do it, and each has a different cost and profile.
The first is to tell them each time, in the moment: before each call, you hand them a note with the data they'll need ("this customer ordered such a product, the shipping policy is this"). It's instantaneous —you change the note and they have the new information— and it requires no prior preparation. But the note has a size limit: you can't hand them a 500-page manual before each call. This is the prompt: the knowledge goes inside the instruction, changes instantly, but is limited by how much fits on the note (the context window).
The second is to give them access to a well-organized file cabinet: they memorize nothing, but when they need a datum, they go to the cabinet, look for the correct folder, and consult it. The cabinet can be enormous (millions of documents) and it's always up to date —when a policy changes, you update the folder and the employee now consults the new version—. The cost is that each consultation takes an extra moment (going to the cabinet and back). This is RAG: the knowledge lives in an external index the model consults in the moment; excellent freshness, unlimited knowledge, in exchange for a retrieval hop per query.
The third is to send them to an intensive training course: for weeks, you drill all the company's knowledge and style into them until they have it internalized and answer fast without consulting anything. After the course, they're lightning-fast —they need neither note nor cabinet, they already know it—. But the course was expensive and slow, and here's the problem: the day a policy changes, the employee keeps teaching the old one, because what they learned in the course became fixed. To update them, you have to send them to another course. This is fine-tune: the knowledge and style are baked into the model —fast and cheap inference, short prompts— but frozen at the moment of training; updating requires re-training.
The architectural decision is exactly choosing between these three ways to give the employee information, and the answer depends on your situation. Does the information change often? The note (prompt) or the cabinet (RAG), never the course (fine-tune, which goes obsolete). Is it a great deal of live information? The cabinet (RAG). Is it little and you want to start today? The note (prompt). Is the style stable, the volume enormous, and neither the note nor the cabinet achieves the behavior you need? There, and only there, the course (fine-tune) is justified.
Worked example: the profile of each lever, and the recommender
We're not going to say when each lever is worth it: we're going to model its architectural profile and execute a recommender that chooses per feature. We model each option with its properties —extra latency, tokens per query, update effort, data freshness— and compute its monthly cost; then a recommender decides, for four Mercado scenarios, which lever is best. Notice that data freshness usually decides first.
# Lesson 06 (M7) — prompt vs RAG vs fine-tune as an ARCHITECTURAL DECISION. SIMULATED.
# Zero network, zero API, zero keys. Deterministic.
#
# We do NOT teach the mechanics (indexing embeddings, training) — that's AI Engineering.
# We model the ARCHITECTURAL PROPERTIES of each option and decide which one fits
# according to the feature's requirements: latency, cost, data FRESHNESS and maintenance.
# --- Architectural profile of each option (illustrative numbers, not mechanics). ---
# added_latency_ms : extra latency the option adds per request
# freshness_lag : how long a NEW datum takes to be reflected in the responses
# update_effort : what it costs to incorporate new information (0=trivial, 10=very expensive)
# setup_effort : what it costs to stand it up the first time
# per_query_tokens : typical tokens per request (more tokens = more cost/latency)
APPROACHES = {
"prompt": dict(
added_latency_ms=0, freshness_lag="instant (you edit the prompt)",
update_effort=1, setup_effort=1, per_query_tokens=1200,
note="the knowledge goes INSIDE the prompt; limited by the context window"),
"rag": dict(
added_latency_ms=120, freshness_lag="minutes (you update the index)",
update_effort=2, setup_effort=5, per_query_tokens=1500,
note="retrieves fresh data on every query; adds a retrieval hop"),
"finetune": dict(
added_latency_ms=-40, freshness_lag="days (must RE-TRAIN)",
update_effort=9, setup_effort=9, per_query_tokens=400,
note="the knowledge/style is BAKED IN; short prompts, but frozen"),
}
print("=== Architectural profile of each option ===")
print(f"{'option':<10}{'lat_extra':>10}{'tok/query':>11}{'update':>8}{'setup':>7} freshness")
for name, p in APPROACHES.items():
print(f"{name:<10}{p['added_latency_ms']:>+9}ms{p['per_query_tokens']:>11}"
f"{p['update_effort']:>8}{p['setup_effort']:>7} {p['freshness_lag']}")
print()
# --- The estimated monthly cost per option (tokens -> USD, consistent with M2). ---
USD_PER_1K = 0.006 # blended in/out price per 1000 tokens (illustrative)
QUERIES_MONTH = 100_000
print(f"=== Estimated monthly cost ({QUERIES_MONTH:,} queries/month) ===")
for name, p in APPROACHES.items():
cost = QUERIES_MONTH * (p["per_query_tokens"] / 1000) * USD_PER_1K
print(f" {name:<10}: ${cost:>8,.0f}/mo ({p['per_query_tokens']} tok/query)")
print(" (fine-tune is cheaper PER QUERY with short prompts, but its real cost is")
print(" in the RE-TRAINING every time a datum changes — not in the inference.)")
print()
# --- The recommender: given the feature's NEED, it chooses the option. ---
# The hard design rule: if the data CHANGES often, fine-tune is ruled out
# (its freshness is "days / re-train"). If the knowledge is LARGE and live, RAG.
# If it's small and you want to iterate fast, prompt. fine-tune only if the
# behavior is STABLE, HIGH volume and prompt+RAG fall short.
def recommend(need):
if need["data_changes"] in ("hours", "days"):
# Freshness rules: fine-tune is out (it freezes when trained).
return "rag" if need["knowledge_size"] == "large" else "prompt"
# Stable data:
if need["knowledge_size"] == "large":
return "rag"
if need["stable_behavior"] and need["volume"] == "high":
return "finetune" # stable style + high volume amortizes the training
return "prompt"
SCENARIOS = {
"Support agent (shipping/return policies change every week)":
dict(data_changes="days", knowledge_size="medium", stable_behavior=False, volume="high"),
"Search over a live catalog (millions of products, change daily)":
dict(data_changes="hours", knowledge_size="large", stable_behavior=False, volume="high"),
"'Describe your product' with a FIXED brand tone and huge volume":
dict(data_changes="never", knowledge_size="small", stable_behavior=True, volume="high"),
"Quick prototype of a ticket classifier (iterate today)":
dict(data_changes="never", knowledge_size="small", stable_behavior=False, volume="low"),
}
print("=== The recommender: the option according to the need ===")
for scenario, need in SCENARIOS.items():
choice = recommend(need)
print(f" -> {choice.upper():<9} | {scenario}")
print()
print("Architectural rule: DATA freshness usually decides first.")
print(" data that changes often -> prompt or RAG (NEVER fine-tune: it freezes)")
print(" large and live knowledge -> RAG")
print(" stable behavior + high volume + prompt/RAG fall short -> fine-tune")
What to expect. When you run it, the output is exactly this:
=== Architectural profile of each option ===
option lat_extra tok/query update setup freshness
prompt +0ms 1200 1 1 instant (you edit the prompt)
rag +120ms 1500 2 5 minutes (you update the index)
finetune -40ms 400 9 9 days (must RE-TRAIN)
=== Estimated monthly cost (100,000 queries/month) ===
prompt : $ 720/mo (1200 tok/query)
rag : $ 900/mo (1500 tok/query)
finetune : $ 240/mo (400 tok/query)
(fine-tune is cheaper PER QUERY with short prompts, but its real cost is
in the RE-TRAINING every time a datum changes — not in the inference.)
=== The recommender: the option according to the need ===
-> PROMPT | Support agent (shipping/return policies change every week)
-> RAG | Search over a live catalog (millions of products, change daily)
-> FINETUNE | 'Describe your product' with a FIXED brand tone and huge volume
-> PROMPT | Quick prototype of a ticket classifier (iterate today)
Architectural rule: DATA freshness usually decides first.
data that changes often -> prompt or RAG (NEVER fine-tune: it freezes)
large and live knowledge -> RAG
stable behavior + high volume + prompt/RAG fall short -> fine-tune
Read the output in three parts, because each table installs a part of the decision.
The profile reveals the central trade-off: latency/cost per query vs freshness and maintenance. Look at the first table carefully. The prompt adds no extra latency (+0 ms), is trivial to update (update 1) and to set up (setup 1), and its freshness is instant —you edit the prompt and that's it—. The RAG adds a retrieval hop (+120 ms) and some setup (5), but its freshness is minutes (you update the index) and it handles unlimited knowledge. The fine-tune is the strangest: it subtracts latency (-40 ms, because the baked knowledge allows short prompts and even a smaller model) and uses far fewer tokens per query (400 vs 1200-1500), but its update_effort is 9 and its setup 9 —very expensive—, and its freshness is days because updating requires re-training. There's the whole trade-off: fine-tune optimizes the inference (fast, cheap per query) at the cost of freshness and maintenance (rigid, expensive to update).
The cost per query deceives; the real cost of fine-tune is in the re-training. The second table seems to give fine-tune the win: $240/mo against $720 for the prompt and $900 for RAG, because its short prompts consume fewer tokens. And it's true —per query, fine-tune is the cheapest—. But read the note: that calculation doesn't include the cost of re-training, which is where the real cost of fine-tune lives. Every time a policy changes, you have to assemble a new dataset and run a training (engineering time + compute), and that doesn't appear in the cost per query. For a feature whose data changes often, the cost of re-training over and over far exceeds the per-query savings. That's why the cost per query is a trap: it makes fine-tune look cheap exactly when its hidden cost (maintenance) makes it expensive.
The recommender demonstrates that freshness decides first. The third table is the decision in action. The support agent (policies that change every week) → prompt: the data changes, so fine-tune is ruled out, and since the knowledge is medium (not enormous), it fits in the prompt. The search over a live catalog (millions of products, change daily) → RAG: the data changes and is enormous, so RAG is the only one that gives freshness without a size limit. The "describe your product" generator with a fixed brand tone and huge volume → fine-tune: here yes, because the behavior (the tone) is stable —it doesn't change— and the volume is enormous, so the baked style is justified and amortizes the cost of training. The classifier prototype to iterate today → prompt: fast to set up, fast to change. Notice the pattern: in three of the four cases, the first thing asked was "does the data change?", and that question alone ruled out or chose the lever. Data freshness is, almost always, the first gate of the decision.
The profile of each lever, at the design level
The example modeled the profiles; it's worth understanding what governs each lever at the architectural level, without going into its mechanics.
Prompt (in-context): the default lever. Putting the knowledge and instructions in the prompt is the cheapest option, the fastest to change, and the most reversible. Its freshness is perfect (you edit the prompt and the change is instant) and its maintenance cost is minimal. Its limit is the size: only what fits in the context window fits, and putting a lot of text in each prompt raises the tokens (cost and latency). When it's the answer: small or medium knowledge, frequent changes, fast iteration, or simply as a first attempt —it's almost always worth trying the prompt before anything else—. It's the equivalent of the note you hand the employee before each call.
RAG (retrieval-augmented): the freshness and volume lever. RAG retrieves, at query time, the relevant documents from an external index and gives them to the model. Its great architectural virtue is twofold: freshness (you update the index and the change is reflected in the next query, without touching the model) and volume (the index can be enormous, far more than would fit in a prompt). Its cost is a retrieval hop per query (extra latency) and a few more tokens (the retrieved documents go into the prompt). When it's the answer: large and/or often-changing knowledge —a catalog, a base of policy documents, a live wiki—. It's the always-updated file cabinet. How the index is built (embeddings, chunking, retrieval) is AI Engineering; that RAG is the correct lever when you need freshness over a lot of knowledge is from here.
Fine-tune: the stable-behavior and high-volume lever. Fine-tune adjusts the model itself with examples, baking knowledge or —more usefully— a style/behavior into it. Its virtue is the inference: short prompts (no need to include the knowledge or many examples), low latency, low cost per query, and sometimes the possibility of using a smaller model. Its great flaw is the rigidity: what it learns is frozen at training, so updating it requires re-training —expensive and slow—. When it's the answer, and only then: the behavior is stable (doesn't change often), the volume is high (to amortize the cost of training), and prompt+RAG can't achieve the behavior you need (a very specific style, a very consistent format, a specialized task). It's the training course: expensive, slow, but leaves the employee internalizing the style. How it's trained (dataset, hyperparameters, evaluation) is AI Engineering; that fine-tune is justified only with stable behavior + high volume + insufficient prompt/RAG is from here.
And the hierarchy that summarizes the decision, which is the antidote against the error of jumping to fine-tune:
The ladder of levers (go up only when the previous rung falls short):
1. PROMPT ── start ALWAYS here. Cheap, instant, reversible.
│ Not enough? (very large or very changing knowledge)
▼
2. RAG ── go up if you need freshness over LOTS of knowledge.
│ Still not enough? (you need a style/behavior
│ that neither the prompt nor the retrieved context achieve, and the
▼ behavior is stable and the volume high)
3. FINE-TUNE ── ONLY here, and knowing you freeze it and it'll be costly to update.
The gravity of the decision: go up a rung only with an architectural reason,
never for prestige. Most features live on rung 1 or 2.
Common mistakes
Jumping to fine-tune when prompt or RAG were enough (of over-engineering). What happens: the team has a quality problem and its first instinct is "let's train a model with our data", skipping the prompt and RAG. It invests weeks in assembling a dataset and running trainings, when a prompt adjustment (half an hour) or a RAG over its documents (a few days) would have solved the problem —and with better freshness—. Worse: the fine-tune goes obsolete as soon as a policy changes, and you have to re-train. Why it happens: fine-tune has prestige ("we really trained the model") while the prompt sounds like a patch; the effort hierarchy is inverted relative to the prestige one. How to detect it: if you're planning a fine-tune and haven't exhausted the prompt nor RAG, you're almost certainly over-investing. How to fix it: go up the ladder in order —prompt first, RAG if you need freshness/volume, fine-tune only if the previous two fall short and the behavior is stable and high-volume—.
Choosing fine-tune for data that changes (of freshness). What happens: the team trains a model with the company's knowledge —policies, catalog, prices— and deploys it. It works well... until a policy changes, and the model keeps answering with the old one, because what it learned was frozen at training. Now every policy change requires a re-training, and the system is always behind reality. Why it happens: freshness wasn't considered when choosing the lever; the cost per query was optimized without seeing the maintenance cost. How to detect it: if your knowledge changes more often than you can re-train, fine-tune will always leave you out of date. How to fix it: for knowledge that changes, use prompt (if it fits) or RAG (if it's large) —the levers with freshness—; reserve fine-tune for what doesn't change (a style, a format, a stable task). Data freshness is the first question of the decision.
Looking only at the cost per query and believing fine-tune is "the cheapest" (of incomplete accounting). What happens: the team compares the options by their cost per query, sees that fine-tune uses fewer tokens (short prompts), and concludes it's the most economical option. It ignores the cost of re-training every time something changes, which is where the real cost of fine-tune lives. For a feature with changing data, that hidden cost far exceeds the per-query savings, and the "cheapest option" turns out to be the most expensive. Why it happens: the cost per query is visible and easy to compare; the maintenance cost is diffuse and overlooked. How to detect it: if your cost comparison doesn't include "how much does it cost to update this when something changes, and how often does it change?", it's incomplete. How to fix it: count the total cost —inference plus maintenance— and weight it by the frequency of change; there fine-tune usually loses its apparent cost advantage.
Exercises
Exercise 1 — The new employee. For each of the three ways to give the employee information, say which lever it represents, what its main virtue is, and what its limit is. Then, for Mercado's support agent —whose shipping and return policies change every week— say which way you'd choose and why, connecting it to the recommender's result.
See solution
- Handing them a note before each call → the prompt. Virtue: instantaneous (you change the note and they have the new thing), no prior preparation. Limit: the size of the note (the context window) —you can't hand them a whole manual—.
- Giving them access to an organized file cabinet → RAG. Virtue: unlimited and always-updated knowledge (you update the folder and they consult the new version). Limit: each consultation takes an extra moment (the retrieval hop).
- Sending them to an intensive course → fine-tune. Virtue: very fast afterward (they already know it, they consult nothing), cheap inference. Limit: the course was expensive and slow, and it stays frozen —when a policy changes, they keep teaching the old one until you send them to another course—.
For the support agent whose policies change every week, I'd choose the prompt (the note), and it matches the recommender. The reason is freshness: since the policies change often, fine-tune (the course) is ruled out —it would freeze with the old policies and you'd have to re-train every week—. Between prompt and RAG, the agent's knowledge is medium (it fits in the prompt), so the note is enough and is the cheapest and fastest to update; if the knowledge were enormous (millions of policy documents), I'd go up to RAG (the cabinet). Freshness decided first (ruled out fine-tune), and size decided between the two remaining.
Exercise 2 — The cost-per-query trap. A colleague argues: "the recommender says to use prompt for the support agent, but fine-tune costs $240/mo against $720 for the prompt —fine-tune is three times cheaper, we should train it—". Explain why the argument is incomplete and what cost it fails to consider, given that the agent's policies change every week.
See solution
The argument is incomplete because it compares only the cost per query (inference) and ignores the maintenance cost (re-training), which is where the real cost of fine-tune lives. Fine-tune's $240/mo are true for inference —its short prompts consume fewer tokens—, but that number doesn't include what it costs to update the model when the policies change.
And here's the detail that kills the argument: the agent's policies change every week. With fine-tune, each policy change requires assembling a new dataset and running a re-training —engineering time plus compute—, and that would happen every week. That recurring maintenance cost far exceeds the $480/mo fine-tune "saves" in inference relative to the prompt. With the prompt, on the other hand, updating a policy is editing text (minutes, ~zero cost) and the change is instant. So, counting the total cost (inference + maintenance, weighted by the weekly change frequency), the prompt is cheaper, not more expensive —and it also gives instant freshness, which fine-tune can't—. The cost per query is a trap that makes fine-tune look cheap exactly when its rigidity makes it expensive. Fine-tune would be the cheap option if the policies didn't change (like the description generator with a fixed tone), but they change every week, and that disqualifies it.
Exercise 3 — Which lever for each new feature? For each new Mercado feature, say which lever (prompt, RAG, or fine-tune) you'd recommend and why, following the ladder. (a) An assistant that answers questions about a product's thousands of reviews, which arrive new every hour. (b) A formatter that converts sellers' descriptions to the exact style and structure of the Mercado brand, a style that hasn't changed in years and that applies to millions of products. (c) A classifier that labels a support ticket into one of five categories, for an experiment that starts this week.
See solution
- (a) Assistant over thousands of reviews that arrive every hour → RAG. The knowledge is large (thousands of reviews per product, millions total) and changes constantly (new ones every hour). That rules out fine-tune (it would freeze, and re-training every hour is absurd) and rules out the prompt (thousands of reviews don't fit in the context window). RAG is the only one that gives freshness over a lot of knowledge: you index the reviews and retrieve the relevant ones on each query. Freshness and volume rule.
- (b) Formatter to brand style, stable, millions of products → fine-tune (the case that justifies it). Here yes. The behavior (the brand's style and structure) is stable —it hasn't changed in years—, the volume is enormous (millions of products, high amortization), and it's a style/format that a prompt with examples could achieve but less consistently and with long prompts (costly at that volume). Fine-tune bakes the style: short prompts, cheap inference at massive scale, and since the style doesn't change, the rigidity doesn't hurt. It meets the three requirements: stable, high volume, and suboptimal prompt/RAG. (Even so, it would be worth trying the prompt with examples first and measuring whether it suffices —going up the ladder in order—.)
- (c) 5-category classifier, this week's experiment → prompt. Little knowledge (five categories with their definitions fit easily in the prompt), and above all, you want to iterate today. The prompt is instant to set up and to adjust while you experiment with the categories. Setting up RAG or fine-tune for this week's experiment would be over-engineering —start with the prompt, and only go up the ladder if the experiment matures and the prompt falls short—.
The pattern: (a) freshness+volume → RAG; (b) stable+volume+needs the baked style → fine-tune; (c) small+iterate fast → prompt. And in all three, the ladder says: don't go up a rung without an architectural reason.
Summary and next step
In this lesson you learned to make the architectural decision prompt vs RAG vs fine-tune: each lever has a different profile of latency, cost, data freshness, and maintenance, and that profile —not the prestige— is what should decide. You saw it with the new employee analogy (the instant note = prompt, the always-updated file cabinet = RAG, the intensive course that freezes = fine-tune) and you measured it: the profile of each option, its cost per query (where fine-tune deceives because it hides the cost of re-training), and a recommender that chose per feature —prompt for the changing-policies agent, RAG for the live catalog, fine-tune only for the stable high-volume style—. You understood the ladder of levers (start with the prompt, go up to RAG for freshness/volume, fine-tune only as a last resort) and the antidote against the most expensive error: don't jump to fine-tune for prestige; go up only with an architectural reason, knowing you freeze it.
Before moving on you should be able to: describe the profile of each lever (latency, cost, freshness, maintenance); explain why data freshness usually decides first; recognize the cost-per-query trap (fine-tune hides the cost of re-training); apply the ladder of levers; and place the hard boundary (the mechanics of RAG/fine-tune/embeddings is AI Engineering; the choice is from here).
What follows is the module's synthesis: routing the feedback signal to the correct lever. You already know how to capture the feedback (lesson 4), close it toward the eval (lesson 5), and which the levers are (this lesson). In lesson 7 you'll bring it all together: the feedback is grouped by failure type, each group is routed to its lever —missing datum → retrieval, systematic tone → prompt, stable class prompt+RAG don't cover → consider fine-tune—, and every failure also goes to the eval-set. You'll execute the complete loop and see the score rise from 0.43 to 0.86 by applying the correct levers. It's the step from "I know the levers" to "I know which lever to move for each type of failure, and how to verify it worked".
Resources
- Anthropic — Claude documentation — the conceptual guide to when to put the knowledge in the prompt, when to retrieve it (retrieval), and when to consider fine-tuning the model; the reference for the properties of each approach, without pinning a model version. In English.
- Chip Huyen — AI Engineering (O'Reilly) — the in-depth treatment of prompt engineering, RAG, and fine-tuning with their latency, cost, and freshness trade-offs; the book for the mechanics of the levers this lesson only teaches to choose (the boundary with AI Engineering).
- martinfowler.com — "Emerging Patterns in Building GenAI Applications", Bharani Subramaniam and Martin Fowler — the pattern catalog places RAG and fine-tuning as architecture decisions with their consequences; the frame of this lesson. In English.
- AI Engineering ecosystem (referral) — for building each lever: how RAG is indexed and retrieved (embeddings, chunking), how the dataset is assembled and a fine-tune is run, how serious prompt engineering is done. This module teaches how to choose the lever; AI Engineering teaches how to build it.