5. RAG vs Fine-tuning: When to Use Each

Overview

RAG and fine-tuning are two different strategies for adapting LLMs to your domain. This capsule compares them in depth: advantages, limitations, costs, and a decision matrix.

---

What each one is

RAG (Retrieval-Augmented Generation)

Real-time search + a base LLM (unmodified)

Query → Search the docs → Augment the prompt → The LLM generates

---

Fine-tuning

Retraining the LLM on your specific data

Your data → Retraining → A customized LLM

---

Full comparison

Aspect	RAG	Fine-tuning
Upfront cost	Low ($50-500)	High ($500-5000+)
Setup time	Hours	Days/weeks
Updating the data	Immediate (add docs)	Slow (retrain)
Hallucinations	Reduced (if the search works)	Reduced but not eliminated
Explainability	High (it shows the sources)	Low (the model is a black box)
Latency	High (2-5s)	Low (1-2s)
Scalability	Scales with the docs	Scales with the model
Task precision	Medium-high	High (if trained well)

---

Advantages and disadvantages

RAG

Advantages:

1. ✅ Always up to date: Adding a new document = immediate
2. ✅ Explainable: It shows which documents it used
3. ✅ Low upfront cost: You only index the docs
4. ✅ No ML expertise required: Implement it with LangChain/LlamaIndex
5. ✅ Flexible: Swap the vector DB, embeddings, or LLM easily

Disadvantages:

1. ❌ High latency: Search + generation (2-5s)
2. ❌ Dependence on search: If the search fails, the answer is wrong
3. ❌ Cost per query: Embedding + LLM every time
4. ❌ The context limit: Only the top-K chunks fit in the prompt
5. ❌ It doesn't learn style: The LLM doesn't adapt to your tone/vocabulary

---

Fine-tuning

Advantages:

1. ✅ Lower latency: Inference only (1-2s)
2. ✅ It learns style: The model adapts to your specific tone/vocabulary
3. ✅ Higher precision on specific tasks: (e.g. classification, extraction)
4. ✅ No search required: The knowledge is "internalized" in the weights

Disadvantages:

1. ❌ High cost: $500-5000+ per fine-tuning run (depending on model size)
2. ❌ Slow: Days/weeks of training
3. ❌ Going stale: If the data changes, you retrain
4. ❌ It doesn't eliminate hallucinations: It only reduces the frequency
5. ❌ Less explainable: You can't see what it "learned"
6. ❌ Requires expertise: Preparing the data, tuning the hyperparameters

---

Ideal use cases

Use RAG when:

1. The data changes frequently

   Example: Product documentation (updated weekly)
   → RAG: Adding a new doc = instant
   

2. You need explainability

   Example: A legal/medical system
   → RAG: It shows which article/paper it cited
   

3. The dataset is large and diverse

   Example: 10K documents across multiple topics
   → RAG: It searches only what's relevant per query
   

4. The budget is limited

   Example: An early-stage startup
   → RAG: $50-500 setup, $0.02/query
   

5. You need a fast prototype

   Example: Validating an idea in 1 week
   → RAG: Implement it in hours/days
   

---

Use Fine-tuning when:

1. The task is very specific

   Example: Classifying support tickets into 20 exact categories
   → Fine-tuning: The model learns the specific patterns
   

2. The style/tone is very particular

   Example: An assistant that must speak like Shakespeare
   → Fine-tuning: The model internalizes the style
   

3. The data does NOT change (or changes slowly)

   Example: A historical corpus of 19th-century literature
   → Fine-tuning: Static knowledge
   

4. Latency is critical

   Example: Live chat with answers in < 1 second
   → Fine-tuning: Inference only (no search)
   

5. The dataset is small and structured

   Example: 500 examples of sales emails
   → Fine-tuning: It learns the specific patterns
   

---

Combining them: RAG + Fine-tuning

Can they be combined? Yes!

The architecture:

1. Fine-tune the LLM on your specific style/tone
2. Use that fine-tuned LLM as the generator in RAG
   → Search (RAG) + a customized style (fine-tuning)

Example:

A startup with:
- Technical documentation (it changes weekly)
- A very specific brand tone (casual, friendly)

The solution:
1. Fine-tune GPT-3.5 with examples of your tone
   → The LLM learns the style
2. RAG over the documentation
   → The search is always up to date
3. The fine-tuned LLM generates the answer in your style

The trade-off: Greater complexity and cost (both methods).

---

Decision matrix

Your situation	Recommended method
The data changes frequently	RAG ✅
A very specific task (classification, extraction)	Fine-tuning ✅
A large dataset (10K+ docs)	RAG ✅
A small dataset (500 examples)	Fine-tuning ✅
You need explainability	RAG ✅
Latency < 1s is critical	Fine-tuning ✅
A limited budget (< $1000)	RAG ✅
A very particular style/tone	Fine-tuning ✅
A fast prototype (days)	RAG ✅
Production with static data	Fine-tuning ✅
Multiple domains/topics	RAG ✅
A single, specific domain	Fine-tuning ✅

---

Cost comparison (1 year)

The scenario: An internal Q&A system

RAG:

Setup: $200 (the initial indexing)
Queries: 10K/month × $0.02 = $200/month
Vector DB: $70/month (Pinecone Starter)

Total for year 1: $200 + ($270 × 12) = $3440

Fine-tuning:

Setup: $2000 (fine-tuning GPT-3.5)
Retraining: $2000 × 4 (every 3 months) = $8000
Inference: 10K/month × $0.002 = $20/month

Total for year 1: $10,000 + ($20 × 12) = $10,240

The result: RAG is 3x cheaper in this scenario.

---

Summary

Key points:

• RAG: Immediate updates, explainable, low upfront cost
• Fine-tuning: It learns style, lower latency, higher cost
• The decision: It depends on update frequency, budget, latency
• Combining them: RAG + fine-tuning for the best of both worlds

---

Next capsule: 06-rag-limitations.md — Common problems and solutions.