Module 7: RAG and Semantic Search

2. The Problem RAG Solves

Next →

Overview

Before you can understand RAG, you need to understand why it exists. This capsule covers the fundamental problems of pure LLMs: hallucinations, outdated knowledge, and the lack of private context.

Problem 1: Hallucinations

What are hallucinations?
The LLM generates information that looks correct but is made up or wrong.

Example:

User: "When was the company XYZ Inc. founded?"

LLM (without RAG): "XYZ Inc. was founded in 2015 by John Smith."
→ An invented answer (the LLM has no data about XYZ Inc.)

Why it happens:

  • The LLM was trained on public data (up to a cutoff date)
  • It has no information about private/recent companies
  • The model "fills in the gaps" with plausible but false information

Problem 2: Outdated knowledge

Example:

User: "Who is the current CEO of Twitter?"

LLM (trained up to 2023): "Jack Dorsey"
→ Incorrect (Elon Musk bought Twitter in 2022)

Why it happens:

  • The training cutoff date (e.g. GPT-4 up to Sep 2023)
  • The real world changes constantly
  • Retraining the model is expensive and slow

Problem 3: No access to private data

Example:

User: "What does the internal handbook say about the vacation policy?"

LLM: "I don't have access to your company's internal documents."
→ A correct answer, but not a useful one

Why it happens:

  • The LLM was trained on public data (the web, books)
  • It has no access to:
    • Companies' internal documents
    • Private databases
    • Personal files

Problem 4: The context limit

Example:

User: "Summarize these 100 PDFs about financial regulations"

LLM: [Error] "The context exceeds the 128K token limit"

Why it happens:

  • LLMs have a context limit (e.g. GPT-4 Turbo = 128K tokens)
  • 100 PDFs can be 1M+ tokens
  • You can't send it all to the LLM

The traditional solution: Fine-tuning

What is it?
Retraining the LLM on your specific data.

Limitations:

  1. Cost: $1000+ per fine-tuning run (for large models)
  2. Time: Hours/days of training
  3. Going stale: If the data changes, you retrain
  4. Hallucinations: They still happen (it only reduces the frequency)

The modern solution: RAG

The key idea:
Instead of retraining the LLM, give it access to search in real time.

The flow:

1. The user asks: "What does the handbook say about vacation?"

2. The system searches the internal documents (semantic search)
   → It finds 5 relevant sections

3. The system sends the question + the context to the LLM:
   "Context: [5 sections of the handbook]
    Question: What does it say about vacation?"

4. The LLM generates an answer based on the context
   → An accurate answer (it doesn't make things up)

Comparison: With and without RAG

Without RAG (a pure LLM):

User: "How do I configure Pinecone for production?"

LLM: "To configure Pinecone in production, you should:
1. Create an index with the correct dimension
2. Configure sharding
3. Tune the throughput
..."

The problem: A generic answer (it may be outdated)

With RAG:

User: "How do I configure Pinecone for production?"

The system:
1. Searches the Pinecone docs (semantic search)
   → It finds the official setup guide (up to date)

2. Sends this to the LLM:
   Context: [Pinecone's official guide]
   Question: How do I configure it for production?

LLM: "According to Pinecone's official documentation:
1. Create an index with 'pod_type=p1' for production
2. Set 'replicas=3' for high availability
3. Use 'metric=cosine' for OpenAI embeddings
..."

The result: An accurate, up-to-date answer ✅

Advantages of RAG

  1. It eliminates hallucinations (answers grounded in real documents)
  2. It's always up to date (it searches the current docs)
  3. Access to private data (it searches your documents)
  4. No retraining required (adding docs = updating the index)
  5. Explainable (you can see which docs it used)

Limitations of RAG

(We'll look at these in depth in capsule 06)

  1. Higher latency (search + generation)
  2. Dependence on search (if the search fails, the answer is wrong)
  3. Cost per query (embedding + LLM)
  4. The context limit (only the top-K docs fit in the prompt)

Summary

Key points:

  • LLM problems: Hallucinations, going stale, no private data
  • Fine-tuning: Expensive, slow, and it still hallucinates
  • RAG: Real-time search + an LLM = accurate answers
  • Advantages: No hallucinations, up to date, access to private data

Next capsule: 03-rag-architecture.md — RAG's components and flows.

← Previous1. Module Introduction: RAG and Semantic SearchNext →3. RAG Architecture: Components and Flows