Module 7: RAG and Semantic Search
6. RAG's Limitations and How to Mitigate Them
Overview
RAG isn't perfect. This capsule covers its common limitations (bad retrieval, the context limit, latency, residual hallucinations) and mitigation strategies.
Limitation 1: Retrieval failure
The problem:
If semantic search returns irrelevant chunks, the LLM generates a wrong answer.
Example:
User: "What are the opening hours?"
The system searches (semantic search):
→ It returns chunks about "customer service" (a general topic)
→ It does NOT return the chunk with the specific hours (8am-5pm)
LLM: "Our team is available for personalized service."
→ A generic answer (it doesn't answer the question)
Mitigation 1.1: Hybrid search (keyword + semantic)
Query: "opening hours"
Keyword search:
→ Documents that literally contain "hours"
Semantic search:
→ Documents about "service" conceptually
Fusion (RRF):
→ Combine both → it finds the chunk with the hours ✅
Mitigation 1.2: Reranking with a cross-encoder
1. kNN returns the top-100 candidates
2. The cross-encoder reorders them specifically for the query
3. A more precise final top-10
Example:
Query: "refund policy"
kNN top-3:
1. "Company policies..." (0.85)
2. "Refund policy: 30 days..." (0.83) ← Relevant
3. "Privacy policy..." (0.81)
Cross-encoder:
1. "Refund policy: 30 days..." (0.95) ✅
2. "Company policies..." (0.70)
3. "Privacy policy..." (0.40)
Mitigation 1.3: Query expansion
Expanding the query with synonyms:
The original query: "hours"
The expanded query: "hours OR schedule OR opening times"
→ Greater recall (it finds more variants)
Limitation 2: The context window limit
The problem:
LLMs have a context limit (e.g. GPT-4 Turbo = 128K tokens). If you need more chunks, they don't fit.
Example:
Query: "Summarize the entire employee handbook" (200 pages)
The complete handbook: 250K tokens
GPT-4's limit: 128K tokens
→ You can't send it all ❌
Mitigation 2.1: Two-stage retrieval (coarse → fine)
1. The first search (coarse):
→ It returns the top-50 chunks (25K tokens)
2. The second search (fine):
→ Reranking → the top-5 (2.5K tokens)
→ They fit in the context ✅
Mitigation 2.2: The map-reduce pattern
For queries that require multiple chunks:
1. Split the query into sub-queries
"Summarize the handbook" → ["Summarize section 1", "Summarize section 2", ...]
2. For each sub-query:
An individual RAG pass → a partial answer
3. Combine the partial answers:
A final LLM call: "Combine these summaries: [summary 1] [summary 2]..."
Mitigation 2.3: Use an LLM with a long context
GPT-4 Turbo: 128K tokens
Claude 3 Opus: 200K tokens
Gemini 1.5 Pro: 1M tokens
→ More context = more simultaneous chunks
The trade-off: Higher cost.
Limitation 3: High latency
The problem:
RAG is slower than a pure LLM (search + generation).
Typical latency:
Query embedding: 50-200ms
kNN search: 10-50ms
Reranking (optional): 100-300ms
LLM generation: 2-5 seconds
Total: 2.2-5.5 seconds
Mitigation 3.1: Streaming the response
The user sees the answer as the LLM generates it:
"To set up..." [appears]
"the development..." [appears]
"environment, follow..." [appears]
The perception of lower latency (even though the total is the same)
Mitigation 3.2: Caching frequent queries
Query: "What is RAG?"
→ Cache hit → a pre-generated answer
→ Latency: 0ms ✅
A new query:
→ Cache miss → the full RAG flow
→ Latency: 3s
→ Save it to the cache for next time
Mitigation 3.3: A faster model
GPT-4 Turbo: 2-5s generation
GPT-3.5 Turbo: 1-2s generation
Claude 3 Haiku: 0.5-1s generation
The trade-off: Lower quality (but faster)
Limitation 4: Residual hallucinations
The problem:
Even though RAG reduces hallucinations, the LLM can still "make things up" if the context is ambiguous.
Example:
Context: "The product costs $99. We offer discounts."
Query: "How much is the discount?"
LLM: "The discount is 20%."
→ Made up (the context does NOT mention 20%) ❌
Mitigation 4.1: Strict prompt engineering
An instruction to the LLM:
"Answer ONLY if the context contains the answer EXPLICITLY.
If the context does NOT contain the answer, say:
'I don't have enough information to answer this.'"
The result:
LLM: "I don't have enough information about the discount percentage.
The context only mentions that we offer discounts,
but it doesn't specify the amount." ✅
Mitigation 4.2: Citation forcing
An instruction to the LLM:
"ALWAYS cite the exact source.
Format: 'According to [document], page [X]: [answer]'"
The result:
LLM: "According to pricing-guide.pdf, page 3: The product costs $99.
The discount percentage is NOT specified in this document." ✅
Mitigation 4.3: Post-generation verification
1. The LLM generates an answer
2. The system verifies:
Does the answer contain facts from the context? ✅
Does the answer contain information NOT in the context? ❌
3. If there's unverified information:
→ Reject the answer or flag it as "unverified"
Limitation 5: Accumulated cost
The problem:
RAG has a cost per query (embedding + LLM). With many queries, the cost grows.
Example:
10K queries/month × $0.022 = $220/month
100K queries/month × $0.022 = $2200/month
Mitigation 5.1: Aggressive caching
Cache the frequent answers (TTL: 1 week)
→ 30% of queries are repeats
→ Savings: 30% × $2200 = $660/month
Mitigation 5.2: A cheaper model
GPT-4 Turbo: $0.01/1K input tokens
GPT-3.5 Turbo: $0.0005/1K input tokens (20x cheaper)
Cost/query:
- GPT-4: $0.022
- GPT-3.5: $0.0011
Savings with GPT-3.5: 95%
The trade-off: Lower quality (but 20x cheaper).
Mitigation 5.3: A local model
Llama 3 (8B):
- Inference cost: $0 (self-hosted)
- Setup: A GPU server ($500/month)
Break-even:
$500/month = 22,727 queries with GPT-4
If you have > 22K queries/month → a local model is cheaper
A summary of the limitations and solutions
| Limitation | Solution |
|---|---|
| Retrieval failure | Hybrid search, reranking, query expansion |
| The context limit | Two-stage retrieval, map-reduce, an LLM with a long context |
| High latency | Streaming, caching, a faster model |
| Residual hallucinations | A strict prompt, citation forcing, verification |
| Accumulated cost | Aggressive caching, a cheaper model, self-hosting |
Summary
Key points:
- RAG is NOT perfect: Retrieval, context, latency, cost
- Mitigation: Hybrid search, reranking, caching, prompt engineering
- Trade-offs: Precision vs latency vs cost
- A robust design: Combine multiple techniques
Next capsule: 07-capstone-exercise-7.md — Diagnosing and fixing RAG problems.