Module 7: RAG and Semantic Search
7. Capstone Exercise: Diagnosing RAG Problems
Exercise overview
This is the capstone exercise for Module 7. Here you diagnose 5 common problems in RAG systems and propose specific solutions.
Problem 1: Generic answers
Context:
A RAG system for API documentation. Users report generic answers that don't address their specific question.
Example:
User: "How do I authenticate requests with JWT?"
The system (RAG):
"Our API requires authentication to protect your data.
You can authenticate using several available methods."
→ A generic answer (it does NOT say how to use JWT)
Diagnosis: What's the problem?
See solution
The problem: Retrieval failure (the search returns general chunks, not specific ones about JWT)
Possible causes:
- The chunk with JWT is poorly embedded
- The query embedding doesn't capture "JWT" as an important keyword
- Top-K is too low (e.g. top-3, not enough to include the relevant chunk)
Solutions:
Solution 1: Hybrid search (keyword + semantic)
Keyword search: "JWT" must appear literally
Semantic search: "authentication" conceptually
Fusion: RRF → it guarantees "JWT" is in the results
Solution 2: Increase top-K
Current top-K: 3
New top-K: 10
→ A greater chance of including the relevant chunk
Solution 3: Query expansion
The original query: "JWT"
Expanded: "JWT OR JSON Web Token OR token authentication"
→ Greater recall
Problem 2: Unacceptable latency
Context:
A RAG system for live support. Average latency: 8 seconds. Users abandon before seeing the answer.
The latency breakdown:
- Query embedding: 150ms
- kNN search: 50ms
- Reranking (Cohere): 300ms
- LLM generation (GPT-4): 7500ms
Total: 8000ms
Diagnosis: How do you reduce the latency?
See solution
The problem: LLM generation is 94% of the latency (7.5s out of 8s)
Solutions:
Solution 1: Switch to a faster model
GPT-4 Turbo → GPT-3.5 Turbo
Latency: 7500ms → 1500ms
Savings: 6 seconds ✅
The new total latency: 2000ms (acceptable)
Solution 2: Streaming
The user sees the answer as it's generated:
- Perceived latency: ~1 second
- Actual latency: the same (8s)
- But the UX is improved
Solution 3: Caching frequent queries
The top-20 questions (30% of queries):
→ Pre-generated answers
→ Latency: 0ms for those queries
The remaining 70%:
→ Latency: 8s (but the average drops to ~5.6s)
Solution 4: Drop reranking (if that's acceptable)
Reranking: 300ms
If precision stays high without reranking:
→ New latency: 7700ms (a smaller saving, but something)
Problem 3: Hallucinations in the answers
Context:
A RAG system for medical information. Sometimes the LLM "makes up" medication dosages that are NOT in the context.
Example:
The retrieved context:
"Medication X is used to treat Y. Consult your doctor."
Query: "What is the dosage of X?"
LLM: "The recommended dosage is 500mg twice a day."
→ MADE UP (the context does NOT mention a dosage) ❌
Diagnosis: How do you prevent this?
See solution
The problem: The LLM hallucinates when the context is insufficient
Solutions:
Solution 1: Strict prompt engineering
The system prompt:
"You are a medical assistant. CRITICAL RULE:
- If the context does NOT contain the explicit answer,
say EXACTLY: 'I don't have enough information to answer this.'
- NEVER invent dosages, diagnoses, or treatments.
- ALWAYS cite the exact source."
Solution 2: Citation forcing
"Answer in this format:
'According to [document], page [X]: [answer]'
If you CAN'T cite a source, do NOT answer."
Solution 3: Post-generation verification
1. The LLM generates an answer
2. The system extracts the "facts" from the answer:
- "500mg" (dosage)
- "twice a day" (frequency)
3. The system checks whether those facts are in the context
→ "500mg" is NOT in the context ❌
4. The system rejects the answer:
"Answer blocked: it contains unverified information"
Solution 4: Temperature = 0
An LLM with temperature=0.1 (very low):
→ More deterministic answers
→ Less creativity = fewer hallucinations
Problem 4: Cost out of control
Context:
A RAG system with 500K queries/month. Current cost: $11,000/month. Budget: $3000/month.
The cost breakdown:
- Embeddings (query): 500K × $0.000004 = $2/month
- LLM (GPT-4): 500K × $0.022 = $11,000/month
- Vector DB (Pinecone): $70/month
Total: $11,072/month (3.7x over budget)
Diagnosis: How do you get the cost down to $3000/month?
See solution
The problem: The LLM is 99% of the cost ($11K out of $11K)
Solutions:
Solution 1: Switch to GPT-3.5 Turbo
GPT-4: $0.022/query
GPT-3.5: $0.0011/query (20x cheaper)
The new LLM cost: 500K × $0.0011 = $550/month
Total: $622/month ✅ (within budget)
Solution 2: Aggressive caching (50% of queries are repeats)
Cache hits: 250K queries → cost $0
Cache misses: 250K queries → cost $5,500
The new total: $5,572/month
→ Still over budget; combine it with GPT-3.5:
250K × $0.0011 = $275/month ✅
Solution 3: A self-hosted model (Llama 3)
Setup: A GPU server ($500/month)
Inference: $0/query (self-hosted)
The new cost: $500/month (just the server) + $72 (Pinecone) = $572/month ✅
Solution 4: Tiered pricing (critical vs non-critical queries)
Critical queries (10%): GPT-4 → 50K × $0.022 = $1,100/month
Non-critical queries (90%): GPT-3.5 → 450K × $0.0011 = $495/month
Total: $1,667/month ✅
Problem 5: Insufficient context
Context:
A RAG system for legal analysis. Query: "What do articles 12, 15, and 23 say, taken together, about this case?"
The problem:
The top-5 chunks only include articles 12 and 15. Article 23 is NOT in the context.
The result:
LLM: "According to articles 12 and 15... (it doesn't mention article 23)"
→ An incomplete answer ❌
Diagnosis: How do you guarantee that all the relevant articles are in the context?
See solution
The problem: Top-K is insufficient, or the search doesn't capture every article mentioned
Solutions:
Solution 1: Query decomposition
The original query: "articles 12, 15, and 23"
→ Detect the multiple entities
Run 3 searches:
1. "article 12" → Top-3
2. "article 15" → Top-3
3. "article 23" → Top-3
Combine: 9 chunks in total (no duplicates)
→ It guarantees they're all in the context ✅
Solution 2: Metadata filtering
Search with a filter:
metadata.article_number IN [12, 15, 23]
→ It returns only the chunks from those specific articles
Solution 3: Increase top-K
Current top-K: 5
New top-K: 20
→ A greater chance of including article 23
Solution 4: Hybrid search (keyword + semantic)
Keyword: "article 23" (literal)
Semantic: The related concept
Fusion → it guarantees that "23" appears exactly
Exercise summary
What you did:
- ✅ You diagnosed 5 common problems in RAG
- ✅ You identified the root causes (retrieval, latency, hallucinations, cost, context)
- ✅ You proposed specific solutions (hybrid search, caching, prompt engineering, etc.)
The intuition, consolidated:
"RAG problems almost always reduce to 5 categories: retrieval failure, latency, hallucinations, cost, or insufficient context. Diagnosing correctly is the first step; the solutions are known and applicable."
Module 7 conclusion
Congratulations on completing Module 7: RAG and Semantic Search. 🎉
What you achieved:
- ✅ You understand the problem RAG solves (hallucinations, going stale)
- ✅ You know RAG's architecture (vector DB, embeddings, LLM)
- ✅ You can analyze the complete flow (retrieval → augmentation → generation)
- ✅ You can compare RAG vs fine-tuning (when to use each)
- ✅ You can identify RAG's limitations (and how to mitigate them)
The module's key intuition:
RAG connects semantic search with LLMs to eliminate hallucinations and keep knowledge up to date. It's the practical application of everything you've learned: vectors, similarity, ANN, and ranking. RAG isn't perfect, but with careful design (hybrid search, reranking, prompt engineering), it's the most robust solution for production Q&A systems.
Next module: Module 8: Final Project — Designing a complete end-to-end RAG system.