Module 1: The Complete RAG Pipeline (Architecture Overview)

RAG vs Traditional Search (Keywords)

Capsule overview

RAG isn't always the right answer. Sometimes traditional search (keyword search, SQL queries, grep) is simpler, faster and cheaper. This capsule teaches you when to use RAG and when not to, with concrete trade-off examples.

Understanding when NOT to use RAG matters just as much as knowing how to use it well. If your dataset has 50 structured documents with exact IDs, SQL search is 10x faster and 100x cheaper than RAG. If your query is "find transaction_id=12345", keyword search answers in 5ms while RAG takes 500ms.

This capsule gives you: (1) a technical comparison of RAG vs keywords, (2) decision criteria based on your dataset and query type, (3) the cases where RAG wins, (4) the cases where keywords win, (5) the hybrid cases where you combine both.

🔍 Traditional search (keywords)

How does keyword search work?

It looks for exact or partial matches of words in text.

# Example 1: SQL search (structured data)
query = "SELECT * FROM docs WHERE title LIKE '%FastAPI%'"

# Example 2: grep (unstructured text)
query = "grep -i 'fastapi' docs/*.txt"

# Example 3: BM25 (term frequency ranking)
from rank_bm25 import BM25Okapi

corpus = ["FastAPI is a web framework", "Django is a web framework", "Flask is lightweight"]
tokenized_corpus = [doc.split() for doc in corpus]

bm25 = BM25Okapi(tokenized_corpus)
query = "fastapi framework"
scores = bm25.get_scores(query.split())

# Output: [high_score, low_score, low_score]

The characteristics of keyword search:

Aspect	Description	Example
Match type	Exact or partial match of terms	"FastAPI" → literal match
Context-aware	❌ Doesn't understand synonyms or context	"framework" doesn't match "library"
Latency	✅ Very fast (5-50ms)	Indexed search
Cost	✅ Almost zero (local compute)	No API calls
Setup	✅ Simple (SQL, grep, BM25)	pip install rank-bm25

🤖 Semantic search (RAG)

How does semantic search work?

It converts text into embeddings (vectors) and searches by similarity.

from openai import OpenAI
import chromadb

# 1. Indexing: convert the docs into embeddings
client = OpenAI()
docs = ["FastAPI is a web framework", "Django is a web framework"]

embeddings = [
    client.embeddings.create(model="text-embedding-ada-002", input=doc).data[0].embedding
    for doc in docs
]

# 2. Storage: store them in a vector DB
chroma_client = chromadb.Client()
collection = chroma_client.create_collection("docs")
collection.add(documents=docs, embeddings=embeddings, ids=["1", "2"])

# 3. Retrieval: semantic query
query = "python web framework"
query_embedding = client.embeddings.create(model="text-embedding-ada-002", input=query).data[0].embedding

results = collection.query(query_embeddings=[query_embedding], n_results=2)

# Output: matches "FastAPI" and "Django" even though the query says "python" (which isn't in the docs)

The characteristics of semantic search:

Aspect	Description	Example
Match type	Semantic similarity (context-aware)	"framework" matches "library"
Context-aware	✅ Understands synonyms and concepts	"python web" matches "FastAPI"
Latency	⚠️ Slower (200-500ms)	Embedding + vector search
Cost	⚠️ API calls ($0.0001/1K tokens)	OpenAI embeddings
Setup	⚠️ More complex (embeddings + vector DB)	ChromaDB + OpenAI

⚖️ Head to head

Example: searching for "FastAPI tutorial"

Keyword search (BM25):

from rank_bm25 import BM25Okapi

corpus = [
    "FastAPI is a modern web framework",
    "Django tutorial for beginners",
    "Building APIs with FastAPI",
    "Python web development guide"
]

tokenized_corpus = [doc.split() for doc in corpus]
bm25 = BM25Okapi(tokenized_corpus)

query = "FastAPI tutorial"
scores = bm25.get_scores(query.split())

# Ranking:
# 1. "Building APIs with FastAPI" (0.85) ← Contains "FastAPI"
# 2. "FastAPI is a modern web framework" (0.72) ← Contains "FastAPI"
# 3. "Django tutorial for beginners" (0.42) ← Contains "tutorial"
# 4. "Python web development guide" (0.10) ← No direct match

The result: the top-2 are relevant because they contain "FastAPI". But "Django tutorial" ranks third purely because of "tutorial", even though it isn't about FastAPI.

Semantic search (embeddings):

from openai import OpenAI
import chromadb

client = OpenAI()
chroma_client = chromadb.Client()
collection = chroma_client.create_collection("docs")

# Index the corpus
embeddings = [
    client.embeddings.create(model="text-embedding-ada-002", input=doc).data[0].embedding
    for doc in corpus
]

collection.add(documents=corpus, embeddings=embeddings, ids=["1", "2", "3", "4"])

# Query
query = "FastAPI tutorial"
query_embedding = client.embeddings.create(model="text-embedding-ada-002", input=query).data[0].embedding

results = collection.query(query_embeddings=[query_embedding], n_results=4)

# Ranking (cosine similarity):
# 1. "Building APIs with FastAPI" (0.92) ← Topic + tool both match
# 2. "FastAPI is a modern web framework" (0.89) ← Tool matches
# 3. "Python web development guide" (0.78) ← Related topic (web dev)
# 4. "Django tutorial for beginners" (0.75) ← Similar topic but a different tool

The result: the top-2 are relevant (FastAPI). "Python web development" ranks third because it's conceptually related, even though it never says "FastAPI" literally.

The key difference:

Keywords: literal match on "FastAPI" → misses "Python web development" (which is relevant)
Semantic: conceptual match → finds "Python web development" (relevant by context)

🎯 When to use keywords

Case 1: Exact-match queries

Example: looking up a transaction ID, a user ID, an error code.

# Query: "Find transaction_id=12345"

# Keywords (SQL): ✅ Perfect
query = "SELECT * FROM transactions WHERE id = 12345"
# Latency: 5ms
# Cost: Zero

# Semantic (RAG): ❌ Overkill
query_embedding = create_embedding("transaction 12345")
results = collection.query(query_embeddings=[query_embedding])
# Latency: 500ms
# Cost: $0.0001
# The problem: the embedding can confuse "12345" with "12346" (semantically similar)

The decision: keywords win. Exact match is faster and more precise.

Case 2: Structured data

Example: searching a database with a defined schema.

# Dataset: users with fields (name, email, age, city)

# Query: "Find users in San Francisco older than 30"

# Keywords (SQL): ✅ Perfect
query = "SELECT * FROM users WHERE city = 'San Francisco' AND age > 30"
# Latency: 10ms
# Precision: 100%

# Semantic (RAG): ❌ Doesn't work well
# The problem: embeddings don't understand ranges (">30") or structured filters

The decision: keywords win. Structured data is SQL's home turf.

Case 3: Small dataset (<100 docs)

Example: searching 20 internal policy documents.

# Dataset: 20 documents, 1-2 pages each

# Keywords (grep): ✅ Enough
grep -i "vacation policy" docs/*.txt
# Latency: 50ms
# Cost: Zero

# Semantic (RAG): ❌ Overkill
# Setup: create the embeddings (20 API calls), a vector DB, etc.
# Latency: 500ms
# Cost: $0.002
# Benefit: minimal (small dataset, keywords are enough)

The decision: keywords win. Setting up RAG isn't worth it for 20 docs.

Case 4: Code search (IDs, function names)

Example: finding a function definition in a codebase.

# Query: "Find function calculate_total"

# Keywords (grep): ✅ Perfect
grep -r "def calculate_total" src/
# Latency: 100ms
# Precision: 100%

# Semantic (RAG): ❌ It gets confused
query_embedding = create_embedding("calculate total")
# The problem: it can match "compute_sum", "get_total", etc. (semantically similar)
# But we want EXACTLY "calculate_total"

The decision: keywords win. Function names need an exact match.

🤖 When to use RAG

Case 1: Conceptual queries

Example: "How does authentication work in FastAPI?"

# Dataset: the FastAPI documentation (500 pages)

# Keywords (grep): ❌ Misses the context
grep -i "authentication" docs/*.txt
# The problem: it returns 100+ literal "authentication" matches
# But it doesn't rank them by conceptual relevance

# Semantic (RAG): ✅ Perfect
query = "How does authentication work in FastAPI"
query_embedding = create_embedding(query)
results = collection.query(query_embeddings=[query_embedding], n_results=5)

# Output: the top-5 documents about auth in FastAPI, ranked by semantic relevance
# It finds "OAuth2", "JWT", "security" even though the query never mentions those terms

The decision: RAG wins. It understands related concepts.

Case 2: Synonyms and paraphrasing

Example: "python web library" should match "FastAPI framework".

# Keywords: ❌ No match
corpus = ["FastAPI is a web framework"]
query = "python web library"
# BM25 score: 0.0 (no term overlap)

# Semantic: ✅ Match
query_embedding = create_embedding("python web library")
doc_embedding = create_embedding("FastAPI is a web framework")
similarity = cosine_similarity(query_embedding, doc_embedding)
# Similarity: 0.85 (a strong contextual match)

The decision: RAG wins. It understands that "library" ≈ "framework", "python web" ≈ "FastAPI".

Case 3: A large unstructured dataset (>10,000 docs)

Example: searching 100,000 Wikipedia documents.

# Keywords (grep): ❌ Doesn't scale well
grep -i "machine learning" wikipedia/*.txt
# Latency: 30 seconds (it scans 100K files)
# Output: 10,000+ matches with no ranking

# Semantic (RAG): ✅ Scales + ranks
results = collection.query(query_embeddings=[query_embedding], n_results=10)
# Latency: 200ms (vector search is fast)
# Output: the top-10 most relevant documents (ranked)

The decision: RAG wins. Vector search scales better than grep for large datasets.

Case 4: Question answering

Example: "What is the capital of France?"

# Keywords: ❌ Doesn't answer the question
grep -i "capital france" docs/*.txt
# Output: text fragments containing "capital" and "france"
# The user has to read them and find the answer themselves

# Semantic RAG: ✅ Answers directly
query = "What is the capital of France?"
docs = retrieve_relevant_docs(query)  # Retrieval
answer = llm.invoke(context=docs, query=query)  # Generation
# Output: "The capital of France is Paris."

The decision: RAG wins. The LLM generates a direct answer instead of just retrieving.

🔀 Hybrid cases (keywords + semantic)

Case: searching technical documentation

The problem: queries contain exact names (API endpoints, function names) but also concepts.

Example: "How to use FastAPI OAuth2PasswordBearer class?"

Approach 1: Keywords only (BM25)

# BM25 searches for exact terms
query = "FastAPI OAuth2PasswordBearer class"

# Pros:
# - Exact match on "OAuth2PasswordBearer" (the class name)

# Cons:
# - Misses documents that explain OAuth2 without mentioning "OAuth2PasswordBearer"
# - Doesn't understand that "how to use" implies looking for examples/tutorials

Approach 2: Semantic only (embeddings)

# Embeddings search by context
query_embedding = create_embedding("How to use FastAPI OAuth2PasswordBearer class")

# Pros:
# - Finds OAuth2 examples even if they never say "OAuth2PasswordBearer" exactly
# - Understands "how to use" → prioritizes tutorials

# Cons:
# - It can confuse "OAuth2PasswordBearer" with "OAuth2AuthorizationCodeBearer" (similar)
# - Misses the exact match when the user is looking for a specific class

Approach 3: Hybrid (BM25 + embeddings) ✅

from rank_bm25 import BM25Okapi

# 1. BM25 keyword search (top-50)
bm25_scores = bm25.get_scores(query.split())
bm25_top_50 = get_top_k(bm25_scores, k=50)

# 2. Semantic search (top-50)
semantic_results = collection.query(query_embeddings=[query_embedding], n_results=50)
semantic_top_50 = semantic_results['ids'][0]

# 3. Reciprocal Rank Fusion (merge)
def reciprocal_rank_fusion(bm25_results, semantic_results, k=60):
    """Merges BM25 + Semantic with RRF"""
    
    scores = {}
    
    # BM25 scores
    for rank, doc_id in enumerate(bm25_results, 1):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    
    # Semantic scores
    for rank, doc_id in enumerate(semantic_results, 1):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    
    # Sort by combined score
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

merged = reciprocal_rank_fusion(bm25_top_50, semantic_top_50)
top_10 = [doc_id for doc_id, score in merged[:10]]

The hybrid result:

✅ Exact match on "OAuth2PasswordBearer" (from BM25)
✅ Finds related examples even when they don't name the exact class (from semantic)
✅ Better recall and precision than either one alone

Performance:

Approach	Precision@10	Recall@50	Latency
BM25 only	75%	60%	50ms
Semantic only	78%	70%	200ms
Hybrid (RRF)	85%	80%	250ms

The decision: hybrid wins. +10% precision, +20% recall, for only +50ms of latency.

Technique: Module 5 teaches hybrid search in detail.

📊 Decision matrix: keywords or RAG?

The decision factors:

def decide_search_method(
    dataset_size: int,
    data_type: str,  # "structured" or "unstructured"
    query_type: str,  # "exact_match", "conceptual", "mixed"
    budget: float,
    latency_target: int  # ms
):
    """
    A decision tree for selecting the search method.
    """
    
    # Factor 1: Data type
    if data_type == "structured":
        return "keywords_sql"  # SQL always wins on structured data
    
    # Factor 2: Query type
    if query_type == "exact_match":
        return "keywords_bm25"  # Exact match → keywords
    
    # Factor 3: Dataset size
    if dataset_size < 100:
        return "keywords_grep"  # Small dataset → grep is enough
    
    # Factor 4: Budget
    if budget == 0:
        if latency_target < 500:
            return "keywords_bm25"  # Free + fast
        else:
            return "hybrid"  # Local embeddings + BM25
    
    # Factor 5: The default for conceptual queries + a large dataset
    if query_type == "conceptual":
        if latency_target < 300:
            return "hybrid"  # Balance speed/quality
        else:
            return "rag_semantic"  # Maximum quality
    
    # Factor 6: Mixed queries (technical docs)
    if query_type == "mixed":
        return "hybrid"  # Always hybrid for technical content
    
    return "rag_semantic"  # Default

# Examples
print(decide_search_method(50, "unstructured", "conceptual", 0, 1000))
# Output: "keywords_grep"

print(decide_search_method(100_000, "unstructured", "mixed", 500, 500))
# Output: "hybrid"

print(decide_search_method(10_000, "unstructured", "conceptual", 500, 2000))
# Output: "rag_semantic"

📋 The final comparison table

Aspect	Keywords (BM25)	Semantic (RAG)	Hybrid (BM25 + Semantic)
Exact match	✅ Excellent	❌ Can get confused	✅ Excellent
Conceptual queries	❌ Misses synonyms	✅ Excellent	✅ Excellent
Latency	✅ 50ms	⚠️ 200-500ms	⚠️ 250ms
Cost	✅ Zero	⚠️ $0.0001/query	⚠️ $0.0001/query
Setup	✅ Simple	⚠️ Complex	⚠️ Complex
Small dataset (<100)	✅ Enough	❌ Overkill	❌ Overkill
Large dataset (>10K)	⚠️ Slow without an index	✅ Scales well	✅ Scales well
Structured data	✅ Perfect (SQL)	❌ Doesn't work	❌ Doesn't work
Technical docs	⚠️ Misses context	⚠️ Misses exact matches	✅ Best of both

🎯 Summary

Key concepts:

✅ Keywords (BM25): fast, free, exact match → perfect for IDs, structured data, small datasets
✅ Semantic (RAG): context-aware, synonyms, conceptual → perfect for Q&A, large datasets, unstructured content
✅ Hybrid (BM25 + semantic): best of both → perfect for technical docs, mixed queries
✅ The decision factors: dataset size, data type, query type, budget, latency target
✅ Where keywords win: exact match, structured data, small datasets, code search
✅ Where RAG wins: conceptual queries, synonyms, large datasets, question answering
✅ Where hybrid wins: technical docs, API reference, mixed queries

The typical decision:

if query_has_exact_ids or data_is_structured:
    use_keywords()  # SQL, BM25, grep
elif dataset_size < 100:
    use_keywords()  # Grep is enough
elif query_is_conceptual and budget_available:
    use_rag_semantic()  # Embeddings
elif query_is_mixed or technical_docs:
    use_hybrid()  # BM25 + Semantic (Module 5)
else:
    use_keywords()  # Simple default

What's next:

Capsule 07 shows you the complete architecture of modules 2-8: how chunking (M2), query optimization (M3), re-ranking (M4), hybrid search (M5), metadata filtering (M6), vector DBs (M7) and evaluation (M8) connect to build production-ready RAG.

📚 Additional resources

BM25 Explained - The BM25 algorithm in detail
Keyword vs Semantic Search - A technical comparison
Hybrid Search Best Practices - Elasticsearch hybrid search
Reciprocal Rank Fusion - The original RRF paper
When to Use SQL vs Vector DB - A decision guide
RAG vs Traditional Search - A LlamaIndex comparison

Created: February 6, 2026
Version: 1.0

← PreviousReal-World RAG Use Cases in Production Next →Architecture Overview: Modules 2-8