Module 3: Essential Features for RAG

Capsule 03: Hybrid Search (Keyword + Semantic)

Understand WHY pure semantic search fails on certain queries, how hybrid search (BM25 + vector) improves accuracy 15-20%, and when to use each strategy.

By the end of this capsule:

✅ You'll explain the limitations of pure semantic search
✅ You'll understand how hybrid search combines keyword + semantic
✅ You'll compare ranking strategies (RRF, weighted fusion)
✅ You'll decide when to use hybrid vs pure semantic

Estimated time: 10-12 minutes

🔍 Problem: Pure Semantic Search fails on specific queries

What is semantic search?

Semantic search = Searching by meaning (embeddings) instead of exact words.

Example:

Query: "automobile repair"
Embedding: [0.2, 0.5, 0.8, ...]

Top results (semantic):
1. "Car maintenance guide" ✅ (similar meaning)
2. "Vehicle troubleshooting" ✅ (similar meaning)
3. "How to fix your vehicle" ✅ (similar meaning)

Advantage: Finds conceptually similar documents (not just exact keyword matches).

But... semantic search fails on specific queries

Problem 1: Proper nouns

Query: "GPT-4 API documentation"
Embedding: [0.1, 0.7, 0.3, ...]

Pure semantic results:
1. "OpenAI models overview" ⚠️ (mentions GPT-3, Claude)
2. "API integration guide" ⚠️ (general, no GPT-4)
3. "Language model comparison" ⚠️ (GPT-3 vs PaLM)

Expected:
1. "GPT-4 API reference" ← Missing because "GPT-4" didn't dominate the embedding

Why it fails: "GPT-4" is a proper noun (a specific token), but the embedding captures the general concept "language model API".

Problem 2: IDs, codes, exact numbers

Query: "Invoice #INV-2024-001234"

Pure semantic results:
1. "Invoice management guide" ❌
2. "Billing documentation" ❌
3. "Payment processing" ❌

Expected:
1. Invoice #INV-2024-001234 ← Missing because the embedding doesn't capture the exact ID

Why it fails: Embeddings aren't designed to capture exact strings (they're semantic representations).

Problem 3: Queries with typos or exact variants

Query: "PostgreSQL"

Pure semantic results:
1. "Database management" ✅ (general)
2. "SQL tutorial" ✅ (general)
3. "MySQL guide" ⚠️ (a different DB!)

Missing:
- "PostgreSQL installation" ← Should be #1

Why it fails: The embeddings of "PostgreSQL" and "MySQL" are similar (both are SQL databases).

🔀 Solution: Hybrid Search (BM25 + Vector)

What is hybrid search?

Hybrid search = Combining keyword search (BM25) + semantic search (vector) to get the best of both worlds.

Keyword search (BM25):

Finds exact keyword matches
Excellent for proper nouns, IDs, codes
Terrible for synonyms, concepts

Semantic search (Vector):

Finds similar concepts
Excellent for conceptual queries
Terrible for exact matches

Hybrid = Keyword + Semantic:

Combines results from both
Ranks using a fusion algorithm (RRF, weighted)

Hybrid Search architecture

User Query: "GPT-4 API documentation"
        │
        ├─────────────────┬─────────────────┐
        ↓                 ↓                 ↓
  [Keyword Search]  [Semantic Search]
    (BM25)            (Vector HNSW)
        │                 │
  1. GPT-4 docs      1. OpenAI API guide
  2. API reference   2. LLM integration
  3. OpenAI guide    3. GPT-3 docs
        │                 │
        └─────────────────┴─────────────────┐
                          ↓
                   [Fusion Algorithm]
                   (RRF or Weighted)
                          ↓
                   Merged Top-10:
                   1. GPT-4 API docs ✅
                   2. OpenAI API guide ✅
                   3. GPT-4 reference ✅

Key: Keyword captures the exact "GPT-4", Semantic captures the concept "API documentation".

📊 Benchmark: Pure Semantic vs Hybrid

Scenario: Technical Documentation RAG

Setup:

Database: 100K technical docs
Queries: 1000 test queries (mix of conceptual + specific)
Metric: MRR (Mean Reciprocal Rank) + NDCG@10

Test A: Pure Semantic Search

results = db.query(
    query_embedding=embed(query),
    k=10
)

Results:

Conceptual queries: 92% accuracy ✅
Specific queries (names, IDs): 68% accuracy ❌
Overall: 78% accuracy

Typical failures:

"GPT-4 docs" → Returns GPT-3 docs
"React 18 features" → Returns general React
"Invoice #12345" → Not found

Test B: Pure Keyword Search (BM25)

results = bm25_search(query, k=10)

Results:

Specific queries: 95% accuracy ✅
Conceptual queries: 55% accuracy ❌
Overall: 72% accuracy

Typical failures:

"automobile repair" → Doesn't find "car maintenance" (synonym)
"password reset" → Doesn't find "credential recovery" (similar concept)

Test C: Hybrid Search (BM25 + Vector)

# Keyword results
keyword_results = bm25_search(query, k=20)

# Semantic results
semantic_results = db.query(
    query_embedding=embed(query),
    k=20
)

# Fusion (RRF)
final_results = reciprocal_rank_fusion(
    [keyword_results, semantic_results],
    k=10
)

Results:

Conceptual queries: 91% accuracy ✅ (almost equal to pure semantic)
Specific queries: 94% accuracy ✅ (almost equal to pure keyword)
Overall: 92% accuracy ✅

Gain: 14% improvement vs pure semantic, 20% vs pure keyword.

🔧 Fusion Algorithms

1. Reciprocal Rank Fusion (RRF)

Definition: Assign a score based on position (rank) in each list.

Formula:

RRF_score(doc) = Σ (1 / (k + rank_i))

where:
- k = constant (typically 60)
- rank_i = position of the doc in list i (1-indexed)

Example:

Keyword results:
1. Doc A (rank=1)
2. Doc B (rank=2)
3. Doc C (rank=3)

Semantic results:
1. Doc B (rank=1)
2. Doc D (rank=2)
3. Doc A (rank=3)

RRF scores:
Doc A: 1/(60+1) + 1/(60+3) = 0.0164 + 0.0159 = 0.0323
Doc B: 1/(60+2) + 1/(60+1) = 0.0161 + 0.0164 = 0.0325 ← Highest
Doc C: 1/(60+3) + 0 = 0.0159
Doc D: 0 + 1/(60+2) = 0.0161

Final ranking: B, A, D, C

Advantages:

✅ No need to normalize scores (only ranks)
✅ Robust to outliers (one very high score doesn't dominate)
✅ Simple to implement

Disadvantages:

❌ Doesn't consider the magnitude of scores (only position)
❌ Can penalize docs with a high score but low rank

Used by: Weaviate (default)

2. Weighted Score Fusion

Definition: Combine normalized scores with weights.

Formula:

Final_score(doc) = α × keyword_score + (1-α) × semantic_score

where:
- α = weight for keyword (typically 0.3-0.7)
- keyword_score, semantic_score = normalized [0,1]

Example:

Keyword results:
Doc A: score=0.9
Doc B: score=0.7
Doc C: score=0.5

Semantic results:
Doc A: score=0.6
Doc B: score=0.9
Doc D: score=0.8

Weighted (α=0.5):
Doc A: 0.5×0.9 + 0.5×0.6 = 0.75
Doc B: 0.5×0.7 + 0.5×0.9 = 0.80 ← Highest
Doc D: 0.5×0 + 0.5×0.8 = 0.40
Doc C: 0.5×0.5 + 0.5×0 = 0.25

Final ranking: B, A, D, C

Advantages:

✅ Considers the magnitude of scores
✅ Flexible (adjust α based on query type)
✅ Intuitive

Disadvantages:

❌ Requires normalizing scores (BM25 and cosine similarity have different ranges)
❌ Sensitive to outliers

Used by: Pinecone (custom), Elasticsearch

RRF vs Weighted: Which to choose?

Criterion	RRF	Weighted
Simplicity	✅ Simple	⚠️ Requires normalization
Robustness	✅ Robust to outliers	❌ Sensitive
Flexibility	❌ Fixed (not adjustable)	✅ Adjustable (α)
Performance	✅ Similar	✅ Similar
Recommended	Default (Weaviate)	Custom tuning (Pinecone)

Recommendation: Start with RRF (simple, robust). Move to weighted if you need fine control.

🎯 When to use Hybrid vs Pure Semantic

Decision Tree

┌─────────────────────────────────────────┐
│ Does your query contain proper nouns,   │
│ IDs, codes, or very specific terms?     │
│                                          │
└─────────────────────────────────────────┘
                  │
       ┌──────────┴──────────┐
       │                     │
      Yes                    No
       │                     │
       ↓                     ↓
┌──────────────┐      ┌─────────────────┐
│ Hybrid Search│      │ Pure Semantic   │
│ (BM25 + Vec) │      │ (Vector only)   │
└──────────────┘      └─────────────────┘

Examples:            Examples:
- "GPT-4 docs"       - "car repair"
- "Invoice #123"     - "fix bug"
- "React 18"         - "improve performance"
- "user_id: abc"     - "reduce latency"

Use cases: Hybrid Search

1. Technical Documentation

Query: "Kubernetes 1.28 ingress"
→ Hybrid captures exact "Kubernetes 1.28" + the concept "ingress"

2. E-commerce Product Search

Query: "Nike Air Max red size 10"
→ Hybrid captures exact "Nike Air Max" + the concept "red shoes"

3. Legal Document Search

Query: "Case #2024-CV-12345"
→ Hybrid captures the exact case number

4. Code Search

Query: "def authenticate_user()"
→ Hybrid captures the exact function name

Use cases: Pure Semantic

1. Conceptual Queries

Query: "How to improve database performance?"
→ Semantic captures the concept (optimization, indexes, caching)

2. Ambiguous Queries

Query: "fix connection issues"
→ Semantic captures multiple interpretations (network, DB, API)

3. Synonyms Expected

Query: "automobile maintenance"
→ Semantic finds "car repair", "vehicle service"

🏭 Hybrid Search in Vector Databases

Weaviate (native Hybrid Search)

# Weaviate supports hybrid search out-of-the-box
results = client.query.get("Document", ["content"])\
    .with_hybrid(
        query="GPT-4 API documentation",
        alpha=0.5  # 0=pure keyword, 1=pure vector
    )\
    .with_limit(10)\
    .do()

# Fusion: RRF (default)

Advantage: Built-in, no custom code.

Pinecone + Keyword Search (Custom)

# Pinecone doesn't have native hybrid
# Option 1: Combine with Elasticsearch

# 1. Keyword search in Elasticsearch
keyword_results = es.search(
    index="docs",
    body={"query": {"match": {"content": query}}}
)

# 2. Semantic search in Pinecone
semantic_results = pinecone_index.query(
    vector=embed(query),
    top_k=20
)

# 3. Manual fusion
final_results = weighted_fusion(keyword_results, semantic_results, alpha=0.5)

Disadvantage: Requires 2 systems (Pinecone + ES).

ChromaDB + BM25 (Custom implementation)

# ChromaDB doesn't have native hybrid
# Implement BM25 manually

from rank_bm25 import BM25Okapi

# 1. Index documents with BM25
corpus = [doc['content'] for doc in documents]
tokenized_corpus = [doc.split() for doc in corpus]
bm25 = BM25Okapi(tokenized_corpus)

# 2. Keyword search
tokenized_query = query.split()
keyword_scores = bm25.get_scores(tokenized_query)

# 3. Semantic search
semantic_results = collection.query(
    query_embeddings=[embed(query)],
    n_results=20
)

# 4. Fusion (RRF)
final_results = rrf_fusion(keyword_scores, semantic_results)

Disadvantage: Requires a custom implementation.

📊 Impact on Accuracy: A real example

Scenario: API Documentation RAG

Dataset: 50K API documentation pages (OpenAI, Anthropic, AWS, Azure)

Test queries (100):

50 conceptual: "How to handle rate limits?"
50 specific: "GPT-4 turbo pricing"

Pure Semantic

Conceptual queries: 94% accuracy
Specific queries: 72% accuracy ← Fails on "GPT-4 turbo"
Overall: 83%

Hybrid (α=0.5)

Conceptual queries: 92% accuracy (similar)
Specific queries: 96% accuracy ← Improves 24%
Overall: 94%

Gain: 11% overall, 24% on specific queries.

✅ Comprehension checklist

Verify that you understood this capsule:

Why does pure semantic search fail on "GPT-4 API docs"?
- Answer: The embedding captures the general concept "API documentation" but doesn't prioritize the exact name "GPT-4". Keyword search captures the exact "GPT-4".
What is hybrid search?
- Answer: Combining keyword search (BM25) + semantic search (vector) using a fusion algorithm (RRF or weighted).
What is RRF?
- Answer: Reciprocal Rank Fusion. It assigns a score based on position (rank) in each list. Formula: 1/(60+rank).
When to use hybrid vs pure semantic?
- Answer: Hybrid if the query has proper nouns, IDs, codes. Pure semantic if the query is conceptual/ambiguous.
What's the typical accuracy improvement of hybrid?
- Answer: 15-20% overall, 20-30% on specific queries.

If you answered 4-5/5 correctly → ✅ Ready for Capsule 04 (Multi-tenancy)

🔗 Connection with RAG

How does hybrid search improve your RAG?

Case A: Technical Docs Chatbot

Pure semantic:

User: "Show me Kubernetes 1.28 networking changes"
RAG: Returns general Kubernetes networking (mix of versions)
LLM: Response contaminated with info from old versions

Hybrid:

User: "Show me Kubernetes 1.28 networking changes"
RAG: Returns docs specific to v1.28 (keyword match) + networking concepts
LLM: Precise and up-to-date response

Case B: E-commerce Search

Pure semantic:

User: "Nike Air Jordan 1 Retro High OG"
RAG: Returns general Nike shoes (Jordan 1, 3, 4...)

Hybrid:

User: "Nike Air Jordan 1 Retro High OG"
RAG: Returns exactly "Jordan 1 Retro High OG" (exact keyword match)

🚀 Next step

You now know the 2 most critical features: Metadata filtering and Hybrid search. Now you'll learn feature #3 for SaaS: Multi-tenancy.

Next capsule: 04 - Multi-tenancy (Data isolation)

You'll learn:

Multi-tenancy strategies (collection per tenant, metadata filtering, namespace)
Security considerations (data leakage prevention)
Performance trade-offs (horizontal vs vertical scale)
When it's critical (SaaS, enterprise RAG)

Key: Multi-tenancy is essential for SaaS RAG (isolating data per customer).

Reading time: 10-12 minutes
Next: 04-multi-tenancy.md

← PreviousCapsule 02: Metadata Filtering (Where Clauses)Next →Capsule 04: Multi-tenancy (Data isolation)

Capsule 03: Hybrid Search (Keyword + Semantic)

🎯 Capsule objective

🔍 Problem: Pure Semantic Search fails on specific queries

What is semantic search?

But... semantic search fails on specific queries

🔀 Solution: Hybrid Search (BM25 + Vector)

What is hybrid search?

Hybrid Search architecture

📊 Benchmark: Pure Semantic vs Hybrid

Scenario: Technical Documentation RAG

Test A: Pure Semantic Search

Test B: Pure Keyword Search (BM25)

Test C: Hybrid Search (BM25 + Vector)

🔧 Fusion Algorithms

1. Reciprocal Rank Fusion (RRF)

2. Weighted Score Fusion

RRF vs Weighted: Which to choose?

🎯 When to use Hybrid vs Pure Semantic

Decision Tree

Use cases: Hybrid Search

Use cases: Pure Semantic

🏭 Hybrid Search in Vector Databases

Weaviate (native Hybrid Search)

Pinecone + Keyword Search (Custom)

ChromaDB + BM25 (Custom implementation)

📊 Impact on Accuracy: A real example

Scenario: API Documentation RAG

Pure Semantic

Hybrid (α=0.5)

✅ Comprehension checklist

🔗 Connection with RAG

Case A: Technical Docs Chatbot

Case B: E-commerce Search

🚀 Next step