Module 5: Hybrid Search — combining keyword + semantic for queries that need both

Module description

So far, all of your retrieval depends on embeddings and cosine similarity. It works very well for semantic queries — questions about concepts, paraphrased with words different from the document's, in different languages. But there's a class of queries where semantic search fails systematically: queries that require an exact match on specific tokens.

"how do I configure OAuth2PasswordBearer with custom scopes?" "error code: ERR_NETWORK_TIMEOUT_504" "difference between pd.merge and pd.concat"

These queries carry exact identifiers (class names, error codes, specific functions) that the user knows and wants to find literally. Embeddings paraphrase — OAuth2PasswordBearer and "OAuth2 dependency with username/password" end up close in vector space. For conceptual queries that's great. For queries with exact identifiers, it's the difference between finding the right document or one that merely "talks about the topic".

Hybrid search solves this by combining two methods in parallel: BM25 (classic keyword search, which prioritizes exact match) + semantic search (embeddings, which prioritize meaning), fusing the rankings with techniques like Reciprocal Rank Fusion (RRF). The result: precision +15-18% and recall +10-15% over semantic-only, especially on technical content.

This module teaches you to implement hybrid search from scratch, calibrate the balance between keyword and semantic with your specific dataset, and recognize when hybrid search is justified vs when pure semantic is enough.

By the end of this module you'll be able to:

• ✅ Identify the queries where semantic search fails and BM25 wins
• ✅ Implement BM25 over your corpus with rank_bm25 or Elasticsearch
• ✅ Combine BM25 + semantic rankings with Reciprocal Rank Fusion
• ✅ Tune the balance between the two methods with a weight α (weighted hybrid)
• ✅ Decide when hybrid search adds value vs when it's needless complexity
• ✅ Build a production-ready hybrid search engine with A/B testing metrics

Estimated module time: 2-3 hours (8 capsules).

---

Why hybrid search exists: the queries that break semantic search

Let's make the problem concrete. Three real queries, and what happens with each method:

Query 1 — purely semantic

"how do I handle user authentication?"

Method	Top result	Good?
Semantic (cosine)	"FastAPI authentication tutorial: OAuth2 flows, JWT tokens, password hashing..."	✅ Perfect match
BM25 (keyword)	"Handling authentication: basic principles..." (vaguer)	🟡 Decent match but generic

For semantic queries, semantic wins. The user wants concepts, and embeddings understand them well.

Query 2 — exact identifier

"how do I use OAuth2PasswordBearer?"

Method	Top result	Good?
Semantic (cosine)	"For username/password authentication in FastAPI, use the appropriate dependency..." (cosine 0.85)	❌ It paraphrases. Never mentions OAuth2PasswordBearer literally
BM25	"OAuth2PasswordBearer is the FastAPI security class for password flow. Import from..."	✅ Exact match on the identifier

For queries with identifiers, BM25 wins. Semantic search "normalized" OAuth2PasswordBearer into a similar concept but lost the specificity.

Query 3 — error code

"error: ERR_NETWORK_TIMEOUT_504"

Method	Top result	Good?
Semantic	"Network errors and how to handle timeouts in production..." (generic)	❌ No match on the specific code
BM25	"Troubleshooting ERR_NETWORK_TIMEOUT_504: this error indicates..."	✅ Finds exactly the right guide

For errors and codes, BM25 is essential. Embeddings treat them as "weird words" with no meaning; BM25 searches for them literally.

The insight: neither one always wins

                Semantic (cosine)    BM25 (keyword)
─────────────────────────────────────────────────────
Conceptual queries:        ✅ wins           🟡 OK
Paraphrased queries:       ✅ wins           ❌ fails
Cross-language queries:    ✅ wins           ❌ fails
Queries with identifiers:  ❌ loses          ✅ wins
Queries with codes:        ❌ fails          ✅ wins
Exact-match queries:       ❌ fails          ✅ wins

Hybrid search runs both in parallel and combines the rankings. When semantic wins, its results dominate. When BM25 wins, its results do. When both agree (the ideal case), they reinforce the ranking of the right document.

---

The technical stack of hybrid search

                    Query
                     │
        ┌────────────┴────────────┐
        │                         │
        ▼                         ▼
┌───────────────┐        ┌───────────────────┐
│ BM25          │        │ Semantic search   │
│ (keyword)     │        │ (cosine)          │
│               │        │                   │
│ rank_bm25 or  │        │ ChromaDB,         │
│ Elasticsearch │        │ Pinecone, etc.    │
│               │        │                   │
│ Top-K BM25    │        │ Top-K semantic    │
└───────┬───────┘        └─────────┬─────────┘
        │                          │
        └──────────────┬───────────┘
                       │
                       ▼
              ┌──────────────────┐
              │ Fusion (RRF / α) │
              │ Combines rankings│
              └────────┬─────────┘
                       │
                       ▼
              Final hybrid top-K

The components:

1. BM25 — the classic keyword search algorithm. A modern variant of TF-IDF, it considers term frequency and global rarity. Implementation: rank_bm25 in Python (in-memory, simple) or Elasticsearch (production scale).

2. Semantic search — the one you already have. Cosine similarity over embeddings.

3. Fusion — how to combine two rankings:

   • RRF (Reciprocal Rank Fusion): combines rankings while ignoring absolute scores. A reasonable default.
   • Weighted (α blending): weights both sets of scores with an α parameter. More control, requires calibration.

---

Module roadmap

Capsule	Topic	Why it matters	Time
01 (you're here)	Module introduction	Why hybrid search exists and an overview of the module	10-15 min
02	Limitations of semantic-only	When and why cosine similarity fails with exact keywords	25-30 min
03	BM25 keyword search	How BM25 works, implementation with rank_bm25	30-35 min
04	Reciprocal Rank Fusion (RRF)	Combining rankings without tuning weights	25-30 min
05	Weighted hybrid blending (α)	Tuning the keyword vs semantic balance	30-35 min
06	Elasticsearch integration	BM25 at scale with Elasticsearch	30-35 min
07	Strategy comparison	The decision framework: when to use each technique	25-30 min
08	Capstone project	A Hybrid Search Engine with A/B testing	45-60 min

Estimated total: 3-4 hours. It's a code-heavy module — it's worth doing in 2-3 sessions.

---

Expected improvement with hybrid search

On datasets with mixed queries (semantic + identifiers), typical benchmarks:

Strategy	Precision@5	Recall@50	Latency p95	When to use it
Semantic only	87%	72%	200ms	Purely conceptual queries (rare in production)
BM25 only	81%	65%	50ms	Purely exact-match queries (rare in production)
Hybrid (RRF)	94%	84%	240ms	The default for technical content
Hybrid + reranking	96%	86%	400ms	Critical cases

The key reading:

• Hybrid isn't the "average" of the two — it's better than either one alone.
• The extra latency is ~40ms (BM25 is very fast).
• Recall climbs noticeably — you recover docs that neither method found on its own.

---

When hybrid search IS the right call

Case	Hybrid?
Technical documentation with class/function names	✅ Yes
An error code / message search engine	✅ Yes
A product catalog with SKUs / codes	✅ Yes
Multilingual support with exact identifiers	✅ Yes
Code search (Stack Overflow, GitHub)	✅ Yes
Purely conceptual questions (humanities, philosophy)	❌ No, semantic is enough
A narrative corpus (literature, journalism)	❌ No, semantic is enough
An MVP where you haven't measured keyword problems yet	❌ No, add it later if you need it

A reasonable default: if your corpus has any technical content (code, errors, IDs, specific names), hybrid search is worth the extra complexity.

---

Connection with the previous and following modules

Relevant previous modules:
  ├─ Module 1: The complete RAG pipeline
  │   └─ Hybrid search slots in here, in the retrieval phase
  ├─ Module 2: Chunking strategies
  │   └─ Chunking affects how BM25 indexes keywords
  ├─ Module 3: Query optimization
  │   └─ Query expansion + hybrid search complement each other
  └─ Module 4: Re-ranking
      └─ Hybrid + re-ranking is the "state of the art" pattern

This module prepares you for:
  ├─ Module 6: Metadata filtering
  │   └─ Combining hybrid + metadata filters = advanced retrieval
  ├─ Module 7: Production with Pinecone
  │   └─ Pinecone supports hybrid search natively
  └─ Module 8: RAG evaluation
      └─ How to measure whether hybrid is improving things or just adding complexity

The "state of the art" production-ready RAG pattern (May 2026):

Query
  ↓
Query optimization (expansion, rewriting) [M03]
  ↓
Hybrid search (BM25 + semantic + RRF) [this module]
  ↓
Metadata filtering [M06]
  ↓
Re-ranking (cross-encoder or Cohere) [M04]
  ↓
LLM generation

Each component adds 5-15% of precision. Combined, the system goes from "working MVP" (75% precision) to "serious production" (94%+ precision).

---

Prerequisites before starting the module

Make sure you have:

• ✅ A working RAG pipeline with cosine retrieval (M01)
• ✅ Re-ranking implemented (M04) — hybrid + rerank is the complete pattern
• ✅ Your own eval set with at least 50 queries with ground truth, ideally with a category assigned
• ✅ ChromaDB or a vector DB with embeddings already generated
• ✅ Python 3.10+ with pip install rank-bm25

If your eval set doesn't distinguish "queries with identifiers" from "semantic queries", build that in first — without that segmentation you won't be able to measure whether BM25 is contributing where it should.

---

Technical setup for the module

# Capsules 03-05 (rank_bm25 in-memory)
pip install rank-bm25

# Capsule 06 (Elasticsearch, optional for scale)
pip install elasticsearch

# For a local Elasticsearch in Docker
# Create docker-compose.yml as shown in capsule 06

Environment variables:

# .env
OPENAI_API_KEY=sk-...
# If you use ES in capsule 06:
ES_HOST=http://localhost:9200

---

Self-check before moving on

Before starting capsule 02, make sure you can answer:

1. Why does semantic search fail with queries that carry exact identifiers like OAuth2PasswordBearer?
2. What does fusion (RRF or weighted) do in hybrid search?
3. When is it NOT worth adding hybrid search to an existing RAG system?

Answers

1. Embeddings normalize paraphrases. OAuth2PasswordBearer and "OAuth2 dependency with username/password" end up in nearby regions of vector space — the model "understands" they're the same thing. For conceptual queries that's great, but when the user knows the exact name and wants to find that specific identifier, the normalization works against them. BM25, which prioritizes exact token matches, literally finds the mention of the identifier.

2. Fusion combines two rankings (one from BM25, one from semantic) into a unified ranking. RRF does it while ignoring the absolute scores (because BM25 and semantic have different, non-comparable scales) — it only looks at each document's position in each ranking. Weighted blending does look at the scores, but it requires normalizing them and tuning an α parameter (the weight between the two methods).

3. When your corpus is purely conceptual (literature, philosophy, narrative) and exact identifiers aren't part of the domain. Adding hybrid there is needless complexity — semantic is enough. When you're at the MVP stage and you haven't yet measured concrete keyword problems. Adding components preemptively without data leads to over-engineered systems.

---

Next step: Capsule 02

The next capsule digs into the limitations of semantic-only. You'll see more cases where cosine similarity concretely fails, with benchmarks over real datasets. It's the full pedagogical justification for the module — without understanding why semantic fails in these cases, the technical capsules that follow sound like "adding complexity just because".

---

Resources

1. BM25 — The Probabilistic Relevance Framework (Robertson & Zaragoza) — The foundational paper on BM25
2. Reciprocal Rank Fusion (Cormack et al., 2009) — The original RRF paper
3. Pinecone — Hybrid Search Guide — An accessible overview
4. Elasticsearch — Hybrid Search — Official documentation
5. rank_bm25 Python Library — A simple in-memory implementation
6. Anthropic — Contextual Retrieval — A technique that complements hybrid search

---

Estimated time: 10-15 minutes Next: 02-limitations-of-semantic-only-search.md

---

Final notes

This module assumes you're going to integrate hybrid search into an existing RAG pipeline. If you're only just building your first RAG, consider starting with M01 (the basic pipeline) before adding hybrid — the extra complexity is only justified once the problem "queries with identifiers fail" is measurable in your product.

The final pattern you'll have at the close of the module is: semantic + BM25 → RRF → cross-encoder rerank → LLM. It's the "state of the art" architecture for production RAG in May 2026, validated by teams like Anthropic, OpenAI, Pinecone and the main RAG-as-a-service providers.