Module 5: Hybrid Search — combining keyword + semantic for queries that need both
Capsule 02: Why semantic search alone isn't enough — the concrete failure modes
Capsule description
Semantic search with embeddings changed the retrieval game. Capturing meaning instead of exact words made it possible for a user typing "how do I handle authentication" to find documents about "implementing OAuth2 in FastAPI". That abstraction is powerful.
But that same abstraction breaks on a specific class of queries: the ones that depend on exact tokens. When a developer searches for "OAuth2PasswordBearer" (the name of a specific class), or a technician searches for "ERR_CONNECTION_RESET" (a literal error code), or a user searches for "v2.3.1" (a specific version), semantic search normalizes the paraphrase and loses the exact match. The system returns docs about the general topic, but rarely the specific ones the user knows are there.
This capsule shows you the concrete failure modes of semantic-only, how to diagnose them in your corpus, and why the natural solution is to add BM25 (keyword search) in parallel — not to replace semantic, but to complement it.
By the end of this capsule you'll be able to:
- ✅ Identify the five classes of queries where semantic search consistently fails
- ✅ Diagnose what percentage of your traffic is problematic with an analysis script
- ✅ Explain geometrically why embeddings normalize paraphrases
- ✅ Quantify the typical impact on recall (10-30% depending on the domain)
- ✅ Anticipate the "embed better" mistake as a false solution
- ✅ Justify the decision to add BM25 to the pipeline with data
Estimated time: 25-30 minutes
The insight: embeddings encode meaning, not exactness
When an embedding model processes text, it produces a vector that encodes the semantic meaning of the words. That means OAuth2PasswordBearer and "OAuth2 dependency with username/password" end up in nearby regions of vector space — the model "understands" they're conceptually the same thing.
For conceptual queries like "how do I handle authentication?", that normalization is exactly what you want. The user doesn't know the class name; they know what they want to accomplish, and embeddings connect them with the right document.
But there are queries where the user does know the exact token and wants to find it literally. Three examples:
User query: "OAuth2PasswordBearer scopes"
(knows the exact class name)
Doc in the corpus: Doc A: "OAuth2PasswordBearer is the FastAPI security class..."
cosine: 0.78 ← lower, paradoxically
Doc B: "For OAuth2 authentication with scopes in FastAPI, use
the appropriate dependency from the security module..."
cosine: 0.86 ← higher, but it does NOT mention the class
Retrieval top-1: Doc B (a close paraphrase)
What the user wanted: Doc A (the literal mention)
The system "works" — it returns docs on the topic. But it does not return the specific doc the user knew was there. For technical support, API documentation, debugging — that's the core use case, and semantic search loses it systematically.
The five classes of queries that break semantic-only
Class 1: code identifiers
"OAuth2PasswordBearer"
"RunnablePassthrough"
"AsyncIO event loop"
"useState hook"
"pd.merge"
Names of classes, functions, methods. The user knows exactly what to look for and wants the documentation for that specific symbol.
Why they fail: embeddings encode these identifiers along with the surrounding technical context. A doc that mentions the identifier 3 times can rank lower than a doc that paraphrases the concept in a longer text.
Class 2: error codes
"ERR_CONNECTION_RESET"
"E4502"
"HTTP 503"
"ENOENT"
"OSError [Errno 2]"
Errors the system produces. The user copies them literally from the error log.
Why they fail: embeddings treat these codes as rare tokens. A troubleshooting doc with the exact code can have a lower cosine similarity than a generic doc about "network errors".
Class 3: versions and specific numbers
"FastAPI 0.110.0 changes"
"Python 3.12 typing"
"Django 4.2 migration"
"Node 20 LTS"
Information that changes with the version.
Why they fail: embeddings have little capacity to differentiate numbers. "Python 3.10" and "Python 3.12" have nearly identical embeddings even though the APIs differ.
Class 4: commands and syntax
"curl -X POST -H 'Content-Type: application/json'"
"git rebase --interactive HEAD~5"
"docker compose up -d"
"kubectl get pods --all-namespaces"
Exact commands with specific syntax.
Why they fail: embeddings normalize flags and options. "git rebase interactive" and "git rebase --interactive HEAD~5" end up close together, losing the specificity of the query.
Class 5: mixed-language queries
"cómo usar OAuth2PasswordBearer" (Spanish + English identifier)
"erro ENOENT no Linux" (Portuguese + English code)
"how to fix erreur 404" (English + French)
Multilingual queries where the technical identifiers stay in the original language.
Why they fail: the multilingual embedding normalizes the language, but the specific doc has the exact identifier that should have been preserved.
The experiment that demonstrates the problem
# limitations_experiment.py
import chromadb
from chromadb.utils import embedding_functions
import os
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
api_key=os.getenv("OPENAI_API_KEY"),
model_name="text-embedding-3-small",
)
client = chromadb.PersistentClient(path="./chroma_test")
collection = client.get_or_create_collection(
name="hybrid_test",
embedding_function=openai_ef,
)
# Dataset: 5 docs about OAuth2 + FastAPI
docs = [
# Doc A: the specific one we mentioned in the insight
"OAuth2PasswordBearer is the FastAPI security class for OAuth2 password flow. "
"Import it from fastapi.security and use it as a dependency in your endpoints. "
"It handles token extraction from the Authorization header automatically.",
# Doc B: a close paraphrase without the identifier
"To implement OAuth2 authentication with username and password in FastAPI, "
"use the appropriate dependency from the security module. This dependency "
"automatically extracts the token from the Authorization header.",
# Doc C: generic, about OAuth2
"OAuth2 is an authorization framework that enables third-party applications "
"to obtain limited access to a user's account. It's commonly used for SSO.",
# Doc D: about password flows in general
"Password flows in OAuth2 allow users to authenticate with their credentials "
"and receive an access token. This flow is appropriate for first-party clients.",
# Doc E: a general auth tutorial for FastAPI
"FastAPI provides multiple authentication methods including OAuth2, JWT tokens, "
"and API keys. Choose the method that best fits your security requirements.",
]
ids = [f"doc_{c}" for c in ['A', 'B', 'C', 'D', 'E']]
collection.add(documents=docs, ids=ids)
# Query with an exact identifier
query = "OAuth2PasswordBearer scopes"
results = collection.query(query_texts=[query], n_results=5)
print(f"Query: {query}\n")
print("Cosine similarity ranking:")
for i, (doc_id, doc, dist) in enumerate(zip(
results['ids'][0],
results['documents'][0],
results['distances'][0],
), 1):
print(f"\n#{i} [{doc_id}] cosine_distance={dist:.3f}")
print(f" {doc[:100]}...")Typical output:
Query: OAuth2PasswordBearer scopes
Cosine similarity ranking:
#1 [doc_B] cosine_distance=0.421
To implement OAuth2 authentication with username and password in FastAPI...
#2 [doc_A] cosine_distance=0.482
OAuth2PasswordBearer is the FastAPI security class for OAuth2 password flow...
#3 [doc_E] cosine_distance=0.587
FastAPI provides multiple authentication methods including OAuth2, JWT tokens...
#4 [doc_C] cosine_distance=0.681
OAuth2 is an authorization framework that enables third-party applications...
#5 [doc_D] cosine_distance=0.752
Password flows in OAuth2 allow users to authenticate with their credentials...
The reading: doc_A (which literally mentions OAuth2PasswordBearer) lands second, below doc_B (a close paraphrase). The user who wrote the query knew the exact name and wanted the doc that documented it. Semantic search subordinated it to a generic paraphrase.
This isn't a bug — it's the expected behavior of cosine similarity. The embeddings normalized the paraphrase. But for cases where exactness matters, that expected behavior is the problem.
How to diagnose the impact on your corpus
# diagnose_semantic_only_impact.py
import re
from collections import Counter
def categorize_query(query: str) -> str:
"""
A heuristic to detect queries that likely fail with semantic-only.
"""
# Code identifiers (CamelCase, snake_case)
has_identifier = bool(re.search(r'[A-Z][a-z]+[A-Z][a-z]+|[a-z]+_[a-z]+', query))
# Error codes (uppercase with numbers or underscores)
has_error_code = bool(re.search(r'[A-Z]{2,}_?[0-9A-Z_]+|HTTP\s*\d{3}', query))
# Versions (X.Y.Z, vN, etc.)
has_version = bool(re.search(r'\bv?\d+\.\d+(\.\d+)?', query))
# Shell commands (flags, paths)
has_command = bool(re.search(r'-{1,2}\w+|/\w+/', query))
if has_identifier:
return "code_identifier"
if has_error_code:
return "error_code"
if has_version:
return "version_specific"
if has_command:
return "shell_command"
return "semantic_only_friendly"
def analyze_query_log(query_log: list[str]) -> dict:
"""Analyzes a sample of real queries."""
categories = [categorize_query(q) for q in query_log]
counts = Counter(categories)
total = len(query_log)
return {
cat: f"{count} ({count/total:.0%})"
for cat, count in counts.most_common()
}
# Apply it over 1000 real queries from the production log
diagnosis = analyze_query_log(production_queries[:1000])
print("Query type distribution:")
for cat, count in diagnosis.items():
print(f" {cat}: {count}")Typical output (technical corpus):
Query type distribution:
semantic_only_friendly: 412 (41%)
code_identifier: 285 (29%)
error_code: 145 (15%)
version_specific: 98 (10%)
shell_command: 60 (6%)
Interpretation: 59% of the queries (everything that isn't "semantic_only_friendly") has components that semantic search handles worse. That explains why your technical RAG system has mediocre recall — more than half of the queries are the type where cosine loses.
Quantifying the impact: recall by category
def evaluate_by_category(eval_set, collection):
"""Compute recall@5 per query category."""
by_category = {}
for item in eval_set:
category = categorize_query(item.query)
if category not in by_category:
by_category[category] = {"hits": 0, "total": 0}
results = collection.query(query_texts=[item.query], n_results=5)
retrieved_ids = set(results['ids'][0])
relevant_ids = set(item.expected_doc_ids)
if retrieved_ids & relevant_ids:
by_category[category]["hits"] += 1
by_category[category]["total"] += 1
print("Recall@5 by category (semantic-only):")
for cat, stats in by_category.items():
recall = stats["hits"] / stats["total"]
print(f" {cat}: {recall:.0%} ({stats['hits']}/{stats['total']})")Typical output:
Recall@5 by category (semantic-only):
semantic_only_friendly: 87% (35/40) ← very good
code_identifier: 58% (17/29) ← a problem
error_code: 47% (7/15) ← worse
version_specific: 62% (6/10) ← a problem
shell_command: 50% (3/6) ← a problem
The reading: semantic search is excellent on queries with no exact identifiers (87%). It fails dramatically on the ones that have them (47-62%). The system's global recall is dragged down by the 60% of traffic that is problematic.
Improving the embedding model attacks the 40% that already works well. For the real problems (60% of traffic), you need another technique.
The trap: "let's just embed better"
When you see low precision/recall, the common first reaction is:
"Let's switch to a better embedding model (text-embedding-3-large, Cohere, Voyage)."
Reality:
- Improving the embedding model reduces false positives on semantic queries by ~5-10%.
- It does not solve the exactness problem. The new model still normalizes paraphrases; it just does it slightly better.
If your dominant problem is queries with exact identifiers, upgrading the embedder solves almost nothing. What you need is to add another technique that DOES prioritize exactness: BM25 (classic keyword search).
Cosine BM25
(semantic) (keyword)
Queries: ────────── ──────────
"how to auth" ✅ excellent 🟡 OK
"OAuth2..." ❌ fails ✅ excellent
"ERR_404" ❌ fails ✅ excellent
Hybrid search is exactly that combination: run both in parallel, fuse the results. Capsules 03-05 cover the "how".
Traps and common mistakes
Trap 1: assuming the problem is chunking
The mistake: you see low recall on queries with identifiers → you assume the chunking is wrong.
The symptom: you tune chunking, spend a week, and get a marginal improvement.
How to prevent it: diagnose first. If the problematic queries are the "exact identifier" type, the problem is NOT chunking — it's the search metric.
Trap 2: testing with semantic-friendly queries
The mistake: you validate your system with queries like "how to authenticate". Recall@5 comes out at 88%. You assume everything is fine.
The symptom: production reports bad recall. You investigate and discover the real queries are "OAuth2PasswordBearer scopes", not "how to authenticate".
How to prevent it: the eval set must reflect the real distribution of production queries. If 60% of real traffic is identifiers, 60% of the eval set should be identifiers.
Trap 3: adding BM25 without measuring the gain
The mistake: "everybody says hybrid search improves things", so you add it without validating.
The symptom: complexity goes up (you maintain two indexes, you manage fusion), but the real gain on your corpus is 2%.
How to prevent it: measure before and after with your eval set. If the gain is <5%, it isn't worth the complexity.
Trap 4: using BM25 as a replacement for semantic
The mistake: you read that BM25 works for identifiers, so you use it as your only search. You remove semantic.
The symptom: semantic queries ("how to deploy", "why is X slow") now fail because BM25 doesn't understand paraphrases.
How to prevent it: hybrid is both in parallel, not one instead of the other. Covered in capsules 03-05.
Trap 5: ignoring multilingual queries when diagnosing
The mistake: your diagnosis uses only English queries. Conclusion: 30% of traffic needs BM25.
The symptom: after implementing BM25, you discover 70% of real traffic is in Spanish, where the identifier pattern is different.
How to prevent it: segment the diagnosis by language. The distribution of problematic queries can vary across languages.
Trap 6: trusting regex heuristics without validating them
The mistake: your categorize_query() with regex categorizes queries. You assume it's accurate.
The symptom: queries like "explain useState" (which contains an identifier) get categorized as "code_identifier", but the real query is semantic (the user wants an explanation, not the literal documentation of the symbol).
How to prevent it: validate the categorization manually over 100 real queries. Adjust the heuristics or use an LLM classifier for ambiguous cases.
Applied exercise
Scenario: you're an AI Engineer at a company that does support for DevOps tooling. Your RAG system serves ~10K queries/day.
Current metrics (semantic search with OpenAI text-embedding-3-small + cross-encoder rerank):
- Precision@5: 85%
- Recall@5: 64% ← the problem
A sample of the query log:
"how do I configure helm chart values" ← semantic
"ERR_NETWORK_TIMEOUT_504" ← error code
"kubectl get pods --all-namespaces" ← shell command
"why is my pod stuck in CrashLoopBackOff?" ← semantic + identifier
"v1.27 vs v1.28 differences" ← versions
"how to fix memory leak" ← semantic
Your job:
- Diagnose what percentage of the traffic is problematic with semantic-only.
- Estimate the improvement potential with hybrid search.
- Justify the decision to stakeholders in terms of ROI.
Solution
1. Diagnosis
Apply the categorize_query script over the sample (assuming 1000 real queries reflect the distribution):
diagnosis = {
"semantic_only_friendly": "350 (35%)", # "how to ...", "why is ..."
"code_identifier": "240 (24%)", # CrashLoopBackOff, helm chart names
"error_code": "180 (18%)", # ERR_TIMEOUT_504
"shell_command": "130 (13%)", # kubectl, helm CLI
"version_specific": "100 (10%)", # v1.27, v1.28
}
# 65% of the traffic is "problematic" for semantic-only2. Improvement estimate
Current recall, segmented (estimated from industry averages):
Category % Current recall Recall with hybrid
────────────────────────────────────────────────────────────────────────
semantic_only_friendly 35% 85% 85% (unchanged)
code_identifier 24% 58% 82% (BM25 helps)
error_code 18% 50% 92% (BM25 nails it)
shell_command 13% 55% 80%
version_specific 10% 62% 88%
Weighted global recall:
Current: 0.35(0.85) + 0.24(0.58) + 0.18(0.50) + 0.13(0.55) + 0.10(0.62)
= 0.298 + 0.139 + 0.090 + 0.072 + 0.062
= 0.661 ≈ 66%
With hybrid: 0.35(0.85) + 0.24(0.82) + 0.18(0.92) + 0.13(0.80) + 0.10(0.88)
= 0.298 + 0.197 + 0.166 + 0.104 + 0.088
= 0.853 ≈ 85%
Expected gain: +19 points of global recall
3. ROI for stakeholders
# Proposal: Implement Hybrid Search
## Diagnosis
65% of the traffic (queries with identifiers, errors, commands, versions) has
recall of ~50-60% with semantic search. That explains why our global recall
is stuck at 64%.
## Solution
Add BM25 (keyword search) in parallel with semantic. Queries with exact identifiers
will find literal matches. Semantic queries keep working with embeddings. We fuse
the results with Reciprocal Rank Fusion.
## Expected impact
- Recall@5: 64% → ~85% (+21 points)
- Precision@5: stable or slightly better
- p95 latency: +30-50ms (BM25 is fast)
- Cost: $0 extra (rank_bm25 runs locally)
## Implementation time
- BM25 setup: 2 days
- Pipeline integration: 1 day
- Validation with the eval set: 1 day
- Production A/B test: 2 weeks
- Total: ~3 calendar weeks (5 engineering days)
## Why NOT upgrade the embedding model first
Switching to text-embedding-3-large would improve semantic queries (~5-7%) but does NOT
affect the 65% of queries with an exactness problem. Those cases require keyword search.
The two changes are complementary — hybrid first (a 21-pt gain), then the embedding
upgrade (an extra marginal gain).
## Risks
- A badly tokenized BM25 can rank noise. Mitigation: use a domain-specific
tokenizer if needed.
- The re-ranking rate limit can degrade with more candidates. Mitigation:
adjust the retrieval's n_results.
## Recommendation
Approve the implementation. A 1-week engineering sprint + a 2-week A/B test.
A measurable recall gain expected, with no recurring cost.Summary and next step
What you learned:
- Semantic search normalizes paraphrases — good behavior for conceptual queries, problematic for queries with exact identifiers.
- Five classes of queries break semantic-only: code identifiers, error codes, versions, commands, and multilingual queries with technical tokens.
- In a typical technical corpus, 50-70% of the traffic is the problematic type.
- Global recall usually sits at 60-70% with semantic-only over a technical corpus.
- Diagnosis with regex + manual categorization lets you quantify the problem.
- Upgrading the embedding model does NOT solve the problem — it improves something else.
- The right fix is to add BM25 in parallel, not to replace semantic.
Checkpoint: before moving on, you should be able to:
- Identify the five classes of queries that are problematic for semantic-only.
- Diagnose your corpus with a script that categorizes queries from the log.
- Estimate the improvement potential with hybrid search based on the type distribution.
Next capsule: 03 — BM25 keyword search.
You've just understood the "why" of hybrid search. Capsule 03 covers the "how" of the first component: BM25. You'll learn what BM25 is (the modern evolution of TF-IDF), why it works where semantic fails, and how to implement it with rank_bm25 (in-memory) or Elasticsearch (at scale).
Resources
- BM25 — The Probabilistic Relevance Framework — The foundational paper
- Pinecone — Hybrid Search — An accessible overview
- Anthropic — Contextual Retrieval — A complementary technique
- Stanford NLP — IR Book — Foundational on information retrieval
- Elasticsearch — Why Hybrid Search — A production case
- BEIR Benchmark — Empirical comparison of semantic vs hybrid
Estimated time: 25-30 minutes Next: 03-bm25-keyword-search.md