Module 5: Hybrid Search — combining keyword + semantic for queries that need both
Capsule 07: The decision framework — which hybrid search strategy to choose
Capsule description
We covered the individual components: BM25 (capsule 03), RRF (04), weighted blending (05), Elasticsearch (06). The operational question that closes the module: given a new project, which combination do you choose?
This capsule consolidates what you learned into a reproducible decision framework. You'll learn to make the call in under 10 minutes: semantic-only? simple hybrid? hybrid with weighted? in-memory or Elasticsearch? Every choice has concrete trade-offs you'll be able to defend with data.
By the end of this capsule you'll be able to:
- ✅ Compare the four retrieval architectures across six dimensions
- ✅ Apply a 5-question flowchart to pick a strategy
- ✅ Justify the choice to a Tech Lead with numerical arguments
- ✅ Design an evolution plan (start simple, scale gradually)
- ✅ Identify the common anti-patterns of "choosing everything at once"
- ✅ Decide when to migrate between strategies as your product grows
Estimated time: 25-30 minutes
The four possible architectures
Architecture A: Semantic-only
Query → Embedding → Vector DB → Top-K → LLM
- When: purely conceptual queries, an MVP, a team with no resources.
- Typical recall: 85-90% on semantic queries, 50-70% on queries with identifiers.
- Latency: 100-300ms.
- Cost: $0-50/month for 100K queries (embeddings only).
- Maintenance: minimal.
Architecture B: BM25-only
Query → BM25 (rank_bm25 or ES) → Top-K → LLM
- When: a purely exact-match corpus (error codes, code search, SKUs).
- Typical recall: 90%+ on exact queries, 40-50% on semantic queries.
- Latency: 5-30ms.
- Cost: $0-30/month (minimal compute).
- Maintenance: low (a careful tokenizer).
Architecture C: Simple hybrid (semantic + BM25 + RRF)
Query ──┬→ Embedding → Vector DB → Top-30 ──┐
│ │→ RRF → Top-K → LLM
└→ BM25 → Top-30 ─────────────────────┘
- When: a mixed corpus (technical + narrative), typical production. The reasonable default.
- Typical recall: 88-95%.
- Latency: 150-400ms (parallelizable).
- Cost: $30-100/month for 100K queries.
- Maintenance: medium (two engines).
Architecture D: Advanced hybrid (weighted + routing + ES)
Query → Type detector → dynamic α
├→ Elasticsearch BM25 (top-30) ──┐
└→ Vector DB (top-30) ──┼→ Weighted blending → Top-K → LLM
(optional rerank afterwards)
- When: a corpus at scale (>1M docs), a variety of query types, a critical product.
- Typical recall: 92-97%.
- Latency: 200-500ms.
- Cost: $200-1000/month.
- Maintenance: high (an ES cluster, α tuning, routing).
Quantitative comparison
Architecture Recall@5 Latency p95 Setup time Cost/month (100K queries)
─────────────────────────────────────────────────────────────────────────────────────────
A: Semantic-only 85% 200ms 1 day $20
B: BM25-only 82% 15ms 2 days $5
C: Simple hybrid 92% 320ms 3-5 days $50
D: Advanced hybrid 95% 400ms 2-3 weeks $300
The reading: options C and D give the best recall, but the biggest "jump" is from A/B to C (+10 points). From C to D the jump is marginal (+3 points) for far more effort.
A practical rule: most projects can stay at architecture C. D is only justified with a robust eval set that proves C isn't enough.
The 5-question decision framework
1. Is your corpus 100% narrative text (no identifiers, errors, commands)?
→ Yes: Architecture A (semantic-only). Save yourself the complexity.
→ No: continue to 2.
2. Are 95%+ of your product's queries built on exact identifiers?
→ Yes: Architecture B (BM25-only). It's probably code search or a lookup.
→ No: continue to 3 (you need hybrid).
3. Is your corpus <1M documents?
→ Yes: continue to 4 with in-memory `rank_bm25`.
→ No: you're going to need Elasticsearch (capsule 06).
4. Do you have an eval set of ≥50 queries with ground truth?
→ No: architecture C (simple hybrid with RRF). Without an eval set you can't tune α.
→ Yes: continue to 5.
5. Does your eval set show that one signal is notably better (>10 pts) for some query type?
→ No: architecture C is enough.
→ Yes: architecture D with weighted + routing.
Applied to real cases
| Project | Recommendation | Why |
|---|---|---|
| A SaaS chatbot MVP | A: semantic-only | Iterate fast, optimize later |
| A Stack Overflow search engine | C: simple hybrid | A mix of identifiers + semantic queries |
| A product catalog with SKUs | B: BM25-only | Exact codes dominate |
| A legal system with technical queries | C: simple hybrid | Specific vocabulary + narrative queries |
| An enterprise internal search (5M docs, large team) | D: advanced hybrid | The scale justifies the complexity |
| An academic paper assistant | A: semantic-only | Purely conceptual queries |
| A SaaS company's help desk | C: simple hybrid | The typical mix |
The gradual evolution plan
Don't start at architecture D. Build up progressively:
┌─────────────────────────────────────────────────────────────────┐
│ Phase 1: MVP │
│ Architecture A (semantic-only) │
│ Time: 1 day │
│ Metrics to validate: are precision/recall acceptable? │
└────────────────────────┬─────────────────────────────────────────┘
│ If recall < 80% on technical queries
▼
┌─────────────────────────────────────────────────────────────────┐
│ Phase 2: Simple hybrid │
│ Architecture C (semantic + BM25 + RRF) │
│ Time: 3-5 additional days │
│ Metrics to validate: does recall go up +10 pts? │
└────────────────────────┬─────────────────────────────────────────┘
│ If you need more quality or you hit 1M+ docs
▼
┌─────────────────────────────────────────────────────────────────┐
│ Phase 3: ES + tuning │
│ Architecture D, partially (ES + weighted blending) │
│ Time: 2-3 additional weeks │
│ Metrics to validate: does recall go up another +3-5 pts? │
└────────────────────────┬─────────────────────────────────────────┘
│ If you have varied traffic and specific queries
▼
┌─────────────────────────────────────────────────────────────────┐
│ Phase 4: Dynamic routing │
│ Architecture D, complete (routing by query type) │
│ Time: 1 additional week │
│ Metrics to validate: does recall improve per query type? │
└─────────────────────────────────────────────────────────────────┘
The benefit of the gradual plan:
- Each phase has a clear validation before you move to the next.
- If phase 1 works, you don't spend time on phase 2.
- Each phase is reversible (rollback with a feature flag).
The anti-pattern: starting directly at architecture D. You triple the setup time, you maintain unnecessary complexity, and sometimes you discover A or C would have been enough.
When NOT to use hybrid search
Despite the benefits, there are cases where hybrid search is overkill:
| Case | Hybrid? | Reason |
|---|---|---|
| A purely narrative corpus (literature, journalism) | ❌ No | Semantic is enough; BM25 adds noise |
| An early-stage MVP with <5K docs | ❌ No | The dominant problem is something else (probably chunking) |
| An ultra-strict latency budget (<100ms total) | ❌ No | The parallelization + fusion overhead breaks the SLA |
| A team of 1-2 people with no DevOps | ⚠️ Maybe | Maintaining two engines adds operational overhead |
| A product where "slow but perfect answer" beats "fast and mediocre" | ⚠️ Look deeper | The problem may be in generation, not retrieval |
Migrating between architectures
From A to C (adding BM25 + RRF)
The trigger: recall@5 < 80% on queries with identifiers.
The effort: 3-5 engineering days.
The risk: low. RRF works out of the box, with no tuning.
The protective metric: precision (it shouldn't drop >2% if BM25 is well tokenized).
From C to D (Elasticsearch + weighted + routing)
The trigger: recall@5 stuck at 90% and you need 95%+, OR the corpus passed 1M docs.
The effort: 2-3 engineering weeks + 1 week of tuning.
The risk: medium. It requires a robust eval set to tune α and validate the routing.
The protective metric: operational complexity. If the team doesn't have the capacity to maintain ES, fall back to C.
Migrating BACKWARD (from C to A)
Sometimes the right move is to simplify. The cases:
- The product pivoted to a purely narrative corpus. BM25 no longer contributes.
- The corpus shrank to <50K docs. The complexity isn't justified.
- You discovered 95% of the queries are semantic, and BM25 rarely fires.
Migrating to a simpler architecture is legitimate if the data justifies it.
Common anti-patterns
Anti-pattern 1: implementing everything at once
The mistake: first sprint of the project, the team decides "let's go with advanced hybrid from the start".
The symptom: 3 weeks of setup before you've even validated that the product has fit. You discover half the work was unnecessary.
How to prevent it: the gradual plan. Every phase with a validation criterion.
Anti-pattern 2: copying another product's architecture
The mistake: "Company X uses hybrid with weighted blending → so will we".
The symptom: the complexity isn't justified for your case. Your corpus, your volume, your queries are different.
How to prevent it: justify every component with your own data. If company X does something, find out why they do it before copying it.
Anti-pattern 3: ignoring the operational cost
The mistake: choosing architecture D without considering that a team of 3 can't maintain ES in production.
The symptom: ES ends up misconfigured, there are outages, and someone has to learn ES from scratch at 3am when something breaks.
How to prevent it: weigh the team's operational capacity in the decision, not just the technical quality.
Anti-pattern 4: tuning α without a robust eval set
The mistake: weighted blending with α picked by intuition.
The symptom: after subtle changes to the corpus, the optimal α shifted, but nobody notices. Quality degrades slowly.
How to prevent it: α must be determined by a grid search over the eval set. Re-evaluate quarterly.
Anti-pattern 5: not measuring precision when you add BM25
The mistake: you turn on BM25, recall goes up 10%, you deploy. Nobody looked at precision.
The symptom: BM25 added noise. The top-5 now has 1-2 irrelevant docs that the LLM doesn't ignore well. Answer quality gets worse.
How to prevent it: measure precision AND recall together. If precision drops >3%, BM25 is badly calibrated (probably the tokenizer).
Applied exercise
Scenario: evaluate three projects and decide which architecture to recommend.
Project X: a technical support SaaS. 80K documentation articles. Queries from the log:
- 30% conceptual ("how to deploy")
- 50% identifiers ("kubectl get pods")
- 20% mixed
Current stack: semantic only. Recall@5 = 65%. Team: 4 engineers.
Project Y: a literature platform. 500K novels + literary analyses. Queries: "novels with unreliable narrators", "the water metaphor in Argentine literature".
Current recall@5 with semantic = 87%. Team: 2 engineers.
Project Z: e-commerce with 10M products. Queries: 70% exact SKUs and models, 30% product descriptions.
Current recall@5 with semantic = 48%. Team: 8 engineers with dedicated DevOps.
Your job:
- Apply the framework to each project.
- Justify the decision.
- Define a migration plan with the order of the phases.
Solution
Project X — Recommendation: Architecture C (simple hybrid with RRF)
Applying the framework:
- Is the corpus 100% narrative? No (50% are identifiers).
- 95%+ identifiers? No (only 50%).
- Corpus <1M? Yes (80K docs). →
rank_bm25is enough. - An eval set of ≥50 queries? Assume they can build one. → step 4 says architecture C.
The justification:
- 50% of the traffic is queries with identifiers where semantic fails. Hybrid is necessary.
- A small corpus (80K) → you don't need ES.
- A team of 4 → they can maintain two engines with no trouble.
The plan:
- Phase 1 (3 days): implement BM25 with
rank_bm25+ a technical tokenizer. - Phase 2 (1 day): integrate RRF (capsule 04).
- Phase 3 (1 week): validate over an eval set of 80 queries (proportional to the log's percentages).
- Phase 4 (1 week): production A/B test.
Expected improvement: recall 65% → 85-90%.
Project Y — Recommendation: Architecture A (semantic-only). Do NOT migrate.
Applying the framework:
- Is the corpus 100% narrative? Yes (literature). → architecture A.
The justification:
- Recall is already at 87% — high. There's no exactness problem for BM25 to solve.
- The queries are purely semantic ("the water metaphor") — BM25 would add noise.
- A team of 2 → hybrid's complexity isn't justified.
The plan:
- Don't touch the architecture. If the team wants to improve, consider:
- Better embeddings (text-embedding-3-large): +3-5%
- Re-ranking with a cross-encoder: +5-8%
- Semantic chunking (M02): +3-5%
Expected recall with those improvements: 87% → 92-95%. No need for hybrid.
Project Z — Recommendation: Architecture D (ES + weighted + routing)
Applying the framework:
- Is the corpus 100% narrative? No (70% are SKUs).
- 95%+ identifiers? Almost (70%). → consider architecture B, but the remaining 30% is semantic → you need hybrid.
- Corpus <1M? No (10M docs). → you need Elasticsearch.
- A robust eval set? Assume yes (a large team can build one).
- Is one signal clearly better? Yes — BM25 is far better for SKUs (70% of the traffic). → weighted with α=0.3 for queries with a SKU.
The justification:
- 48% recall is unacceptable. It needs a big jump, not a marginal one.
- 10M docs justifies ES.
- A team of 8 with DevOps → they can maintain a complex architecture.
- 70% of queries carry explicit SKUs → weighted with a dynamic α improves things notably.
The plan:
- Phase 1 (1 week): ES cluster setup + index the 10M docs.
- Phase 2 (1 week): integrate ES into the pipeline (in parallel with the existing vector DB).
- Phase 3 (1 week): an RRF baseline + validate.
- Phase 4 (2 weeks): weighted blending with a grid search over α + routing by query type.
- Phase 5 (1 week): production A/B test.
Expected improvement: recall 48% → 90%+. Extra cost: ~$300/month (Elastic Cloud) + $50/month incremental in embedding costs.
ROI: if the e-commerce site processes 1M queries/day, improving recall by 40 points can translate into a measurable conversion gain, amply justifying the cost.
Summary and next step
What you learned:
- Four architectures: semantic-only (A), BM25-only (B), simple hybrid (C), advanced hybrid (D).
- A 5-question decision framework resolves the choice in <10 minutes.
- The gradual evolution plan: start with A, scale to C when the problem justifies it, scale to D only with a robust eval set.
- Most projects can stay at architecture C. D only when D > C is demonstrable with data.
- Migrating BACKWARD is legitimate if the complexity isn't justified.
- The anti-patterns: implementing everything at once, copying architectures without justifying them, ignoring the operational cost, tuning without an eval set, not measuring precision when you add BM25.
Checkpoint: before closing the module, you should be able to:
- Apply the framework to a new project in <10 min.
- Justify the choice to a Tech Lead with numerical arguments.
- Design a gradual evolution plan with validation criteria.
Next capsule: 08 — The capstone hybrid search engine project.
The close of the module: you're going to build an end-to-end hybrid search system with BM25 (rank_bm25 or ES) + semantic + RRF, A/B testing against the baseline, and a comparison report. It's the project that goes into your portfolio — and it's the closest thing to the "state of the art" pattern in production RAG.
Resources
- Pinecone — Hybrid Search Decision Guide — Applied patterns
- LangChain — EnsembleRetriever — Reference implementations
- LlamaIndex — Hybrid Retrieval — The pattern in LlamaIndex
- BEIR Benchmark — Empirical comparisons
- Elasticsearch Hybrid Search — The native implementation
- Anthropic — Contextual Retrieval — A complementary technique
Estimated time: 25-30 minutes Next: 08-project-hybrid-search-engine.md