Module 5: Hybrid Search — combining keyword + semantic for queries that need both

Capsule 06: Elasticsearch — hybrid search at production scale

Capsule description

rank_bm25 (capsule 03) is perfect for learning and for datasets up to ~1M docs. But when your corpus grows to 10M, 100M or more, in-memory BM25 stops being viable: the RAM isn't enough, the queries get slow, and there's no horizontal distribution. The industry's solution for the past ~15 years is Elasticsearch (or its fork Apache Solr): a distributed engine that implements BM25 natively and scales to billions of documents.

This capsule teaches you how to integrate Elasticsearch as the keyword search layer in a RAG pipeline, the two possible architectures (dual: ES + vector DB; or single: ES with native vector search), and the operational details that matter in production: ID consistency across engines, query parallelization, network latency.

By the end of this capsule you'll be able to:

✅ Decide when to migrate from rank_bm25 to Elasticsearch
✅ Set up a basic local Elasticsearch with Docker
✅ Index documents in ES with structured metadata
✅ Run BM25 queries and combine them with vector DB results
✅ Compare the dual architecture (ES + Pinecone/Chroma) vs a single engine (ES with dense vectors)
✅ Anticipate the operational traps: inconsistent IDs, network latency, a misconfigured ES

Estimated time: 30-35 minutes

When to migrate from `rank_bm25` to Elasticsearch

Symptom	Migrate?
Corpus < 500K docs, RAM to spare	❌ Stay with `rank_bm25`
Corpus 1-5M docs, BM25 latency > 200ms p95	⚠️ Evaluate it
Corpus > 10M docs	✅ Yes, ES or equivalent
You need a cluster with HA and replication	✅ Yes
Multiple instances of the app need to share the index	✅ Yes (or use BM25 with Redis)
The team already has ES deployed for logs/metrics	⚠️ Probably yes (leverage the existing infra)
You want an advanced query DSL (filters, aggregations, fuzzy)	✅ Yes

The cost of migrating to ES:

Operational setup: ~3-5 days (local Docker is fine; a production cluster is more complex).
Re-indexing: depends on the size. ES indexes ~10K docs/second on reasonable hardware.
Recurring cost: $50-500/month for hosted (Elastic Cloud), or the cost of your own infra.

The cost of staying with rank_bm25:

If your corpus grows and you don't migrate: unacceptable latencies, OOM, an unstable app.
If it never grows past 1M: the cost of migrating to ES isn't justified.

Basic setup with Docker

For local development, Docker Compose is the fastest route:

# docker-compose.yml
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false  # dev only; in prod use TLS + auth
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data

volumes:
  es_data:

docker-compose up -d
# Verify
curl http://localhost:9200

Expected output:

{
  "name": "...",
  "cluster_name": "docker-cluster",
  "version": {"number": "8.13.0", ...},
  "tagline": "You Know, for Search"
}

Indexing documents with structured metadata

# es_setup.py
from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk
import os


es = Elasticsearch(
    "http://localhost:9200",
    # For production add:
    # api_key=os.getenv("ES_API_KEY"),
    # verify_certs=True,
    # ca_certs="/path/to/ca.crt",
)


# Define the mapping (the index's schema)
INDEX_NAME = "rag_docs"

mapping = {
    "mappings": {
        "properties": {
            "doc_id": {"type": "keyword"},     # exact match
            "content": {
                "type": "text",
                "analyzer": "standard",         # tokenizes, lowercases, etc.
            },
            "title": {"type": "text"},
            "category": {"type": "keyword"},
            "language": {"type": "keyword"},
            "created_at": {"type": "date"},
            "metadata": {"type": "object", "enabled": False},
        }
    }
}


def create_index():
    """Creates the index with the mapping. Idempotent: drops and recreates if it exists."""
    if es.indices.exists(index=INDEX_NAME):
        es.indices.delete(index=INDEX_NAME)
    es.indices.create(index=INDEX_NAME, body=mapping)
    print(f"Index '{INDEX_NAME}' created")


def index_documents(docs: list[dict]):
    """Indexes documents in bulk."""
    actions = [
        {
            "_index": INDEX_NAME,
            "_id": doc["doc_id"],
            "_source": doc,
        }
        for doc in docs
    ]
    success, errors = bulk(es, actions, chunk_size=500)
    print(f"Indexed: {success}, errors: {errors}")


# Usage example
docs = [
    {
        "doc_id": "doc_001",
        "title": "FastAPI OAuth2 Implementation",
        "content": "OAuth2PasswordBearer is the FastAPI security class for password flow...",
        "category": "auth",
        "language": "en",
        "created_at": "2026-01-15T00:00:00",
    },
    {
        "doc_id": "doc_002",
        "title": "Autenticación OAuth2 en FastAPI",
        "content": "Para implementar autenticación OAuth2 en FastAPI usar la dependency...",
        "category": "auth",
        "language": "es",
        "created_at": "2026-02-20T00:00:00",
    },
    # ...
]

create_index()
index_documents(docs)

The key point: Elasticsearch's _id should be the same doc_id you use in your vector DB. That's what lets you fuse the rankings with no extra lookups.

BM25 queries with filters

def bm25_search_es(query: str, top_k: int = 30, filters: dict = None) -> list[dict]:
    """
    Runs a BM25 search in Elasticsearch with optional filters.

    Args:
        query: the query text
        top_k: the maximum number of results
        filters: e.g. {"category": "auth", "language": "en"}
    """
    must_clauses = [
        {
            "multi_match": {
                "query": query,
                "fields": ["title^2", "content"],   # title gets 2x the weight
                "type": "best_fields",
            }
        }
    ]

    filter_clauses = []
    if filters:
        for field, value in filters.items():
            filter_clauses.append({"term": {field: value}})

    query_body = {
        "query": {
            "bool": {
                "must": must_clauses,
                "filter": filter_clauses,
            }
        },
        "size": top_k,
    }

    response = es.search(index=INDEX_NAME, body=query_body)

    results = []
    for hit in response["hits"]["hits"]:
        results.append({
            "doc_id": hit["_id"],
            "score": hit["_score"],         # ES returns the BM25 score
            "content": hit["_source"]["content"],
            "metadata": hit["_source"],
        })
    return results


# Try it
results = bm25_search_es(
    query="OAuth2PasswordBearer scopes",
    top_k=10,
    filters={"category": "auth", "language": "en"},
)
for r in results[:3]:
    print(f"Score: {r['score']:.2f}")
    print(f"  {r['content'][:120]}")

Note:

multi_match with fields=["title^2", "content"] gives the title double weight. Useful when the title carries the most representative keywords.
filter clauses are fast (they don't contribute to the score, they only filter). Appropriate for metadata like category, language, dates.
ES's _score is the raw BM25 score, ready to fuse.

The dual architecture: Elasticsearch + ChromaDB/Pinecone

The most common pattern in production:

                          Query
                            │
              ┌─────────────┴─────────────┐
              │                           │
              ▼                           ▼
      ┌──────────────┐           ┌──────────────────┐
      │ Elasticsearch│           │  Vector DB       │
      │ (BM25)       │           │ (ChromaDB,       │
      │              │           │  Pinecone, etc.) │
      │ Top-30       │           │  Top-30          │
      └──────┬───────┘           └────────┬─────────┘
             │                            │
             └────────────┬───────────────┘
                          │
                          ▼
                   ┌──────────────┐
                   │ RRF / Weighted│
                   │  fusion       │
                   └──────┬───────┘
                          │
                          ▼
                     Final top-K

The implementation:

import chromadb
from chromadb.utils import embedding_functions
from concurrent.futures import ThreadPoolExecutor


# ChromaDB setup (semantic)
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small",
)
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_collection("rag_docs", embedding_function=openai_ef)


def hybrid_search_es_chroma(query: str, top_k: int = 5, filters: dict = None):
    """Hybrid search with parallel queries to ES and ChromaDB."""

    # Run both queries in parallel (they're I/O bound)
    with ThreadPoolExecutor(max_workers=2) as executor:
        future_bm25 = executor.submit(bm25_search_es, query, 30, filters)

        # ChromaDB: convert the filters to ChromaDB's format
        chroma_filters = {k: v for k, v in (filters or {}).items()}

        future_semantic = executor.submit(
            lambda: collection.query(
                query_texts=[query],
                n_results=30,
                where=chroma_filters or None,
            )
        )

        bm25_results = future_bm25.result()
        sem_results = future_semantic.result()

    # Extract the rankings (a sorted list of doc_ids)
    bm25_ranking = [r["doc_id"] for r in bm25_results]
    semantic_ranking = sem_results["ids"][0]

    # Fuse with RRF
    fused = reciprocal_rank_fusion([bm25_ranking, semantic_ranking], k=60)
    top_ids = [doc_id for doc_id, score in fused[:top_k]]

    # Retrieve the full content (from ChromaDB, which holds the embeddings and metadata)
    return collection.get(ids=top_ids)

The advantages of this architecture:

Each engine is optimized for its task: ES for keyword, the vector DB for semantic.
Parallel queries → total latency = max(ES, vector DB), not the sum.
Changing one doesn't affect the other.

The disadvantages:

Two systems to maintain (operational overhead).
Data synchronization: every document has to live in both.
Network latency: two connections per query.

The single architecture: Elasticsearch with dense vectors

Since Elasticsearch 8.x, ES has native support for dense vectors and allows hybrid search in a single query. If you're already on ES or you want to simplify, this is a valid option.

# Mapping with a vector field
mapping_with_vectors = {
    "mappings": {
        "properties": {
            "doc_id": {"type": "keyword"},
            "content": {"type": "text"},
            "embedding": {
                "type": "dense_vector",
                "dims": 1536,                   # OpenAI text-embedding-3-small
                "index": True,
                "similarity": "cosine",
            },
            "category": {"type": "keyword"},
        }
    }
}


def index_with_embedding(doc_id: str, content: str, metadata: dict):
    """Indexes the doc + its embedding in ES."""
    # Generate the embedding
    embedding = openai_ef([content])[0]  # assumes the embedding_function is configured

    es.index(
        index=INDEX_NAME,
        id=doc_id,
        document={
            "doc_id": doc_id,
            "content": content,
            "embedding": embedding,
            **metadata,
        }
    )


def hybrid_search_native_es(query: str, top_k: int = 5):
    """Native hybrid search in ES (BM25 + kNN)."""
    query_embedding = openai_ef([query])[0]

    # ES 8+ supports hybrid with native RRF
    body = {
        "size": top_k,
        "query": {
            "bool": {
                "should": [
                    {"match": {"content": query}},  # BM25
                ]
            }
        },
        "knn": {
            "field": "embedding",
            "query_vector": query_embedding,
            "k": 30,
            "num_candidates": 100,
        },
        "rank": {
            "rrf": {
                "rank_window_size": 50,
                "rank_constant": 60,             # RRF's k
            }
        },
    }

    response = es.search(index=INDEX_NAME, body=body)
    return [
        {
            "doc_id": hit["_id"],
            "score": hit["_score"],
            "content": hit["_source"]["content"],
        }
        for hit in response["hits"]["hits"]
    ]

The advantages:

A single engine. Simpler operations.
Native RRF (you don't need to implement the fusion).
Consistent filters across BM25 and vector search.

The disadvantages:

ES isn't the fastest engine for vector search compared to Pinecone/Qdrant.
Resources: ES with dense vectors needs a lot of RAM.
Migrating later is hard if you want to change stacks.

The decision: dual vs single

Criterion	Dual (ES + vector DB)	Single (ES with dense vectors)
Vector search performance	Better (a specialized vector DB)	OK (ES isn't optimal for vectors)
Operational simplicity	Worse (2 systems)	Better (1 system)
Infra cost	Higher	Lower
Expertise the team needs	ES + a vector DB	ES only
Flexibility to swap components	High	Low (lock-in)
Appropriate for	>10M docs, large teams	<10M docs, small teams

The practical recommendation:

If you already have ES deployed for logs/metrics: go single (leverage the infra).
If you don't have ES and you're starting from zero: go dual with a specialized vector DB (Pinecone, Qdrant).
If your corpus is <1M: you don't even need ES — rank_bm25 + ChromaDB is enough.

Traps and common mistakes

Trap 1: different IDs across ES and the vector DB

The mistake: ES indexes with _id="abc123". ChromaDB indexes with _id="doc_abc123".

The symptom: the RRF fusion finds no common docs. Each engine "votes" for different docs, and the final ranking is noise.

How to prevent it: always the same doc_id in both engines. If you have to reformat, do it before indexing, not after.

Trap 2: sequential queries in a dual architecture

The mistake:

bm25 = bm25_search_es(query)        # 50ms
semantic = collection.query(query)   # 100ms
# Total: 150ms

The symptom: total latency = the sum of the two queries.

How to prevent it: run them in parallel with ThreadPoolExecutor. They're I/O bound, so parallelism is trivial. Total: 100ms (the max of the two).

Trap 3: Elasticsearch without TLS/auth in production

The mistake: you copy the development setup (xpack.security.enabled=false) into production.

The symptom: ES exposed with no auth. Any network scan can read and modify your index.

How to prevent it: production always with TLS + API keys + firewall rules.

Trap 4: re-indexing everything when the mapping changes

The mistake: ES doesn't allow changing a field's type in place. If you change category: keyword to category: text, you have to re-index.

The symptom: after changing the mapping, queries return 0 results or errors.

How to prevent it: plan the mapping up front. For future changes, use the reindex API or create a new index and migrate.

Trap 5: forgetting score normalization if you're NOT using RRF

The mistake: a dual architecture, and you want to combine ES scores (BM25) with ChromaDB's (cosine distance) without RRF.

The symptom: incompatible ranges. Same result as in capsule 05.

How to prevent it: either use RRF (capsule 04, robust to ranges), or normalize the scores with min-max (capsule 05).

Trap 6: `num_candidates` set too low in ES's native kNN

The mistake:

"knn": {"k": 30, "num_candidates": 30}  # the default if you don't specify it

The symptom: low recall. ES discards valid candidates when num_candidates is capped.

How to prevent it: num_candidates >= 5x k. For k=30, use num_candidates=150-200.

Applied exercise

Scenario: you're an AI Engineer at a technical support company. The current stack:

5M chunks of documentation + resolved tickets
ChromaDB with OpenAI embeddings (semantic)
rank_bm25 in-memory (BM25)
Hybrid with RRF

The symptoms:

BM25 latency climbs from 30ms to 250ms at peak hour (in-memory doesn't scale further).
The app's memory grows to 12GB → instances sit on the edge of OOM.
You need 3-4 replicas of the app, but each one has its own BM25 (expensive and out of sync).

Your job:

Decide between the dual architecture (ES + Chroma) and the single one (ES with dense vectors).
Design the migration plan.
Estimate the operational impact.

Solution

1. The decision: the dual architecture (ES + ChromaDB)

The reasons:

Vector search is critical and ChromaDB is already optimized for it. Migrating to ES with dense vectors would degrade vector performance.
The team already has ChromaDB expertise. Switching the vector engine adds an unnecessary learning cost.
5M docs justifies ES for BM25 (vs rank_bm25), but it does NOT require unifying the stack.
Future flexibility: dual lets you migrate one without touching the other.

2. The migration plan (3-4 weeks)

Week 1: Elasticsearch setup

Deploy ES in a cluster (3 nodes for HA, or managed Elastic Cloud).
Configure TLS, API keys, monitoring.
Create the rag_docs index with the defined mapping.

Week 2: index the existing corpus in ES

A migration script: read the chunks from ChromaDB (collection.get(...)) and bulk index them into ES.
5M docs × 100ms per bulk of 500 = ~17 minutes of pure work. Realistically 30-60 minutes.
Verify that ES has the same docs as ChromaDB (matching count).
Guarantee that the _id in ES = the id in ChromaDB (for the later RRF fusion).

Week 3: implement parallelization in the pipeline

Refactor the RAG pipeline: parallel queries to ES and ChromaDB with ThreadPoolExecutor.
Replace the rank_bm25 calls with calls to ES.
A/B test over the eval set: compare the results pre/post migration.

Week 4: gradual rollout

Feature flag: 10% of traffic to the new pipeline. Monitor latency + recall.
If the metrics hold, scale to 50% → 100%.
Remove rank_bm25 from the code once you're deployed at 100%.

3. Operational impact

Expected improvements:

BM25 latency: 250ms p95 → 30-50ms p95 (ES is fast for BM25).
The app's memory: 12GB → 4GB (no more in-memory BM25).
App replicas can share the same ES, with no desynchronization.
Availability: ES with HA has better uptime than an app with in-memory BM25.

Extra cost:

ES infra: $200-500/month (managed) or the cost of operating your own cluster.
Latency added by the network hop to ES: ~5-10ms (negligible).
Operational: monitoring + ES backups + version upgrades.

ROI:

If dropping latency from 250ms to 50ms improves user retention/satisfaction, the $300/month is justified.
If the app goes from 4 replicas (12GB each) to 4 replicas (4GB each), you save 32GB of RAM in infra → potentially $100-200/month.

Plan B if the migration hits problems:

A fast rollback is possible with the feature flag (1 hour).
If ES has quality problems: investigate the mapping (is the analyzer right? is the ^2 boost being applied to the title?).
If the latency is worse than expected: review the ES configuration (refresh_interval, replicas, shards).

Metrics to monitor post-migration:

ES latency vs before (it should drop 5x).
Recall@5 (it should hold or improve).
5xx errors in ES (they should be 0).
The app's memory (it should drop 50%+).

Summary and next step

What you learned:

Elasticsearch is the industry-standard solution for BM25 at scale (>1M docs).
Local setup with Docker is fast (5 minutes). Production requires TLS, auth, a cluster.
Indexing with the bulk API: ~10K docs/second on reasonable hardware.
Multi-match with a boost (^2) for titles. Filters for metadata (fast, and they don't contribute to the score).
Two architectures: dual (ES + a specialized vector DB) or single (ES with native dense vectors).
For dual queries: parallelize with ThreadPoolExecutor so latency = the max, not the sum.
Consistent IDs across engines is critical for the later RRF fusion.
ES 8+ has native RRF via rank.rrf — useful if you go with the single architecture.

Checkpoint: before moving on, you should be able to:

Decide between rank_bm25 and Elasticsearch based on scale.
Set up ES locally with Docker in 10 minutes.
Implement a dual hybrid pipeline with parallel queries.
Tell the dual and single architectures apart and choose based on context.

Next capsule: 07 — The hybrid search decision framework.

We covered the components (BM25, semantic, RRF, weighted, ES). Capsule 07 consolidates it all: when each technique wins, which pattern to choose based on your scale/domain/team, and a reproducible decision framework.

Resources

Elasticsearch — Official Docs — The complete documentation
Elasticsearch Python Client — The official SDK
Hybrid Search with Elasticsearch (8.x) — The official guide
Elasticsearch RRF — The native implementation
Qdrant — Hybrid Search Alternative — A comparison with another option
BEIR Benchmark — Empirical comparison of BM25 vs hybrid

Estimated time: 30-35 minutes Next: 07-strategy-comparison-2.md

← PreviousCapsule 05: Weighted hybrid blending — when one signal is clearly better than the other Next →Capsule 07: The decision framework — which hybrid search strategy to choose