Capsule 03: BM25 — the classic algorithm that still wins for exact keywords

Capsule description

BM25 (Best Match 25, also called Okapi BM25) is the modern evolution of TF-IDF that dominated information retrieval for decades before the embeddings boom. The question in 2026 isn't whether BM25 works — it's when to add it to your RAG pipeline. After capsule 02 you already know that cosine similarity fails with exact identifiers; BM25 is exactly the opposite: it prioritizes literal lexical matching over abstract semantics.

This capsule teaches you what BM25 does under the hood (no advanced math), how to implement it with rank_bm25 for small or mid-sized datasets, and how to avoid the typical failure modes: tokenization badly calibrated to the domain, aggressive preprocessing that erases the very identifiers you want to preserve, and the mistake of comparing BM25 vs embeddings as if they were competitors instead of complements.

By the end of this capsule you'll be able to:

• ✅ Explain what BM25 is and why it's the right choice for keyword search in 2026
• ✅ Implement BM25 over a corpus with rank_bm25 (in-memory) in under 30 lines
• ✅ Tokenize correctly to preserve technical identifiers (camelCase, snake_case)
• ✅ Diagnose preprocessing problems that destroy exactness
• ✅ Decide when to use BM25 standalone vs always combined with semantic
• ✅ Anticipate the scale limit: when to migrate to Elasticsearch (capsule 06)

Estimated time: 30-35 minutes

The insight: BM25 measures how "improbably common" the match is

The conceptual core of BM25 is a simple idea: a document that contains your query's keywords is more relevant than one that doesn't. But the subtlety is in how BM25 models "contains":

Query:       "OAuth2PasswordBearer FastAPI"

Doc A:       "OAuth2PasswordBearer is the FastAPI security class for password flow.
              It handles token extraction from Authorization headers."

Doc B:       "FastAPI is a modern web framework. It supports OAuth2 authentication
              through various dependency classes including OAuth2PasswordBearer."

Doc C:       "Authentication frameworks like OAuth2 are common in modern APIs."

BM25 score:
  Doc A: 8.42  ← exact match, high density of the query's terms
  Doc B: 6.18  ← exact match, but the terms are diluted
  Doc C: 1.94  ← only "OAuth2" matches; "PasswordBearer" and "FastAPI" are absent

BM25 considers three factors for each term in the query:

1. TF (Term Frequency): how many times the term appears in the document. Docs that mention OAuth2PasswordBearer 3 times rank higher than docs that mention it once.

2. IDF (Inverse Document Frequency): how rare the term is across the whole corpus. OAuth2PasswordBearer appears in few docs → big weight. is appears in all of them → negligible weight.

3. Document length normalization: short docs containing all the terms rank higher than long docs where the terms are "diluted" among thousands of words.

The formula is mathematically precise, but you don't need to derive it to use BM25. What matters is understanding the mental model: it ranks documents where the query's terms are simultaneously frequent (in that doc) and rare (in the corpus).

Why BM25 already wins where semantic fails

Back to the example from capsule 02 (the OAuth2PasswordBearer identifier). Where semantic search subordinated it to a close paraphrase, BM25 works the other way around:

Query: "OAuth2PasswordBearer scopes"

Semantic (cosine distance):
  Doc B (paraphrase):   0.421  ← close by meaning
  Doc A (identifier):   0.482  ← farther away, even though it has the exact match

BM25:
  Doc A (identifier):   8.42   ← wins because it contains the exact terms
  Doc B (paraphrase):   1.20   ← low, because it does NOT contain "OAuth2PasswordBearer"

The conceptual difference:

• Semantic asks: "how similar is the meaning?"
• BM25 asks: "how present are the exact tokens?"

For queries with identifiers, BM25 is structurally superior. It isn't "better in general" — it's better for that specific class. That's why hybrid search combines them instead of choosing one.

A basic implementation with rank_bm25

rank_bm25 is the most widely used Python library for in-memory BM25. It works well up to ~1M documents on a reasonable machine. Beyond that, consider Elasticsearch (capsule 06).

Setup

pip install rank-bm25

The minimal code

# bm25_basic.py
from rank_bm25 import BM25Okapi
from dataclasses import dataclass


@dataclass
class BM25Result:
    document: str
    score: float
    original_index: int


class BM25Index:
    """A wrapper over rank_bm25 with a configurable tokenizer."""

    def __init__(self, documents: list[str], tokenizer=None):
        self.documents = documents
        self.tokenizer = tokenizer or self._default_tokenizer

        # Tokenize every document
        tokenized_corpus = [self.tokenizer(doc) for doc in documents]
        self.bm25 = BM25Okapi(tokenized_corpus)

    @staticmethod
    def _default_tokenizer(text: str) -> list[str]:
        """A simple tokenizer that preserves camelCase and identifiers."""
        return text.lower().split()

    def search(self, query: str, top_k: int = 5) -> list[BM25Result]:
        """Search for the top-K documents by BM25 score."""
        query_tokens = self.tokenizer(query)
        scores = self.bm25.get_scores(query_tokens)

        # Top-K by descending score
        top_indices = sorted(
            range(len(scores)),
            key=lambda i: -scores[i],
        )[:top_k]

        return [
            BM25Result(
                document=self.documents[i],
                score=float(scores[i]),
                original_index=i,
            )
            for i in top_indices
        ]


# Try it
docs = [
    "OAuth2PasswordBearer is the FastAPI security class for OAuth2 password flow.",
    "For OAuth2 authentication with username/password in FastAPI, use the appropriate dependency.",
    "OAuth2 is an authorization framework that enables third-party application access.",
    "FastAPI provides multiple authentication methods including OAuth2 and JWT tokens.",
    "Password flows in OAuth2 allow users to authenticate with their credentials.",
]

index = BM25Index(docs)
results = index.search("OAuth2PasswordBearer scopes", top_k=3)

for i, r in enumerate(results, 1):
    print(f"#{i} (BM25 score={r.score:.2f})")
    print(f"   {r.document[:100]}...")

Output:

#1 (BM25 score=1.12)
   OAuth2PasswordBearer is the FastAPI security class for OAuth2 password flow....

#2 (BM25 score=0.00)
   For OAuth2 authentication with username/password in FastAPI, use the appropriate depe...

#3 (BM25 score=0.00)
   OAuth2 is an authorization framework that enables third-party application access....

Note: the doc with the literal OAuth2PasswordBearer lands first, and it's the only one with a non-zero score. Every other doc scores exactly 0.00 — none of them contains the token oauth2passwordbearer, and scopes doesn't appear anywhere in the corpus. BM25 is that literal: no exact token, no score. That brutality is exactly what makes it the right complement to cosine.

Tokenization: the detail that makes or breaks BM25

The library's default (text.split()) is very simple and usually works OK. But there are cases where it needs fine-tuning — specifically when your corpus has identifiers in CamelCase, snake_case, or with special characters.

The problem: CamelCase

# Default tokenizer
"OAuth2PasswordBearer".lower().split()
# → ["oauth2passwordbearer"]

The tokenizer treats OAuth2PasswordBearer as a single token. If the user types OAuth2 password bearer (separated), it doesn't match.

Solution 1: tokenize by casing transitions

import re


def code_aware_tokenizer(text: str) -> list[str]:
    """
    A tokenizer that splits CamelCase and snake_case into individual tokens
    while also preserving the full version.
    """
    text = text.lower() if not _has_camelcase(text) else text

    # Split by casing (camelCase → camel, Case)
    tokens = re.findall(r'[A-Z][a-z]+|[a-z]+|\d+', text)

    # Lowercase
    tokens = [t.lower() for t in tokens]

    # Also preserve the full original token
    if '_' in text or any(c.isupper() for c in text):
        tokens.extend(text.lower().split())

    return tokens


def _has_camelcase(text: str) -> bool:
    return bool(re.search(r'[a-z][A-Z]', text))


# Try it
tokens = code_aware_tokenizer("OAuth2PasswordBearer")
print(tokens)
# → ['o', 'auth', '2', 'password', 'bearer', 'oauth2passwordbearer']

Now OAuth2PasswordBearer matches queries for oauth2, password, bearer, AND oauth2passwordbearer. Better coverage.

Solution 2: include token n-grams

For queries with error codes that have a specific structure:

def ngram_tokenizer(text: str, max_n: int = 3) -> list[str]:
    """Individual tokens + n-grams up to max_n."""
    words = text.lower().split()

    tokens = list(words)
    for n in range(2, max_n + 1):
        for i in range(len(words) - n + 1):
            ngram = "_".join(words[i:i+n])
            tokens.append(ngram)

    return tokens


# Try it
tokens = ngram_tokenizer("ERR NETWORK TIMEOUT 504", max_n=3)
print(tokens)
# → ['err', 'network', 'timeout', '504',
#    'err_network', 'network_timeout', 'timeout_504',
#    'err_network_timeout', 'network_timeout_504']

Useful so that the query "ERR NETWORK TIMEOUT 504" finds the document that has the exact phrase as a unit.

The recommended tokenizer for a technical corpus

def technical_tokenizer(text: str) -> list[str]:
    """
    A tokenizer appropriate for a technical corpus (code + text).
    Preserves identifiers and splits casing.
    """
    # Lowercase for consistent matching
    text_lower = text.lower()

    # Standard tokens (split on whitespace and common punctuation)
    tokens = re.findall(r'\b\w+\b', text_lower)

    # Plus: identifiers with underscores or numbers preserved
    underscore_tokens = re.findall(r'\b\w*_\w+\b', text_lower)
    tokens.extend(underscore_tokens)

    # Plus: error codes (uppercase with underscores or numbers)
    code_tokens = re.findall(r'[A-Z][A-Z_0-9]+', text)  # NOT lowercase here
    tokens.extend([t.lower() for t in code_tokens])

    return tokens

Preprocessing: what NOT to do

A common trap with BM25 is applying "classic NLP" preprocessing that breaks exactly what you want to preserve:

Trap 1: aggressive stemming

# ❌ This erases the exact match
from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
tokens = [stemmer.stem(t) for t in tokens]
# "OAuth2PasswordBearer" → "oauth2passwordbear"
# The user types "OAuth2PasswordBearer" and it doesn't match

How to prevent it: avoid stemming for a technical corpus. It's useful for narrative English text (e.g. matching running with run), but it destroys identifiers.

Trap 2: global stopword removal

# ❌ Removing stopwords from the document AND from the query
STOPWORDS = {"a", "an", "the", "of", "in", "to", "is"}
tokens = [t for t in tokens if t not in STOPWORDS]

The symptom: queries like "how to use" end up with no tokens. BM25 can't rank anything.

How to prevent it: stopword removal is NOT necessary for BM25 — the IDF formula already gives negligible weight to very common tokens. Stopword removal is a leftover from eras with less memory; it doesn't add quality.

Trap 3: aggressive lowercasing

# ❌ Lowercasing everything, losing the case-sensitivity of error codes
text = text.lower()
# "ERR_NETWORK_TIMEOUT" → "err_network_timeout"
# The user types "ERR_NETWORK_TIMEOUT", and it matches OK by luck

The symptom: uppercase tokens (error codes, constants) get treated like ordinary text. Most of the time it works, but you lose signal when the query does preserve the uppercase.

How to prevent it: lowercase ordinary tokens, but preserve uppercase for tokens that are clearly identifiers (regex match [A-Z]{2,}).

Trap 4: stripping punctuation inside identifiers

# ❌ Replacing everything that isn't alphanumeric
text = re.sub(r'[^\w\s]', '', text)
# "v2.3.1" → "v231"
# The user types "v2.3.1", and it doesn't match

How to prevent it: preserve . inside versions, - inside identifiers, : inside timestamps. Only clean the punctuation at the end of sentences.

When to use BM25 standalone (without semantic)

BM25 can be enough on its own in these cases:

A practical rule: if 90%+ of your queries are exact-match, BM25 standalone is enough. If there's a mix with conceptual queries, hybrid is better.

The limits of rank_bm25 (in-memory)

rank_bm25 loads the whole corpus into memory. It works well up to a point:

Beyond 1M docs, in-memory BM25 stops being practical. Capsule 06 covers Elasticsearch as the solution at scale — same algorithm, distributed.

Traps and common mistakes

Trap 1: applying aggressive preprocessing "because it's standard NLP"

Covered above. Stemming, stopword removal and aggressive lowercasing destroy what BM25 needs.

Trap 2: using the default tokenizer without checking

The mistake: a technical corpus with CamelCase, and the default tokenizer treats OAuth2PasswordBearer as a single token.

The symptom: queries with separated words (oauth2 password bearer) don't match docs that have the identifier joined.

How to prevent it: manually validate the tokenization with representative identifiers before populating the index.

Trap 3: forgetting to re-index when the tokenization changes

The mistake: you tune the tokenizer. You don't re-index. Queries use the new tokenizer while the docs in the index use the old one.

The symptom: queries that should match find nothing.

How to prevent it: any change to the tokenizer requires rebuilding the entire BM25 index. Version the tokenizer + the cache key.

Trap 4: comparing BM25 scores with cosine similarity

The mistake: you add BM25 scores to cosine scores to "fuse" them.

The symptom: the scores have completely different ranges. BM25 can go from 0 to 50+, cosine from 0 to 2. The sum produces meaningless rankings.

How to prevent it: use Reciprocal Rank Fusion (capsule 04), which combines rankings while ignoring absolute scores.

Trap 5: not eliminating the long-document bias

The mistake: some docs in your corpus are very long (10K+ chars). BM25 ranks them systematically higher on short queries.

The symptom: retrieval always returns the longest, irrelevant docs.

How to prevent it: BM25 already has length normalization (the b parameter in the formula). Verify it's enabled (default b=0.75). If very long docs still dominate, consider chunking more finely before indexing.

Trap 6: indexing text without useful metadata

The mistake: you index only the chunk's content in BM25. The metadata (title, section, author) is lost.

The symptom: queries that match metadata (e.g. the author's name) find nothing.

How to prevent it: concatenate the relevant metadata with the content before tokenizing:

def doc_for_bm25(chunk: dict) -> str:
    parts = [
        chunk.get("title", ""),
        chunk.get("section", ""),
        chunk.get("text", ""),
    ]
    return " ".join(filter(None, parts))


index = BM25Index([doc_for_bm25(c) for c in chunks])

Applied exercise

Scenario: you're an AI Engineer at a DevOps SaaS company. The data:

• 80K chunks of technical documentation (a mix of code + narrative text)
• Production log queries: 65% are the "code identifier + description" type (kubectl get pods, helm chart values, OAuth2PasswordBearer scopes)
• Current system: semantic search only, with OpenAI embeddings + cross-encoder rerank

Metrics:

• Precision@5: 84%
• Recall@5: 62%

Your job:

1. Decide whether BM25 would help here.
2. Design the implementation, including an appropriate tokenizer.
3. Estimate the expected impact.

# bm25_setup.py
import re
from rank_bm25 import BM25Okapi


def technical_tokenizer(text: str) -> list[str]:
    """A tokenizer for a DevOps technical corpus."""
    # Lowercase for ordinary tokens
    text_lower = text.lower()

    # 1. Standard tokens
    tokens = re.findall(r'\b\w+\b', text_lower)

    # 2. Identifiers with underscores (helm_chart, oauth2_password_bearer)
    underscore_tokens = re.findall(r'\b\w+_\w+\b', text_lower)
    tokens.extend(underscore_tokens)

    # 3. CamelCase split (OAuth2PasswordBearer → oauth, 2, password, bearer + the full token)
    camel_matches = re.findall(r'[A-Z][a-z]+|[A-Z]+(?=[A-Z][a-z])|[A-Z]+|\d+', text)
    tokens.extend([m.lower() for m in camel_matches])

    # 4. Error codes (ERR_TIMEOUT, HTTP 503)
    error_codes = re.findall(r'[A-Z][A-Z_0-9]{2,}|HTTP\s+\d{3}', text)
    tokens.extend([c.lower().replace(' ', '_') for c in error_codes])

    # 5. Versions (v1.27.3)
    versions = re.findall(r'\bv?\d+\.\d+(?:\.\d+)?\b', text_lower)
    tokens.extend(versions)

    # 6. Shell commands (kubectl, helm, etc. + flags)
    commands = re.findall(r'\b(?:kubectl|helm|docker|git)\s+\w+', text_lower)
    tokens.extend(commands)

    return tokens


def build_bm25_index(chunks: list[dict]) -> BM25Okapi:
    """Build a BM25 index over chunks with metadata."""
    docs_for_indexing = []
    for chunk in chunks:
        # Concatenate the relevant metadata + the content
        text = f"{chunk.get('title', '')} {chunk.get('section', '')} {chunk['content']}"
        docs_for_indexing.append(text)

    tokenized = [technical_tokenizer(d) for d in docs_for_indexing]
    return BM25Okapi(tokenized)

Category                   %       Current recall   Recall with BM25 hybrid
─────────────────────────────────────────────────────────────────────────
Identifiers                65%     50%              85% (BM25 head-on)
Shell commands             15%     55%              90% (BM25 nails it)
Versions                   10%     58%              92%
Purely semantic            10%     85%              85%

Expected global recall:
  0.65(0.85) + 0.15(0.90) + 0.10(0.92) + 0.10(0.85)
= 0.5525 + 0.135 + 0.092 + 0.085
= 0.865 ≈ 86%

Summary and next step

What you learned:

• BM25 is the modern evolution of TF-IDF: it ranks by the presence of exact tokens, normalizing for rarity and length.
• An in-memory implementation with rank_bm25 works up to ~1M docs. Beyond that, Elasticsearch.
• Tokenization is the most sensitive decision. For a technical corpus, a custom tokenizer that preserves CamelCase, snake_case, error codes and versions.
• Aggressive preprocessing (stemming, stopwords, lowercasing) destroys exactness. Avoid it in BM25.
• BM25 standalone is useful for very specific corpora (error codes, code search, SKUs). In most cases, hybrid (BM25 + semantic) wins.
• BM25 scores are NOT comparable with cosine similarity. You need Reciprocal Rank Fusion (capsule 04) to combine them.

Checkpoint: before moving on, you should be able to:

• Implement BM25 with rank_bm25 and a tokenizer appropriate for your domain.
• Diagnose tokenization problems with specific identifiers.
• Decide when BM25 standalone is enough vs when you need hybrid.

Next capsule: 04 — Reciprocal Rank Fusion (RRF).

You have two rankings (BM25 and semantic). How do you combine them into a single unified ranking? RRF is the standard algorithm — it combines rankings while ignoring absolute scores, robust to the range difference between techniques. Capsule 04 covers it with the implementation, tuning of the k parameter, and a comparison with the alternatives.

Resources

1. BM25 Paper — Robertson & Zaragoza (2009) — The complete foundational text
2. Okapi BM25 — Wikipedia — The formula and its variants
3. rank_bm25 — Python Library — The library's documentation
4. Elasticsearch BM25 Implementation — For scaling up later
5. Pinecone — Hybrid Search Theory — Context on fusion
6. Anthropic — Contextual Retrieval — A complementary technique

Estimated time: 30-35 minutes Next: 04-reciprocal-rank-fusion.md