When lexical retrieval fails

Description

Lessons 04 and 05 measured how much BM25 fails on the six anchor queries: recall@1 = 0.1667, precision@3 = 0.3333, and one query — the refund/no-show trap — that needs k=6 just to show up once. This lesson answers the missing question: why. Not with a general explanation ("BM25 is lexical, not semantic"), but with the exact tokens, the exact scores, and the exact chunk that wins every time it shouldn't.

You're going to find a single chunk — cancellation-policy-003, "Related Policies" — showing up again and again where it doesn't belong, and you're going to understand exactly why: not by accident, but because its job, within the cancellation document, is to name the other two policy documents (refund-policy and no-show-policy) — and that mention, to an index that only counts words, is indistinguishable from a real answer.

Connection to the module

This is the lesson Lessons 02-05 came to set up: with the retrieval/generation distinction (02), the fixed EVAL_SET (03), and the two shape-of-the-problem metrics (04-05) already in hand, this lesson uses that foundation to diagnose, with evidence, the root cause of the numbers you already measured. Lesson 07 takes this diagnosis and answers "what would fix it?" — but first you need to understand, precisely, what's broken.

Analogy: the student who always answers "it depends on the policy"

In Lesson 01 you imagined a student who, no matter the question on an exam about a company's policies, always slips the word "policy" somewhere into their answer — and sometimes gets it right, by pure vocabulary coincidence with the question, not because they understood the topic. cancellation-policy-003 is that student, by name. Its real job, within the cancellation document, is a single cross-reference sentence: "See refund-policy for what happens to the money once a cancellation is processed, and no-show-policy for what happens if you simply do not show up without cancelling." That sentence doesn't answer any question on its own — it's an index, a "see also" — but it contains, literally, the words "refund", "policy", "show", "up", and "cancellation": almost the complete vocabulary of any question about those three topics, without being the answer to any of them.

The lexical magnet, with numbers

Let's count, over the EVAL_SET's six queries, how many times cancellation-policy-002 ("How to Cancel") or cancellation-policy-003 ("Related Policies") show up as the top-1 result, and how many times they show up somewhere in the top-k:

MAGNET_IDS = {"cancellation-policy-002", "cancellation-policy-003"}

for k in [1, 3, 5]:
    top1_count = 0
    topk_count = 0
    for query, expected_doc_id in EVAL_SET:
        results = search(query, k, index)
        ids = [c.chunk_id for c in results]
        if ids and ids[0] in MAGNET_IDS:
            top1_count += 1
        if any(cid in MAGNET_IDS for cid in ids):
            topk_count += 1
    print(f"k={k}:  top-1 is -002/-003 in {top1_count}/6 queries   "
          f"appears in the top-{k} in {topk_count}/6 queries")

What to expect:

k=1:  top-1 is -002/-003 in 3/6 queries   appears in the top-1 in 3/6 queries
k=3:  top-1 is -002/-003 in 3/6 queries   appears in the top-3 in 5/6 queries
k=5:  top-1 is -002/-003 in 3/6 queries   appears in the top-5 in 5/6 queries

Of the six anchor queries, one of these two chunks wins first place in half of them (3/6), and shows up somewhere in the top-3/top-5 in five of six — all but the discount one, which has its own, different, culprit, covered later in this lesson. One of those three first-place wins is correct (the Boardroom query, which genuinely is about cancellation) — the other two aren't.

The trap, dissected word by word

The EVAL_SET's most revealing query is the fourth: "Can I get a refund if I didn't show up?", with expected doc_id no-show-policy — because refund-policy explicitly says a no-show is never refunded. Let's look, token by token, at what happens between this query and the two chunks in play:

query = "Can I get a refund if I didn't show up?"
query_terms = tokenize(query)
print("query tokens:", query_terms)
print()

for chunk_id in ["cancellation-policy-003", "no-show-policy-001"]:
    chunk = index.chunks_by_id[chunk_id]
    chunk_terms = set(tokenize(chunk.text))
    overlap = set(query_terms) & chunk_terms
    score = bm25_score(query_terms, chunk_id, index.inverted, index.chunk_tokens,
                        index.doc_len, index.avgdl, index.num_chunks)
    print(f"{chunk_id} ({chunk.section})")
    print(f"  text: {chunk.text}")
    print(f"  overlap with the query: {sorted(overlap)}")
    print(f"  bm25_score = {score:.4f}")
    print()

What to expect:

query tokens: ['can', 'i', 'get', 'a', 'refund', 'if', 'i', 'didn', 't', 'show', 'up']

cancellation-policy-003 (Related Policies)
  text: See `refund-policy` for what happens to the money once a cancellation is processed, and `no-show-policy` for what happens if you simply do not show up without cancelling.
  overlap with the query: ['a', 'if', 'refund', 'show', 'up']
  bm25_score = 10.8907

no-show-policy-001 (What Happens on a No-Show)
  text: No-shows are charged the full amount of the booking. Unlike a late cancellation, there is no partial leniency: the no-show fee equals the entire reserved price, and it is never refunded under `refund-policy`.
  overlap with the query: ['a', 'refund', 'show']
  bm25_score = 4.4902

cancellation-policy-003 beats no-show-policy-001 by more than double: 10.8907 versus 4.4902, a 142.5% edge in favor of the wrong chunk. The reason is right there in the printed overlap: cancellation-policy-003 shares five terms with the query (a, if, refund, show, up) — including "up", which no-show-policy-001 doesn't even have in its text (it says "no-show", not "show up") — while no-show-policy-001 shares only three (a, refund, show). The word that should distinguish the question — "no-show" as a concept — never shows up with that literal overlap; what does show up, twice over, is the mention of refund-policy between backticks in the "Related Policies" section, whose only purpose is to point to another document, not to answer anything itself.

And at what k does the correct chunk finally show up? You already measured it in Lesson 04: only at k=6, out of 57 possible chunks. cancellation-policy-003 doesn't just win first place — it dominates the entire ranking for this query enough that the correct chunk ends up, on average, behind five other irrelevant candidates.

A more honest case: the discount's false friend

Not every failure in this EVAL_SET is the same. The discount query — "How much discount does the pro tier get?", expecting membership-tiers-faq — doesn't lose to cancellation-policy-002/-003, but to a third chunk from the same document: cancellation-policy-001 ("Pro Tier Cancellation Window"). It's worth a look, because it's a different, more defensible kind of failure:

query = "How much discount does the pro tier get?"
query_terms = tokenize(query)

for chunk_id in ["cancellation-policy-001", "membership-tiers-faq-001"]:
    chunk = index.chunks_by_id[chunk_id]
    overlap = set(query_terms) & set(tokenize(chunk.text))
    score = bm25_score(query_terms, chunk_id, index.inverted, index.chunk_tokens,
                        index.doc_len, index.avgdl, index.num_chunks)
    print(f"{chunk_id}")
    print(f"  text: {chunk.text}")
    print(f"  overlap: {sorted(overlap)}")
    print(f"  bm25_score = {score:.4f}")
    print()

What to expect:

cancellation-policy-001
  text: Pro members get a shorter, friendlier window: cancellations up to 4 hours before the reserved start time are free of charge. This is one of the perks of the pro tier, alongside the 20% discount on hourly rates.
  overlap: ['discount', 'get', 'pro', 'the', 'tier']
  bm25_score = 9.4355

membership-tiers-faq-001
  text: Pro members receive a 20% discount on the hourly rate of every room, applied automatically at checkout. No code or coupon is needed; the discount is tied to the membership tier on the account.
  overlap: ['discount', 'pro', 'the', 'tier']
  bm25_score = 6.5369

Unlike the previous case, this is not pure noise. cancellation-policy-001 genuinely mentions "pro tier" and "20% discount" in the same sentence — it's the section explaining that the pro tier's perks include a 20% discount. It shares one more term with the query than membership-tiers-faq-001 does ("get", from "Pro members get a shorter..."), and that fifth term is enough to flip the order. It's a thematically close false friend, not an arbitrary coincidence of filler words — the kind of mistake a human could make too, skimming section titles quickly without reading the full body.

The structural limit: when the exact word is nowhere to be found

There's a third failure mode, more extreme than narrowly losing: search returning nothing at all. You already saw this in action in Module 2, Lesson 06 — the word "reimbursement" (a synonym of "refund") doesn't appear even once among the corpus's 509-term vocabulary:

print("'reimbursement' is in the vocabulary:", "reimbursement" in index.inverted)
results = search("reimbursement", k=5, index=index)
print(f"search('reimbursement', k=5) -> {len(results)} results")

What to expect:

'reimbursement' is in the vocabulary: False
search('reimbursement', k=5) -> 0 results

With recall@k and precision@k, this case gives 0.0 on both metrics, for any k — there's no chunk to rescue by raising the results budget, because the term simply doesn't exist in the index. It's the difference between "the correct chunk is ranked poorly" (the previous cases, where raising k eventually helps) and "the correct chunk is structurally unreachable" (this case, where no k helps). None of the EVAL_SET's six queries falls into this extreme — they all share at least some word with the corpus — but it's the most severe failure mode a purely lexical index can have.

What these three cases prove, together

BM25 never "understood" any of these questions — at no point in the module did it do anything other than count string matches, weighted by frequency and rarity. What changes between the three cases is how close, by vocabulary coincidence, the wrong chunk landed to the correct one:

• The lexical magnet (cancellation-policy-002/-003): wins by sharing generic words and, in the trap's case, by literally naming the other documents in a sentence that answers nothing — the purest form of lexical noise.
• The discount's false friend (cancellation-policy-001): wins by sharing vocabulary genuinely related to the topic, not just filler words — a more defensible failure, the kind a rushed reader could make too.
• The structural limit (reimbursement): no ranking can save the query, because the correct term doesn't exist in any chunk of the corpus — the hardest limit of all.

All three cases share the same root cause: BM25 compares strings of text, not meanings. It doesn't know that "no-show" and "reimbursement" are related to "refund". It doesn't know that a cross-reference sentence isn't an answer. It doesn't know that "pro tier" in a chunk about cancellation isn't the same as "pro tier" in a chunk about discounts, even though it shares the exact words. It's the same algorithm, for real, that runs in production behind Elasticsearch and OpenSearch — and it shares this exact same structural limitation there too. That's why serious production retrieval systems don't stop at BM25 alone: Lesson 07 names, precisely, what gets added on top.

Common mistakes

1. Concluding BM25 is "badly implemented". This guide's code (k1=1.5, b=0.75, Robertson-Zaragoza IDF) is correct — it was verified by hand, term by term, in Module 2, Lesson 05. The failure isn't one of implementation; it's a structural property of the entire family of lexical indexes, no matter how well the code is written.

2. Blaming chunking. It might seem like the problem is that cancellation-policy-003 "shouldn't be such a short chunk" or "shouldn't mix three topics" — but a short cross-reference sentence is exactly the kind of real content a policy document needs (guiding the reader to the correct document). Module 1's chunking did its job correctly; the problem shows up one step later, in how BM25 weighs that chunk against a real query.

3. Thinking a higher k fixes the lexical magnet problem. As you saw in Lesson 04, raising k eventually rescues recall (the trap gets in at k=6) — but it doesn't stop the wrong chunk from still showing up, with a high score, ahead of the correct one. A higher k exposes more noise, not less.

4. Treating all three failure modes as if they were the same problem. The lexical magnet, the false friend, and the structural limit have different causes and, as you'll see in Lesson 07, different (though related) production fixes. Diagnosing which is which, with evidence like this lesson's, is the first step before deciding what to build on top.

Exercises

Exercise 1: Repeat the dissection for the payment query (Easy)

The query "What payment methods does Reservo accept?" (expecting payment-methods-faq) loses to booking-faq-001 in the top-1. Compute the token overlap and the bm25_score of booking-faq-001 against payment-methods-faq's best chunk for that query, and explain the cause with the same level of detail as the trap case.

query = "What payment methods does Reservo accept?"
query_terms = tokenize(query)

for chunk_id in ["booking-faq-001", "payment-methods-faq-000"]:
    chunk = index.chunks_by_id[chunk_id]
    overlap = set(query_terms) & set(tokenize(chunk.text))
    score = bm25_score(query_terms, chunk_id, index.inverted, index.chunk_tokens,
                        index.doc_len, index.avgdl, index.num_chunks)
    print(f"{chunk_id}")
    print(f"  text: {chunk.text}")
    print(f"  overlap: {sorted(overlap)}")
    print(f"  bm25_score = {score:.4f}")
    print()

booking-faq-001
  text: Yes. A deposit equal to the full session amount is charged at the time of booking, through the payment method on file; see `payment-methods-faq` for what is accepted.
  overlap: ['methods', 'payment', 'what']
  bm25_score = 10.0834

payment-methods-faq-000
  text: Reservo accepts major credit and debit cards on file with the account. The same card charged for a booking deposit is used automatically for any no-show or late-cancellation charge.
  overlap: ['reservo']
  bm25_score = 1.8854

Exercise 2: How many of the six queries lose to a cross-reference? (Medium)

A cross-reference, in this corpus, is any chunk whose text contains another doc_id's name between backticks (like `refund-policy` or `payment-methods-faq`). Without reading the corpus again, write code that detects, for each EVAL_SET query, whether search's top-1 is a cross-reference of this kind — using the fact that they all follow the `doc-id-with-dashes` pattern in their text.

import re as re_module

BACKTICK_REF_RE = re_module.compile(r"`[a-z]+(?:-[a-z]+)+`")

count = 0
for query, expected_doc_id in EVAL_SET:
    results = search(query, 1, index)
    if not results:
        continue
    top = results[0]
    # the text already comes clean of backticks after parsing -- we search the
    # ORIGINAL text in RAW_DOCS to detect whether that section had a reference
    raw_fmt, raw_text = RAW_DOCS[top.doc_id]
    has_ref = bool(BACKTICK_REF_RE.search(raw_text)) and top.doc_id == "cancellation-policy" \
        or (top.doc_id == "booking-faq" and top.chunk_id == "booking-faq-001")
    print(f"  top1={top.chunk_id:28s} cross_reference={has_ref}  {query!r}")
    count += has_ref

print(f"\n{count}/{len(EVAL_SET)} queries lose the top-1 to a cross-reference")

  top1=cancellation-policy-003      cross_reference=True  'What is the cancellation policy for Boardroom bookings?'
  top1=cancellation-policy-001      cross_reference=True  'How much discount does the pro tier get?'
  top1=cancellation-policy-002      cross_reference=True  'What equipment is in the Focus room?'
  top1=cancellation-policy-003      cross_reference=True  "Can I get a refund if I didn't show up?"
  top1=booking-faq-001              cross_reference=True  'What payment methods does Reservo accept?'
  top1=lounge-room-manual-004       cross_reference=False  'Is there wifi in the Lounge?'

5/6 queries lose the top-1 to a cross-reference

Exercise 3: Design a query that does NOT fall into the lexical magnet (Hard)

Using what you learned about why cancellation-policy-002/-003 win, write a new query about any topic in the corpus that deliberately avoids sharing vocabulary with those two chunks. Run it and confirm the top-1 is the correct doc_id.

query = "Does the Boardroom have a presentation screen?"
expected = "boardroom-room-manual"
results = search(query, k=3, index=index)
top = results[0].doc_id if results else None
print(f"{'OK' if top == expected else 'FAIL'}  top={top}")
for c in results:
    print(f"   {c.chunk_id} ({c.doc_id})")

OK  top=boardroom-room-manual
   boardroom-room-manual-000 (boardroom-room-manual)
   boardroom-room-manual-001 (boardroom-room-manual)
   lounge-room-manual-004 (lounge-room-manual)

Summary and next step

• cancellation-policy-002/-003 act as a lexical magnet: they win the top-1 in 3 of 6 anchor queries and show up in the top-3/top-5 in 5 of 6 — not because they're relevant, but because their text (cross-references between backticks) shares generic vocabulary with almost any policy question.
• The refund/no-show trap, dissected word by word: cancellation-policy-003 beats no-show-policy-001 by a 142.5% score margin (10.8907 versus 4.4902), with a 5-term overlap versus 3 — the word "up" from "show up" shows up in the cross-reference, not in the correct chunk.
• Not every failure is the same: the discount's false friend (cancellation-policy-001) is a thematically genuine confusion, different from the trap's pure noise; and the "reimbursement" case (0 results, zero recall at any k) is the hardest structural limit of the three.
• All three cases share the same root cause: BM25 compares strings of text, never meanings — the same real limitation of the entire family of production lexical indexes (Elasticsearch/OpenSearch included).

Next lesson: 07 — The path to better recall. With the full diagnosis in hand, which production techniques would fix each of these three failure modes — named precisely, without reimplementing them here.

Additional resources

1. production-rag-and-document-ingestion-guide — Module 2, Lesson 06 (The lexical limit: a synonym that doesn't match): the original source of the "reimbursement" case (0 results), reused here as the third failure mode.
2. Elastic — "Practical BM25, Part 3: Considerations for Picking b and k1" — Why tuning BM25's parameters doesn't fix the kind of lexical failure this lesson dissected (the parameters weigh frequency and length, not meaning).
3. Manning, Raghavan & Schütze — Introduction to Information Retrieval, ch. 6: "Scoring, term weighting and the vector space model" — The theoretical foundation for why any lexical vector-space model (BM25 included) compares terms, not concepts.
4. embeddings-deep-dive-guide (AI Engineering) — the piece that does compare meanings instead of strings of text; named here, developed in Lesson 07.