Module 6: Evaluating Retrieval Quality

The path to better recall

Next →

Description

Lesson 06 left a precise diagnosis, backed by evidence: BM25 fails against the anchor queries in three distinct ways — the lexical magnet (cross-references that win through shared vocabulary, without being the answer), the thematic false friend (a related but incorrect chunk sharing genuine vocabulary), and the structural limit (a word that simply doesn't exist in the index's vocabulary). This lesson answers the question an honest diagnosis always leaves open: what gets built to fix this in production?

You're not going to implement any of what follows — that would break this guide's boundary, fixed since its design. You're going to leave this lesson knowing, precisely, which technique fixes each specific failure mode you measured in Lesson 06, at which stage of the pipeline it acts, and which AI Engineering guide actually builds it.

Connection to the module

This is the module's conceptual closing lesson: it takes Lesson 06's diagnosis and turns it into a map of "what to learn next". Lesson 08 (the mini-project) doesn't build any of these techniques — it assembles the full evaluation harness, exactly as defined across Lessons 03-05, as the module's final deliverable.

The map: three techniques, three pipeline stages

Before the details, the big picture. There are three ways to improve retrieval on top of what this guide already built, and they aren't interchangeable — each one intervenes at a different point in the pipeline:

                    ┌─────────────────────────────────────────────────┐
                    │  query                                          │
                    └───────────────────────┬─────────────────────────┘
                                             ▼
   ┌─────────────────────────────────────────────────────────────────────┐
   │  STAGE 1: RETRIEVAL (candidates)                                     │
   │  BM25 (this guide) ─── or ─── semantic embeddings ─── or ─── hybrid  │
   │  → produces a list of candidates, potentially large (top-50)         │
   └───────────────────────┬────────────────────────────────────────────┘
                            ▼
   ┌─────────────────────────────────────────────────────────────────────┐
   │  STAGE 2: RE-RANKING (optional, over stage 1's candidates)           │
   │  A cross-encoder rereads each candidate (query, chunk) pair with     │
   │  more precision, and reorders them -- never adds candidates that     │
   │  weren't already there                                               │
   └───────────────────────┬────────────────────────────────────────────┘
                            ▼
                    final top-k, what the agent sees

The distinction between the two stages is this lesson's most important piece: re-ranking can only reorder what retrieval already brought in as a candidate. If Stage 1 never considers a chunk — as happened with "reimbursement", which didn't even produce a candidate because the term doesn't exist in the inverted index — no re-ranker, no matter how sophisticated, can rescue it afterward. This isn't a minor technical detail: it determines which technique fixes which failure, as you'll see next.

Technique 1: re-ranking with cross-encoders

What it does: BM25 scores every chunk independently — it never compares the query's full text against the chunk's full text at the same time, it only counts term matches. A cross-encoder is a model that does exactly that: it receives the query and a candidate chunk together, in the same pass, and produces a relevance score that can tell "this chunk mentions the topic" apart from "this chunk answers the question" — because it reads both with the other's full context.

Where it acts: after retrieval, over a short list of candidates (typically BM25's top-20 or top-50) — never over the full corpus, because a cross-encoder is much more expensive to run per candidate than BM25.

Which Lesson 06 failures it would fix:

  • The lexical magnet. A cross-encoder reading "See refund-policy for what happens..." alongside the query "Can I get a refund if I didn't show up?" has the capacity to recognize that sentence is a cross-reference — a "see also" — not an answer, something BM25 structurally can't do because it never reads the two texts together.
  • The discount's false friend. Similarly, a cross-encoder can tell that cancellation-policy-001 mentions the discount only in passing (as an extra perk of the pro cancellation window), while membership-tiers-faq-001 is the section entirely dedicated to answering "how much discount?" — a distinction of emphasis BM25's term counting doesn't capture.

What it would NOT fix: the structural limit. If BM25 never produced a candidate for "reimbursement" — because the term doesn't exist in its inverted index — there's no candidate list for the cross-encoder to act on. Re-ranking improves the order of what got retrieved; it doesn't expand what gets retrieved.

Where it's built: advanced-rag-techniques-guide (AI Engineering) — the full implementation of a cross-encoder re-ranker, with a real model, on a real corpus.

Technique 2: semantic embeddings and hybrid search

What it does: an embedding converts a text (query or chunk) into a numerical vector, trained so that texts with similar meaning end up with nearby vectors in that space — even if they don't share a single word. Unlike BM25, which compares exact strings of text, an embedding compares meaning. Hybrid search combines the two signals — BM25's lexical score and the embedding's semantic similarity — into a single ranking, typically with a weighted sum or with reciprocal rank fusion.

Where it acts: in Stage 1 (retrieval), as a replacement for or complement to BM25 — not as a later step, but as a second source of candidates that gets merged with BM25's before returning any result.

Which Lesson 06 failures it would fix:

  • The structural limit — this is the only one of the three techniques that fixes it. An embedding trained on enough English text places "reimbursement" and "refund" close together in vector space, because they're synonyms in real language use, regardless of sharing zero letters in common. Semantic search would find refund-policy for that query even though the exact word never appears in the corpus — something no BM25 tuning can achieve, because the problem isn't one of ranking, it's that the term isn't even in the indexed vocabulary.
  • The refund/no-show trap, partially. "No-show" and "refund" aren't synonyms — they're related but distinct concepts — so an embedding doesn't automatically guarantee no-show-policy wins; but an embedding of the full query ("I didn't show up, do I get my money back?") can capture the conceptual relationship between "didn't show up" and "no-show" in a way the literal "refund"/"show"/"up" match from cancellation-policy-003 can't imitate as well.

What it would NOT fix as well as re-ranking: the lexical magnet. A pure embedding can also get confused by a short cross-reference sentence if that sentence genuinely mentions the right topics — "refund-policy", "no-show-policy" — even in passing; a cross-encoder's real advantage is that it reads the full relationship between query and chunk, not just general topic closeness.

Where it's built: embeddings-deep-dive-guide (AI Engineering) covers embedding theory — architecture, training, how a model learns that "reimbursement" and "refund" are close together; vector-databases-fundamentals-guide (AI Engineering) covers how a real vector index gets built and queried under the hood (HNSW, IVF, PQ) and how a production vector database gets operated (ChromaDB, Pinecone, Weaviate, Qdrant); advanced-rag-techniques-guide covers the BM25+embeddings hybrid fusion itself.

Technique 3: query expansion and HyDE

What it does: instead of changing how the query gets compared against the corpus, this family of techniques changes the query itself before searching. Query expansion adds synonyms or reformulations to the original query (for example, automatically adding "reimbursement" when the user wrote "refund"). HyDE (Hypothetical Document Embeddings) asks a model to generate a hypothetical answer to the question, and searches with that hypothetical answer's embedding instead of the original question's embedding — the intuition being that a hypothetical answer resembles a real answer, in embedding space, more than the question itself does.

Where it acts: before Stage 1 — it transforms the query, and then passes it to BM25, to an embedding, or to both.

Which Lesson 06 failures it would fix: in principle, both the lexical magnet (if the expansion adds terms that favor the correct chunk) and the structural limit (if the expansion adds the missing synonym) — but with fewer guarantees than the previous two techniques, because it depends on the quality of the expansion or the generated hypothetical answer, which can introduce its own noise.

Where it's built: advanced-rag-techniques-guide (AI Engineering).

Summary table: what fixes what

Failure measured in L06Re-ranking (cross-encoder)Hybrid search (BM25 + embeddings)AI Engineering guide
Lexical magnet (cancellation-policy-002/-003)✅ Strong⚠️ Partialadvanced-rag-techniques-guide
False friend (cancellation-policy-001, discount)✅ Strong⚠️ Partialadvanced-rag-techniques-guide
Refund/no-show trap⚠️ Partial✅ Strongadvanced-rag-techniques-guide + embeddings-deep-dive-guide
Structural limit ("reimbursement", 0 results)❌ Falls short✅ Strongembeddings-deep-dive-guide + vector-databases-fundamentals-guide

No row says "one single technique fixes everything" — and that's the table's real conclusion. A serious production system doesn't choose between BM25 and embeddings: it combines them (hybrid search) and adds re-ranking on top for the final stretch of the ranking, precisely because each technique covers a different blind spot than the others.

What it would look like, in shape (not run)

Just to make the shape clear — this is concept, not code executed in this guide, because sentence-transformers and cross-encoder/embedding models aren't preinstalled and require network access to download weights — here's what integrating a re-ranker on top of this guide's search would look like:

# CONCEPT -- not executed in this guide (requires sentence-transformers + network,
# outside the $0/no-network environment of production-rag-and-document-ingestion-guide).
# Actually built in advanced-rag-techniques-guide (AI Engineering).

def search_with_reranking(query: str, k: int, index: Index, rerank_top_n: int = 20):
    candidates = search(query, k=rerank_top_n, index=index)  # BM25, Stage 1
    reranked = cross_encoder.rerank(query, [c.text for c in candidates])  # Stage 2
    return reranked[:k]

The shape matters more than the code: this guide's lexical retrieval remains Stage 1 — it doesn't get discarded, it's used as the cheap filter that reduces 57 chunks to a handful of reasonable candidates — and the cross-encoder, much more expensive per candidate, only runs on those few, never on the full corpus. It's exactly the same "budget" principle you already saw with k in Module 2: never ask the most expensive piece of the pipeline to process more than necessary.

Why this guide stops here

It's not that these techniques matter less than what actually got built — it's a boundary decision, the same one fixed since this guide's design. sentence-transformers, chromadb, and cross-encoder models aren't preinstalled in this environment and require network access to install or download weights, violating the $0/no-network rule this guide has held since Module 1. Beyond that practical reason, there's a pedagogical one: understanding why BM25 fails, with real numbers like Lesson 06's, is the real prerequisite for using these more expensive techniques well — without that diagnosis, "add embeddings" turns into a production superstition ("they say it helps") instead of an informed decision about which specific failure is being fixed.

Common mistakes

  1. Thinking hybrid search or re-ranking are "optional, just for a little extra improvement". As the table shows, there are failures — the structural limit in particular — that BM25 alone, no matter how well tuned (k1, b, or any search k), structurally cannot fix. It's not an incremental improvement; it's a different class of failure that needs a different piece.

  2. Confusing re-ranking with "a better BM25". Re-ranking doesn't replace BM25 — it operates afterward, on what BM25 already brought in. If BM25 never put the correct chunk on the candidate list, there's no re-ranking possible.

  3. Assuming hybrid search "fixes everything automatically". As seen in the lexical magnet row, a pure embedding can also get confused by a thematically related cross-reference sentence — hybrid search improves the picture, it doesn't solve it with a full guarantee. This lesson's table marks "partial" on purpose, not "solved".

  4. Jumping straight to implementing these techniques without Lesson 06's diagnosis. Choosing between re-ranking, hybrid search, or query expansion without knowing which of the three failure modes dominates your real case is choosing blind — the value of having measured recall@k/precision@k and dissected the failures isn't just academic, it's what informs what to build next.

Exercises

Exercise 1: Classify three new failures (Easy)

For each of these three hypothetical situations on Reservo's corpus, decide which of this lesson's three techniques (re-ranking, hybrid search, query expansion) would be the first reasonable option:

(a) A query uses the word "gratis" instead of "free" and finds nothing, because the corpus is in English. (b) A chunk that only mentions "Boardroom" in a list of rooms with a physical door accidentally wins the top-1 for a query about Boardroom's specific equipment. (c) A query uses "termination fee" instead of "cancellation" and "no-show" — a related concept but with vocabulary completely different from the corpus's.

See solution

(a) Query expansion (or, if product policy allows it, forcing the query's language) — this is a problem of vocabulary completely absent due to language, not a synonym within the same language; neither re-ranking nor an embedding trained only in English would reliably fix this without a prior translation.

(b) Re-ranking with a cross-encoder — it's the same pattern as Lesson 06's lexical magnet/false friend: a chunk that mentions the topic in passing beats one that genuinely develops it, and a cross-encoder that reads query and chunk together is the most direct tool for that distinction.

(c) Hybrid search with semantic embeddings — "termination fee" and "cancellation"/"no-show" are conceptually close but lexically distinct, the same pattern as Lesson 06's "reimbursement"/"refund" — the structural limit only a semantic embedding fixes.

Exercise 2: Explain why stage order matters (Medium)

Explain, in two or three sentences, why running a re-ranking cross-encoder before BM25 (instead of after) would be a bad design idea, even if it were technically possible.

See solution

A cross-encoder is much more expensive to run per candidate than BM25 — it evaluates the query and the chunk together, with a full model, instead of just counting term matches with simple arithmetic. Running it over the corpus's full 57 chunks (or, worse, over a real production corpus with millions of chunks) instead of over a handful of candidates would already be computationally costly at a scale that isn't justified. The correct order — BM25 first, as a cheap filter that narrows the corpus down to a reasonable top-20/50, and the cross-encoder afterward, only over those few candidates — is exactly the same "budget" principle this guide already used with search's k parameter: never spend the most expensive piece of the pipeline on more work than necessary.

Exercise 3: Design the EVAL_SET you'd use to compare BM25 alone against hybrid search (Hard)

Without implementing anything, describe how you'd extend this module's harness (recall_at_k, precision_at_k, evaluate) to compare, with numbers, whether a hypothetical hybrid search system improves on this guide's BM25. What would change, and what would stay exactly the same?

See solution

What stays exactly the same: the EVAL_SET (the same six queries with the same expected doc_id — the ground truth doesn't change because the system being evaluated changed) and the recall_at_k/precision_at_k functions (they measure the same thing, regardless of where the results came from). What would change: instead of a single BM25 index, you'd have two retrieval systems — search_bm25(query, k, index) (this guide's) and a hypothetical search_hybrid(query, k, hybrid_index) — and you'd run evaluate(EVAL_SET, index, k) once against each, over the same range of k, reporting both results side by side (like the recall@k/precision@k table from Lessons 04-05, but with an extra column per system). An honest comparison requires running exactly the same EVAL_SET, with the same ground truth, against both systems — if the EVAL_SET changed between one run and the other, you'd no longer be comparing two systems, but two different evaluations, and the number would stop meaning anything. This is, in essence, how advanced-rag-techniques-guide would evaluate whether a new technique genuinely improves on the baseline — this module's same harness, reused as the measuring stick.

Summary and next step

  • Re-ranking with cross-encoders operates after retrieval, over a short candidate list, and fixes the lexical magnet and the thematic false friend well — but it can never rescue a chunk retrieval never considered a candidate.
  • Hybrid search (BM25 + embeddings) operates within retrieval, and is the only one of this lesson's three techniques that fixes the structural limit (a term, like "reimbursement", absent from the indexed vocabulary).
  • Query expansion/HyDE transforms the query before searching — useful for both kinds of failure, with fewer guarantees than the other two.
  • No single technique fixes everything on its own — a serious production system combines hybrid retrieval with re-ranking, precisely because they cover different blind spots. This guide stops at the diagnosis ($0, no network); advanced-rag-techniques-guide, embeddings-deep-dive-guide, and vector-databases-fundamentals-guide (AI Engineering) actually build all three techniques.

Next lesson: 08 — Mini-project: a retrieval evaluation harness. The module's close: recall_at_k, precision_at_k, and evaluate together, in an end-to-end script, with a final report over the full index.

Additional resources

  1. advanced-rag-techniques-guide (AI Engineering) — Re-ranking with cross-encoders, BM25+embeddings hybrid search, query expansion/HyDE, RAGAS. The full implementation of the three techniques named in this lesson.
  2. embeddings-deep-dive-guide (AI Engineering) — Semantic embedding theory: architecture, training, why "reimbursement" and "refund" end up close together in vector space.
  3. vector-databases-fundamentals-guide (AI Engineering) — How a real vector index gets built and queried under the hood (HNSW, IVF, PQ) and how ChromaDB/Pinecone/Weaviate/Qdrant get operated in production.
  4. Nogueira & Cho — "Passage Re-ranking with BERT" — The paper that originated the pattern of cross-encoder re-ranking over a lexical retriever's candidates, exactly the pattern named in this lesson.
  5. Gao et al. — "Precise Zero-Shot Dense Retrieval without Relevance Labels" (HyDE) — The original HyDE paper, the query expansion technique named in this lesson.
← PreviousWhen lexical retrieval failsNext →Mini-project: a retrieval evaluation harness