Module 4: Re-ranking — the second stage that turns mediocre retrieval into excellent retrieval
Module 4: Re-ranking — the second stage that turns mediocre retrieval into excellent retrieval
Module overview
Up to this point, every technique we covered manipulates the query (M03: optimization), the documents (M02: chunking), or the search metric. Re-ranking is structurally different: it adds a second scoring stage after the initial retrieval. The idea is simple but powerful: retrieval with cosine similarity is fast but noisy. Instead of fighting that limitation, we accept its nature and add a second step that refines the candidates with a more precise model.
Query
│
▼
┌──────────────────────────────────┐
│ Stage 1: retrieval with cosine │
│ Top-30 candidates (fast but │
│ noisy — some are irrelevant) │
└─────────────┬────────────────────┘
│
▼
┌──────────────────────────────────┐
│ Stage 2: re-ranking │
│ Re-score each (query, doc) pair │
│ with a more sophisticated model │
└─────────────┬────────────────────┘
│
▼
Final top-5
(high precision)
This two-stage architecture — a fast bi-encoder + a precise cross-encoder — is the "state of the art" pattern in production RAG. Typical gain: precision goes up 15-25 points. This capsule introduces the module, the three types of re-ranker you'll compare (local cross-encoder, LLM-based, Cohere managed), and how to decide which one to use.
By the end of this module you'll be able to:
- ✅ Explain why cosine similarity has an intrinsic ceiling that justifies re-ranking
- ✅ Implement the three main techniques: local cross-encoder, LLM-based, Cohere Rerank
- ✅ Optimize the operational trade-offs (n_results, batching, caching)
- ✅ Apply a decision framework to pick the right technique for your context
- ✅ Build a production-ready re-ranking system with A/B testing and metrics
Estimated module time: 3-4 hours (8 capsules).
Why re-ranking is the improvement with the best ROI
If your RAG system is in production and quality needs to improve, re-ranking is usually the first intervention with the best gain/effort ratio. Compared with other options:
| Technique | Typical precision gain | Engineering effort | Recurring cost |
|---|---|---|---|
| Switching the embedding model (small → large) | +3-5% | 1 day + re-ingest | More expensive embeddings |
| Improving chunking | +5-10% | 2-3 days + re-ingest | $0 |
| Query optimization | +5-15% (varies) | 1-2 weeks | $20-200/month |
| Re-ranking (cross-encoder) | +15-25% | 1 day | $0 |
| Hybrid search | +5-15% | 1-2 weeks + re-index | $0 |
Re-ranking typically gives more gain for less effort. That's why it's the second most common intervention after reasonable chunking. The reason is structural: cosine similarity has an inherent ceiling, and adding a second scoring stage breaks through it.
The concrete problem: cosine similarity has a ceiling
Picture this query and the top-5 cosine similarity returns:
Query: "how do I implement OAuth2 authentication in FastAPI?"
Top-5 with cosine:
#1 cosine: 0.85 "FastAPI OAuth2 implementation guide"
✅ Relevant: specific to the query
#2 cosine: 0.83 "OAuth2 authentication overview"
❌ Irrelevant: generic, never mentions FastAPI
#3 cosine: 0.82 "FastAPI security dependencies"
✅ Relevant: covers OAuth2 within security
#4 cosine: 0.80 "OAuth2 specification (RFC 6749)"
❌ Irrelevant: a theoretical spec, not implementation
#5 cosine: 0.79 "FastAPI authentication tutorial"
✅ Relevant: implementation-focused
Precision@5: 3/5 = 60%
The cosine scores are all similar (0.79-0.85). The metric doesn't distinguish "FastAPI-specific" from "generic OAuth2" — both talk about the topic, both rank high.
Re-ranking with a cross-encoder:
Top-5 after the rerank (over the top-30 candidates):
#1 score: 9.2 "FastAPI OAuth2 implementation guide" ✅
#2 score: 8.8 "FastAPI security dependencies" ✅
#3 score: 8.5 "FastAPI authentication tutorial" ✅
#4 score: 8.1 "OAuth2 with FastAPI code examples" ✅
#5 score: 7.4 "Securing FastAPI endpoints with OAuth2" ✅
Precision@5: 5/5 = 100%
The cross-encoder analyzes the (query, doc) pair as a single unit. It detects that "FastAPI-specific" answers the query and "generic OAuth2" doesn't. The absolute score reflects real relevance.
Result: the false positives (docs about generic OAuth2, the RFC) get filtered out. The downstream LLM receives focused context.
The three techniques you'll learn
Technique 1: local cross-encoder (a reasonable default)
from sentence_transformers import CrossEncoder
model = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-12-v2')
scores = model.predict([(query, doc) for doc in candidates])- ✅ Free (runs locally)
- ✅ Latency ~150ms for 20 candidates
- ✅ Quality: typically +20% precision
- ❌ Suboptimal multilingual (the model was trained in English)
When: typical production with an English monolingual corpus. It's the default option covering 80% of cases.
Capsule 03 covers it in detail.
Technique 2: LLM-based (maximum quality, expensive)
score = llm_rerank_pair(query, document) # GPT-4o-mini scores 0-10- ✅ Quality: typically +24% precision
- ❌ Latency +1500ms for 20 candidates
- ❌ Cost +$0.002/query
- ✅ Excellent multilingual
When: critical domains (legal, medical, financial) where an extra 3-5% precision justifies the cost.
Capsule 04 covers it.
Technique 3: Cohere Rerank (managed, in between)
import cohere
co = cohere.Client(api_key)
results = co.rerank(query=query, documents=docs, top_n=5)- ✅ Quality: typically +21% precision
- ✅ Latency ~250ms
- ⚠️ Cost ~$0.002 per 1K docs
- ✅ Excellent multilingual
When: teams with no MLE to maintain a local cross-encoder, or a multilingual corpus.
Capsule 05 covers it.
Module map
| Capsule | Topic | Time |
|---|---|---|
| 01 (you are here) | Module introduction | 10-15 min |
| 02 | Why cosine similarity has a ceiling | 30 min |
| 03 | Cross-encoder re-ranking (the default) | 30 min |
| 04 | LLM-based re-ranking (premium) | 30 min |
| 05 | Cohere Rerank API (managed) | 25 min |
| 06 | Trade-offs and operational optimizations | 25 min |
| 07 | Decision framework — which technique to pick | 30 min |
| 08 | Capstone project — a re-ranking system | 45 min |
Estimated total: 3-4 hours. It's one of the best-ROI modules in the path.
Connection with previous and following modules
Previous modules:
├─ M01: RAG Pipeline → re-ranking slots in after the initial retrieval
├─ M02: Chunking → better chunking + re-ranking complement each other
└─ M03: Query Optimization → query optimization improves retrieval; re-ranking refines the output
This module prepares you for:
├─ M05: Hybrid Search → hybrid search + re-ranking is the "state of the art" pattern
├─ M06: Metadata Filtering → re-ranking is applied after the filtering
├─ M07: Production with Pinecone → Pinecone has a native rerank
└─ M08: Evaluation → how to measure whether rerank improves things vs just adds complexity
The "state of the art" pattern in production RAG (May 2026):
Query
↓
Query optimization (M03)
↓
Hybrid search (M05) — BM25 + cosine
↓
Metadata filter (M06)
↓
Re-ranking (this module)
↓
LLM generation
This module is one of the three pillars (along with chunking and hybrid search) that separate an "MVP" RAG from a "production-ready" RAG.
Expected cumulative gain
If you go through every module in the path, the system's quality improves cumulatively:
Starting point (an "MVP" system):
Cosine + recursive chunk + no optimization = ~68% precision@5
After M02 (optimized chunking): +5-10 pts → 73-78%
After M03 (query optimization): +5-10 pts → 78-88%
After M04 (re-ranking) ← YOU ARE HERE: +15-25 pts → 88-95%
After M05 (hybrid search): +3-8 pts → 91-95%
After M06 (metadata filter): +2-5 pts → 93-97%
Re-ranking alone accounts for most of the gain. That's why many teams add it first, before investing in more complex optimizations.
Concrete cases: when each technique wins
To anchor the decisions you'll make at the end of the module, three examples:
Case A: internal support chatbot, 200K docs, a team of 3
- Constraints: English corpus, no dedicated MLE, latency <500ms.
- Expected recommendation: Cohere Rerank ($5-15/month, 0 maintenance, quality close to an LLM).
- Why not a cross-encoder: with no MLE, maintaining models locally adds operational overhead.
- Why not an LLM: $200/month for an internal chatbot isn't justified vs Cohere at $15.
Case B: LATAM legal SaaS assistant, 500K multilingual cases
- Constraints: Spanish + Portuguese + English, precision is critical, generous budget.
- Expected recommendation: a cross-encoder + LLM cascade for queries flagged "high stakes".
- Why a cascade: the cross-encoder filters fast (a multilingual model), the LLM refines the top-15 at maximum quality.
- Why not Cohere alone: a critical domain justifies the LLM's extra cost.
Case C: an internal Stack Overflow search for a startup
- Constraints: 10M docs (large scale), high volume, limited budget.
- Expected recommendation: a local cross-encoder (free at scale).
- Why not Cohere: $30/month is manageable, but it grows with volume. A cross-encoder is fixed at the cost of infra.
- Why not an LLM: prohibitive at 10M docs / 50K daily queries.
These three cases cover the typical patterns. The module teaches you to derive the right decision for your specific case.
Prerequisites before starting the module
Make sure you have:
- ✅ A working RAG pipeline with cosine retrieval (covered in M01)
- ✅ Reasonable chunking (recursive with
chunk_size=500,overlap=50at minimum) — M02 - ✅ Your own eval set with at least 30 queries and ground truth — to validate the gain
- ✅ Access to the OpenAI API (for capsule 04 on LLM rerank)
- ✅ A Cohere account (a free trial is available) — for capsule 05
If you're missing the eval set, build it before starting the module. Without it, the benchmarks you'll run in each capsule are blind.
Technical setup for the module
# Capsule 03 (local cross-encoder)
pip install sentence-transformers
# Capsule 04 (LLM-based)
pip install openai pydantic
# Capsule 05 (Cohere)
pip install cohereEnvironment variables:
# .env
OPENAI_API_KEY=sk-...
COHERE_API_KEY=...Self-check before moving on
Before starting capsule 02, make sure you can answer:
- Why does re-ranking typically give more gain for less effort than other techniques?
- What's the structural difference between a bi-encoder (cosine) and a cross-encoder (rerank)?
- When would you NOT need re-ranking in your RAG system?
Answers
-
Because cosine similarity has an inherent ceiling (it measures semantic overlap, not real relevance). Adding a second scoring stage breaks through that ceiling. The gain comes from the architecture (two stages), not from marginally optimizing one component. Low effort: add a library + 30 lines of code. High benefit: 15-25 points of precision.
-
A bi-encoder (cosine) processes the query and the document independently, producing two vectors that are then compared with cosine similarity. Fast but shallow. A cross-encoder (rerank) processes the (query, doc) pair together, attending to token-by-token interactions. Slower, but it captures the relationship the bi-encoder can't see.
-
An MVP with a small dataset (<10K docs): cosine is enough, the false positives don't pile up. A system with a strict SLA (<200ms total): the rerank's extra latency breaks the SLA. When your current precision is already >92% on the eval set: the marginal gain doesn't justify the extra cost. When false positives don't affect the user (e.g. the downstream LLM is robust enough to ignore them).
Next step: Capsule 02
The next capsule gets into the detailed "why" of cosine similarity's failure modes. Without understanding those modes concretely, re-ranking techniques sound like unnecessary optimization. Once you understand them, the rest of the module becomes obviously useful.
Resources
- Sentence Transformers — Cross-Encoders — Official documentation
- Pinecone — Re-ranking Guide — Visual tutorial
- Cohere Rerank Documentation — For capsule 05
- BEIR Benchmark — Empirical comparisons
- Anthropic — Contextual Retrieval — Complementary technique
- Khattab & Zaharia — ColBERT Paper — Late interaction as the cross-encoder's evolution
Estimated time: 10-15 minutes Next: 02-problems-with-cosine-similarity.md