Capsule 03: Cross-encoder re-ranking — the default option that almost always wins

Capsule overview

In capsule 02 we saw why cosine similarity fails in certain modes. The most efficient, fastest and cheapest solution is cross-encoder re-ranking: a model that takes each (query, document) pair and produces a relevance score by analyzing the two together, instead of comparing independent embeddings.

The cross-encoder is the reasonable default for 80%+ of cases. Compared with LLM-based or Cohere Rerank, the cross-encoder wins on cost (free, runs locally) and latency (~150ms to rerank 20 candidates). Quality is typically 90-92% precision — it doesn't reach the 94% of an LLM rerank, but the zero cost makes up for it on most projects.

This capsule teaches you to pick the right model among the ones available, integrate it efficiently into your RAG pipeline with sentence-transformers, optimize the batching to keep latency low, and compare your gain before and after with an honest benchmark.

By the end of this capsule you'll be able to:

• ✅ Implement cross-encoder re-ranking with sentence-transformers in under 30 lines
• ✅ Choose among the available models (MiniLM L-6 vs L-12, TinyBERT, multilingual) for your case
• ✅ Optimize the batching to minimize latency (typically 50-100ms on CPU, <30ms on GPU)
• ✅ Integrate the re-ranker as a second stage after the cosine retrieval
• ✅ Benchmark precision before and after on your own eval set
• ✅ Anticipate the operational traps: cold start, missing batching, a model misaligned with the domain

Estimated time: 30-35 minutes

How a cross-encoder works under the hood

A cross-encoder is a BERT-like model that receives two texts concatenated as input (the query and the document) and produces a single relevance score. Internally, it attends to how each token of the query relates to each token of the document — capturing interactions that are impossible for bi-encoders (the embedding models).

Bi-encoder (cosine):

    query  ──> encoder ──> vec_q [1536]
    doc    ──> encoder ──> vec_d [1536]

    score = cosine(vec_q, vec_d)

    ─ Processes query and doc independently
    ─ Vec_q never "sees" vec_d


Cross-encoder:

    [query] [SEP] [doc]  ──> encoder with cross-attention ──> score

    ─ Processes query and doc IN PARALLEL
    ─ Every token of the query attends to every token of the doc
    ─ Captures interactions, not just similarity

Why that difference matters:

• The bi-encoder answers: "how similar are these two vectors the model produced independently?"
• The cross-encoder answers: "given this (query, doc) pair, how relevant is the doc to the specific query?"

The second question is what you want in RAG. The bi-encoder is a proxy. The cross-encoder is direct.

The trade-off: the cross-encoder does NOT scale. With 1M docs in your collection, you can't run the cross-encoder over all of them for every query — that would be ~1000 seconds per query. That's why the optimal architecture uses the bi-encoder first (fast, retrieves the top-30) and the cross-encoder afterwards (over those 30, refining down to the top-5).

Available models: which one to pick

The cross-encoders trained on MS MARCO are the standard. Three main variants:

Practical recommendation:

• Start with MiniLM-L-12-v2. The best default balance. The difference in size and latency vs L-6 is small; the quality gain is noticeable.
• Drop to TinyBERT only if latency is a measurable problem. If your pipeline already takes 800ms and you need to get down to 500ms, TinyBERT can save you 130ms. Losing ~3% precision may be acceptable.
• Move up to multilingual only if your corpus needs it. MS MARCO multilingual is 2-3x heavier and slower, but it gains ~10% on non-English queries.

The correct implementation

Base setup

# rerank_with_cross_encoder.py
from sentence_transformers import CrossEncoder
from dataclasses import dataclass
from typing import List
import time


# Load the model ONCE at startup, not per query
_CROSS_ENCODER = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")


@dataclass
class RerankedDoc:
    document: str
    score: float
    original_index: int


def cross_encoder_rerank(
    query: str,
    documents: List[str],
    top_k: int = 5,
) -> List[RerankedDoc]:
    """
    Re-rank documents using a local cross-encoder model.

    Important:
    - The model is loaded ONCE at startup (a global variable).
    - The whole batch of pairs is processed in a single call (efficient).
    - Results are sorted by descending score.
    """
    if not documents:
        return []

    # Build the (query, document) pairs for the whole batch
    pairs = [(query, doc) for doc in documents]

    # Batch prediction (much faster than a loop)
    scores = _CROSS_ENCODER.predict(pairs, batch_size=32, show_progress_bar=False)

    # Sort and map
    indexed = list(enumerate(zip(documents, scores)))
    sorted_results = sorted(indexed, key=lambda x: -x[1][1])

    return [
        RerankedDoc(document=doc, score=float(score), original_index=idx)
        for idx, (doc, score) in sorted_results[:top_k]
    ]

End-to-end usage with ChromaDB

# pipeline_rag_with_rerank.py
import chromadb
from chromadb.utils import embedding_functions
import os

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small",
)

client_chroma = chromadb.PersistentClient(path="./chroma_db")
collection = client_chroma.get_collection("docs", embedding_function=openai_ef)


def rag_pipeline(query: str, top_k: int = 5) -> List[RerankedDoc]:
    """
    Full pipeline: retrieval → cross-encoder rerank → final top-K.
    """
    # Stage 1: broad retrieval (top-20-30 candidates)
    results = collection.query(query_texts=[query], n_results=25)
    candidates = results['documents'][0]

    # Stage 2: cross-encoder rerank
    reranked = cross_encoder_rerank(query, candidates, top_k=top_k)

    return reranked


# Try it out
query = "how do I configure HNSW for 1M vectors with high accuracy?"
top_5 = rag_pipeline(query, top_k=5)

print(f"Top 5 after the rerank:")
for i, doc in enumerate(top_5, 1):
    print(f"\n#{i} (score: {doc.score:.3f}, was #{doc.original_index+1} before)")
    print(f"   {doc.document[:120]}...")

Typical output:

Top 5 after the rerank:

#1 (score: 8.421, was #4 before)
   For production with 1M+ vectors, configure HNSW with M=32 and construction_ef=200...

#2 (score: 7.892, was #1 before)
   HNSW (Hierarchical Navigable Small World) is the default indexing algorithm...

#3 (score: 6.103, was #7 before)
   Configuring HNSW for high accuracy: raising M improves recall but uses more memory...

#4 (score: 4.521, was #2 before)
   Vector databases use a variety of indexing algorithms...

Note the ranking movements: the most specific doc ("M=32 with 1M+ vectors") climbs from position #4 to #1 after the rerank. The most generic doc ("HNSW is the default") drops from #1 to #2.

Batching optimization

The model's internal batch_size parameter determines how many pairs get processed in parallel on the GPU/CPU. Setting it wrong can triple your latency.

# batch_size benchmark
import time

candidates = [f"Document {i} about topic..." for i in range(50)]

for batch_size in [1, 8, 16, 32, 64]:
    start = time.perf_counter()
    pairs = [(query, doc) for doc in candidates]
    scores = _CROSS_ENCODER.predict(pairs, batch_size=batch_size, show_progress_bar=False)
    elapsed = (time.perf_counter() - start) * 1000
    print(f"batch_size={batch_size}: {elapsed:.0f}ms")

Typical output (CPU, MiniLM-L-12):

batch_size= 1: 1240ms   ← terrible (doesn't exploit vectorization)
batch_size= 8:  280ms
batch_size=16:  220ms
batch_size=32:  200ms   ← CPU sweet spot
batch_size=64:  210ms   ← diminishing returns

Typical output (GPU, MiniLM-L-12):

batch_size= 1: 80ms
batch_size= 8: 35ms
batch_size=16: 28ms
batch_size=32: 25ms
batch_size=64: 24ms     ← the GPU loves big batches

Recommendations:

• CPU: batch_size=32 is the sweet spot. Higher gives diminishing returns.
• GPU: batch_size=32-64. If you have plenty of VRAM, you can go higher.
• The sentence-transformers default: usually 32, but verify it.

Validation: measuring the impact on your eval set

Don't trust blog benchmarks. Measure on your own eval set.

# benchmark_rerank_impact.py
from dataclasses import dataclass
import statistics


@dataclass
class EvalQuery:
    query: str
    expected_doc_ids: list[str]  # IDs of the relevant docs (ground truth)


def benchmark_with_vs_without_rerank(eval_set: list[EvalQuery]):
    """
    Compare metrics with and without re-ranking on the same eval set.
    """
    no_rerank = {"precision_at_5": [], "recall_at_5": [], "latency_ms": []}
    with_rerank = {"precision_at_5": [], "recall_at_5": [], "latency_ms": []}

    for item in eval_set:
        # Without re-rank
        start = time.perf_counter()
        results = collection.query(query_texts=[item.query], n_results=5)
        elapsed = (time.perf_counter() - start) * 1000

        retrieved_ids = set(results['ids'][0])
        relevant_in_top5 = retrieved_ids & set(item.expected_doc_ids)

        no_rerank["precision_at_5"].append(len(relevant_in_top5) / 5)
        no_rerank["recall_at_5"].append(len(relevant_in_top5) / len(item.expected_doc_ids))
        no_rerank["latency_ms"].append(elapsed)

        # With re-rank
        start = time.perf_counter()
        results = collection.query(query_texts=[item.query], n_results=25)
        candidates = results['documents'][0]
        candidate_ids = results['ids'][0]
        reranked = cross_encoder_rerank(item.query, candidates, top_k=5)
        elapsed = (time.perf_counter() - start) * 1000

        # Map back to IDs
        reranked_ids = {candidate_ids[r.original_index] for r in reranked}
        relevant_in_top5_reranked = reranked_ids & set(item.expected_doc_ids)

        with_rerank["precision_at_5"].append(len(relevant_in_top5_reranked) / 5)
        with_rerank["recall_at_5"].append(len(relevant_in_top5_reranked) / len(item.expected_doc_ids))
        with_rerank["latency_ms"].append(elapsed)

    # Report
    print(f"\n{'Metric':<20} {'No rerank':<15} {'With rerank':<15} {'Gain'}")
    print("-" * 70)

    for metric in ["precision_at_5", "recall_at_5", "latency_ms"]:
        mean_no = statistics.mean(no_rerank[metric])
        mean_with = statistics.mean(with_rerank[metric])
        diff = mean_with - mean_no
        sign = "+" if diff > 0 else ""

        if "latency" in metric:
            print(f"{metric:<20} {mean_no:>10.0f} ms   {mean_with:>10.0f} ms   {sign}{diff:.0f} ms")
        else:
            print(f"{metric:<20} {mean_no:>13.2%}   {mean_with:>13.2%}   {sign}{diff*100:.1f} pts")

Typical output:

Metric              No rerank      With rerank    Gain
----------------------------------------------------------------------
precision_at_5            72.40%          90.20%   +17.8 pts
recall_at_5               58.30%          76.40%   +18.1 pts
latency_ms                  185 ms           340 ms   +155 ms

How to read it:

• Precision gain: +17.8 points. Excellent.
• Recall gain: +18.1 points. Excellent (which confirms it also surfaced good docs from the top-25 that cosine had pushed beyond the top-5).
• Cost: +155ms of latency. Acceptable for a chatbot, tight for sub-200ms search interfaces.

Traps and common mistakes

Trap 1: loading the model on every query

The mistake:

def rerank(query, docs):
    model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")  # ❌ loads every time
    return model.predict(...)

Symptom: the first query takes 5-10 seconds (cold start). Subsequent queries do too if the app doesn't cache.

How to prevent it: load the model once at startup and reuse it:

# A global variable or a singleton
_CROSS_ENCODER = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")

def rerank(query, docs):
    return _CROSS_ENCODER.predict(...)

In Flask/FastAPI, load it at app startup, not per request.

Trap 2: sequential predict instead of a batch

The mistake:

scores = []
for doc in docs:
    score = model.predict([(query, doc)])  # ❌ one at a time
    scores.append(score)

Symptom: re-ranking 20 docs takes ~3 seconds on CPU. Unusable.

How to prevent it: hand it all the pairs together:

pairs = [(query, doc) for doc in docs]
scores = model.predict(pairs, batch_size=32)  # native batching

The speed difference: 10-50x faster.

Trap 3: using an MS MARCO model with non-English queries

The mistake: your corpus is in Spanish. You use ms-marco-MiniLM-L-12-v2 (trained in English).

Symptom: precision@5 on Spanish queries is ~75%, whereas on English it would be ~90%. Inconsistent.

How to prevent it: for a non-English corpus, use mmarco-mMiniLMv2-L12-H384-v1 (multilingual) or Cohere Rerank multilingual.

Trap 4: re-ranking only the direct top-5

The mistake:

results = collection.query(query_texts=[q], n_results=5)
reranked = cross_encoder_rerank(q, results['documents'][0], top_k=5)

Symptom: re-ranking doesn't improve anything, because you're only reordering the 5 that cosine already filtered.

How to prevent it: retrieval with n_results=20-30, and let the re-rank pick the top-5 from those. The value of the rerank is filtering false positives out of the top-30, not reordering the top-5.

Trap 5: ignoring the score threshold

The mistake: you take the top-5 after the rerank regardless of the absolute score.

Symptom: some queries have no relevant match in the corpus at all, but they still return 5 docs (with very low scores). The LLM gets confused by irrelevant context.

How to prevent it:

reranked = cross_encoder_rerank(query, candidates, top_k=10)
# Only include docs with score >= threshold
threshold = 1.0  # find it empirically on your eval set
relevant = [doc for doc in reranked if doc.score >= threshold]

If fewer than 5 survive the threshold, that's fine. Better to pass the LLM 3 relevant docs than 5 with 2 pieces of garbage.

Trap 6: re-ranking raw text when the chunks have useful metadata

The mistake:

pairs = [(query, raw_chunk_text) for chunk in retrieved]

Symptom: the cross-encoder only sees the chunk's text, not the extra context (doc title, section, date) that could improve the ranking.

How to prevent it: include the relevant metadata in the text you pass to the cross-encoder:

def format_for_rerank(chunk, metadata):
    return f"Source: {metadata['source']}\nSection: {metadata['section']}\n\n{chunk}"

pairs = [(query, format_for_rerank(chunk, meta)) for chunk, meta in zip(chunks, metas)]

Applied exercise

Scenario: you're the AI Engineer at a financial services company. The current RAG pipeline:

• 300K chunks of regulatory documentation + financial analysis, in English
• Cosine similarity for retrieval, n_results=5
• No re-ranking
• Current precision@5: 76%
• p95 latency: 280ms

Stakeholders ask: "reach precision@5 ≥88% without going over 500ms of latency."

Your job:

1. Decide which cross-encoder model to use.
2. Design the integration (what n_results on the initial retrieval, what batch_size, what score threshold).
3. Estimate the expected precision and latency. Does it meet the requirements?

from sentence_transformers import CrossEncoder
import time

# Load the model at startup
_RERANKER = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")

# Configuration
RETRIEVAL_N = 25      # top-25 candidates from the initial retrieval
RERANK_TOP_K = 8      # top-8 after the rerank (more than 5, for margin after the threshold)
SCORE_THRESHOLD = 1.5 # determined empirically on the eval set
BATCH_SIZE = 32       # CPU sweet spot


def rag_pipeline(query: str) -> list[dict]:
    # Stage 1: broad retrieval
    results = collection.query(query_texts=[query], n_results=RETRIEVAL_N)
    candidates = results['documents'][0]
    candidate_metas = results['metadatas'][0]

    # Stage 2: cross-encoder rerank with metadata
    enriched_pairs = [
        (query, f"Source: {meta.get('source','')}\n\n{doc}")
        for doc, meta in zip(candidates, candidate_metas)
    ]
    scores = _RERANKER.predict(enriched_pairs, batch_size=BATCH_SIZE)

    # Stage 3: filter by threshold and take the top-5
    scored = sorted(
        zip(candidates, candidate_metas, scores),
        key=lambda x: -x[2]
    )
    relevant = [
        {"doc": doc, "metadata": meta, "score": float(score)}
        for doc, meta, score in scored[:RERANK_TOP_K]
        if score >= SCORE_THRESHOLD
    ][:5]  # final top-5

    return relevant

Retrieval (n_results=25):    ~100ms (cosine search is fast)
Cross-encoder rerank:        ~180ms (25 docs in a batch of 32, MiniLM-L-12 on CPU)
Filtering + final ranking:   ~5ms (in memory)
─────────────────────────────────
Total p95:                   ~285ms

Recap and next step

What you learned:

• The cross-encoder analyzes (query, doc) pairs in parallel, capturing interactions cosine similarity can't.
• A reasonable default: ms-marco-MiniLM-L-12-v2 for English. ~150ms latency, +20% precision over cosine, free.
• For multilingual, use mmarco-mMiniLMv2-L12-H384-v1 or Cohere Rerank.
• batch_size=32 on CPU is the sweet spot. A GPU allows 64-128 with massive parallelism.
• Load the model ONCE at startup. Each load takes 5-10s.
• Retrieval with n_results=20-30 before the rerank. Re-ranking only the direct top-5 doesn't improve anything.
• A score threshold lets you discard low-relevance results. Find it empirically on your eval set.
• Validation is mandatory: measure precision on your own eval set, don't assume.

Checkpoint: before moving on, you should be able to:

• Implement a cross-encoder rerank with sentence-transformers in under 30 lines.
• Choose n_results and batch_size with justification for your hardware.
• Measure the rerank's impact on your own eval set before deploying.

Next capsule: 04 — LLM-based re-ranking.

The cross-encoder is the default for 80% of cases. But there are domains where the extra 3-4% precision justifies using an LLM as the re-ranker. Capsule 04 covers when the extra cost pays for itself: legal, medical, critical financial — and how to implement it correctly with structured outputs and calibrated prompts.

Resources

1. Sentence Transformers — Cross-Encoders — Complete official documentation
2. MS MARCO — Microsoft Research — The dataset used to train the most popular cross-encoders
3. Pinecone — Cross-Encoder Reranking — Visual tutorial with benchmarks
4. Hugging Face Hub — Cross-encoder models — The full list of available models
5. Khattab & Zaharia — ColBERT Paper — Late interaction as the cross-encoder's evolution
6. BEIR Benchmark — Reproducible empirical comparisons

Estimated time: 30-35 minutes Next: 04-llm-based-reranking.md