Capsule 06: Re-ranking trade-offs and operational optimizations

Capsule overview

The previous capsules covered which re-ranking technique to pick (cross-encoder, LLM, Cohere). But there are operational decisions that affect performance regardless of which technique you use: how many candidates to pass to the re-ranker, when to re-rank vs when to trust the direct retrieval, what to cache, how to parallelize.

These optimizations can improve latency by 30-50% or cut cost by 70% without changing the underlying technique. In production systems, they're what separates "re-ranking works but it's slow" from "re-ranking is fast and cheap". This capsule gives you the operational playbook for getting the most out of whichever technique you choose.

By the end of this capsule you'll be able to:

• ✅ Compute the optimal n_results for the initial retrieval based on the re-ranker type
• ✅ Design a re-ranking result cache for repeated queries
• ✅ Apply score thresholds to discard low-relevance results
• ✅ Identify queries where re-ranking adds no value (dynamic skip)
• ✅ Parallelize correctly when the re-ranker allows it
• ✅ Anticipate the trap: optimizing latency while silently breaking accuracy

Estimated time: 25-30 minutes

Optimization 1: the right n_results for the initial retrieval

The key question: how many candidates do you pass from the retrieval to the re-ranker?

        cosine retrieval                  re-ranker
    [corpus] ────────────────> [top-N candidates] ───────> [final top-K]
                                       N = ?                K = 5

    Too low (N=10):   you lose good docs before the rerank
    Too high (N=200): re-ranking becomes slow and expensive
    Sweet spot:       N = 20-50 (depending on the corpus and the technique)

How to find the optimal N empirically

# benchmark_n_results.py
import time
import statistics
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")


def benchmark_n_results(query, eval_set, n_values=[5, 10, 20, 50, 100, 200]):
    """
    Measure precision@5 vs latency for different n_results values of the retrieval.
    """
    results = []
    for n in n_values:
        latencies = []
        precisions = []

        for item in eval_set:
            # Retrieval with N candidates
            start = time.perf_counter()
            candidates = collection.query(
                query_texts=[item["query"]], n_results=n
            )['documents'][0]

            # Re-rank
            pairs = [(item["query"], doc) for doc in candidates]
            scores = reranker.predict(pairs)

            # Final top 5
            import numpy as np
            top_5_idx = np.argsort(scores)[::-1][:5]
            top_5_docs = [candidates[i] for i in top_5_idx]

            latencies.append((time.perf_counter() - start) * 1000)

            # Precision: are the expected_doc_ids in the top 5?
            # (this assumes your eval set has a doc → id mapping)
            hits = sum(1 for doc in top_5_docs if doc in item["expected_chunks"])
            precisions.append(hits / 5)

        results.append({
            "n_results": n,
            "p95_latency_ms": sorted(latencies)[int(len(latencies) * 0.95)],
            "avg_precision_at_5": statistics.mean(precisions),
        })

    return results

Typical output (local cross-encoder, technical dataset):

n_results=  5: p95=210ms, precision@5=0.74
n_results= 10: p95=240ms, precision@5=0.85
n_results= 20: p95=320ms, precision@5=0.91   ← sweet spot
n_results= 50: p95=580ms, precision@5=0.92   ← diminishing returns
n_results=100: p95=1100ms, precision@5=0.92  ← wasteful
n_results=200: p95=2200ms, precision@5=0.92  ← unusable

How to read it:

• From N=5 to N=20: precision goes up 17 points. Every extra candidate pays for itself.
• From N=20 to N=50: precision goes up just 1 point. Latency almost doubles. A bad trade.
• N>50: zero returns, latency explodes.

The sweet spot per technique:

The rule: start at N=20, tune empirically on your eval set.

Optimization 2: caching results

In production, queries aren't random. There's a Pareto pattern: ~20% of unique queries account for ~80% of the traffic. Caching results for repeated queries is the optimization with the best gain/effort ratio.

Implementation with an LRU cache (in-memory)

# rerank_cache.py
import hashlib
from functools import lru_cache
from typing import Tuple


def query_cache_key(query: str, n_candidates: int) -> str:
    """A stable cache key based on the query and the configuration."""
    raw = f"{query.strip().lower()}|{n_candidates}"
    return hashlib.md5(raw.encode()).hexdigest()


# Cache with a configurable size
_RERANK_CACHE: dict[str, list] = {}
_MAX_CACHE_SIZE = 10_000


def cached_rerank_pipeline(query: str, top_k: int = 5):
    """The full pipeline with a cache for re-ranking results."""
    cache_key = query_cache_key(query, n_candidates=20)

    # Cache hit
    if cache_key in _RERANK_CACHE:
        return _RERANK_CACHE[cache_key][:top_k]

    # Cache miss: run the full pipeline
    candidates = collection.query(query_texts=[query], n_results=20)['documents'][0]
    reranked = cross_encoder_rerank(query, candidates, top_k=top_k * 2)

    # Store in the cache (up to the limit)
    if len(_RERANK_CACHE) >= _MAX_CACHE_SIZE:
        # Evict the oldest (simple FIFO; in production use a real LRU)
        _RERANK_CACHE.pop(next(iter(_RERANK_CACHE)))
    _RERANK_CACHE[cache_key] = reranked

    return reranked[:top_k]

When to invalidate the cache

The cache becomes wrong if:

• The collection's content changes (you insert new documents): the old results may be omitting relevant new docs.
• The embedding model changes (a re-ingest): the whole rankings change.
• The re-ranking model changes (upgrading from Cohere v3 to v4, for example).

def invalidate_cache(reason: str):
    global _RERANK_CACHE
    print(f"Invalidating rerank cache: {reason}")
    _RERANK_CACHE.clear()


# In the ingestion pipeline
def ingest_documents(...):
    collection.add(...)
    invalidate_cache(f"Added {n} new documents")

A distributed cache (Redis) for multiple instances

An in-memory LRU works for a single process. If you have multiple instances of your API, each one has its own cache and hits are rare. The solution: a distributed cache.

import redis
import json

redis_client = redis.Redis(host='localhost', port=6379, db=0)


def cached_rerank_redis(query: str, top_k: int = 5, ttl_seconds: int = 3600):
    cache_key = f"rerank:{query_cache_key(query, n_candidates=20)}"

    # Cache hit
    cached = redis_client.get(cache_key)
    if cached:
        return json.loads(cached)[:top_k]

    # Cache miss
    candidates = collection.query(query_texts=[query], n_results=20)['documents'][0]
    reranked = cross_encoder_rerank(query, candidates, top_k=top_k * 2)

    # Store with a TTL (it auto-invalidates after ttl_seconds)
    redis_client.setex(cache_key, ttl_seconds, json.dumps(reranked))

    return reranked[:top_k]

The typical benefit of caching:

• Typical hit rate in systems with real traffic: 40-70%.
• Latency on hits: ~5ms (a Redis lookup) vs 320ms (the full pipeline).
• Cost reduction in systems with an LLM rerank: up to 70% fewer OpenAI calls.

Optimization 3: score thresholds to discard garbage

Re-ranking doesn't guarantee that every one of the top_k is relevant. Sometimes the top-5 includes 1-2 docs with very low scores that the LLM shouldn't see.

# Without a threshold (everything gets through)
top_5 = reranker.rerank(query, candidates, top_k=5)

# With a threshold (filters out low-relevance results)
all_reranked = reranker.rerank(query, candidates, top_k=20)
top_relevant = [doc for doc in all_reranked if doc.score > 0.5]

# If fewer than 5 survive the threshold, that's fine
# Better to pass the LLM 3 relevant docs than 5 with 2 pieces of garbage

How to pick the threshold:

1. On your eval set, plot the score distribution of relevant vs irrelevant docs.
2. Find the point where the two distributions clearly separate.
3. Set the threshold there, and validate that precision goes up without recall dropping too much.

import matplotlib.pyplot as plt

def find_threshold(eval_set):
    relevant_scores = []
    irrelevant_scores = []

    for item in eval_set:
        candidates = collection.query(query_texts=[item["query"]], n_results=20)['documents'][0]
        reranked = reranker.rerank(item["query"], candidates, top_k=20)

        for doc in reranked:
            if doc.document in item["expected_chunks"]:
                relevant_scores.append(doc.score)
            else:
                irrelevant_scores.append(doc.score)

    # Comparative histogram
    plt.hist(relevant_scores, alpha=0.5, label="Relevant", bins=20)
    plt.hist(irrelevant_scores, alpha=0.5, label="Irrelevant", bins=20)
    plt.legend()
    plt.show()
    # Where the two distributions cross = the candidate threshold

Typical threshold (MS MARCO cross-encoder): 0.4-0.6 Typical threshold (Cohere relevance score): 0.5-0.7 Typical threshold (LLM rerank 0-10): 5.0-7.0

Optimization 4: the dynamic skip — when NOT to re-rank

Not every query needs re-ranking. If the initial cosine retrieval is already very confident (a top-3 with scores far above the rest), re-ranking is a waste.

def needs_reranking(retrieval_scores: list[float], threshold_gap: float = 0.15) -> bool:
    """
    Decide whether re-ranking is worth it based on the score distribution.

    If the gap between the top-3 and the top-10 is large, the retrieval is confident and
    re-ranking won't change much. If the gap is small, there's ambiguity and
    re-ranking is worth it.
    """
    if len(retrieval_scores) < 10:
        return False  # not enough candidates

    # Cosine distance: lower = more similar
    top_3_avg = sum(retrieval_scores[:3]) / 3
    top_10_avg = sum(retrieval_scores[:10]) / 10

    gap = top_10_avg - top_3_avg
    return gap < threshold_gap


def smart_rerank(query: str, top_k: int = 5):
    results = collection.query(query_texts=[query], n_results=20)
    candidates = results['documents'][0]
    distances = results['distances'][0]

    if needs_reranking(distances):
        # Re-rank because there's ambiguity
        return cross_encoder_rerank(query, candidates, top_k=top_k)
    else:
        # Skip the rerank — the retrieval is already confident
        return [(doc, dist) for doc, dist in zip(candidates[:top_k], distances[:top_k])]

The benefit: ~20-30% of queries can skip re-ranking with no measurable quality loss. It saves latency and cost.

The risk: setting threshold_gap wrong can skip re-ranking on queries that did need it. Validate empirically on your eval set.

Optimization 5: parallelizing correctly

Local cross-encoder: use native batching

# ❌ Bad: sequential (a Python loop)
for doc in candidates:
    score = reranker.predict([(query, doc)])

# ✅ Good: native batching
pairs = [(query, doc) for doc in candidates]
scores = reranker.predict(pairs, batch_size=32)

The internal batch_size in sentence-transformers exploits GPU/CPU vectorization. The difference between sequential and batch is 10-50x.

LLM rerank: ThreadPoolExecutor

from concurrent.futures import ThreadPoolExecutor

def parallel_llm_rerank(query, candidates, top_k=5, workers=5):
    def score_one(doc):
        return llm_rerank_pair(query, doc).score

    with ThreadPoolExecutor(max_workers=workers) as executor:
        scores = list(executor.map(score_one, candidates))

    paired = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return paired[:top_k]

5-10 parallel workers turn 1500ms into ~300ms for reranking 20 candidates. The calls are I/O bound (they're waiting on OpenAI), so parallelizing is trivial.

Cohere Rerank: it already comes parallelized server-side

Cohere receives the N documents in a single API call and processes them in parallel internally. No client-side parallelization needed.

# A single call with all the candidates
response = co.rerank(query=query, documents=candidates, top_n=top_k)

The trap: optimizing latency while silently breaking accuracy

Goodhart's Law applied: when you tune n_results=10 to bring latency down, you may be bringing precision down without seeing it on the dashboard.

# Aggressive optimization
n_results = 5  # low, "to save"
top_k_rerank = 3  # low, "to save"
score_threshold = 0.7  # high, "to be strict"

# Result: latency improves 40%, precision drops 8% with no alert

Mitigation:

1. A guardrail metric: continuous precision@K monitoring on the eval set. If it drops, alert.
2. A/B test before deploying: compare the optimized version vs the current one on a fixed eval set.
3. Composite metrics: "latency × (1 / precision)" — it gets worse if either of the two degrades.

Traps and common mistakes

Trap 1: caching without invalidating when new docs are ingested

The mistake: the re-ranking cache is on. Someone inserts 1000 new docs. The cache keeps returning results that don't take the new docs into account.

Symptom: relevant new docs never show up on cached queries. The system looks like it isn't taking advantage of the new information.

How to prevent it: invalidate the cache on every ingestion (or at least at the end of the batch). If the ingestion is continuous, use a short TTL (5-15 min) instead of a permanent cache.

Trap 2: a global threshold when scores have different ranges per query

The mistake: you set a global threshold=0.5. But some queries are ambiguous (all scores between 0.3 and 0.5) and others are specific (scores between 0.6 and 0.9).

Symptom: ambiguous queries return zero results (everything is below the threshold). Specific queries filter nothing (everything is above).

How to prevent it: use a relative threshold instead of an absolute one:

# Relative threshold: only discard docs significantly worse than the top-1
top_score = reranked[0].score
relevant = [doc for doc in reranked if doc.score > top_score * 0.7]

Trap 3: a rerank skip that becomes permanent

The mistake: you implement a "dynamic skip" with logic that drops the rerank when the retrieval is "confident". Because of a bug in the logic, it almost always considers the retrieval confident, and the rerank almost never runs.

Symptom: the operational metrics don't change (rerank latency and Cohere cost drop) but precision falls silently.

How to prevent it: instrument rerank_skipped_count and rerank_executed_count. Monitor that the ratio stays ~20-30%, not >70%.

Trap 4: an in-memory cache with no maximum size

The mistake:

_CACHE = {}  # grows infinitely

Symptom: the app's memory grows without limit, and eventually OOMs.

How to prevent it: always with a MAX_SIZE and an eviction policy (typically LRU).

Trap 5: parallelizing a cross-encoder with threads

The mistake: you use a ThreadPoolExecutor to call cross_encoder.predict with one pair at a time.

Symptom: it doesn't speed anything up. A local cross-encoder is CPU-bound; threads don't help, the GIL serializes them.

How to prevent it: for a local cross-encoder, use the internal batch_size, not threads. For API calls (LLM, Cohere), threads do help (I/O bound).

Trap 6: measuring only the rerank's latency

The mistake: you measured that the cross-encoder rerank takes 150ms. You assume that's the total cost.

Symptom: in production, the latency is 400ms. You forgot to count the retrieval before it, the query's embedding and the generation.

How to prevent it: measure end-to-end, from the moment the user's query arrives until the answer is returned. Every component of the pipeline should log its latency.

Applied exercise

Scenario: a RAG system in production with these baseline metrics:

• Pipeline: cosine retrieval n_results=50 → cross-encoder rerank → top-5 to the LLM
• Current p95 latency: 850ms
• Precision@5: 89%
• Cost: ~$0/query (local cross-encoder)
• Volume: 100K queries/day

The product team asks: "let's get latency down to <500ms p95 without sacrificing precision."

Your job: propose three specific optimizations in priority order, estimate the impact of each, and specify how you'd validate them.

# Pre-warm the cache with the top queries from the last day
def warm_cache_with_top_queries():
    top_queries = get_top_queries_from_logs(limit=1000)
    for query in top_queries:
        cached_rerank_pipeline(query)  # fills the cache

def should_skip_rerank(distances):
    if len(distances) < 10:
        return False
    return distances[9] - distances[2] < 0.10  # a small gap = confident

Recap and next step

What you learned:

• The initial retrieval's n_results gives diminishing returns after 20-30 candidates. The empirical sweet spot depends on the re-ranker.
• Caching results with LRU/Redis exploits the Pareto pattern of repeated queries (a typical 40-70% hit rate).
• Score thresholds let you discard low-relevance results. The optimal threshold is found empirically on your eval set.
• The dynamic skip (skipping re-ranking when the retrieval is already confident) saves 20-30% of the cost with no precision loss.
• Parallelization: native batching for a local cross-encoder, ThreadPoolExecutor for LLM/Cohere, none for Cohere (it does it server-side).
• Optimizing latency without measuring precision leads to Goodhart's Law: the metric improves but quality falls silently. Always use guardrail metrics.

Checkpoint: before moving on, you should be able to:

• Find the optimal n_results for your pipeline with a benchmark on your eval set.
• Implement an LRU/Redis cache with correct invalidation.
• Design a dynamic re-ranking skip with guardrail metrics.

Next capsule: 07 — Comparing re-ranking techniques.

We covered the three techniques (cross-encoder, LLM, Cohere) and the operational optimizations. Capsule 07 is the consolidation: a decision framework for picking the right technique in a given context, benchmarks side by side, and an actionable flowchart you'll use whenever you start a new project. It's the capsule you come back to month after month.

Resources

1. Caching Strategies for ML Systems (Uber Eng) — Caching patterns in ML production
2. Redis Best Practices for Caching — Distributed caching patterns
3. Speed up sentence-transformers inference — Batching and optimization tips
4. Goodhart's Law (Wikipedia) — Why optimizing what you measure can break what you don't
5. The Four Golden Signals (Google SRE) — Latency, traffic, errors, saturation
6. Asyncio vs Threading vs Multiprocessing in Python — Choosing the right concurrency model

Estimated time: 25-30 minutes Next: 07-technique-comparison-2.md