Capsule 07: Decision framework — which reranker to pick and when

Capsule overview

This is the capsule you'll come back to every time you start a new RAG project. It's the consolidation of capsules 03-06: cross-encoder, LLM-based, Cohere Rerank, operational optimizations. Here you put them side by side, compare trade-offs across the dimensions that matter in the real world (quality, latency, cost, multilingual, maintenance), and apply a reproducible decision framework to make the right call for your case.

It isn't a theory capsule — it's operational. When you're done, you'll have a flowchart you can apply in 5 minutes to any project, with no need to reread the previous 4 capsules.

By the end of this capsule you'll be able to:

• ✅ Compare the three re-ranking techniques across six dimensions (quality, latency, cost, language, maintenance, vendor lock-in)
• ✅ Apply a reproducible decision framework to pick a reranker in under 10 minutes
• ✅ Identify the three most common pipeline patterns (no rerank, single-stage, cascade)
• ✅ Justify your reranker choice to a Tech Lead with quantitative data
• ✅ Anticipate when it's worth combining two rerankers in a cascade
• ✅ Identify the signals for migrating from one reranker to another as your product evolves

Estimated time: 30-35 minutes

Consolidated benchmark: the three techniques side by side

Quality and latency

Cost and maintenance

Operational

The decision framework as a flowchart

                    ┌──────────────────────────────────────┐
                    │ Is your current precision@5 <85%?    │
                    └──────────────┬───────────────────────┘
                                   │
                  ┌────────────────┴────────────────┐
                  │ NO                              │ YES
                  ▼                                 ▼
       ┌──────────────────────┐         ┌──────────────────────────┐
       │ Do NOT use re-ranking│         │ Sensitive data that can  │
       │ The system works.    │         │ NOT leave your infra?    │
       └──────────────────────┘         └──────────┬───────────────┘
                                                   │
                                  ┌────────────────┴────────────┐
                                  │ YES                         │ NO
                                  ▼                             ▼
                       ┌──────────────────────┐      ┌────────────────────────┐
                       │ Local cross-encoder. │      │ Is your corpus         │
                       │ Data stays local.    │      │ multilingual?          │
                       └──────────────────────┘      └──────────┬─────────────┘
                                                                │
                                              ┌─────────────────┴───────────┐
                                              │ YES                         │ NO (pure English)
                                              ▼                             ▼
                                   ┌──────────────────────┐    ┌────────────────────────┐
                                   │ Volume >1M           │    │ Does the team have an  │
                                   │ queries/month?       │    │ MLE to maintain it?    │
                                   └──────────┬───────────┘    └──────────┬─────────────┘
                                              │                           │
                                  ┌───────────┴────────┐      ┌───────────┴──────────┐
                                  │ YES                │ NO   │ YES                  │ NO
                                  ▼                    ▼      ▼                      ▼
                       ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
                       │ Consider a       │  │ Cohere Rerank    │  │ Cohere Rerank    │
                       │ local            │  │ multilingual-v3. │  │ multilingual-v3  │
                       │ multilingual     │  │                  │  │ or LLM rerank.   │
                       │ cross-encoder    │  │ Sensible default.│  │                  │
                       │ vs Cohere ($).   │  │                  │  │                  │
                       └──────────────────┘  └──────────────────┘  └──────────────────┘

Applied to real cases

Three pipeline patterns

Pattern 1: no re-ranking (cosine only)

Query ──> Embedding ──> Cosine retrieve ──> Top-5 ──> LLM

When: MVPs, a small dataset (<10K), precision@5 >85% on your eval set.

Don't add re-ranking just because. If the system works, don't touch it. A rerank has a cost and adds complexity.

Pattern 2: single-stage rerank (the most common)

Query ──> Embedding ──> Cosine retrieve top-20-30 ──> Reranker ──> Top-5 ──> LLM

When: typical production. It covers 80% of cases. The reranker = cross-encoder or Cohere depending on your context.

Recommended configuration:

def standard_rag_pipeline(query: str):
    # Stage 1: broad retrieval
    candidates = collection.query(
        query_texts=[query],
        n_results=25  # broader than the final top-5
    )['documents'][0]

    # Stage 2: re-ranking
    reranked = reranker.rerank(query, candidates, top_k=5)

    # Stage 3: filter by a relevance threshold
    relevant = [doc for doc in reranked if doc.score > THRESHOLD]

    return relevant  # 3-5 genuinely relevant docs

Pattern 3: a multi-stage cascade (for critical cases)

Query ──> Embedding ──> Cosine retrieve top-50 ──> Cross-encoder rerank top-15 ──> LLM rerank top-5 ──> LLM
                                                       (fast, free)                  (precise, expensive)

When: critical domains where an extra 3-5% precision is justified + the budget allows the extra spend.

Why a cascade instead of going straight to the LLM:

• The cross-encoder filters out the "obviously irrelevant" ones cheaply.
• The LLM only evaluates the best 15 → fewer calls, the same quality result.
• Total cost ~70% lower than running the LLM directly over 50 candidates.

def critical_rag_pipeline(query: str):
    # Stage 1: broad retrieval
    candidates = collection.query(
        query_texts=[query],
        n_results=50
    )['documents'][0]

    # Stage 2: the cross-encoder filters down to the top-15 (fast and free)
    cross_top_15 = cross_encoder.rerank(query, candidates, top_k=15)
    cross_docs = [item.document for item in cross_top_15]

    # Stage 3: the LLM rerank refines to the top-5 (precise but expensive)
    llm_top_5 = llm_rerank(query, cross_docs, top_k=5)

    return llm_top_5

When to migrate from one to another

Your product evolves, and so do the rerankers. Signals to migrate:

From "no rerank" to "local cross-encoder":

• Precision@5 fell below 85% on your eval set.
• The volume of queries with a vague/wrong answer exceeds ~10%.
• Support tickets mentioning "the bot can't find things" are rising.

From "local cross-encoder" to "Cohere":

• You started receiving queries in other languages and quality dropped.
• The team has no bandwidth to maintain models.
• The volume grew to the point where paying for managed is justified vs the engineering.

From "Cohere" to "a cascading LLM rerank":

• Stakeholders are asking for precision@5 ≥94%.
• The product entered a critical domain (legal, medical).
• The product's margin allows the extra cost.

Migrating BACKWARDS (a downgrade):

Sometimes the best reranker isn't the most expensive one. Cases where a downgrade makes sense:

• The volume grew 10x. Cohere now costs $500/month — a local cross-encoder keeps similar quality for free.
• Compliance changed and data can't leave → forces a local cross-encoder.
• The product pivoted to monolingual English → an MS MARCO cross-encoder is enough, saving $$/month on Cohere.

The complete decision: a 6-question framework

When you start a project, walk through these 6 questions in order. The first relevant "no" or "yes" defines your reranker:

1. Is your current precision@5 with cosine only already ≥90%?
   → Yes: do NOT use a reranker. Your system works.
   → No: go to 2.

2. Can your data NOT leave your infrastructure (compliance)?
   → Yes: a local cross-encoder. No valid alternatives.
   → No: go to 3.

3. Does your corpus have queries in non-English languages (>10% of traffic)?
   → Yes: Cohere multilingual. An MS MARCO cross-encoder fails here.
   → No: go to 4.

4. Is your domain critical (legal, medical, financial, compliance)?
   → Yes: a cross-encoder + LLM rerank cascade. Worth the extra cost.
   → No: go to 5.

5. Does your team have an MLE/engineer to maintain local models?
   → Yes: a local cross-encoder. Free, total control.
   → No: go to 6.

6. Is your volume <500K queries/month and does the budget allow ~$50/month?
   → Yes: Cohere. Good quality with no maintenance.
   → No: a local cross-encoder. High volume justifies the local setup.

80% of cases end up at a local cross-encoder or Cohere. Only special cases justify an LLM rerank.

Traps and common mistakes when choosing

Trap 1: picking an LLM rerank "because it's the best"

The mistake: you see a benchmark saying "GPT-4 gives 96% precision" and pick it without considering that your volume of 100K queries/day generates $1500/month in cost.

How to prevent it: follow the framework. Start with the cheapest option that meets the requirements. Move up only if the metrics justify it.

Trap 2: comparing rerankers with no eval set of your own

The mistake: you copy benchmarks from blog posts. You assume Cohere gives 91% in your case because it gave 91% in the blog's benchmark.

Symptom: your dataset's reality may be different. Some domains favor certain rerankers over others.

How to prevent it: always build your own eval set (30-100 representative queries with ground truth). Measure every candidate reranker on your eval set. Decide with your own data.

Trap 3: switching rerankers without measuring the effect

The mistake: you migrate from a cross-encoder to Cohere because "Cohere is better". You measured neither before nor after.

Symptom: you don't know whether you improved things or made them worse. Stakeholders ask, and you can't answer.

How to prevent it: A/B test before/after with a fixed eval set. If Cohere improves precision by <3%, consider whether it's worth the recurring cost.

Trap 4: ignoring the migration cost

The mistake: you assume switching rerankers is "just changing the code".

Reality: every migration requires:

• Implementation + testing (2-5 days)
• Re-validation on the eval set
• Updating the monitoring dashboards
• Changes to the runbooks (if the API has a different failure mode)
• A rollback plan

How to prevent it: estimate the engineering cost and compare it against the change's recurring cost. Sometimes it's cheaper to pay for Cohere than to migrate to a cross-encoder.

Trap 5: picking an MS MARCO cross-encoder for a non-English corpus

The mistake: your system processes documentation in Spanish. You pick ms-marco-MiniLM-L-12-v2 because "it's the most popular".

Symptom: precision on Spanish queries is 75%, while on English it's 90%. Inconsistent, and bad.

How to prevent it: for a non-English corpus, use a multilingual cross-encoder or Cohere multilingual. Never MS MARCO on non-English languages in production.

Trap 6: vendor lock-in with no plan B

The mistake: all the code assumes Cohere specifically. Cohere raises prices 50%, and you can't migrate quickly.

How to prevent it: abstract the re-ranking interface behind a class. Have implementations for at least 2 rerankers. Be able to switch with a feature flag.

Applied exercise

Scenario: evaluate the following 4 projects and decide which reranker to pick for each. Justify it with the 6-question framework.

Project A: An internal support chatbot for a SaaS company. 80K documentation articles in English. 500 employees are the users. Current precision with cosine only: 76%. Team: 2 backend engineers.

Project B: A medical RAG for a hospital. 200K clinical articles in Spanish. Compliance: data can NOT leave the hospital. Volume: 1000 queries/day.

Project C: A legal SaaS assistant for LATAM lawyers. 500K cases in Spanish/Portuguese/English. Precision target: 95%. We charge $500/month/user and have 200 premium users. A team of 5 engineers + 1 MLE.

Project D: An internal Stack Overflow search for a startup. 10M questions/answers, mostly English with embedded code. Volume: 50K queries/day. Current precision with no rerank: 84%.

def legal_rag_pipeline(query, stakes="normal"):
    # Stage 1: broad retrieval
    candidates = collection.query(query_texts=[query], n_results=50)['documents'][0]

    # Stage 2: Cohere multilingual rerank
    cohere_top_15 = cohere_rerank(query, candidates, top_k=15, model="rerank-multilingual-v3")

    if stakes == "high":
        # Stage 3: LLM rerank with GPT-4o (premium)
        cohere_docs = [item.document for item in cohere_top_15]
        return llm_rerank(query, cohere_docs, top_k=5, model="gpt-4o")
    else:
        # Cohere only for normal queries
        return cohere_top_15[:5]

Recap and next step

What you learned:

• The three techniques (cross-encoder, Cohere, LLM) cover the quality-cost-latency spectrum. None is universally best.
• A 6-question decision framework resolves 80% of cases in under 10 minutes.
• Three pipeline patterns: no rerank (MVP), single-stage (the default), cascade (critical cases).
• Migrating between rerankers has a real engineering cost — don't migrate without measuring the gain.
• Vendor lock-in is mitigated by abstracting the re-ranking interface. Always have a plan B.
• MS MARCO cross-encoders are for English only. For multilingual, use the multilingual version or Cohere.

Checkpoint: before moving on, you should be able to:

• Apply the 6-question framework to a new project and pick a reranker in <10 min.
• Design a cascading pipeline for a critical case, justifying it with cost/precision.
• Identify the signals for migrating between rerankers (including downgrades).

Next capsule: 08 — Re-ranking System project.

The module's closer: you'll build a real re-ranking system with A/B testing between two techniques, on a dataset of your own. You'll apply everything from capsules 02-07: correct implementation, optimizations, the decision framework, validation with an eval set. It's the project that goes in your portfolio.

Resources

1. BEIR Benchmark Leaderboard — Reproducible reranker comparisons across multiple datasets
2. Pinecone — Choosing a Reranker — A visual decision guide
3. LlamaIndex — Reranker Comparisons — Side-by-side implementations
4. Cohere vs Cross-Encoder Benchmarks — A multilingual use case
5. Reciprocal Rank Fusion Paper — For combining results from several rerankers
6. The Cost-Quality Frontier in Retrieval (Anthropic) — Pareto analysis for retrieval decisions

Estimated time: 30-35 minutes Next: 08-project-reranking-system.md