Capsule 04: LLM-based re-ranking — the expensive option when 3% extra precision justifies the cost

Capsule overview

The cross-encoder is the reasonable default for re-ranking. But there are cases where its accuracy isn't enough: legal, where a wrong answer has regulatory consequences; medical, where the cost of a false positive is clinical; financial, where the cited information gets audited. In those cases it's worth stepping up and using the most sophisticated model available — a general-purpose LLM — as the relevance evaluator.

LLM-based re-ranking isn't magic. It takes each candidate (query, document) pair and asks an LLM to assign a relevance score. The model "reads" each pair the way a human would: it understands the question, reads the document, judges whether it answers, and returns a number. It's more expensive and slower than a cross-encoder, but it gains an extra 3-5% precision on the hard cases where the cross-encoder was failing.

This capsule teaches you when to choose it, how to implement it robustly (including how to avoid the obvious traps of a badly designed prompt), and how to justify the extra cost with numbers.

By the end of this capsule you'll be able to:

• ✅ Identify the three scenarios where LLM re-ranking is justified over a cross-encoder
• ✅ Implement LLM re-ranking with OpenAI using structured outputs
• ✅ Design the evaluation prompt correctly (explicit criteria, a calibrated scale)
• ✅ Compute the total monthly cost of the change for a given volume
• ✅ Anticipate the most expensive trap: inconsistent scoring across calls (high variance)
• ✅ Tell apart "LLM as reranker" (this capsule) from "LLM as retriever" (which we do NOT recommend)

Estimated time: 30-35 minutes

When to choose LLM re-ranking over a cross-encoder

A local cross-encoder is the default option for good reasons: free, fast (~150ms), 90%+ precision in typical cases. LLM re-ranking is only justified if at least one of these three factors applies:

Factor 1: a domain where 3% extra precision has a disproportionate impact

In these contexts, going from 91% (cross-encoder) to 94% (LLM-based) means cutting false positives by 33%. If your product processes 10,000 queries a month in a critical context, those 300 fewer queries with a doubtful answer can be the difference between "a trustworthy system" and "a system with legal liability".

Factor 2: semantically complex queries the cross-encoder handles worse

Cross-encoders are trained on datasets like MS MARCO — short, factual queries, in English. They work excellently for those cases. But they degrade when the queries are:

• Multi-step or reasoning-based: "which documents discuss X taking constraint Y into account?"
• Comparative: "what's the difference between A and B according to authors Z?"
• Hypothetical: "if I had scenario X, which documents would apply?"
• In non-English languages: legal Spanish, technical Portuguese, etc.

Large LLMs (GPT-4, Claude) reason better about those queries because their training covers more linguistic diversity and more reasoning patterns.

Factor 3: the cost is trivial vs the product's value

If your product charges $500/month per user and the extra cost of LLM re-ranking is $20/month per user, the math is obvious. If you charge $5/month, the extra cost may be prohibitive.

Calculating the extra cost:

# Approximate cost per query with LLM re-ranking
# (assumes reranking the top-20 candidates with GPT-4o-mini)
QUERIES_PER_DAY = 1000
TOP_K_TO_RERANK = 20
TOKENS_PER_PAIR = 500  # query + chunk + prompt overhead
TOKENS_OUT_PER_PAIR = 5  # just the numeric score

GPT_4O_MINI_INPUT_PER_MILLION = 0.15  # USD
GPT_4O_MINI_OUTPUT_PER_MILLION = 0.60  # USD

input_tokens_per_query = TOP_K_TO_RERANK * TOKENS_PER_PAIR
output_tokens_per_query = TOP_K_TO_RERANK * TOKENS_OUT_PER_PAIR

cost_per_query_input = (input_tokens_per_query / 1_000_000) * GPT_4O_MINI_INPUT_PER_MILLION
cost_per_query_output = (output_tokens_per_query / 1_000_000) * GPT_4O_MINI_OUTPUT_PER_MILLION
cost_per_query_total = cost_per_query_input + cost_per_query_output

monthly_cost = cost_per_query_total * QUERIES_PER_DAY * 30

print(f"Cost per query: ${cost_per_query_total:.5f}")
print(f"Monthly cost ({QUERIES_PER_DAY} queries/day): ${monthly_cost:.2f}")

Output:

Cost per query: $0.00154
Monthly cost (1000 queries/day): $46.20

How to read it: ~$46/month in re-ranking for 30K queries. If your product generates more value than that, the decision is trivial. If not, use the cross-encoder.

The correct implementation with structured outputs

The naive implementation (the one the old version of this capsule left behind) has three problems: ambiguous prompts, fragile parsing of the response, inconsistent scoring. Let's do it properly.

Setup with structured outputs

# llm_reranker.py
from openai import OpenAI
from pydantic import BaseModel, Field
from typing import List
import os
import time


class RelevanceScore(BaseModel):
    score: float = Field(
        ge=0.0, le=10.0,
        description="Relevance score from 0 to 10"
    )
    reasoning: str = Field(
        description="One-sentence justification for the score"
    )


client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))


SYSTEM_PROMPT = """You are an expert relevance evaluator for retrieval systems.

Given a user query and a candidate document, score how well the document answers
the query on a scale of 0 to 10:

- 10: Document directly and completely answers the query with specific details
- 7-9: Document answers the query but with some gaps or generality
- 4-6: Document is on-topic but doesn't directly answer this specific query
- 1-3: Document mentions related concepts but isn't useful for this query
- 0: Document is completely unrelated

Be strict. Most documents should score 4-7. Reserve 9-10 for genuinely excellent matches.
Reserve 0-2 for clearly wrong matches.

Return JSON with score (number 0-10) and reasoning (one sentence)."""


def llm_rerank_pair(query: str, document: str, model: str = "gpt-4o-mini") -> RelevanceScore:
    """Score a single (query, document) pair."""
    response = client.beta.chat.completions.parse(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": f"Query: {query}\n\nDocument:\n{document[:1500]}",
            },
        ],
        response_format=RelevanceScore,
        temperature=0.1,  # low for consistency
    )
    return response.choices[0].message.parsed

Why response_format=RelevanceScore: it guarantees the LLM returns parseable JSON with exactly the fields you expect. Without it, the model sometimes replies with free text and the parsing fails.

Why temperature=0.1: re-ranking needs consistency. If the same query+doc gives scores of 8.5 and 6.2 on different runs, the ranking becomes unstable. A low temperature (not exactly 0, which blocks structured outputs on some models) reduces that variance.

Why the explicit guidance in the prompt: without clear criteria, models tend to give inflated scores (everything between 7-9). The system ends up badly calibrated and every doc looks relevant.

The complete re-ranking function

import numpy as np
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass


@dataclass
class RerankedDoc:
    document: str
    score: float
    reasoning: str
    original_rank: int


def llm_rerank(
    query: str,
    documents: List[str],
    top_k: int = 5,
    model: str = "gpt-4o-mini",
    parallel_workers: int = 5,
) -> List[RerankedDoc]:
    """
    Re-rank a list of candidate documents using an LLM.
    Calls are parallelized via ThreadPoolExecutor.
    """
    def score_one(args):
        original_rank, doc = args
        try:
            result = llm_rerank_pair(query, doc, model=model)
            return RerankedDoc(
                document=doc,
                score=result.score,
                reasoning=result.reasoning,
                original_rank=original_rank,
            )
        except Exception as e:
            print(f"Failed to score doc {original_rank}: {e}")
            return RerankedDoc(
                document=doc,
                score=0.0,  # fallback: treat it as irrelevant
                reasoning=f"scoring failed: {e}",
                original_rank=original_rank,
            )

    args = list(enumerate(documents))
    with ThreadPoolExecutor(max_workers=parallel_workers) as executor:
        scored_docs = list(executor.map(score_one, args))

    # Sort by descending score
    scored_docs.sort(key=lambda d: -d.score)
    return scored_docs[:top_k]

End-to-end usage

# pipeline_with_llm_rerank.py
import chromadb
from chromadb.utils import embedding_functions

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small"
)
client_chroma = chromadb.PersistentClient(path="./chroma_db")
collection = client_chroma.get_collection("docs", embedding_function=openai_ef)

query = "How does HNSW handle 10M+ vectors with limited RAM?"

# Stage 1: broad retrieval
results = collection.query(query_texts=[query], n_results=20)
candidates = results['documents'][0]

# Stage 2: re-ranking with an LLM
start = time.perf_counter()
top_5 = llm_rerank(query, candidates, top_k=5, model="gpt-4o-mini")
elapsed_ms = (time.perf_counter() - start) * 1000

print(f"Re-ranking took {elapsed_ms:.0f}ms (with parallel workers)")
print(f"\nTop 5 after LLM re-rank:")
for i, doc in enumerate(top_5, 1):
    print(f"\n#{i} (score: {doc.score}, was rank #{doc.original_rank+1} before)")
    print(f"  Reasoning: {doc.reasoning}")
    print(f"  Doc: {doc.document[:120]}...")

Typical output:

Re-ranking took 1240ms (with parallel workers)

Top 5 after LLM re-rank:

#1 (score: 9.0, was rank #3 before)
  Reasoning: Discusses HNSW memory scaling specifically with examples for 10M+ vectors.
  Doc: HNSW graph memory scales linearly with M parameter. For 10M vectors with M=16...

#2 (score: 8.5, was rank #1 before)
  Reasoning: Covers HNSW tuning for memory but focuses on smaller datasets (<1M).
  Doc: Tuning HNSW parameters for production: lower M values reduce memory at cost...

#3 (score: 7.0, was rank #5 before)
  Reasoning: Mentions IVF+PQ as alternative to HNSW for memory-constrained large datasets.
  Doc: When HNSW exceeds available RAM, consider IVF+PQ which compresses vectors...

Note the ranking movements: the original doc #3 (which cosine placed third) climbs to #1 after the re-ranking because it's the most specific about the "10M+ vectors with limited RAM" case. The original doc #1 (more general about HNSW tuning) drops to #2.

Designing the prompt: what makes the difference

The prompt is the most sensitive piece. Three critical elements:

1. Explicit scoring criteria

❌ Bad: "Score how relevant this is on a scale of 0-10."
✅ Good: "10: directly answers with specifics. 7-9: answers with gaps. 4-6: on-topic
        but doesn't answer this specific query. 1-3: related concepts only. 0: unrelated."

Without criteria, the LLM invents its own scale (typically inflated toward 7-9).

2. Calibrating the severity

✅ "Be strict. Most documents should score 4-7. Reserve 9-10 for genuinely excellent matches."

LLMs are "kind" by default — everything deserves 8/10. Explicitly asking for strictness improves the separation between good and bad candidates.

3. A mandatory short rationale

✅ Ask for reasoning: "score (number) + reasoning (one sentence)"

Forcing the LLM to briefly justify the score improves the score's quality (implicit chain-of-thought) and gives you auditability — you can review why a specific doc ranked high or low.

A prompt for Spanish queries

If your corpus and queries are in Spanish, adjust the prompt:

SYSTEM_PROMPT_ES = """Eres un evaluador experto de relevancia para sistemas de retrieval.

Dada una consulta del usuario y un documento candidato, evalúa qué tan bien el documento
responde la consulta en una escala de 0 a 10:

- 10: El documento responde directa y completamente la consulta con detalles específicos
- 7-9: El documento responde la consulta pero con algunas lagunas o generalidad
- 4-6: El documento está en el tema pero no responde esta consulta específica
- 1-3: El documento menciona conceptos relacionados pero no es útil para esta consulta
- 0: El documento es completamente irrelevante

Sé estricto. La mayoría de documentos deberían puntuar entre 4 y 7. Reserva 9-10 para
matches genuinamente excelentes. Reserva 0-2 para matches claramente equivocados.

Devuelve JSON con score (número 0-10) y reasoning (una oración justificando el score)."""

Note that the "JSON with score and reasoning" instruction stays in Spanish, so the model doesn't mix languages in its output.

Traps and common mistakes

Trap 1: temperature=0 with structured outputs

The mistake:

response = client.beta.chat.completions.parse(
    ...,
    response_format=RelevanceScore,
    temperature=0,
)

Symptom: some models throw an error because temperature=0 disables the sampling that structured outputs need.

How to prevent it: use temperature=0.1 (low enough for consistency, high enough for structured outputs to work).

Trap 2: sequential calls with no parallelism

The mistake:

for doc in candidates:
    score = llm_rerank_pair(query, doc)  # sequential

Symptom: re-ranking 20 candidates takes 20 × 800ms = 16 seconds. Unusable in production.

How to prevent it: a ThreadPoolExecutor with 5-10 workers. The calls are I/O bound (they're waiting on OpenAI), so parallelizing is trivial. Total latency drops to ~1.5 seconds.

Trap 3: a prompt with no concrete criteria

The mistake: a minimalist prompt along the lines of "Score this from 0-10".

Symptom: every doc gets a score between 6 and 9. The separation between relevant and irrelevant disappears. The re-ranking doesn't improve anything because almost everything ends up tied.

How to prevent it: explicit criteria + a strictness instruction (covered above).

Trap 4: passing enormous documents to the LLM

The mistake:

content=f"Query: {query}\n\nDocument:\n{document}"  # a 10K-character document

Symptom: per-query costs spike (more tokens), latency goes up, and sometimes the LLM gets distracted by irrelevant parts of the document.

How to prevent it: truncate the document to 1500-2000 characters. If your chunks are bigger, consider chunking them before re-ranking, or take just the first N characters. The critical info is usually at the start of the chunk.

content=f"Query: {query}\n\nDocument:\n{document[:1500]}"

Trap 5: using GPT-4 to rerank small chunks

The mistake: you pick GPT-4 (a large model) for every rerank "for maximum quality".

Symptom: costs are 10x higher than GPT-4o-mini. The re-ranking quality between the two models is similar (a 5-10% difference), which doesn't justify the 10x cost.

How to prevent it: GPT-4o-mini is the quality/cost sweet spot for re-ranking. GPT-4 only for critical cases where the extra 5-10% precision is justified.

Trap 6: not handling API failures

The mistake: a call fails with a timeout/rate limit. The score comes back as None. The subsequent sort crashes.

Symptom: one failure among the 20 parallel calls takes down the entire re-ranking.

How to prevent it: try/except on each call with a reasonable fallback (score=0 = "treat it as irrelevant, it goes to the end"). The pipeline carries on with the 19 docs that scored correctly.

try:
    result = llm_rerank_pair(query, doc)
    score = result.score
except Exception as e:
    print(f"Failed to score doc: {e}")
    score = 0.0  # fallback

Applied exercise

Scenario: you're the Tech Lead at a digital case-law company (legal case search). The current system:

• 150K chunked legal cases, indexed with OpenAI text-embedding-3-large
• Cosine similarity + cross-encoder re-ranking (ms-marco-MiniLM-L-12-v2)
• 5,000 queries/day, mostly lawyers researching precedent
• Current metrics: precision@5 = 88%, p95 latency = 600ms

The premium client asks: "For critical cases (multi-million-dollar litigation), we need precision@5 ≥ 94%. We're willing to pay more for queries flagged 'high-stakes'."

Your job:

1. Design the solution: do you replace the cross-encoder with an LLM rerank? Combine the two? Route by query type?
2. Estimate the extra monthly cost.
3. Identify a non-obvious risk in your solution.

def rerank(query: str, candidates: list[str], stakes: str = "normal") -> list:
    """
    Re-rank with a tier based on the query's importance.
    
    stakes:
      - "normal": cross-encoder (default, covers 95% of queries)
      - "high": cross-encoder + LLM rerank in a cascade (premium)
    """
    if stakes == "normal":
        # The current pipeline
        return cross_encoder_rerank(query, candidates, top_k=5)
    
    elif stakes == "high":
        # Cascade: cross-encoder first, then the LLM over the top-10
        cross_top_10 = cross_encoder_rerank(query, candidates, top_k=10)
        cross_docs = [item.document for item in cross_top_10]
        
        # LLM-rerank over the cross-encoder's 10 best
        return llm_rerank(query, cross_docs, top_k=5, model="gpt-4o")

@app.post("/search")
async def search(query: str, stakes: str = "normal"):
    # Validate permission for "high stakes" (premium plan only)
    if stakes == "high" and not user.is_premium:
        stakes = "normal"
    
    # Pipeline
    candidates = vector_db.query(query, n_results=30)
    top_5 = rerank(query, candidates, stakes=stakes)
    answer = llm_generate(query, top_5)
    return answer

DAILY_HIGH_STAKES = 500
PAIRS_PER_QUERY = 10
TOKENS_IN_PER_PAIR = 600
TOKENS_OUT_PER_PAIR = 30

# GPT-4o pricing (May 2026)
INPUT_PER_MILLION = 2.50  # USD
OUTPUT_PER_MILLION = 10.00  # USD

input_tokens_daily = DAILY_HIGH_STAKES * PAIRS_PER_QUERY * TOKENS_IN_PER_PAIR
output_tokens_daily = DAILY_HIGH_STAKES * PAIRS_PER_QUERY * TOKENS_OUT_PER_PAIR

cost_input = (input_tokens_daily / 1_000_000) * INPUT_PER_MILLION
cost_output = (output_tokens_daily / 1_000_000) * OUTPUT_PER_MILLION
daily_cost = cost_input + cost_output
monthly_cost = daily_cost * 30

print(f"Daily cost: ${daily_cost:.2f}")
print(f"Monthly cost: ${monthly_cost:.2f}")

Daily cost: $9.00
Monthly cost: $270.00

Recap and next step

What you learned:

• LLM re-ranking gains an extra 3-5% precision over a cross-encoder at the cost of 5-10x more latency and a monetary cost.
• It's only justified in three scenarios: critical domains (legal/medical/financial), semantically complex queries, or when the cost is trivial vs the product's value.
• A correct implementation requires: structured outputs (safe parsing), a prompt with explicit criteria and calibrated strictness, parallelism via ThreadPoolExecutor, failure handling.
• GPT-4o-mini is the quality/cost sweet spot. GPT-4 only if the extra 5% justifies the 10x cost.
• A cross-encoder → LLM cascade is more efficient than the LLM alone: the cross-encoder filters first, the LLM refines the finalists.
• Routing by query type (normal vs high-stakes) lets you offer LLM rerank as a premium feature without blowing up your overall costs.

Checkpoint: before moving on, you should be able to:

• Compute the monthly cost of adding LLM re-ranking for a given volume.
• Design a prompt with explicit criteria and calibrated strictness.
• Identify when a cross-encoder is enough vs when an LLM rerank is justified.

Next capsule: 05 — Cohere Rerank API.

We covered the cross-encoder (default) and LLM-based (premium). Capsule 05 covers the third option: managed rerankers like Cohere Rerank — specialized APIs that aren't general LLMs but offer close-to-LLM quality without requiring local infrastructure. It's the middle option: better than a local cross-encoder in some cases, cheaper than an LLM, with no models to maintain.

Resources

1. OpenAI — Structured Outputs — Official structured outputs documentation with Pydantic
2. OpenAI — Pricing Calculator — Up-to-date pricing
3. Anthropic — Claude as a Reranker — A use case for Claude in re-ranking
4. LlamaIndex — LLM Reranker Implementation — The complete pattern in LlamaIndex
5. Why Use an LLM for Reranking? (Pinecone) — A comparison of methods with benchmarks
6. BEIR Benchmark — Reranking Comparisons — Empirical results for different rerankers

Estimated time: 30-35 minutes Next: 05-cohere-rerank.md