Capsule 05: Cohere Rerank API — the managed option when you don't want to maintain models

Capsule overview

A local cross-encoder is free but requires maintaining a model in your infrastructure: downloading the model (~80-200 MB), inference on CPU/GPU, occasionally updating to new versions. LLM re-ranking is more expensive and slower, and it depends on OpenAI/Anthropic. Cohere Rerank is the third option: a managed API specialized in re-ranking, with no local infrastructure and no general LLM — a model trained specifically for this task.

For many teams, Cohere Rerank is the practical sweet spot: quality close to an LLM rerank, latency close to a cross-encoder, a reasonable cost, and zero infrastructure overhead. It shines especially in two cases: small teams with no resources to maintain models, and multilingual applications where English-trained cross-encoders (MS MARCO) don't perform well.

This capsule teaches you when to choose it over the other options, how to integrate it correctly, and how to compute the total cost of ownership to decide whether the switch is worth it.

By the end of this capsule you'll be able to:

• ✅ Identify the two scenarios where Cohere Rerank beats a local cross-encoder
• ✅ Implement Cohere Rerank with correct API key and error handling
• ✅ Tell apart the available models (rerank-v3.5, rerank-multilingual-v3) and when to pick each
• ✅ Compute the monthly cost and the TCO compared with a local cross-encoder + LLM rerank
• ✅ Anticipate the critical risk: vendor lock-in and a fallback plan
• ✅ Design an automatic fallback pattern API → local cross-encoder when Cohere fails

Estimated time: 25-30 minutes

The three re-ranking options side by side

Cohere Rerank is the "sensible middle": better multilingual quality than a local cross-encoder, much faster and cheaper than an LLM rerank, with no overhead from maintaining models.

When Cohere beats a local cross-encoder

Scenario 1: a multilingual dataset. If your corpus has queries in Spanish, Portuguese, French, German, Japanese — cross-encoders trained on MS MARCO (English) degrade ~10-15% on non-English languages. Cohere's rerank-multilingual-v3 keeps similar quality across 100+ languages because it was trained specifically on multilingual data.

Scenario 2: a team with no bandwidth to maintain models. A cross-encoder requires:

• Downloading the model on every deploy
• Handling the model load on cold-start
• Deciding when to migrate to new versions
• Managing memory if your app has other models loaded

For small teams with no dedicated MLE, that's real operational overhead. Cohere abstracts it away completely — one API call and you're done.

When Cohere does NOT win

• Monolingual English + a solid technical team: a local cross-encoder gives 90%+ precision for free. Cohere adds ~3% quality for $$/month. Not always worth it.
• Compliance that forbids sending data to external APIs: if your chunks are sensitive (legal, private medical), a local cross-encoder keeps the data in your infra.
• A low-priced product with high volume: if you charge $5/month/user and have 100K queries/month, $200/month on re-ranking may not fit the margins.

The correct implementation

Basic setup

# cohere_rerank.py
import cohere
import os
from dataclasses import dataclass
from typing import List

# Initialize the client with the API key from the environment
co = cohere.Client(api_key=os.getenv("COHERE_API_KEY"))


@dataclass
class CohereRerankResult:
    document: str
    score: float
    original_index: int


def cohere_rerank(
    query: str,
    documents: List[str],
    top_k: int = 5,
    model: str = "rerank-v3.5",
) -> List[CohereRerankResult]:
    """
    Re-rank documents using Cohere's managed Rerank API.

    Models:
      - "rerank-v3.5": general purpose, English-optimized
      - "rerank-multilingual-v3": 100+ languages (use for non-English content)
    """
    response = co.rerank(
        model=model,
        query=query,
        documents=documents,
        top_n=top_k,
    )

    results = []
    for result in response.results:
        results.append(CohereRerankResult(
            document=documents[result.index],
            score=result.relevance_score,
            original_index=result.index,
        ))
    return results

End-to-end usage

import chromadb
from chromadb.utils import embedding_functions

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small",
)
client_chroma = chromadb.PersistentClient(path="./chroma_db")
collection = client_chroma.get_collection("docs", embedding_function=openai_ef)

# Stage 1: broad retrieval
query = "¿cómo configuro autenticación OAuth2 en FastAPI?"
results = collection.query(query_texts=[query], n_results=20)
candidates = results['documents'][0]

# Stage 2: re-ranking with Cohere (multilingual, since the query is in Spanish)
top_5 = cohere_rerank(
    query=query,
    documents=candidates,
    top_k=5,
    model="rerank-multilingual-v3",  # ← multilingual because the query is in Spanish
)

print(f"Top 5 after the Cohere rerank:")
for i, item in enumerate(top_5, 1):
    print(f"\n#{i} (score: {item.score:.3f}, was rank #{item.original_index+1})")
    print(f"   {item.document[:120]}...")

Typical output:

Top 5 after the Cohere rerank:

#1 (score: 0.987, was rank #4)
   FastAPI proporciona OAuth2PasswordBearer para autenticación con username/password...

#2 (score: 0.953, was rank #1)
   Para implementar OAuth2 en FastAPI, primero importar las clases de fastapi.security...

#3 (score: 0.842, was rank #6)
   La configuración de JWT con OAuth2 en FastAPI requiere un secret key y un algoritmo...

Picking the right model

A simple rule: when in doubt, use rerank-multilingual-v3. It loses ~1% quality on pure English vs rerank-v3.5 but gains 10-15% in any other language.

Computing the total cost of ownership

Cohere Rerank charges per document processed, not per query:

Pricing (May 2026):
  rerank-v3.5:               $0.002 per 1,000 documents
  rerank-multilingual-v3:    $0.002 per 1,000 documents

A concrete example:

# A typical configuration
QUERIES_PER_DAY = 5_000
TOP_K_TO_RERANK = 25  # candidates per query
DAYS_PER_MONTH = 30

# The calculation
documents_processed_per_day = QUERIES_PER_DAY * TOP_K_TO_RERANK
documents_processed_per_month = documents_processed_per_day * DAYS_PER_MONTH

cost_per_thousand_docs = 0.002  # USD
monthly_cost = (documents_processed_per_month / 1000) * cost_per_thousand_docs

print(f"Documents re-ranked per month: {documents_processed_per_month:,}")
print(f"Monthly Cohere cost: ${monthly_cost:.2f}")

Output:

Documents re-ranked per month: 3,750,000
Monthly Cohere cost: $7.50

$7.50/month for 5K queries/day reranking 25 candidates. Genuinely cheap compared with an LLM rerank (~$200/month in a similar scenario).

TCO compared: the three options for a real case

Volume: 5K queries/day, 25 candidates reranked, a multilingual dataset.

How to read it: at this volume, Cohere Rerank is 15-25x cheaper than the alternatives once you count the cost of maintaining a cross-encoder. And the multilingual quality is noticeably better than an MS MARCO cross-encoder.

When the cross-encoder wins: VERY high volume (millions of queries/month) where Cohere's linear cost exceeds the fixed cost of maintaining a model.

The fallback pattern: when the API fails

A critical risk: if your RAG pipeline depends on Cohere Rerank and Cohere has an outage, your system fails. The solution: an automatic fallback to a local cross-encoder.

from sentence_transformers import CrossEncoder
import logging

logger = logging.getLogger(__name__)

# Load the cross-encoder at startup (a one-off cost, not per query)
fallback_reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")


def rerank_with_fallback(
    query: str,
    documents: List[str],
    top_k: int = 5,
) -> List[CohereRerankResult]:
    """
    Re-rank with Cohere first. If it fails (timeout, rate limit, error), use the cross-encoder.
    This guarantees the RAG pipeline always returns something, even if the API goes down.
    """
    try:
        # Try Cohere
        return cohere_rerank(
            query=query,
            documents=documents,
            top_k=top_k,
            model="rerank-multilingual-v3",
        )
    except (cohere.CohereAPIError, cohere.CohereConnectionError) as e:
        logger.warning(f"Cohere rerank failed ({e}), falling back to local cross-encoder")

        # Fallback: the local cross-encoder
        pairs = [(query, doc) for doc in documents]
        scores = fallback_reranker.predict(pairs)

        # Convert to the same format
        scored = sorted(
            enumerate(scores),
            key=lambda x: -x[1],
        )[:top_k]

        return [
            CohereRerankResult(
                document=documents[idx],
                score=float(score),
                original_index=idx,
            )
            for idx, score in scored
        ]

Why it matters:

• With no fallback, a Cohere outage = your RAG system is down.
• A local cross-encoder loaded into memory at startup guarantees a response in <200ms even during an outage.
• Quality drops ~3-5% during the fallback, but the system keeps working.

Bonus: logging when the fallback triggers lets you spot patterns (a Cohere outage? a local network problem?). If the fallback triggers frequently, consider adding redundancy or switching providers.

Traps and common mistakes

Trap 1: the API key in the source code

The mistake:

co = cohere.Client(api_key="co-abc123...")  # committed to git

Symptom: the key leaks on GitHub, someone uses it for their own queries, you get an unexpected invoice.

How to prevent it: always from the environment. .env in .gitignore. A pre-commit hook that detects the pattern.

Trap 2: using rerank-v3.5 (English) with multilingual queries

The mistake:

co.rerank(model="rerank-v3.5", query="¿cómo configurar OAuth2?", ...)

Symptom: the Spanish query fails to match correctly against English docs. Degraded results with no explicit error.

How to prevent it: if there's any mix of languages, use rerank-multilingual-v3 even if you lose ~1% on purely English queries.

Trap 3: re-ranking enormous documents

The mistake:

co.rerank(query=q, documents=[doc_of_5000_chars], ...)

Symptom: Cohere truncates docs at its maximum tokens (~512 tokens by default). The re-ranking only "sees" the opening part of the document.

How to prevent it: chunk properly before re-ranking (see M02). Each candidate document should be a chunk of 300-1500 characters, not a full document.

Trap 4: not handling rate limits

The mistake: traffic spikes exceed your Cohere tier's rate limit. Queries start failing.

Symptom: 429 errors at peak hours. Affected users.

How to prevent it:

• Know your tier's rate limit (check the Cohere dashboard).
• Implement exponential backoff on retry.
• For high volume, scale up to a higher tier or cache results for common queries.

import time

def rerank_with_retry(query, docs, top_k=5, max_retries=3):
    for attempt in range(max_retries):
        try:
            return cohere_rerank(query, docs, top_k)
        except cohere.CohereRateLimitError:
            wait = (2 ** attempt) * 2  # 2, 4, 8s
            logger.warning(f"Rate limit, waiting {wait}s")
            time.sleep(wait)
    raise RuntimeError("Cohere rerank failed after retries")

Trap 5: vendor lock-in with no migration plan

The mistake: the whole pipeline assumes Cohere. If prices go up, there's no quick alternative.

How to prevent it: abstract the re-ranking interface behind a class whose methods don't expose the vendor:

class Reranker:
    def rerank(self, query: str, documents: List[str], top_k: int) -> List[RerankResult]:
        raise NotImplementedError

class CohereReranker(Reranker):
    def rerank(self, query, documents, top_k):
        # Cohere implementation
        ...

class CrossEncoderReranker(Reranker):
    def rerank(self, query, documents, top_k):
        # local implementation
        ...

# The rest of the pipeline uses the abstract Reranker
reranker: Reranker = CohereReranker()  # you can swap implementations with a single line

Trap 6: assuming Cohere's order is final

The mistake: you ignore the scores. You take the top_k Cohere returns and that's that.

Symptom: some results with a very low score (e.g. 0.15) get included even though they're barely relevant. They pollute the LLM's context.

How to prevent it: filter by a score threshold after the rerank:

results = cohere_rerank(query, candidates, top_k=10)
# Only include docs with score > 0.5
relevant = [r for r in results if r.score > 0.5]

The optimal threshold is determined empirically on your eval set.

Applied exercise

Scenario: you're the AI Engineer at an e-commerce startup with a presence in LATAM and Spain. The product: a chatbot that answers catalog questions for users.

• 200K chunked products (description, specs, reviews) in Spanish, Portuguese and English
• 30K queries/day (a mix of languages)
• Current pipeline: cosine + cross-encoder ms-marco-MiniLM-L-12-v2
• Metrics: precision@5 = 78% (low, it should be ≥90%)
• Team: 3 engineers, no dedicated MLE

Your job:

1. Diagnose why precision@5 is so low.
2. Decide between the three re-ranking options. Justify it with numbers.
3. Design the implementation plan including a fallback.

queries_per_day = 30_000
top_k_rerank = 25
docs_per_month = queries_per_day * top_k_rerank * 30  # 22.5M
cost = (docs_per_month / 1000) * 0.002  # $45/month

# reranker.py
import cohere
import os
from sentence_transformers import CrossEncoder
from dataclasses import dataclass
from typing import List, Protocol


class Reranker(Protocol):
    def rerank(self, query: str, documents: List[str], top_k: int) -> List[dict]: ...


# Cohere client
_cohere_client = cohere.Client(api_key=os.getenv("COHERE_API_KEY"))

# Fallback cross-encoder loaded at startup
_fallback_model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")


def rerank_with_fallback(query: str, documents: List[str], top_k: int = 5):
    try:
        response = _cohere_client.rerank(
            model="rerank-multilingual-v3",
            query=query,
            documents=documents,
            top_n=top_k,
        )
        return [
            {"document": documents[r.index], "score": r.relevance_score}
            for r in response.results
        ]
    except Exception as e:
        # Fallback: local cross-encoder (we know it gives 78%, but the system works)
        log_event("cohere_rerank_fallback", error=str(e))
        pairs = [(query, doc) for doc in documents]
        scores = _fallback_model.predict(pairs)
        scored = sorted(enumerate(scores), key=lambda x: -x[1])[:top_k]
        return [
            {"document": documents[i], "score": float(s)}
            for i, s in scored
        ]

Recap and next step

What you learned:

• Cohere Rerank is the managed option that offers quality close to an LLM rerank at a cost close to a local cross-encoder.
• It shines in two cases: a multilingual corpus and teams with no resources to maintain models.
• Models: rerank-v3.5 for pure English, rerank-multilingual-v3 for any mix of languages.
• Pricing per document processed: $0.002 per 1K docs. Typical volume = $5-50/month.
• A robust implementation requires: the API key in the environment, a fallback to a local cross-encoder, rate limit handling, and a score threshold to discard low-relevance results.
• The automatic fallback to a local cross-encoder protects you against outages — the pipeline always works, though at slightly lower quality.
• Vendor lock-in is a real risk; abstracting the interface makes it easier to migrate between providers.

Checkpoint: before moving on, you should be able to:

• Decide between a local cross-encoder, an LLM rerank and Cohere for a given scenario.
• Compute Cohere's monthly cost for a specific query volume.
• Implement an automatic fallback to a local cross-encoder if Cohere fails.

Next capsule: 06 — Re-ranking trade-offs and optimizations.

We covered the three re-ranking techniques. But there are finer operational decisions all three share: how many candidates do you pass to the re-ranker? How do you cache the results? When do you re-rank vs when do you trust the direct retrieval? Capsule 06 covers these optimizations, which can improve performance 10-20% without changing the underlying technique.

Resources

1. Cohere Rerank Documentation — Complete official documentation
2. Cohere Rerank Models Guide — A comparison of the available models
3. Cohere Pricing — Up-to-date pricing
4. Multilingual Reranking — Cohere Blog — Multilingual use cases
5. Voyage AI Rerank — A managed alternative to Cohere
6. Anthropic — Contextual Retrieval with Reranking — Patterns combined with rerank

Estimated time: 25-30 minutes Next: 06-tradeoffs-optimizations.md