Capsule 02: Cosine similarity is good, not perfect — the limit that justifies re-ranking

Capsule overview

Cosine similarity made RAG possible. Without it, we couldn't compare millions of vectors in milliseconds. Up to here, everything you've learned about vector databases, embeddings and retrieval depends on cosine similarity as the metric for "how similar are two texts".

But cosine similarity has a limit. It measures semantic overlap between embeddings — how close two vectors are in the space. What it does not measure is real relevance — whether a document actually answers the user's question. Those two concepts coincide 70-80% of the time, but the remaining 20-30% is where your RAG starts returning answers that technically "fit" the query but fail pedagogically.

This capsule shows you the three structural limits of cosine similarity, why they can't be solved by simply "embedding better", and why the natural solution is to add a second scoring stage (re-ranking) over the candidates cosine similarity retrieved. It's the "why" of the whole module — without understanding this limit, the techniques that follow sound like unnecessary optimization.

By the end of this capsule you'll be able to:

• ✅ Explain the difference between semantic similarity and relevance to a query
• ✅ Identify cosine similarity's three failure modes in retrieval
• ✅ Tell apart bi-encoder (cosine) vs cross-encoder (re-ranking) architectures
• ✅ Compute the typical impact on precision (cosine alone vs with re-ranking)
• ✅ Justify the decision to add re-ranking to an existing RAG pipeline
• ✅ Anticipate the "more data fixes retrieval" mistake — sometimes more data alone isn't enough

Estimated time: 30-35 minutes

The insight: similarity ≠ relevance

This is the distinction that defines the whole module. Let's make it concrete with an example.

The user's query: "how do I implement OAuth2 authentication in FastAPI?"

Your vector database has three documents about OAuth2. Cosine similarity ranks them like this:

Doc A — "FastAPI OAuth2 Implementation Guide"                    cosine: 0.85
        "To implement OAuth2 in FastAPI, use OAuth2PasswordBearer
         from fastapi.security..."

Doc B — "OAuth2 Authentication Overview"                         cosine: 0.83
        "OAuth2 is an authorization framework that enables third-party
         applications limited access to user resources..."

Doc C — "OAuth2 Specification (RFC 6749)"                        cosine: 0.81
        "The OAuth 2.0 authorization framework enables a third-party
         application to obtain limited access to an HTTP service..."

All three have similar scores (0.81-0.85). To cosine similarity, all three are "relevant". But let's see what each one actually answers for the real query:

If you pass all three to the LLM, two of the three chunks dilute the context. The LLM sees a lot of text about OAuth2 in general and little about FastAPI specifically. The resulting answer will be correct but generic — exactly the opposite of what the user asked for.

This is the central problem: cosine similarity confused "talks about OAuth2" with "answers how to implement it in FastAPI". Those are related topics, but they're not the same thing.

Cosine similarity's three failure modes

Failure 1: specificity vs generality

The case in the previous example. When a query is specific (FastAPI + OAuth2), but the corpus has both specific and generic documents about the topic, cosine similarity tends to rank them similarly. The difference between "FastAPI OAuth2 guide" and "OAuth2 overview" is ~3-5% in cosine similarity, but it's the difference between "answers the question" and "doesn't".

Why it happens: embeddings encode semantic content, not specificity. A technical document about FastAPI OAuth2 and an abstract document about OAuth2 are close in the vector space because they share vocabulary and core concepts. The difference in specificity (one mentions FastAPI, the other doesn't) gets diluted.

Failure 2: exact keywords vs paraphrase

Query: "how do I use OAuth2PasswordBearer?" (with the exact class name)

Documents in the corpus:

Doc X: "OAuth2PasswordBearer is the FastAPI security class for OAuth2 password flow.
        Import it from fastapi.security..."
        cosine: 0.78  ← lower

Doc Y: "For OAuth2 authentication with username/password in FastAPI, use the password
        flow with the appropriate dependency from the security module..."
        cosine: 0.83  ← higher

Doc Y ranks higher even though it never mentions the exact class name the user asked about. Why? Because embeddings normalize paraphrase — "OAuth2PasswordBearer" and "the appropriate dependency from the security module" end up in nearby regions of the vector space.

For a user who knows the exact name and wants documentation for that specific class, doc X is the answer. Cosine similarity buries it.

This gets worse on queries with identifiers: function names, product IDs, version numbers, error codes. Cosine similarity rarely prioritizes the exact match over a close paraphrase.

Failure 3: query-document relationship vs embedding similarity

This is the subtlest one. Cosine similarity compares two embeddings that were created independently:

# Bi-encoder (what cosine similarity does under the hood)
query_emb = encoder.encode("how do I implement OAuth2 in FastAPI?")    # vector A
doc_emb = encoder.encode("FastAPI OAuth2 guide: use OAuth2Password...")  # vector B

similarity = cosine(query_emb, doc_emb)  # 0.85

The encoder processed the query and the document separately. The document's embedding has no idea it was going to be compared against this specific query. It's like handing an evaluator two arbitrary texts and asking "how similar are these?" with no further context.

Cross-encoders (the basis of re-ranking) do something different:

# Cross-encoder
score = cross_encoder.predict([
    ("how do I implement OAuth2 in FastAPI?",
     "FastAPI OAuth2 guide: use OAuth2Password...")
])  # 0.94 — a higher and better-calibrated score

The cross-encoder analyzes the query-document pair as a single unit. Internally, it attends to how each word of the query relates to each word of the document. It picks up things like "the query asks for an implementation, the document shows an implementation" — relationships cosine similarity can never capture, because the embeddings never "saw" each other.

Bi-encoder vs cross-encoder, side by side

┌────────────────────────────────────────────────────────────────────┐
│                                                                    │
│  BI-ENCODER (cosine similarity in retrieval)                       │
│                                                                    │
│   query  ──┬──>  encoder ──>  vector A                             │
│            │                                                       │
│   doc    ──┴──>  encoder ──>  vector B                             │
│                                                                    │
│                          score = cosine(A, B)                      │
│                                                                    │
│   ✅ Fast (you embed docs ONCE, queries in milliseconds)           │
│   ✅ Scales to millions of docs                                    │
│   ❌ Doesn't analyze the query-doc interaction                     │
│   ❌ Failure modes: specificity, exactness, relevance              │
│                                                                    │
├────────────────────────────────────────────────────────────────────┤
│                                                                    │
│  CROSS-ENCODER (re-ranking)                                        │
│                                                                    │
│   query, doc ──>  encoder (processes BOTH together) ──>  score     │
│                                                                    │
│   ✅ Analyzes the full query-doc interaction                       │
│   ✅ Much better precision                                         │
│   ❌ Slow (re-runs the model for every pair)                       │
│   ❌ Does NOT scale to millions — only the top-K candidates        │
│                                                                    │
└────────────────────────────────────────────────────────────────────┘

The pedagogical insight: they're not competing techniques, they're complementary.

RAG pipeline with re-ranking:
  ┌─────────────────────┐    ┌──────────────────────┐    ┌──────┐
  │ Bi-encoder retrieve │ -> │ Cross-encoder rerank │ -> │ LLM  │
  │   (cosine)          │    │                      │    │      │
  │                     │    │                      │    │      │
  │  1M docs → top-50   │    │  top-50 → top-5      │    │ ...  │
  └─────────────────────┘    └──────────────────────┘    └──────┘
       Fast                       Precise                Generates
       (milliseconds)             (hundreds of ms)

The bi-encoder does the massive first pass (fast but noisy). The cross-encoder refines the top-K candidates (slower but precise). The LLM receives the best 5 chunks instead of 50.

Without re-ranking: the LLM gets the top-5 straight from the cosine retrieve. If 2-3 are false positives, generation gets diluted.

With re-ranking: the LLM gets the top-5 after the second pass. Almost all of them are relevant. Generation improves measurably.

Quantified impact: why the extra cost is worth it

# Typical benchmark over a technical dataset
metrics = {
    "Cosine only (top-5 direct)": {
        "precision@5": 0.70,
        "false_positive_rate": 0.30,
        "latency_p95_ms": 180,
        "cost_per_query_usd": 0.0002,
    },
    "Cosine + cross-encoder rerank": {
        "precision@5": 0.91,
        "false_positive_rate": 0.09,
        "latency_p95_ms": 320,        # +140ms for the re-rank
        "cost_per_query_usd": 0.0002, # the cross-encoder runs locally, no extra cost
    },
    "Cosine + LLM-as-reranker": {
        "precision@5": 0.94,
        "false_positive_rate": 0.06,
        "latency_p95_ms": 850,        # +670ms (LLM API call)
        "cost_per_query_usd": 0.0008, # +$0.0006 per query
    },
}

How to read it:

1. The cross-encoder is the default option. +21% precision, +140ms latency, zero extra cost. Capsule 03 covers this in detail.

2. LLM-as-reranker is for critical cases. +24% precision, but +670ms latency and an additional cost. Use it when a 3% gain justifies the trade-off (e.g. legal, medical). Capsule 04 covers it.

3. False positives drop 70-80%. From 1 out of every 3 docs in the top-5 being garbage, to 1 out of 11. That transforms the system's perceived quality more than any chunking or embedding optimization.

Why "embedding better" doesn't solve these problems

A reasonable question: if cosine similarity fails in these three modes, couldn't you just use a better embedding model?

The short answer: not entirely, because the problems are structural to the bi-encoder approach.

When a better embedding helps:

• Newer models (text-embedding-3-large vs ada-002) cut false positives by ~5-10%
• Domain-specific embeddings (legal-bert, biobert) help in their niche
• Multilingual embeddings reduce cross-language matching errors

When a better embedding is NOT enough:

• The "specific vs generic" distinction requires attending to context that an individual embedding doesn't capture
• Exact matching of identifiers requires comparison at the token level, not at the aggregated embedding level
• The query-document relationship can only be captured by comparing the two in the same pass (a cross-encoder)

The pedagogical conclusion: improving embeddings complements re-ranking, it doesn't replace it. The bi-encoder + cross-encoder architecture is the right combination — no single component reaches the quality of the two together.

When should you add re-ranking?

It isn't always the answer. Re-ranking adds 100-700ms of latency and, depending on the technique, a monetary cost. Consider adding it when:

A reasonable default: you start with no re-ranking, measure precision on your eval set, and add re-ranking when precision@5 falls below 85%.

Traps and common mistakes

Trap 1: confusing a high cosine score with high relevance

The mistake: you see cosine 0.85 and assume "it's very relevant". You pass it to the LLM without filtering.

Symptom: the LLM's answers are consistently correct but generic. They don't address the specific aspect the user asked about.

How to prevent it: measure precision on an eval set. If recall is high (it finds docs about the topic) but precision is low (the docs don't answer the specific query), add re-ranking.

Trap 2: optimizing the embedding model and forgetting re-ranking

The mistake: the team spends weeks comparing text-embedding-3-small vs large vs Cohere vs Voyage. Each one improves things marginally. Nobody tried re-ranking, which would give a 20% gain with one day of work.

Symptom: a terribly unbalanced gain-vs-effort Pareto.

How to prevent it: re-ranking first (big win for little effort), then embedding optimization (small win for a lot of effort).

Trap 3: re-ranking the direct top-5

The mistake: you do a cosine retrieval with n_results=5, then re-rank those 5.

Symptom: re-ranking doesn't improve anything, because the candidates were already filtered by cosine. You're re-ordering 5 docs that are probably the right ones, but some false positives were discarded before the re-rank ever ran.

How to prevent it: retrieval with n_results=20-50, and let re-ranking pick the final top-5. Capsule 03 covers the pattern.

Trap 4: comparing a cosine score with a cross-encoder score

The mistake: you try to "rank them jointly" by summing cosine + cross-encoder scores.

Symptom: the scores have different ranges (cosine ~0-1, cross-encoder logits ~-10 to 10). Summing them without normalizing produces meaningless rankings.

How to prevent it: either normalize both to [0,1] before combining, or use Reciprocal Rank Fusion (RRF), which combines rankings while ignoring the absolute scores. Covered in M05.

Trap 5: assuming re-ranking "always improves things"

The mistake: you add re-ranking without measuring. You assume precision goes up.

Symptom: in some domains (a very specific corpus, very direct queries) re-ranking barely improves anything because cosine was already fine.

How to prevent it: always A/B test. A fixed eval set, measure precision@K before and after. If the gain is <3%, it isn't worth the extra latency.

Trap 6: slow re-ranking with no batching

The mistake: you call the cross-encoder once per query-document pair.

Symptom: re-ranking 50 docs takes 5 seconds instead of 200ms.

How to prevent it: cross-encoders process batches efficiently. Hand them the full list of pairs at once:

# ❌ Slow
for doc in candidates:
    score = model.predict([(query, doc)])

# ✅ Fast (batch)
scores = model.predict([(query, doc) for doc in candidates])

Applied exercise

Scenario: you're the AI Engineer at a cloud computing company. Your RAG system has:

• 800K chunks of technical documentation indexed with OpenAI text-embedding-3-small
• Cosine similarity for retrieval
• n_results=5 passed straight to the LLM
• No re-ranking

Current metrics over an eval set of 100 real queries:

• Precision@5: 71%
• Recall@5: 88% (it finds docs about the topic)
• p95 latency: 280ms total

The product team asks: "we want precision@5 of at least 88% by end of month. Is that reachable?"

Your job:

1. Diagnose whether the problem is retrieval (recall) or relevance (precision).
2. Propose a concrete solution and estimate the impact.
3. Identify the main trade-off and how you'd justify it to the product team.

# Before
results = collection.query(query_texts=[query], n_results=5)
top_5_chunks = results['documents'][0]
# Pass straight to the LLM

# After
# Stage 1: broad retrieval
results = collection.query(query_texts=[query], n_results=30)
candidates = results['documents'][0]

# Stage 2: re-ranking with a cross-encoder
from sentence_transformers import CrossEncoder
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
pairs = [(query, doc) for doc in candidates]
scores = reranker.predict(pairs)

import numpy as np
top_5_idx = np.argsort(scores)[::-1][:5]
top_5_chunks = [candidates[i] for i in top_5_idx]
# Pass to the LLM

Recap and next step

What you learned:

• Cosine similarity measures semantic overlap, not real relevance to a specific query.
• Three failure modes: specificity (generic vs specific), exactness (paraphrase vs keywords), relationship (independent embeddings vs query-doc interaction).
• Bi-encoder (cosine) and cross-encoder (re-ranking) are complementary, not competing. The optimal architecture combines them.
• Re-ranking adds 100-700ms of latency in exchange for 15-25% more precision. A typically favorable trade-off.
• Improving the embedding model helps marginally; adding re-ranking helps dramatically. Start with re-ranking.
• Re-ranking isn't always necessary — it depends on the dataset, the query types and the SLA.

Checkpoint: before moving on, you should be able to:

• Distinguish semantic similarity from relevance with an example of your own.
• Explain why cosine similarity fails on specific queries or on exact identifiers.
• Diagnose whether a RAG system needs re-ranking based on its metrics (precision vs recall).

Next capsule: 03 — Cross-encoder re-ranking.

You've just understood the module's "why". Now comes the "how". The cross-encoder is the default technique — good precision, reasonable latency, zero cost. Capsule 03 teaches you to implement it with sentence-transformers, compare the available models (MiniLM L-6 vs L-12 vs TinyBERT), and benchmark the impact on your eval set. It's the first concrete improvement you'll apply to your RAG pipeline.

Resources

1. Sentence Transformers — Cross-Encoders Documentation — Official documentation
2. Pinecone — Re-ranking Explained — Bi-encoder vs cross-encoder comparison
3. BEIR Benchmark — Empirical retrieval comparisons with and without re-ranking
4. Khattab & Zaharia — ColBERT Paper — Late interaction as an alternative to the cross-encoder
5. Anthropic — Contextual Retrieval — Another technique that complements re-ranking
6. MS MARCO — The dataset behind ms-marco-MiniLM — The dataset used to train the most popular cross-encoders

Estimated time: 30-35 minutes Next: 03-cross-encoder-reranking.md