Module 3: Embedding Models Compared

MTEB: Massive Text Embedding Benchmark

Next →

Capsule overview

MTEB (Massive Text Embedding Benchmark) is the de facto standard for evaluating embedding models. Created by HuggingFace in 2022, it evaluates 58+ datasets across 8 different tasks (retrieval, classification, clustering, etc.). MTEB scores are the main metric you use to compare models.

In this capsule you'll learn what MTEB is, how to interpret scores, the 8 tasks it evaluates, the limitations of benchmarks, and how to run MTEB on your own models. You'll also see the current leaderboard and which models dominate.

By the end, you'll be able to objectively evaluate which model to choose based on MTEB scores.

What is MTEB?

Definition:

MTEB (Massive Text Embedding Benchmark): A framework for evaluating embeddings across 58 datasets over 8 different tasks. It produces an average score (0-100) that summarizes performance.

Developed by: HuggingFace (2022)

Goal: Standardize embedding evaluation (before, every paper used different metrics).

8 tasks evaluated:

TaskDescriptionDatasetsMetric
RetrievalFind relevant documents for a query15nDCG@10
ClassificationClassify text into categories12Accuracy
ClusteringGroup similar texts11V-measure
Pair ClassificationAre two texts similar?3AP
RerankingReorder docs by relevance4MAP
STS (Semantic Similarity)Predict a similarity score10Spearman
SummarizationFind the best summary1Spearman
BitextMiningAlign translations2F1

Total: 58 datasets

MTEB Leaderboard (2026)

Top models (English):

Ranking | Model                     | MTEB Score | Dims | Type
--------|---------------------------|------------|------|----------
#1      | gte-Qwen2-7B-instruct     | 72.3       | 3584 | Open-source
#2      | text-embedding-3-large    | 64.6       | 3072 | OpenAI (API)
#3      | bge-large-en-v1.5         | 64.2       | 1024 | Open-source
#4      | instructor-xl             | 63.9       | 768  | Open-source
#5      | e5-mistral-7b-instruct    | 63.7       | 4096 | Open-source
#6      | text-embedding-3-small    | 62.3       | 1536 | OpenAI (API)
#7      | bge-base-en-v1.5          | 62.8       | 768  | Open-source
#8      | e5-large-v2               | 62.0       | 1024 | Open-source
#9      | all-mpnet-base-v2         | 57.8       | 768  | Open-source
#10     | all-MiniLM-L6-v2          | 56.3       | 384  | Open-source

Source: MTEB Leaderboard (HuggingFace)

Interpreting MTEB scores

Score range:

MTEB Score | Performance | Recommended use
-----------|-------------|------------------
70-100     | Excellent   | Critical production (RAG, legal)
65-70      | Very good   | Standard production (RAG)
60-65      | Good        | Semantic search, prototyping
55-60      | Acceptable  | MVP, demos
<55        | Basic       | Testing only

Breakdown by task:

Example: OpenAI text-embedding-3-large

Task                  | Score | Interpretation
----------------------|-------|----------------
Retrieval             | 60.0  | Very good (RAG-ready)
Classification        | 70.1  | Excellent (better than retrieval)
Clustering            | 51.9  | Acceptable (not specialized)
STS                   | 84.0  | Excellent (semantic similarity)
Reranking             | 64.2  | Very good
Pair Classification   | 88.4  | Excellent
Summarization         | 30.8  | Weak (not its strength)
BitextMining          | 85.7  | Excellent (multilingual)

MTEB Average: 64.6

Insights:

  • Excellent for semantic similarity (STS 84.0)
  • Good for retrieval (60.0) → RAG-ready
  • Weak in summarization (30.8) → Don't use it for that

Tasks in detail

1. Retrieval (most important for RAG):

What it evaluates:

# Given a query, find relevant documents
Query: "How to install Python?"
Corpus: 10,000 documents
Task: Rank documents by relevance

Metric: nDCG@10 (normalized Discounted Cumulative Gain)
- 1.0 = Perfect (relevant docs in top 10)
- 0.0 = Terrible

Datasets:

  • NQ (Natural Questions)
  • MS MARCO
  • HotpotQA
  • FiQA
  • ...15 total

2. Classification:

What it evaluates:

# Classify text into categories
Input: "This movie was amazing!"
Output: "Positive"

Metric: Accuracy
- % of correct classifications

Datasets:

  • Amazon Reviews
  • IMDB
  • Banking77
  • ...12 total

3. Clustering:

What it evaluates:

# Automatically group similar texts
Input: 1000 texts without labels
Output: Clusters (e.g.: [Tech, Sports, Politics])

Metric: V-measure
- How well the clusters match the true labels

4. STS (Semantic Textual Similarity):

What it evaluates:

# Predict similarity between 2 texts (score 0-5)
Text A: "The cat sat on the mat"
Text B: "A feline rested on a rug"
Expected: 4.0 (very similar)

Metric: Spearman correlation
- Correlation between predicted and true scores

Limitations of MTEB

❌ 1. Bias toward English

# MTEB mainly evaluates English datasets
# Performance in Spanish, Chinese, etc. can be different

Solution: MTEB Multilingual (evaluates 100+ languages).

❌ 2. Doesn't evaluate latency/cost

# MTEB only evaluates performance (accuracy, nDCG)
# It doesn't consider:
# - Latency (API 100ms vs local 5ms)
# - Cost ($0.13/1M tokens vs $0)
# - Model size (80MB vs 1.2GB)

Solution: Complement with latency/cost benchmarks (Capsules 05-06).

❌ 3. Datasets may not represent your domain

# MTEB uses public datasets (Wikipedia, Reddit, etc.)
# Your domain may be different (legal, medical)

Solution: Evaluate on your own datasets (domain-specific).

❌ 4. Overfitting to the benchmark

# Models can be "tuned" specifically for MTEB
# Production performance may differ

Solution: A/B testing in production (validate with real users).

Run MTEB on your models

Installation:

pip install mteb

Basic code:

from mteb import MTEB
from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Define the evaluation (only 1 task for the demo)
evaluation = MTEB(tasks=["Banking77Classification"])

# Run it
results = evaluation.run(model, output_folder="results/")

print(results)

Output (example):

{
  "Banking77Classification": {
    "test": {
      "accuracy": 0.7234,
      "f1": 0.7123
    }
  }
}

Full evaluation (58 datasets):

# WARNING: Takes ~8 hours on a GPU
evaluation = MTEB(tasks=MTEB(task_langs=['en']).tasks)
results = evaluation.run(model, output_folder="results/")

Selective evaluation (retrieval only):

# Only retrieval tasks (most relevant for RAG)
retrieval_tasks = [
    "NFCorpus",
    "SciFact",
    "FiQA2018",
    "ArguAna",
    "TRECCOVID"
]

evaluation = MTEB(tasks=retrieval_tasks)
results = evaluation.run(model)

Practical comparison: MTEB vs Real-World

Experiment:

# Model A: MTEB 64 (OpenAI 3-large)
# Model B: MTEB 58 (all-mpnet-base-v2)

# Question: Is Model A 10% better in production?

# Answer: NOT necessarily.
# Reasons:
# 1. The domain may be different (MTEB is general)
# 2. Latency matters (model B is 10x faster)
# 3. Cost matters (model B is $0/query)

Conclusion: MTEB is a guide, not absolute truth. Validate in your domain.

Exercises

Exercise 1: Interpret MTEB

Model X has these scores:

Retrieval: 58
Classification: 72
Clustering: 45
STS: 80
MTEB Average: 62

Which task is this model best at?

See solution

Answer: Semantic Similarity (STS: 80)

Interpretation:

  • Excellent for comparing similarity between texts
  • Very good for classification (72)
  • Acceptable for retrieval (58) → OK for basic RAG
  • Weak in clustering (45) → Don't use it to group documents

Recommended use case: Semantic search, duplicate detection, paraphrase detection.

Exercise 2: Choose a model based on MTEB

You have 2 options:

Model A: MTEB 64, Cost $0.13/1M tokens, Latency 100ms
Model B: MTEB 58, Cost $0 (self-hosted), Latency 10ms

Your case: A RAG system with 100K queries/month, budget $50/month.

Which one do you choose?

See solution

Analysis:

Model A (OpenAI 3-large):

  • Performance: Excellent (MTEB 64)
  • Cost: (100K × 50 tokens) / 1M × $0.13 = $0.65/month ✅
  • Latency: 100ms (acceptable)

Model B (all-mpnet-base-v2):

  • Performance: Good (MTEB 58, enough for basic RAG)
  • Cost: $0/query (+ GPU $200/month) ❌ (exceeds budget)
  • Latency: 10ms (excellent)

Recommendation: Model A (OpenAI 3-large)

Reasons:

  1. Total cost <$1/month (well below budget)
  2. Superior performance (MTEB 64 vs 58)
  3. Zero maintenance
  4. Self-hosting isn't justified for 100K queries/month

Summary

What you learned:

  • ✅ MTEB: Standard benchmark (58 datasets, 8 tasks)
  • ✅ Score range: 70+ excellent, 60-70 good, <60 acceptable
  • ✅ Tasks: Retrieval (RAG), Classification, STS (similarity)
  • ✅ Limitations: English bias, doesn't evaluate latency/cost, general domain
  • ✅ Execution: Code with the mteb library

Key concepts:

  1. MTEB is a guide, not absolute truth
  2. The retrieval score is most important for RAG
  3. Complement with latency, cost, and specific domain

Additional resources

  1. MTEB Leaderboard - Up-to-date rankings
  2. MTEB Paper - Original paper
  3. MTEB GitHub - Code
  4. MTEB Tasks - Datasets

In the next capsule

Capsule 05: Latency and Throughput

You'll learn:

  • Measure latency (API vs local)
  • Throughput (docs/second)
  • Batch processing impact
  • Benchmarking code

From performance to speed.

Module 3 - Embeddings Deep Dive Guide MTEB: the standard for evaluating embeddings

← PreviousOpen-Source Embeddings: Self-Hosted AlternativesNext →Latency and Throughput: Real Performance