Module 3: Embedding Models Compared

Introduction to Module 3: Embedding Models Compared

Module welcome

You already know what embeddings are (Module 1) and how they work internally (Module 2: Transformers, tokenization, pooling). Now comes the critical moment for AI Engineers: choosing the right model.

In this module you'll learn to compare embedding models (OpenAI, Sentence-BERT, BGE, Instructor, Cohere), run benchmarks with MTEB, use open-source models locally, optimize for your specific domain, and handle multiple languages. By the end, you'll be able to make informed decisions about which model to use depending on your use case, budget, and performance requirements.

Module objectives

By completing this module, you'll be able to:

✅ Compare models: OpenAI vs open-source (SBERT, BGE, Instructor)
✅ Run benchmarks: MTEB scores, latency, costs
✅ Use open-source: Sentence-Transformers (HuggingFace)
✅ Optimize for a domain: Fine-tuning (conceptual) vs zero-shot
✅ Handle multiple languages: Multilingual embeddings (mBERT, XLM-R)
✅ Decide trade-offs: Accuracy vs cost vs latency
✅ Implement fallback: Multi-provider strategy
✅ Project: A benchmark framework to compare models

Module roadmap

Phase 1: Model landscape (Capsules 01-03)

Capsule 01: Introduction (this capsule)

Objectives, roadmap, context

Capsule 02: OpenAI Embeddings Models

text-embedding-3-small vs 3-large
Performance, costs, dimensions
When to use each one

Capsule 03: Open-Source Embeddings Overview

Sentence-BERT (SBERT)
BGE Models
Instructor Embeddings
Cohere (API)
E5 Models

Phase 2: Evaluation and benchmarking (Capsules 04-06)

Capsule 04: MTEB Benchmark

What MTEB is (Massive Text Embedding Benchmark)
Interpreting scores
Limitations of benchmarks

Capsule 05: Latency and Throughput

Measuring latency (API vs local)
Throughput (documents/second)
Batch processing performance

Capsule 06: Cost Analysis

Comparing costs (OpenAI vs self-hosted)
TCO (Total Cost of Ownership)
When self-hosted makes sense

Phase 3: Specialization and project (Capsules 07-08)

Capsule 07: Domain-Specific and Multilingual

Domain-specific embeddings (legal, medical)
Multilingual embeddings (100+ languages)
When to fine-tune vs zero-shot

Capsule 08: Mini-Project - Benchmark Framework

Compare 3+ models
Metrics: MTEB, latency, cost
Automatic reports
Model recommendation

Connection with the AI Engineering Path

Prerequisites (completed):

✅ Module 1 (Embeddings Deep Dive): Embedding fundamentals ✅ Module 2 (Embeddings Deep Dive): Internal architecture

This module in the path:

AI Semantics Guide (vector concepts)
          ↓
Embeddings Deep Dive - M1 (What are they?)
          ↓
Embeddings Deep Dive - M2 (How do they work?)
          ↓
Embeddings Deep Dive - M3 (Model comparison) ← YOU ARE HERE
          ↓
Embeddings Deep Dive - M4 (Evaluation and chunking)
          ↓
Vector Databases Comparison Guide

Why comparing models is critical

A real scenario:

Task: Implement semantic search for 1M documents

Options:
1. OpenAI text-embedding-3-large:
   - Performance: Excellent (MTEB ~64)
   - Cost: $130/1M tokens
   - Latency: ~100ms (API)
   
2. Sentence-BERT (all-MiniLM-L6-v2):
   - Performance: Good (MTEB ~58)
   - Cost: $0 (self-hosted)
   - Latency: ~5ms (local GPU)

3. BGE-large-en:
   - Performance: Excellent (MTEB ~63)
   - Cost: $0 (self-hosted)
   - Latency: ~20ms (local GPU)

Question: Which one do you choose?

Answer: It depends on:

Budget (API vs self-hosted)
Required performance (RAG vs basic semantic search)
Tolerable latency (100ms vs 5ms)
Volume (1K docs vs 1M docs)
Language (English vs multilingual)

This module gives you tools to decide.

What you'll learn (in detail)

1. Model landscape:

# OpenAI (API):
- text-embedding-3-small (1536 dims, $0.02/1M tokens)
- text-embedding-3-large (3072 dims, $0.13/1M tokens)

# Open-source (HuggingFace):
- Sentence-BERT (all-MiniLM-L6-v2, all-mpnet-base-v2)
- BGE (BAAI/bge-large-en, bge-base-en)
- Instructor (hkunlp/instructor-large)
- E5 (intfloat/e5-large-v2)

# Commercial API:
- Cohere (embed-english-v3.0)
- Voyage AI (voyage-large-2)

2. Systematic evaluation:

MTEB scores (performance):

Model                | MTEB Score | Typical use
---------------------|------------|------------------
text-embedding-3-large | ~64      | RAG production
BGE-large-en          | ~63      | Self-hosted RAG
text-embedding-3-small | ~62      | Prototyping
all-mpnet-base-v2     | ~58      | Basic semantic search
all-MiniLM-L6-v2      | ~56      | Demos, MVP

Latency (speed):

Model                | Latency  | Where
---------------------|----------|------------
all-MiniLM-L6-v2     | ~5ms    | Local CPU
BGE-large-en (GPU)   | ~20ms   | Local GPU
OpenAI API           | ~100ms  | Cloud

Cost (economics):

Model                | Cost/1M tokens  | Break-even
---------------------|-----------------|-------------
OpenAI 3-large       | $130           | -
BGE (self-hosted)    | $0 + GPU ($2/h)| ~65 hours of use

3. Real trade-offs:

Case 1: Startup on a limited budget

Recommendation: Sentence-BERT (all-MiniLM-L6-v2)
- Cost: $0 (self-hosted)
- Performance: Enough for an MVP
- Latency: Excellent (5ms local)

Case 2: Company with production-ready RAG

Recommendation: OpenAI text-embedding-3-large
- Cost: $130/1M tokens (budget OK)
- Performance: Top-tier (MTEB 64)
- Latency: Acceptable (100ms)
- Zero maintenance

Case 3: E-commerce with 10M products

Recommendation: BGE-large-en (self-hosted GPU)
- Cost: $0/token + GPU ($500/month)
- Performance: Excellent (MTEB 63)
- Latency: 20ms (local)
- Break-even after 1 month vs OpenAI

Theory/practice balance (50/50)

Theory (50%):

Model comparison (architectures, performance)
MTEB benchmark (how to interpret scores)
Cost analysis (TCO, break-even)
Domain-specific and multilingual (when to use them)

Practice (50%):

Run benchmarks with MTEB
Measure latency (code)
Calculate costs (spreadsheet)
Compare models (final project)

This module is less "hands-on code" than M1/M2 (more evaluation and strategic decisions).

What you will NOT learn (out of scope)

❌ Fine-tuning embeddings:

Reason: Fine-tuning is advanced (requires ML Engineering).
Coverage: Conceptual only (when to consider fine-tuning).

❌ Training embeddings from scratch:

Reason: Requires a massive corpus + weeks of GPU + ML expertise.
Coverage: Not applicable for AI Engineers (use pre-trained models).

❌ Custom architectures (modifying Transformers):

Reason: Out of scope for AI Engineering (that's Research/ML Engineering).
Coverage: Only using existing models.

❌ Advanced visualization (UMAP, t-SNE):

Reason: Not critical for production (more Data Science).
Coverage: Conceptual only (no matplotlib code).

Module tools

Python libraries:

# Sentence-Transformers (HuggingFace)
from sentence_transformers import SentenceTransformer

# MTEB (benchmarking)
from mteb import MTEB

# OpenAI (already seen)
from openai import OpenAI

# Cost tracking
import time

Required setup:

# Install Sentence-Transformers
pip install sentence-transformers

# Install MTEB
pip install mteb

# OpenAI (already installed)
pip install openai

Module structure

module-03-embedding-models-compared/
└── en/
    ├── 01-module-introduction-3.md      ← You are here
    ├── 02-openai-models.md
    ├── 03-open-source-overview.md
    ├── 04-mteb-benchmark.md
    ├── 05-latency-and-throughput.md
    ├── 06-cost-analysis.md
    ├── 07-domain-multilingual.md
    └── 08-project-benchmark-framework.md

Professional skills

By completing this module, you'll demonstrate:

1. Strategic thinking:

Evaluate trade-offs (cost vs performance)
TCO analysis (not just API price)
Break-even calculations

2. Technical evaluation:

Interpret benchmarks (MTEB)
Measure latency (code)
Compare models objectively

3. Decision-making:

Choose a model based on the use case
Justify decisions with data
Fallback strategies (multi-provider)

Real use cases

1. RAG system (production):

Requirements:

High precision (correct context)
Latency <200ms
Budget: $500/month
Volume: 100K queries/month

Decision:

Model: OpenAI text-embedding-3-large
Reason:
- Top-tier performance (RAG is critical)
- Latency OK (100ms)
- Cost OK ($130/1M tokens = $13/100K queries)

2. E-commerce search (10M products):

Requirements:

Good performance (not critical)
Latency <50ms
Budget: $200/month
Volume: 1M queries/month

Decision:

Model: BGE-large-en (self-hosted GPU)
Reason:
- Excellent performance (MTEB 63)
- Excellent latency (20ms)
- Cost OK (GPU $200/month, not per query)

3. Startup MVP (proof-of-concept):

Requirements:

Enough performance
Any latency
Budget: $0
Volume: 1K queries/month

Decision:

Model: Sentence-BERT (all-MiniLM-L6-v2)
Reason:
- Performance OK for an MVP (MTEB 56)
- Excellent latency (5ms local CPU)
- Cost: $0 (self-hosted)

Module methodology

Comparative learning:

Step 1: Know the models (landscape)
         ↓
Step 2: Evaluate (MTEB, latency, cost)
         ↓
Step 3: Decide (trade-offs, use case)
         ↓
Step 4: Implement (benchmark project)

Initial exercise (reflection)

Scenario:

You're building a semantic search system for your company's internal knowledge base (10,000 technical documents in Spanish).

Questions:

Which model would you choose? Why?
Which metrics would you use to evaluate it?
What is your maximum monthly budget?
What latency is acceptable?

There's no single correct answer (it depends on constraints).

By the end of the module, you'll be able to answer with concrete data.

Module resources

Benchmarks:

MTEB Leaderboard - Official rankings
Sentence-Transformers - Open-source embeddings
OpenAI Embeddings Guide - Official

Tools:

HuggingFace Model Hub - Pre-trained models
MTEB GitHub - Benchmarking code

In the next capsule

Capsule 02: OpenAI Embeddings Models

You'll learn:

Differences between 3-small and 3-large
Compared performance (MTEB scores)
Detailed costs
Configurable dimensions
When to use each model

From introduction to a deep dive into OpenAI.

Module 3 - Embeddings Deep Dive Guide Choosing the right model: performance, cost, and trade-offs

← PreviousMini-Project: Robust API Client for Production Next →OpenAI Embeddings Models: Deep Dive Comparison