Open-Source Embeddings: Self-Hosted Alternatives

Capsule overview

Beyond OpenAI, there are excellent open-source embedding models you can run locally: Sentence-BERT (SBERT), BGE Models, Instructor Embeddings, and E5. These models offer $0 cost per query (after infrastructure), ultra-low latency (<20ms local), and full control.

In this capsule you'll learn the main open-source models, how to use them with HuggingFace Sentence-Transformers, their advantages/disadvantages vs OpenAI, and when self-hosting makes sense. You'll also see practical code to run models locally.

By the end, you'll be able to choose between an API (OpenAI) and self-hosted (open-source) depending on your use case.

The open-source embeddings landscape

Main models (2026):

Top pick: BGE models (best MTEB/performance).

Sentence-BERT (SBERT)

What it is:

A framework for generating sentence embeddings using BERT-based models. A pioneer of quality embeddings for semantic search (2019).

Developed by: UKP Lab (TU Darmstadt)

Popular models:

1. all-MiniLM-L6-v2 (lightweight)

Dimensions: 384
MTEB Score: ~56
Size: 80 MB
Speed: ~5ms (CPU), ~1ms (GPU)
Use: MVP, prototyping, demos

2. all-mpnet-base-v2 (balanced)

Dimensions: 768
MTEB Score: ~58
Size: 420 MB
Speed: ~15ms (CPU), ~3ms (GPU)
Use: General purpose, light production

Installation:

pip install sentence-transformers

Basic code:

from sentence_transformers import SentenceTransformer
import numpy as np

# Load the model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Generate embeddings
texts = [
    "Python is a programming language",
    "JavaScript is a web language",
    "The cat sleeps on the couch"
]

embeddings = model.encode(texts)

print(f"Model: all-MiniLM-L6-v2")
print(f"Embeddings shape: {embeddings.shape}")  # (3, 384)
print(f"Type: {type(embeddings)}")  # numpy.ndarray

# Calculate similarity
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

sim = cosine_similarity(embeddings[0], embeddings[1])
print(f"\nSimilarity (Python vs JavaScript): {sim:.4f}")

Output:

Model: all-MiniLM-L6-v2
Embeddings shape: (3, 384)
Type: <class 'numpy.ndarray'>

Similarity (Python vs JavaScript): 0.4385

Advantages of SBERT:

✅ 1. $0 cost per query

# You only pay for GPU/CPU (fixed per month)
# There's no cost per query (vs OpenAI $0.02/1M tokens)

✅ 2. Ultra-low latency

# Local CPU: ~5ms
# Local GPU: ~1ms
# vs OpenAI API: ~100ms

✅ 3. No rate limits

# OpenAI: 500 RPM (tier 1)
# Self-hosted: No limits (only hardware)

✅ 4. Full privacy

# Data does NOT leave your infrastructure
# Critical for GDPR, HIPAA, etc.

Disadvantages of SBERT:

❌ 1. Requires infrastructure

# You need a GPU ($200-$500/month in the cloud)
# Or CPU (slower)

❌ 2. Maintenance

# Update models, monitor, scale
# OpenAI: Zero maintenance

❌ 3. Slightly lower performance

# MTEB ~56-58 (SBERT)
# vs MTEB ~62-64 (OpenAI, BGE)

BGE Models (BAAI)

What it is:

State of the art in open-source embeddings (2023). Developed by the Beijing Academy of Artificial Intelligence (BAAI). Competes with OpenAI on MTEB.

Models:

1. bge-small-en-v1.5

Dimensions: 384
MTEB Score: ~62
Size: 120 MB
Speed: ~8ms (CPU), ~2ms (GPU)
Use: Self-hosted RAG (budget-friendly)

2. bge-base-en-v1.5

Dimensions: 768
MTEB Score: ~63
Size: 430 MB
Speed: ~15ms (CPU), ~4ms (GPU)
Use: Standard production

3. bge-large-en-v1.5

Dimensions: 1024
MTEB Score: ~64
Size: 1.2 GB
Speed: ~30ms (CPU), ~8ms (GPU)
Use: Maximum precision (self-hosted)

Basic code:

from sentence_transformers import SentenceTransformer

# Load a BGE model
model = SentenceTransformer('BAAI/bge-large-en-v1.5')

# Generate embeddings
texts = [
    "Python is a programming language",
    "JavaScript is a web language"
]

embeddings = model.encode(texts, normalize_embeddings=True)

print(f"Model: BGE-large-en-v1.5")
print(f"Shape: {embeddings.shape}")  # (2, 1024)
print(f"Normalized: {np.linalg.norm(embeddings[0]):.4f}")  # ~1.0

Why BGE dominates:

✅ 1. Top-tier performance (MTEB ~64)

# Competes with OpenAI text-embedding-3-large
# But $0/query after setup

✅ 2. Instruction-based training

# BGE was trained with queries + passages
# Better for retrieval (RAG)

✅ 3. Optimized for semantic search

# Specifically designed for search
# Not just general similarity

Instructor Embeddings

What it is:

Embeddings with explicit instructions. You can tell the model what task it's going to do (retrieval, classification, clustering).

Code with instructions:

from InstructorEmbedding import INSTRUCTOR

# Load the model
model = INSTRUCTOR('hkunlp/instructor-large')

# Embeddings WITH instructions
query_emb = model.encode([
    ["Represent the question for retrieving supporting documents: ",
     "How to install Python?"]
])[0]

doc_emb = model.encode([
    ["Represent the document for retrieval: ",
     "To install Python, download from python.org"]
])[0]

# Calculate similarity
similarity = np.dot(query_emb, doc_emb)
print(f"Similarity: {similarity:.4f}")

Advantage: Task-specialized embeddings (retrieval query vs document).

E5 Models (Microsoft)

What it is:

Embeddings from Microsoft (2022-2023). Competitive with BGE. Trained with contrastive learning.

Models:

e5-small-v2:  MTEB ~60
e5-base-v2:   MTEB ~62
e5-large-v2:  MTEB ~64

Usage similar to BGE:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('intfloat/e5-large-v2')
embeddings = model.encode(texts)

Comparison: OpenAI vs Open-Source

Comparison table:

Break-even analysis:

# Assumptions:
# - OpenAI 3-large: $0.13/1M tokens
# - Self-hosted GPU: $200/month (cloud)
# - Queries/month: X

# OpenAI cost (X queries, 50 tokens/query):
cost_openai = (X * 50 / 1_000_000) * 0.13

# Break-even:
# cost_openai = $200
# X * 50 * 0.13 / 1_000_000 = 200
# X = 30.8M queries/month

print("Break-even: ~31M queries/month")
print("If <31M queries → OpenAI")
print("If >31M queries → Self-hosted")

When to use open-source

Self-hosted makes sense when:

✅ 1. High volume (>10M queries/month)

# Break-even at 31M queries
# But savings start earlier

✅ 2. Critical privacy (GDPR, HIPAA)

# Data CANNOT leave your infrastructure

✅ 3. Ultra-low latency required (<20ms)

# Local GPU: ~8ms
# OpenAI API: ~100ms

✅ 4. No budget for APIs

# Startups without funding
# You only have a GPU (already paid for)

OpenAI makes sense when:

✅ 1. Low-to-medium volume (<10M queries/month)

# Total cost <$200/month
# Doesn't justify self-hosting

✅ 2. Zero maintenance required

# You don't have a DevOps team
# OpenAI handles everything

✅ 3. Fast prototyping

# API key → code works
# vs Self-hosting: Docker, GPU, monitoring

Exercises

Exercise 1: Use SBERT

Install sentence-transformers and generate embeddings with all-MiniLM-L6-v2:

# Install:
# pip install sentence-transformers

# Implement:
# 1. Load the model
# 2. Generate embeddings for ["Python", "Java", "Cat"]
# 3. Calculate the similarity Python vs Java

from sentence_transformers import SentenceTransformer
import numpy as np

# Load the model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Generate embeddings
texts = ["Python", "Java", "Cat"]
embeddings = model.encode(texts)

print(f"Shape: {embeddings.shape}")  # (3, 384)

# Similarity Python vs Java
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

sim = cosine_similarity(embeddings[0], embeddings[1])
print(f"Python vs Java: {sim:.4f}")  # ~0.45

sim_cat = cosine_similarity(embeddings[0], embeddings[2])
print(f"Python vs Cat: {sim_cat:.4f}")  # ~0.33

Exercise 2: Compare OpenAI vs SBERT

Compare embeddings from both for the same text:

# 1. Generate an embedding with OpenAI (3-small)
# 2. Generate an embedding with SBERT (all-MiniLM-L6-v2)
# 3. Are they similar? (hint: different dimensions, not directly comparable)

from openai import OpenAI
from sentence_transformers import SentenceTransformer
import os
from dotenv import load_dotenv

load_dotenv()
client_openai = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

text = "Python is popular"

# OpenAI
emb_openai = client_openai.embeddings.create(
    model="text-embedding-3-small",
    input=text
).data[0].embedding

# SBERT
model_sbert = SentenceTransformer('all-MiniLM-L6-v2')
emb_sbert = model_sbert.encode([text])[0]

print(f"OpenAI dims: {len(emb_openai)}")  # 1536
print(f"SBERT dims: {len(emb_sbert)}")    # 384

# We can't compare directly (different dimensions)
# But both capture the semantics of "Python is popular"

Summary

What you learned:

• ✅ Open-source landscape: SBERT, BGE, Instructor, E5
• ✅ BGE dominates: MTEB ~64 (same as OpenAI 3-large)
• ✅ Self-hosted advantages: $0/query, latency <20ms, privacy
• ✅ Disadvantages: Requires infra, maintenance
• ✅ Break-even: ~31M queries/month

Key concepts:

1. BGE-large-en-v1.5 → Best open-source (MTEB 64)
2. Self-hosted → High volume, privacy, latency
3. OpenAI → Low volume, zero maintenance

Additional resources

1. Sentence-Transformers Docs - Official SBERT
2. BGE GitHub - Code and models
3. Instructor Embeddings - Official repo
4. HuggingFace Model Hub - Pre-trained models

In the next capsule

Capsule 04: MTEB Benchmark

You'll learn:

• What MTEB is (Massive Text Embedding Benchmark)
• How to interpret scores
• Limitations of benchmarks
• Run MTEB on your models

From models to systematic evaluation.

Module 3 - Embeddings Deep Dive Guide Open-source embeddings: quality self-hosted alternatives