Capsule 09: Embeddings with OpenAI — When to switch from the default

Capsule description

Until now you've used ChromaDB without thinking about how the embeddings are generated. When you called collection.add(documents=[...]), ChromaDB silently converted each text into a vector using a model you never named. That let you learn ChromaDB without distractions — and it was the right pedagogical decision until now.

But that default has a name and has limits. And when you build RAG for production, you'll have to consciously decide whether you stick with it or pay for something better. This capsule teaches you to make that decision with data, not intuition. You'll compare the default against OpenAI text-embedding-3-small over the same dataset, measure the impact on accuracy and cost, and learn when the switch is justified.

When you finish, when a Tech Lead asks you "why do we pay OpenAI if ChromaDB already generates embeddings for free?", you'll have a defensible numeric answer — and you'll also know when the right answer is "we don't need to pay".

By the end of this capsule, you'll be able to:

• ✅ Identify which model ChromaDB uses by default and what its real technical limits are
• ✅ Compare accuracy between the default and text-embedding-3-small with the same dataset
• ✅ Calculate the monthly cost of OpenAI embeddings for a given volume
• ✅ Apply a decision framework to choose between the two models
• ✅ Implement the OpenAI + ChromaDB integration with OpenAIEmbeddingFunction
• ✅ Anticipate the most expensive mistake: switching models without re-embedding existing data

Estimated time: 35-45 minutes

What you've been using without knowing it

When you ran this in previous capsules:

import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.create_collection("docs")
collection.add(
    documents=["password reset instructions", "billing FAQ"],
    ids=["1", "2"]
)

ChromaDB did two things you didn't see:

1. It downloaded an embedding model the first time you ran add() (~80 MB).
2. It generated 384-dimensional embeddings for each document using that model, locally on your machine, with no calls to external APIs.

The model it used is all-MiniLM-L6-v2, part of the Sentence Transformers library. It's free, runs offline, and is reasonably good for general semantic similarity tasks. That's why ChromaDB chose it as the default: zero friction to get started.

The default's real characteristics

That score of 56 on MTEB is the key data point most tutorials don't mention. MTEB (Massive Text Embedding Benchmark) is the industry-standard benchmark for comparing embedding models — it measures accuracy on retrieval, classification, clustering, and semantic similarity tasks.

The question that matters: how far is 56 from the ceiling? And how much does it cost to get higher?

OpenAI text-embedding-3-small: what changes

The most-used OpenAI model for RAG in 2026 is text-embedding-3-small. Compared to ChromaDB's default:

Three important changes:

1. Better quality, but not double. Going from 56 to 62 on MTEB sounds modest, but in RAG queries it translates to 5-15% better recall@10 — the difference between "the bot gives the right answer 78% of the time" and "91%". That difference is huge in production.

2. Real multilingual. If your RAG will receive queries in Spanish, the difference amplifies. all-MiniLM-L6-v2 was trained mostly in English; OpenAI text-embedding-3-small was trained on multilingual data and has reasonable parity between major languages.

3. Long tokens. The default silently truncates any text beyond 256 tokens (~1000 characters). If you have 2000-character documents, you're losing more than half the content without knowing it. OpenAI handles up to 8191 tokens per input.

But all of this has a price: $0.02 per million tokens embedded. If that sounds cheap, let's calculate.

Calculating the real cost

An average technical document has ~500 tokens. Let's estimate the cost for three scenarios:

# OpenAI text-embedding-3-small cost calculator
PRICE_PER_1M_TOKENS = 0.02  # USD

scenarios = {
    "MVP / personal project": {
        "docs": 1_000,
        "avg_tokens_per_doc": 500,
        "queries_per_month": 1_000,
        "avg_tokens_per_query": 30,
    },
    "Startup in production": {
        "docs": 50_000,
        "avg_tokens_per_doc": 500,
        "queries_per_month": 100_000,
        "avg_tokens_per_query": 30,
    },
    "Enterprise mid-size": {
        "docs": 1_000_000,
        "avg_tokens_per_doc": 500,
        "queries_per_month": 10_000_000,
        "avg_tokens_per_query": 30,
    },
}

for name, s in scenarios.items():
    ingestion_tokens = s["docs"] * s["avg_tokens_per_doc"]
    monthly_query_tokens = s["queries_per_month"] * s["avg_tokens_per_query"]

    ingestion_cost = (ingestion_tokens / 1_000_000) * PRICE_PER_1M_TOKENS
    monthly_query_cost = (monthly_query_tokens / 1_000_000) * PRICE_PER_1M_TOKENS

    print(f"\n=== {name} ===")
    print(f"  One-time ingestion cost: ${ingestion_cost:.2f}")
    print(f"  Monthly query cost:      ${monthly_query_cost:.2f}")
    print(f"  Total first month:       ${ingestion_cost + monthly_query_cost:.2f}")

Expected output:

=== MVP / personal project ===
  One-time ingestion cost: $0.01
  Monthly query cost:      $0.00
  Total first month:       $0.01

=== Startup in production ===
  One-time ingestion cost: $0.50
  Monthly query cost:      $0.06
  Total first month:       $0.56

=== Enterprise mid-size ===
  One-time ingestion cost: $10.00
  Monthly query cost:      $6.00
  Total first month:       $16.00

Key reading: for reasonable volumes, OpenAI embeddings is astonishingly cheap. The "startup in production" case costs less than a coffee per month. What's expensive about OpenAI isn't the embeddings — it's generation with GPT-4. But I won't teach it to you as free: the cost scales linearly with volume, and if you do frequent re-ingestion or have huge documents, the bill grows.

Practical comparison: the same dataset with two models

So far you've seen tables. Now let's go to the worked example. You'll load the same set of 100 technical documents into two collections — one with the default, one with OpenAI — and compare the result of the same query in both.

Experiment setup

# experiment_embeddings.py
import os
import chromadb
from chromadb.utils import embedding_functions
from openai import OpenAI

# Verify the API key (you should have it in .env)
assert os.getenv("OPENAI_API_KEY"), "Set OPENAI_API_KEY in your environment"

# Dataset: 100 technical documents about vector databases
docs = [
    "HNSW (Hierarchical Navigable Small World) is a graph-based algorithm for approximate nearest neighbor search.",
    "ChromaDB uses HNSW by default with M=16 and construction_ef=100 for new collections.",
    "Cosine similarity measures the angle between two vectors, ignoring magnitude.",
    "Metadata filtering reduces search space before similarity computation, improving latency.",
    "Pinecone is a managed vector database service with serverless and pod-based deployment options.",
    "Embedding dimensionality affects both retrieval quality and storage cost.",
    "RAG (Retrieval Augmented Generation) combines vector search with LLM generation.",
    "Re-ranking models like cross-encoders improve top-K results at the cost of latency.",
    # ... (in practice, 100 documents about the topic)
]
ids = [f"doc_{i:03d}" for i in range(len(docs))]

client = chromadb.PersistentClient(path="./chroma_experiment")

# Collection 1: using ChromaDB's default
collection_default = client.get_or_create_collection(
    name="vectordb_docs_default"
    # No embedding_function → uses all-MiniLM-L6-v2
)
collection_default.add(documents=docs, ids=ids)

# Collection 2: using OpenAI text-embedding-3-small
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small"
)
collection_openai = client.get_or_create_collection(
    name="vectordb_docs_openai",
    embedding_function=openai_ef
)
collection_openai.add(documents=docs, ids=ids)

print(f"Default collection: {collection_default.count()} docs")
print(f"OpenAI collection:  {collection_openai.count()} docs")

Key point: the only difference between the two collections is the embedding_function. Same dataset, same ChromaDB, same HNSW. What changes is how the texts were converted into vectors.

Query 1: a direct question in English

query_en = "How does HNSW work for approximate nearest neighbor search?"

print("\n=== DEFAULT (all-MiniLM-L6-v2) ===")
results_default = collection_default.query(query_texts=[query_en], n_results=3)
for i, (doc, dist) in enumerate(zip(results_default['documents'][0], results_default['distances'][0])):
    print(f"  #{i+1} (dist={dist:.3f}): {doc[:80]}...")

print("\n=== OPENAI (text-embedding-3-small) ===")
results_openai = collection_openai.query(query_texts=[query_en], n_results=3)
for i, (doc, dist) in enumerate(zip(results_openai['documents'][0], results_openai['distances'][0])):
    print(f"  #{i+1} (dist={dist:.3f}): {doc[:80]}...")

Typical output (both get it right, different distances):

=== DEFAULT (all-MiniLM-L6-v2) ===
  #1 (dist=0.412): HNSW (Hierarchical Navigable Small World) is a graph-based algorithm...
  #2 (dist=0.689): ChromaDB uses HNSW by default with M=16 and construction_ef=100...
  #3 (dist=0.852): Embedding dimensionality affects both retrieval quality and storage cost.

=== OPENAI (text-embedding-3-small) ===
  #1 (dist=0.187): HNSW (Hierarchical Navigable Small World) is a graph-based algorithm...
  #2 (dist=0.341): ChromaDB uses HNSW by default with M=16 and construction_ef=100...
  #3 (dist=0.498): Re-ranking models like cross-encoders improve top-K results...

Both models correctly identify the two most relevant capsules. The difference: OpenAI separates them more clearly (distances 0.187 and 0.341) while the default groups them closer together (0.412 and 0.689). This matters when you apply a score_threshold to discard low-quality results — the default is noisier.

Query 2: the same question in Spanish

This is where the difference becomes dramatic.

query_es = "¿Cómo funciona HNSW para búsqueda aproximada de vecinos cercanos?"

print("\n=== DEFAULT with a Spanish query ===")
results_default_es = collection_default.query(query_texts=[query_es], n_results=3)
for i, (doc, dist) in enumerate(zip(results_default_es['documents'][0], results_default_es['distances'][0])):
    print(f"  #{i+1} (dist={dist:.3f}): {doc[:80]}...")

print("\n=== OPENAI with a Spanish query ===")
results_openai_es = collection_openai.query(query_texts=[query_es], n_results=3)
for i, (doc, dist) in enumerate(zip(results_openai_es['documents'][0], results_openai_es['distances'][0])):
    print(f"  #{i+1} (dist={dist:.3f}): {doc[:80]}...")

Typical output:

=== DEFAULT with a Spanish query ===
  #1 (dist=0.751): Cosine similarity measures the angle between two vectors...   ❌ Irrelevant
  #2 (dist=0.792): RAG (Retrieval Augmented Generation) combines vector search... ❌ Irrelevant
  #3 (dist=0.812): HNSW (Hierarchical Navigable Small World) is a graph-based... ⚠️ Third place

=== OPENAI with a Spanish query ===
  #1 (dist=0.243): HNSW (Hierarchical Navigable Small World) is a graph-based... ✅ Correct
  #2 (dist=0.401): ChromaDB uses HNSW by default with M=16 and construction_ef... ✅ Correct
  #3 (dist=0.589): Embedding dimensionality affects both retrieval quality...    ⚠️ Tangential

What happened: the default was trained mostly in English. When you give it a query in Spanish, its query embeddings don't fall close to the embeddings of English documents about the same topic. The result: irrelevant answers in positions 1-2, the correct answer in position 3.

OpenAI handles the language barrier effortlessly: a query in Spanish → matches an English document about the same concept.

Experiment conclusion: if your audience is 100% English-speaking and your documents are short, the default may be enough. In any other case (multilingual, long documents, production RAG), OpenAI wins measurably.

Decision: a criteria framework

There's no universal answer. But there is a reproducible framework. For each project, evaluate these five criteria:

Practical three-step rule

1. Are you prototyping or learning? → Default. Don't pay for what you don't need to validate.
2. Is your RAG going to production and quality matters? → OpenAI, with very high probability.
3. Sensitive data or a regulation that prohibits sending it to an external API? → Default, or a stronger open-source model (bge-large, e5-large, etc. — out of scope for this guide).

Implementation: integrating OpenAI with ChromaDB

ChromaDB accepts embedding_function in create_collection() and uses that function automatically for all operations (add, query, update).

Complete setup

# embeddings_openai_setup.py
import os
import chromadb
from chromadb.utils import embedding_functions
from dotenv import load_dotenv

load_dotenv()  # Reads OPENAI_API_KEY from .env

# OpenAI embedding function
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small"
)

# Persistent client (data survives restart)
client = chromadb.PersistentClient(path="./chroma_openai_db")

# Collection with OpenAI embeddings
collection = client.get_or_create_collection(
    name="rag_production",
    embedding_function=openai_ef,
    metadata={"hnsw:space": "cosine"}  # Recommended metric for OpenAI embeddings
)

# Insertion: ChromaDB calls OpenAI internally
collection.add(
    documents=[
        "ChromaDB integrates with OpenAI embeddings via the embedding_function parameter.",
        "Cosine similarity is the recommended distance metric for OpenAI text embeddings.",
        "Always set OPENAI_API_KEY in environment variables, never in source code."
    ],
    metadatas=[
        {"category": "integration", "source": "docs"},
        {"category": "best_practice", "source": "guide"},
        {"category": "security", "source": "guide"}
    ],
    ids=["doc_1", "doc_2", "doc_3"]
)

# Query: ChromaDB embeds the query with OpenAI before searching
results = collection.query(
    query_texts=["What's the best distance metric for OpenAI embeddings?"],
    n_results=2
)

print(f"Top result: {results['documents'][0][0]}")
print(f"Distance: {results['distances'][0][0]:.3f}")

Expected output:

Top result: Cosine similarity is the recommended distance metric for OpenAI text embeddings.
Distance: 0.187

Environment variables (proper handling of the API key)

# .env (NEVER commit this to git)
OPENAI_API_KEY=sk-...your-real-key...

# .gitignore (make sure .env is included)
.env
.env.local
chroma_openai_db/

# Validation at the start of your app
from dotenv import load_dotenv
import os

load_dotenv()
api_key = os.getenv("OPENAI_API_KEY")

if not api_key:
    raise RuntimeError(
        "OPENAI_API_KEY is not set. "
        "Create a .env file with your API key."
    )
if not api_key.startswith("sk-"):
    raise RuntimeError(
        f"OPENAI_API_KEY looks invalid (doesn't start with 'sk-')."
    )

Handling rate limits

OpenAI imposes rate limits (RPM and TPM). For large batch ingestion, it's worth controlling concurrency:

import time
from openai import OpenAI

# The OpenAI SDK already does automatic retry on 429 errors
# but it's worth controlling the pace in large batches

def ingest_with_backoff(collection, docs, ids, batch_size=100, sleep_between=0.5):
    """Insert in batches with a pause between calls to avoid rate limits."""
    for i in range(0, len(docs), batch_size):
        batch_docs = docs[i:i + batch_size]
        batch_ids = ids[i:i + batch_size]
        collection.add(documents=batch_docs, ids=batch_ids)
        print(f"  Inserted batch {i//batch_size + 1}: {len(batch_docs)} docs")
        if i + batch_size < len(docs):
            time.sleep(sleep_between)

# For 10K documents:
# 100 batches × 0.5s pause = 50s overhead
# vs the risk of hitting a rate limit and losing progress
ingest_with_backoff(collection, my_docs, my_ids, batch_size=100)

Traps and common mistakes

Trap 1: Switching models without re-embedding existing data

The mistake:

# Day 1: you created the collection with the default
collection = client.create_collection("docs")
collection.add(documents=thousand_docs, ids=thousand_ids)

# Day 30: you decide to switch to OpenAI
openai_ef = embedding_functions.OpenAIEmbeddingFunction(api_key=key, model_name="text-embedding-3-small")
collection_v2 = client.create_collection("docs_v2", embedding_function=openai_ef)
# ✅ So far so good

# ❌ Fatal error: running queries with OpenAI over the old collection
collection.query(query_texts=["...new query..."])  
# ChromaDB uses the default model for the query
# But the docs in "docs" were embedded with the default too
# Result: the query works, but with default quality, not OpenAI

Why it happens: the embeddings already existing in the collection were generated with the old model. If you change embedding_function and run queries, ChromaDB embeds the query with the new model — but searches over vectors generated with the old model. That returns results, but technically you're comparing apples to oranges.

How to detect it: the system "works" but the results are strange or worse than before. There's no explicit error.

How to fix it: create a new collection with the new model and re-insert ALL the documents. There's no shortcut. Migration:

def migrate_collection(client, old_name, new_name, new_embedding_function):
    """Migrates all docs from one collection to another with a new embedding."""
    old = client.get_collection(old_name)
    new = client.create_collection(new_name, embedding_function=new_embedding_function)

    # Extract docs and metadata (in batches if it's large)
    batch_size = 500
    total = old.count()
    for offset in range(0, total, batch_size):
        batch = old.get(limit=batch_size, offset=offset, include=['documents', 'metadatas'])
        new.add(
            documents=batch['documents'],
            metadatas=batch['metadatas'],
            ids=batch['ids']
        )
        print(f"  Migrated {offset + len(batch['ids'])}/{total}")

    print(f"Migration complete. Verify with new.count() == {total}")

Trap 2: Dimension mismatch between collections

The mistake:

# Generate a query with OpenAI (1536 dim)
query_embedding = openai_client.embeddings.create(
    input="my query",
    model="text-embedding-3-small"
).data[0].embedding  # 1536 dimensions

# Search in a collection with the default (384 dim)
collection_default.query(query_embeddings=[query_embedding], n_results=5)
# ❌ ChromaError: Embedding dimension 1536 does not match collection dimensionality 384

Why it happens: each model produces fixed-size vectors. ChromaDB rejects embeddings of a size different from the collection's.

How to prevent it: whenever you pass query_embeddings directly, make sure to use the same model that was used for add. If you let ChromaDB embed the query with query_texts and the collection has an embedding_function configured, ChromaDB uses the same function — no mismatch is possible.

Trap 3: Silently truncated documents

The mistake: you insert a 5000-character document using ChromaDB's default. ChromaDB doesn't fail, but internally it only embeds the first ~256 tokens (~1000 characters). The other 4000 characters are stored as documents[i] but don't influence the embedding.

Symptom: queries that should match the final part of the document don't find it. The system "works" but fails on recall with no explanation.

How to prevent it:

• If you're going to use the default, split long documents into chunks of ~800 characters before inserting (that's covered in M4/10).
• If you use OpenAI, you have margin up to 8191 tokens (~32K characters), but it's still worth chunking for more precise retrieval.

Trap 4: Paying for OpenAI when it adds no value

The mistake: copy-pasting a "production-ready RAG" tutorial that uses OpenAI from day one, without your case justifying it.

How to detect it: a portfolio prototype with 200 documents, all in English, no quality SLA, $0 budget — and you're generating $50/month in API calls.

How to fix it: use the default. Switch to OpenAI when you measure that you need it (inadequate recall@10 on an eval set), not out of cultural default.

Trap 5: API key in the source code

# ❌ NEVER do this
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="sk-proj-abc123...",  # committed to git → leak on GitHub
    model_name="text-embedding-3-small"
)

Why it happens: haste, copy-paste, lack of habit.

How to fix it: always from an environment variable. Set up a pre-commit hook (gitleaks, detect-secrets) that detects the sk- pattern in commits.

Trap 6: Not measuring before choosing

The mistake: choosing between the default and OpenAI by intuition ("OpenAI is better, right?") without ever measuring on your own dataset.

How to fix it: create a small eval set (20-50 queries with labeled relevant documents) and measure recall@10 with both models. If the difference is <5%, the default is enough. If it's >10%, you justify OpenAI with data. That takes you 30 minutes and saves you months of evidence-free discussion.

Applied exercise

Scenario: You've been hired as an AI Engineer at a technical support startup for SaaS companies in LATAM. Project specifications:

• 8,000 help article documents (mixed Spanish + English)
• Average document size: 1,200 words (~1,800 tokens)
• Expected volume: 200,000 queries per month
• SLA: correct answer in top-3 ≥85% of the time
• Monthly budget for AI infrastructure: $200
• Constraint: documents can be sent to external APIs (they're not sensitive)

Question: Do you choose ChromaDB's default or OpenAI text-embedding-3-small? Justify with numbers, not intuition.

# Ingestion (one time)
docs = 8_000
tokens_per_doc = 1_800
ingestion_tokens = docs * tokens_per_doc  # 14.4M tokens
ingestion_cost = (ingestion_tokens / 1_000_000) * 0.02
# = $0.288 (a single payment)

# Monthly queries
queries_per_month = 200_000
tokens_per_query = 30  # typical short queries
monthly_query_tokens = queries_per_month * tokens_per_query  # 6M tokens
monthly_query_cost = (monthly_query_tokens / 1_000_000) * 0.02
# = $0.12 / month

Summary and next step

What you learned:

• ChromaDB's default is all-MiniLM-L6-v2: 384 dim, free, local, but with limits on multilingual queries and documents >256 tokens.
• OpenAI text-embedding-3-small is the standard upgrade for production: 1536 dim, robust multilingual, up to 8191 tokens per input, $0.02 per million tokens.
• The decision isn't default vs OpenAI in the abstract — it depends on five criteria: dataset size, languages, document length, quality SLA, and budget.
• Integrating OpenAI with ChromaDB is one line: embedding_function=OpenAIEmbeddingFunction(...).
• The most expensive mistake is switching models without re-embedding all existing data — the old embeddings are still there, silently mixing apples with oranges.

Checkpoint: before moving on, you should be able to:

• Explain to a colleague what ChromaDB does by default when you call collection.add(documents=...) without configuring embedding_function.
• Calculate the monthly cost of OpenAI embeddings for a given scenario (ingestion + queries).
• Justify with a criterion which of the two models you'd choose for a project someone describes to you.

Next capsule: 10 — Document chunking.

You just learned to generate quality embeddings. But there's a problem the capsule didn't solve: what happens with a 5000-token document? OpenAI accepts it whole (it fits in 8191), but embedding a whole document as a single vector destroys retrieval precision. If a user's question points to a specific section of the document, the "average" vector of the whole document doesn't find it.

The solution is chunking: splitting long documents into ~500-token pieces and embedding each chunk separately. It sounds simple, but the decisions (what size? what overlap? split by paragraphs or characters?) dramatically affect the quality of your RAG. That's what M4/10 covers.

Resources

1. OpenAI Embeddings Documentation — Official model specifications
2. MTEB Leaderboard — Standard benchmark for comparing embedding models
3. ChromaDB Embedding Functions — Official list of available functions
4. Sentence Transformers — all-MiniLM-L6-v2 — Documentation of the default model
5. OpenAI Pricing — Updated prices for embeddings and other models
6. MTEB: Massive Text Embedding Benchmark (paper) — Methodology behind the MTEB benchmark

Estimated time: 35-45 minutes Next: 10-chunking-documents.md