Cost Analysis: The Economics of Embeddings

Capsule overview

MTEB and latency are important, but cost determines viability. OpenAI charges per token; self-hosting requires GPU/CPU. When does each option make sense economically? What's the break-even point?

In this capsule you'll learn to calculate the TCO (Total Cost of Ownership) of embeddings (API vs self-hosted), break-even analysis, hidden costs (bandwidth, storage), and cost optimizations. You'll also build a cost calculator to make informed decisions.

By the end, you'll be able to justify model choices with rigorous cost analysis.

Cost models

API-based (OpenAI, Cohere):

Cost = (Tokens processed / 1,000,000) × Price_per_1M_tokens

# OpenAI example:
# 10M tokens × $0.020/1M = $0.20

Characteristics:

• ✅ Variable cost (you pay for usage)
• ✅ Zero infrastructure
• ❌ Scales linearly (more queries = more cost)

Self-hosted (SBERT, BGE):

Cost = GPU_rental + Storage + Bandwidth + Maintenance

# Example:
# GPU (NVIDIA T4): $200/month
# Storage (1TB SSD): $10/month
# Bandwidth: $5/month
# Total: $215/month (fixed)

Characteristics:

• ✅ Fixed cost (no matter the # of queries)
• ❌ Requires infrastructure
• ❌ Maintenance (DevOps)

OpenAI Pricing (2026)

Models and costs:

Assumptions: *50 tokens on average per text.

Calculate cost:

def calculate_openai_cost(n_queries, tokens_per_query, model="3-small"):
    """
    Calculate the monthly cost of OpenAI embeddings
    
    Args:
        n_queries: Queries per month
        tokens_per_query: Average tokens per query
        model: "3-small" or "3-large"
    
    Returns:
        Monthly cost (USD)
    """
    pricing = {
        "3-small": 0.020,
        "3-large": 0.130
    }
    
    total_tokens = n_queries * tokens_per_query
    cost = (total_tokens / 1_000_000) * pricing[model]
    
    return cost

# Example: RAG system
# 100K queries/month, 50 tokens/query
cost = calculate_openai_cost(100_000, 50, "3-small")
print(f"Monthly cost: ${cost:.2f}")  # $0.10

Self-Hosting Pricing

Infrastructure options:

1. Cloud GPU (AWS, GCP, Azure):

# NVIDIA T4 (16GB):
# AWS: $0.526/hour = $379/month (24/7)
# GCP: $0.35/hour = $252/month (24/7)

# NVIDIA A10G (24GB):
# AWS: $1.01/hour = $727/month (24/7)

# Spot instances (interruptible):
# T4: ~$0.15/hour = $108/month (70% savings)

2. Dedicated servers (Hetzner, OVH):

# GPU server:
# Hetzner: $150-$300/month (GPU included)
# OVH: $200-$400/month

# Pros: Cheaper than cloud
# Cons: Less flexible (no auto-scaling)

3. Local hardware (on-premise):

# NVIDIA RTX 3090 (24GB):
# Purchase: $1,500 (one-time)
# Amortized: $1,500 / 36 months = $42/month
# Electricity: $20/month
# Total: ~$62/month

# Pros: Cheapest long-term
# Cons: Upfront cost, no redundancy

Calculate self-hosted TCO:

def calculate_selfhosted_tco(
    gpu_cost_monthly=200,
    storage_gb=100,
    bandwidth_gb=500,
    maintenance_hours=5,
    devops_hourly=50
):
    """
    Calculate the monthly TCO of self-hosting
    
    Returns:
        Monthly cost (USD)
    """
    # Infrastructure
    gpu_cost = gpu_cost_monthly
    storage_cost = storage_gb * 0.10  # $0.10/GB/month
    bandwidth_cost = bandwidth_gb * 0.01  # $0.01/GB
    
    # Maintenance
    maintenance_cost = maintenance_hours * devops_hourly
    
    # Total
    total = gpu_cost + storage_cost + bandwidth_cost + maintenance_cost
    
    return {
        'gpu': gpu_cost,
        'storage': storage_cost,
        'bandwidth': bandwidth_cost,
        'maintenance': maintenance_cost,
        'total': total
    }

# Example
tco = calculate_selfhosted_tco()
print(f"Monthly TCO: ${tco['total']:.2f}")

Output:

Monthly TCO: $465.00
- GPU: $200
- Storage: $10
- Bandwidth: $5
- Maintenance: $250 (5 hrs × $50/hr)

Break-Even Analysis

Formula:

Break-even point (queries/month) = TCO_selfhosted / Cost_per_query_API

# Example:
# TCO self-hosted: $465/month
# OpenAI 3-small: $0.000001 per query (50 tokens)

break_even = 465 / 0.000001
# = 465M queries/month

print(f"Break-even: {break_even/1_000_000:.0f}M queries/month")

Break-even table:

Conclusion: Break-even ~500M queries/month (with maintenance included).

Without maintenance:

# If you DON'T count maintenance (in-house DevOps):
# TCO self-hosted: $215/month (infra only)

break_even = 215 / 0.000001
# = 215M queries/month

Break-even drops to 215M queries/month.

Hidden Costs

OpenAI (API):

# ✅ Obvious:
# - Cost per token ($0.020/1M)

# ❌ Hidden:
# - Retry logic (re-calls double the cost)
# - Testing/development (each test costs)
# - Scaling (cost scales linearly)

Self-Hosted:

# ✅ Obvious:
# - GPU rental ($200/month)
# - Storage ($10/month)

# ❌ Hidden:
# - DevOps time ($250/month if 5 hrs/month)
# - Downtime (SLA? backup GPU?)
# - Monitoring/logging (Grafana, Prometheus)
# - Updates (new models, dependencies)
# - Security (patches, firewalls)

Total hidden: +$300-$500/month

Cost Optimizations

OpenAI:

1. Aggressive caching

# Cache embeddings you already generated
# Reuse = $0 cost

# Example:
# 1M queries, 80% cache hit rate
# Real queries to the API: 200K
# Savings: 80%

2. Reduced dimensions

# 1536 dims → 512 dims
# Same cost per query, but:
# - 3x less storage
# - 3x faster search

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="...",
    dimensions=512  # Reduce
)

3. Batch processing

# Batching reduces HTTP overhead
# Fewer requests = lower latency
# (but same cost per token)

Self-Hosted:

1. Spot instances (cloud)

# GPU spot instances: 70% cheaper
# T4: $0.526/hr → $0.15/hr
# Trade-off: Can be interrupted

2. Smaller model

# BGE-large: 1024 dims, 30ms
# BGE-small: 384 dims, 8ms
# Trade-off: -1 MTEB point, +4x faster

3. Model quantization (FP16)

# FP32 → FP16
# - 2x less VRAM (you can use a cheaper GPU)
# - 1.5x faster
# Trade-off: Minimal precision loss

Decision Framework

Choose the API (OpenAI) if:

✅ Queries/month < 100M
✅ You don't have a DevOps team
✅ Fast prototyping
✅ Budget <$500/month

Choose Self-Hosted if:

✅ Queries/month > 500M
✅ You have in-house DevOps
✅ Critical latency (<20ms)
✅ Critical privacy (GDPR, HIPAA)
✅ Fixed budget (not variable)

Exercises

Exercise 1: Calculate OpenAI cost

Your RAG system has:

• 500K queries/month
• 80 tokens on average/query

How much does it cost with OpenAI 3-small and 3-large?

queries_per_month = 500_000
tokens_per_query = 80

# 3-small
total_tokens = queries_per_month * tokens_per_query
cost_small = (total_tokens / 1_000_000) * 0.020
print(f"3-small: ${cost_small:.2f}/month")  # $0.80/month

# 3-large
cost_large = (total_tokens / 1_000_000) * 0.130
print(f"3-large: ${cost_large:.2f}/month")  # $5.20/month

Exercise 2: Break-even analysis

Calculate the break-even between OpenAI 3-small and self-hosted (TCO $400/month):

# Self-hosted TCO: $400/month
# OpenAI 3-small: $0.020/1M tokens
# Assumptions: 50 tokens/query

# At how many queries/month does self-hosting make sense?

tco_selfhosted = 400  # USD/month
openai_cost_per_1m_tokens = 0.020
tokens_per_query = 50

# Cost per query (OpenAI)
cost_per_query = (tokens_per_query / 1_000_000) * openai_cost_per_1m_tokens
print(f"Cost per query: ${cost_per_query:.8f}")  # $0.000001

# Break-even
break_even_queries = tco_selfhosted / cost_per_query
print(f"Break-even: {break_even_queries/1_000_000:.0f}M queries/month")

Summary

What you learned:

• ✅ API cost: Variable, scales with usage
• ✅ Self-hosted TCO: Fixed, includes GPU + maintenance
• ✅ Break-even: ~200-500M queries/month (depends on assumptions)
• ✅ Hidden costs: Maintenance, downtime, testing
• ✅ Optimizations: Caching (API), spot instances (self-hosted)

Key concepts:

1. OpenAI → Low-to-medium volume (<100M/month)
2. Self-hosted → High volume (>500M/month)
3. TCO includes hidden costs (maintenance +$300/month)

Additional resources

1. OpenAI Pricing - Up-to-date
2. AWS EC2 GPU Pricing - Comparison
3. Cloud Cost Calculator - AWS, GCP, Azure

In the next capsule

Capsule 07: Domain-Specific and Multilingual

You'll learn:

• Domain-specific embeddings (legal, medical)
• Multilingual embeddings (100+ languages)
• When to fine-tune vs zero-shot
• Trade-offs

From costs to specialization.

Module 3 - Embeddings Deep Dive Guide Cost analysis: when to use an API, when to self-host