Latency and Throughput: Real Performance

Capsule overview

MTEB tells you how accurate a model is, but not how fast. In production, latency (time per query) and throughput (queries/second) are critical. A model with MTEB 70 but 500ms latency can be worse than MTEB 65 with 50ms latency for UX.

In this capsule you'll learn to measure the latency and throughput of embeddings (API vs local), the impact of batch processing, how a GPU speeds up inference, and real benchmarks of popular models. You'll also implement a speed benchmarking framework.

By the end, you'll be able to choose a model considering not only accuracy (MTEB) but also speed.

Latency vs Throughput

Definitions:

Latency:

# Time to process 1 query
latency = total_time / 1_query

Measured in: milliseconds (ms)
Example: 100ms per embedding

Throughput:

# Number of queries processed per second
throughput = queries_processed / total_time

Measured in: queries/second (QPS)
Example: 50 embeddings/second

Relationship:

# If latency = 100ms per query:
throughput = 1 / 0.1 = 10 QPS

# If latency = 10ms per query:
throughput = 1 / 0.01 = 100 QPS

# Lower latency → Higher throughput

Benchmark: OpenAI API

Measure OpenAI latency:

import time
from openai import OpenAI
import os
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def measure_latency(model, text, n_runs=10):
    """Measure average latency"""
    latencies = []
    
    for _ in range(n_runs):
        start = time.time()
        
        response = client.embeddings.create(
            model=model,
            input=text
        )
        
        latency = (time.time() - start) * 1000  # ms
        latencies.append(latency)
    
    avg_latency = sum(latencies) / len(latencies)
    return avg_latency

# Test
text = "Python is a programming language"

latency_small = measure_latency("text-embedding-3-small", text)
latency_large = measure_latency("text-embedding-3-large", text)

print(f"OpenAI 3-small: {latency_small:.0f}ms")
print(f"OpenAI 3-large: {latency_large:.0f}ms")

Typical output:

OpenAI 3-small: 87ms
OpenAI 3-large: 112ms

Observation: 3-large is ~25% slower (more dimensions).

Factors that affect API latency:

# 1. Network latency (your location → OpenAI servers)
# Typical: 20-50ms

# 2. Model size (3-small vs 3-large)
# 3-large: +25% latency

# 3. OpenAI load (time of day)
# Peak hours: +20-50ms

# 4. Text length
# 10 tokens: ~80ms
# 1000 tokens: ~120ms

Benchmark: Self-Hosted (Local)

Measure Sentence-BERT latency:

import time
from sentence_transformers import SentenceTransformer

def measure_local_latency(model_name, text, n_runs=100):
    """Measure local latency"""
    model = SentenceTransformer(model_name)
    
    # Warmup (the first inference is slow)
    _ = model.encode([text])
    
    latencies = []
    for _ in range(n_runs):
        start = time.time()
        _ = model.encode([text])
        latency = (time.time() - start) * 1000
        latencies.append(latency)
    
    avg_latency = sum(latencies) / len(latencies)
    return avg_latency

# Test
text = "Python is a programming language"

latency_mini = measure_local_latency("all-MiniLM-L6-v2", text)
latency_mpnet = measure_local_latency("all-mpnet-base-v2", text)

print(f"all-MiniLM-L6-v2: {latency_mini:.1f}ms")
print(f"all-mpnet-base-v2: {latency_mpnet:.1f}ms")

Typical output (CPU):

all-MiniLM-L6-v2: 4.8ms
all-mpnet-base-v2: 14.2ms

Speedup vs OpenAI: 18x faster (local CPU vs API).

GPU vs CPU:

# CPU (Intel i7):
# all-MiniLM-L6-v2: ~5ms
# all-mpnet-base-v2: ~15ms
# bge-large-en: ~30ms

# GPU (NVIDIA T4):
# all-MiniLM-L6-v2: ~1ms    (5x speedup)
# all-mpnet-base-v2: ~3ms   (5x speedup)
# bge-large-en: ~8ms        (4x speedup)

# GPU speeds up ~4-5x

Throughput Benchmarking

Measure throughput (QPS):

import time
from sentence_transformers import SentenceTransformer

def measure_throughput(model_name, n_texts=1000):
    """Measure throughput (queries/second)"""
    model = SentenceTransformer(model_name)
    
    texts = [f"Text {i}" for i in range(n_texts)]
    
    start = time.time()
    _ = model.encode(texts, show_progress_bar=False)
    elapsed = time.time() - start
    
    throughput = n_texts / elapsed
    return throughput

# Test
throughput_mini = measure_throughput("all-MiniLM-L6-v2", n_texts=1000)
throughput_mpnet = measure_throughput("all-mpnet-base-v2", n_texts=1000)

print(f"all-MiniLM-L6-v2: {throughput_mini:.0f} QPS")
print(f"all-mpnet-base-v2: {throughput_mpnet:.0f} QPS")

Typical output (CPU):

all-MiniLM-L6-v2: 215 QPS
all-mpnet-base-v2: 68 QPS

Batch Processing Impact

Comparison: Single vs Batch:

import time
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = ["Python is popular"] * 100

# Single processing (1 text at a time)
start = time.time()
for text in texts:
    _ = model.encode([text])
single_time = time.time() - start

# Batch processing (all at once)
start = time.time()
_ = model.encode(texts)
batch_time = time.time() - start

print(f"Single: {single_time:.2f}s")
print(f"Batch: {batch_time:.2f}s")
print(f"Speedup: {single_time / batch_time:.1f}x")

Typical output:

Single: 0.48s
Batch: 0.09s
Speedup: 5.3x

Batch processing → 5x speedup.

Optimal batch size:

import time
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = ["Text " + str(i) for i in range(1000)]

batch_sizes = [1, 8, 16, 32, 64, 128, 256]
results = []

for batch_size in batch_sizes:
    start = time.time()
    
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i+batch_size]
        _ = model.encode(batch)
    
    elapsed = time.time() - start
    throughput = len(texts) / elapsed
    
    results.append({
        'batch_size': batch_size,
        'time': elapsed,
        'throughput': throughput
    })
    
    print(f"Batch size {batch_size}: {throughput:.0f} QPS")

# Optimal batch size
best = max(results, key=lambda x: x['throughput'])
print(f"\nOptimal batch size: {best['batch_size']}")

Typical output:

Batch size 1: 208 QPS
Batch size 8: 620 QPS
Batch size 16: 890 QPS
Batch size 32: 1024 QPS  ← Optimal
Batch size 64: 1015 QPS  (plateau)
Batch size 128: 990 QPS  (memory overhead)
Batch size 256: 950 QPS

Optimal batch size: 32

Sweet spot: Batch size 32-64 (depends on hardware).

Comparison Table: Models

Latency (ms/query):

Throughput (QPS):

OpenAI API: ~10 QPS (limited by RPM rate limits).

Latency in production

Typical requirements:

# Semantic search (e-commerce):
# Tolerable latency: <100ms
# Models OK: All (local <30ms, API ~90ms)

# Chatbot (real-time):
# Tolerable latency: <50ms
# Models OK: Self-hosted (CPU <30ms, GPU <10ms)

# Batch processing (nightly):
# Tolerable latency: Any
# Priority: Throughput (GPU >> CPU)

# RAG system:
# Tolerable latency: <200ms
# Models OK: All (embedding latency is only part of the pipeline)

Latency optimizations

1. Use a GPU (if you have one):

# CPU: 15ms
# GPU: 3ms
# Speedup: 5x

from sentence_transformers import SentenceTransformer

# Load on GPU automatically (if available)
model = SentenceTransformer("all-mpnet-base-v2", device='cuda')

2. Batch processing:

# Single: 1000 queries × 15ms = 15s
# Batch (32): 1000 queries / 32 batch × 50ms = 1.6s
# Speedup: 9x

3. Model quantization (FP16):

# FP32 (default): 15ms, 420MB
# FP16 (half precision): 10ms, 210MB
# Speedup: 1.5x, Storage: 2x less

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-mpnet-base-v2")
model.half()  # Convert to FP16

4. Smaller model:

# all-mpnet-base-v2: 15ms, MTEB 58
# all-MiniLM-L6-v2: 5ms, MTEB 56
# Trade-off: -2 MTEB points, +3x speedup

Exercises

Exercise 1: Measure latency

Measure the latency of all-MiniLM-L6-v2 on your machine:

# Implement the measure_latency() function
# Run it 100 times
# Report the average latency

import time
from sentence_transformers import SentenceTransformer

def measure_latency(model_name, text, n_runs=100):
    model = SentenceTransformer(model_name)
    
    # Warmup
    _ = model.encode([text])
    
    latencies = []
    for _ in range(n_runs):
        start = time.time()
        _ = model.encode([text])
        latency = (time.time() - start) * 1000
        latencies.append(latency)
    
    return sum(latencies) / len(latencies)

# Test
latency = measure_latency("all-MiniLM-L6-v2", "Python is popular")
print(f"Average latency: {latency:.1f}ms")

Exercise 2: Compare batch sizes

Find the optimal batch size for all-MiniLM-L6-v2:

# Test batch sizes: [1, 8, 16, 32, 64]
# Which one has the best throughput?

import time
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = ["Text " + str(i) for i in range(500)]

batch_sizes = [1, 8, 16, 32, 64]

for batch_size in batch_sizes:
    start = time.time()
    
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i+batch_size]
        _ = model.encode(batch)
    
    elapsed = time.time() - start
    throughput = len(texts) / elapsed
    
    print(f"Batch {batch_size}: {throughput:.0f} QPS")

Batch 1: 210 QPS
Batch 8: 620 QPS
Batch 16: 890 QPS
Batch 32: 1020 QPS  ← Optimal
Batch 64: 1010 QPS

Summary

What you learned:

• ✅ Latency: Time/query (ms)
• ✅ Throughput: Queries/second (QPS)
• ✅ API: OpenAI ~90-110ms
• ✅ Local: CPU ~5-30ms, GPU ~1-8ms
• ✅ Batch processing: 5-10x speedup
• ✅ Optimal batch size: 32-64

Key concepts:

1. Self-hosted >> API in latency (18x faster)
2. GPU >> CPU (5x faster)
3. Batch processing is critical for throughput

Additional resources

1. Sentence-Transformers Speed - Performance data
2. GPU Benchmarking - HuggingFace
3. Batch Processing Guide - SBERT

In the next capsule

Capsule 06: Cost Analysis

You'll learn:

• Compare costs (OpenAI vs self-hosted)
• TCO (Total Cost of Ownership)
• Break-even calculations
• When self-hosting makes sense

From speed to economics.

Module 3 - Embeddings Deep Dive Guide Latency and throughput: real production performance