Latencia y Throughput: Performance Real

Descripción de la cápsula

MTEB te dice qué tan preciso es un modelo, pero no qué tan rápido. En producción, latencia (tiempo por query) y throughput (queries/segundo) son críticos. Un modelo con MTEB 70 pero latencia 500ms puede ser peor que MTEB 65 con latencia 50ms para UX.

En esta cápsula aprenderás a medir latencia y throughput de embeddings (API vs local), el impacto de batch processing, cómo GPU acelera inferencia, y benchmarks reales de modelos populares. También implementarás un framework de benchmarking de velocidad.

Al final, podrás elegir modelo considerando no solo precisión (MTEB) sino también velocidad.

Latencia vs Throughput

Definiciones:

Latencia:

# Tiempo para procesar 1 query
latencia = tiempo_total / 1_query

Medida: milisegundos (ms)
Ejemplo: 100ms por embedding

Throughput:

# Cantidad de queries procesadas por segundo
throughput = queries_procesadas / tiempo_total

Medida: queries/segundo (QPS)
Ejemplo: 50 embeddings/segundo

Relación:

# Si latencia = 100ms por query:
throughput = 1 / 0.1 = 10 QPS

# Si latencia = 10ms por query:
throughput = 1 / 0.01 = 100 QPS

# Latencia más baja → Throughput más alto

Benchmark: OpenAI API

Medir latencia de OpenAI:

import time
from openai import OpenAI
import os
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def measure_latency(model, text, n_runs=10):
    """Medir latencia promedio"""
    latencies = []
    
    for _ in range(n_runs):
        start = time.time()
        
        response = client.embeddings.create(
            model=model,
            input=text
        )
        
        latency = (time.time() - start) * 1000  # ms
        latencies.append(latency)
    
    avg_latency = sum(latencies) / len(latencies)
    return avg_latency

# Test
text = "Python es un lenguaje de programación"

latency_small = measure_latency("text-embedding-3-small", text)
latency_large = measure_latency("text-embedding-3-large", text)

print(f"OpenAI 3-small: {latency_small:.0f}ms")
print(f"OpenAI 3-large: {latency_large:.0f}ms")

Output típico:

OpenAI 3-small: 87ms
OpenAI 3-large: 112ms

Observación: 3-large es ~25% más lento (más dimensiones).

Factores que afectan latencia API:

# 1. Network latency (tu ubicación → OpenAI servers)
# Típico: 20-50ms

# 2. Tamaño del modelo (3-small vs 3-large)
# 3-large: +25% latencia

# 3. Load de OpenAI (hora del día)
# Peak hours: +20-50ms

# 4. Longitud del texto
# 10 tokens: ~80ms
# 1000 tokens: ~120ms

Benchmark: Self-Hosted (Local)

Medir latencia de Sentence-BERT:

import time
from sentence_transformers import SentenceTransformer

def measure_local_latency(model_name, text, n_runs=100):
    """Medir latencia local"""
    model = SentenceTransformer(model_name)
    
    # Warmup (primera inferencia es lenta)
    _ = model.encode([text])
    
    latencies = []
    for _ in range(n_runs):
        start = time.time()
        _ = model.encode([text])
        latency = (time.time() - start) * 1000
        latencies.append(latency)
    
    avg_latency = sum(latencies) / len(latencies)
    return avg_latency

# Test
text = "Python es un lenguaje de programación"

latency_mini = measure_local_latency("all-MiniLM-L6-v2", text)
latency_mpnet = measure_local_latency("all-mpnet-base-v2", text)

print(f"all-MiniLM-L6-v2: {latency_mini:.1f}ms")
print(f"all-mpnet-base-v2: {latency_mpnet:.1f}ms")

Output típico (CPU):

all-MiniLM-L6-v2: 4.8ms
all-mpnet-base-v2: 14.2ms

Speedup vs OpenAI: 18x más rápido (local CPU vs API).

GPU vs CPU:

# CPU (Intel i7):
# all-MiniLM-L6-v2: ~5ms
# all-mpnet-base-v2: ~15ms
# bge-large-en: ~30ms

# GPU (NVIDIA T4):
# all-MiniLM-L6-v2: ~1ms    (5x speedup)
# all-mpnet-base-v2: ~3ms   (5x speedup)
# bge-large-en: ~8ms        (4x speedup)

# GPU acelera ~4-5x

Throughput Benchmarking

Medir throughput (QPS):

import time
from sentence_transformers import SentenceTransformer

def measure_throughput(model_name, n_texts=1000):
    """Medir throughput (queries/segundo)"""
    model = SentenceTransformer(model_name)
    
    texts = [f"Texto {i}" for i in range(n_texts)]
    
    start = time.time()
    _ = model.encode(texts, show_progress_bar=False)
    elapsed = time.time() - start
    
    throughput = n_texts / elapsed
    return throughput

# Test
throughput_mini = measure_throughput("all-MiniLM-L6-v2", n_texts=1000)
throughput_mpnet = measure_throughput("all-mpnet-base-v2", n_texts=1000)

print(f"all-MiniLM-L6-v2: {throughput_mini:.0f} QPS")
print(f"all-mpnet-base-v2: {throughput_mpnet:.0f} QPS")

Output típico (CPU):

all-MiniLM-L6-v2: 215 QPS
all-mpnet-base-v2: 68 QPS

Batch Processing Impact

Comparación: Single vs Batch:

import time
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = ["Python es popular"] * 100

# Single processing (1 texto a la vez)
start = time.time()
for text in texts:
    _ = model.encode([text])
single_time = time.time() - start

# Batch processing (todos de una vez)
start = time.time()
_ = model.encode(texts)
batch_time = time.time() - start

print(f"Single: {single_time:.2f}s")
print(f"Batch: {batch_time:.2f}s")
print(f"Speedup: {single_time / batch_time:.1f}x")

Output típico:

Single: 0.48s
Batch: 0.09s
Speedup: 5.3x

Batch processing → 5x speedup.

Optimal batch size:

import time
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = ["Texto " + str(i) for i in range(1000)]

batch_sizes = [1, 8, 16, 32, 64, 128, 256]
results = []

for batch_size in batch_sizes:
    start = time.time()
    
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i+batch_size]
        _ = model.encode(batch)
    
    elapsed = time.time() - start
    throughput = len(texts) / elapsed
    
    results.append({
        'batch_size': batch_size,
        'time': elapsed,
        'throughput': throughput
    })
    
    print(f"Batch size {batch_size}: {throughput:.0f} QPS")

# Optimal batch size
best = max(results, key=lambda x: x['throughput'])
print(f"\nOptimal batch size: {best['batch_size']}")

Output típico:

Batch size 1: 208 QPS
Batch size 8: 620 QPS
Batch size 16: 890 QPS
Batch size 32: 1024 QPS  ← Optimal
Batch size 64: 1015 QPS  (plateau)
Batch size 128: 990 QPS  (memory overhead)
Batch size 256: 950 QPS

Optimal batch size: 32

Sweet spot: Batch size 32-64 (depende de hardware).

Comparison Table: Models

Latencia (ms/query):

Throughput (QPS):

OpenAI API: ~10 QPS (limitado por RPM rate limits).

Latencia en producción

Requisitos típicos:

# Semantic search (e-commerce):
# Latencia tolerable: <100ms
# Modelos OK: Todos (local <30ms, API ~90ms)

# Chatbot (real-time):
# Latencia tolerable: <50ms
# Modelos OK: Self-hosted (CPU <30ms, GPU <10ms)

# Batch processing (nightly):
# Latencia tolerable: Cualquiera
# Prioridad: Throughput (GPU >> CPU)

# RAG system:
# Latencia tolerable: <200ms
# Modelos OK: Todos (latencia embedding solo parte del pipeline)

Optimizaciones de latencia

1. Usar GPU (si tienes):

# CPU: 15ms
# GPU: 3ms
# Speedup: 5x

from sentence_transformers import SentenceTransformer

# Cargar en GPU automáticamente (si disponible)
model = SentenceTransformer("all-mpnet-base-v2", device='cuda')

2. Batch processing:

# Single: 1000 queries × 15ms = 15s
# Batch (32): 1000 queries / 32 batch × 50ms = 1.6s
# Speedup: 9x

3. Model quantization (FP16):

# FP32 (default): 15ms, 420MB
# FP16 (half precision): 10ms, 210MB
# Speedup: 1.5x, Storage: 2x menos

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-mpnet-base-v2")
model.half()  # Convert to FP16

4. Modelo más pequeño:

# all-mpnet-base-v2: 15ms, MTEB 58
# all-MiniLM-L6-v2: 5ms, MTEB 56
# Trade-off: -2 MTEB puntos, +3x speedup

Ejercicios

Ejercicio 1: Medir latencia

Mide latencia de all-MiniLM-L6-v2 en tu máquina:

# Implementa función measure_latency()
# Ejecuta 100 veces
# Reporta latencia promedio

import time
from sentence_transformers import SentenceTransformer

def measure_latency(model_name, text, n_runs=100):
    model = SentenceTransformer(model_name)
    
    # Warmup
    _ = model.encode([text])
    
    latencies = []
    for _ in range(n_runs):
        start = time.time()
        _ = model.encode([text])
        latency = (time.time() - start) * 1000
        latencies.append(latency)
    
    return sum(latencies) / len(latencies)

# Test
latency = measure_latency("all-MiniLM-L6-v2", "Python es popular")
print(f"Latencia promedio: {latency:.1f}ms")

Ejercicio 2: Comparar batch sizes

Encuentra el batch size óptimo para all-MiniLM-L6-v2:

# Test batch sizes: [1, 8, 16, 32, 64]
# ¿Cuál tiene mejor throughput?

import time
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = ["Texto " + str(i) for i in range(500)]

batch_sizes = [1, 8, 16, 32, 64]

for batch_size in batch_sizes:
    start = time.time()
    
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i+batch_size]
        _ = model.encode(batch)
    
    elapsed = time.time() - start
    throughput = len(texts) / elapsed
    
    print(f"Batch {batch_size}: {throughput:.0f} QPS")

Batch 1: 210 QPS
Batch 8: 620 QPS
Batch 16: 890 QPS
Batch 32: 1020 QPS  ← Optimal
Batch 64: 1010 QPS

Resumen

Qué aprendiste:

• ✅ Latencia: Tiempo/query (ms)
• ✅ Throughput: Queries/segundo (QPS)
• ✅ API: OpenAI ~90-110ms
• ✅ Local: CPU ~5-30ms, GPU ~1-8ms
• ✅ Batch processing: 5-10x speedup
• ✅ Optimal batch size: 32-64

Conceptos clave:

1. Self-hosted >> API en latencia (18x faster)
2. GPU >> CPU (5x faster)
3. Batch processing crítico para throughput

Recursos adicionales

1. Sentence-Transformers Speed - Performance data
2. GPU Benchmarking - HuggingFace
3. Batch Processing Guide - SBERT

En la siguiente cápsula

Cápsula 06: Cost Analysis

Aprenderás:

• Comparar costos (OpenAI vs self-hosted)
• TCO (Total Cost of Ownership)
• Break-even calculations
• Cuándo self-hosting hace sentido

De velocidad a economía.

Módulo 3 - Embeddings Deep Dive Guide Latencia y throughput: performance real en producción