Comparación de Providers

Descripción

Comparación side-by-side de providers principales (OpenAI, Anthropic, Google, Mistral) usando OpenRouter.

Tiempo: 15 minutos
Dificultad: Baja

---

🎯 Objetivos

• ✅ Benchmark múltiples providers
• ✅ Comparar latency
• ✅ Comparar calidad
• ✅ Comparar costos

---

📊 Benchmark Script

import time
from openai import OpenAI
import os

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.getenv("OPENROUTER_API_KEY")
)

def benchmark_model(model: str, prompt: str) -> dict:
    """Benchmark un modelo."""
    
    start = time.time()
    
    try:
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}]
        )
        
        latency = time.time() - start
        
        return {
            "model": model,
            "response": response.choices[0].message.content,
            "latency": latency,
            "tokens": response.usage.total_tokens,
            "success": True
        }
    
    except Exception as e:
        return {
            "model": model,
            "error": str(e),
            "success": False
        }

# Modelos a comparar
models = [
    "openai/gpt-3.5-turbo",
    "anthropic/claude-3-haiku",
    "google/gemini-pro",
    "mistralai/mixtral-8x7b-instruct"
]

prompt = "Explica qué es Python en 30 palabras"

print("=== Benchmark Results ===\n")

results = []
for model in models:
    print(f"Testing {model}...")
    result = benchmark_model(model, prompt)
    results.append(result)
    
    if result["success"]:
        print(f"  ✅ Latency: {result['latency']:.2f}s")
        print(f"  Tokens: {result['tokens']}")
    else:
        print(f"  ❌ Failed: {result['error']}")
    print()

# Ranking por latency
print("=== Ranking (Latency) ===")
successful = [r for r in results if r["success"]]
successful.sort(key=lambda x: x["latency"])

for i, r in enumerate(successful, 1):
    print(f"{i}. {r['model']}: {r['latency']:.2f}s")

---

📊 Comparación Detallada

Provider	Modelo	Latency	Cost ($/1M)	Context	Strengths
OpenAI	GPT-3.5	1.5s	$0.50	16k	Balance general
OpenAI	GPT-4	3.2s	$10.00	128k	Máxima calidad
Anthropic	Claude 3 Haiku	1.2s	$0.25	200k	Rápido + barato
Anthropic	Claude 3 Opus	4.5s	$15.00	200k	Textos largos
Google	Gemini Pro	2.1s	$0.50	32k	Multimodal
Mistral	Mixtral 8x7B	2.3s	$0.24	32k	Costo bajo

---

🎯 Use Cases por Provider

OpenAI (GPT):

Mejor para:

• General purpose chat
• Well-documented API
• Máxima adoption (community)

Evita si:

• Costo es crítico
• Context window >16k (usa Claude)

---

Anthropic (Claude):

Mejor para:

• Textos muy largos (200k context)
• Análisis de documentos
• Bajo costo (Haiku)

Evita si:

• Necesitas máxima velocidad
• Multimodal (imágenes)

---

Google (Gemini):

Mejor para:

• Multimodal (texto + imágenes)
• Integración Google ecosystem
• Multilingual

Evita si:

• Solo texto simple

---

Mistral (Mixtral):

Mejor para:

• Costo ultra-bajo
• Español nativo
• Open source compatible

Evita si:

• Necesitas máxima calidad (usa GPT-4)

---

💰 Cost Comparison (1M tokens)

pricing = {
    "OpenAI GPT-3.5": 0.50,
    "Anthropic Claude Haiku": 0.25,
    "Google Gemini Pro": 0.50,
    "Mistral Mixtral": 0.24,
    "OpenAI GPT-4": 10.00,
    "Anthropic Claude Opus": 15.00
}

# Para 1M queries (500 tokens promedio)
queries = 1_000_000
tokens_per_query = 500
total_tokens = queries * tokens_per_query  # 500M tokens

print("=== Cost for 1M queries (500 tokens avg) ===\n")

for model, price_per_million in pricing.items():
    total_cost = (total_tokens / 1_000_000) * price_per_million
    print(f"{model:30} ${total_cost:,.0f}")

Output:

OpenAI GPT-3.5                 $250
Anthropic Claude Haiku         $125  (50% cheaper!)
Google Gemini Pro              $250
Mistral Mixtral                $120  (52% cheaper!)
OpenAI GPT-4                   $5,000  (20x más caro!)
Anthropic Claude Opus          $7,500  (30x más caro!)

---

🔄 Provider Diversity Strategy

class DiversifiedRouter:
    """Router que distribuye carga entre providers."""
    
    def __init__(self):
        self.client = OpenAI(
            base_url="https://openrouter.ai/api/v1",
            api_key=os.getenv("OPENROUTER_API_KEY")
        )
        self.provider_pool = [
            "openai/gpt-3.5-turbo",
            "anthropic/claude-3-haiku",
            "google/gemini-pro"
        ]
        self.current_index = 0
    
    def chat(self, prompt: str) -> str:
        """Round-robin entre providers."""
        
        model = self.provider_pool[self.current_index]
        self.current_index = (self.current_index + 1) % len(self.provider_pool)
        
        response = self.client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}]
        )
        
        print(f"[Used: {model}]")
        return response.choices[0].message.content

# Test
router = DiversifiedRouter()
for _ in range(6):
    router.chat("Test")
# Output: Alterna OpenAI → Anthropic → Google → OpenAI...

Ventaja: Distribuyes riesgo entre múltiples providers

---

✅ Resumen

Comparación providers:

• OpenAI: Estándar, calidad alta
• Anthropic: Context largo, barato (Haiku)
• Google: Multimodal
• Mistral: Ultra-barato

Estrategia: Usa múltiples providers para diversificar riesgo y optimizar costo.

---

Siguiente: 08-proyecto-smart-chatbot.md

Proyecto final: Chatbot que auto-selecciona modelo y tiene fallback robusto.

Tiempo: 60 min