Provider Comparison

Overview

A side-by-side comparison of the main providers (OpenAI, Anthropic, Google, Mistral) using OpenRouter.

Time: 15 minutes
Difficulty: Low

---

🎯 Objectives

• ✅ Benchmark multiple providers
• ✅ Compare latency
• ✅ Compare quality
• ✅ Compare costs

---

📊 Benchmark Script

import time
from openai import OpenAI
import os

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.getenv("OPENROUTER_API_KEY")
)

def benchmark_model(model: str, prompt: str) -> dict:
    """Benchmark a model."""
    
    start = time.time()
    
    try:
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}]
        )
        
        latency = time.time() - start
        
        return {
            "model": model,
            "response": response.choices[0].message.content,
            "latency": latency,
            "tokens": response.usage.total_tokens,
            "success": True
        }
    
    except Exception as e:
        return {
            "model": model,
            "error": str(e),
            "success": False
        }

# Models to compare
models = [
    "openai/gpt-3.5-turbo",
    "anthropic/claude-3-haiku",
    "google/gemini-pro",
    "mistralai/mixtral-8x7b-instruct"
]

prompt = "Explain what Python is in 30 words"

print("=== Benchmark Results ===\n")

results = []
for model in models:
    print(f"Testing {model}...")
    result = benchmark_model(model, prompt)
    results.append(result)
    
    if result["success"]:
        print(f"  ✅ Latency: {result['latency']:.2f}s")
        print(f"  Tokens: {result['tokens']}")
    else:
        print(f"  ❌ Failed: {result['error']}")
    print()

# Ranking by latency
print("=== Ranking (Latency) ===")
successful = [r for r in results if r["success"]]
successful.sort(key=lambda x: x["latency"])

for i, r in enumerate(successful, 1):
    print(f"{i}. {r['model']}: {r['latency']:.2f}s")

---

📊 Detailed Comparison

Provider	Model	Latency	Cost ($/1M)	Context	Strengths
OpenAI	GPT-3.5	1.5s	$0.50	16k	General balance
OpenAI	GPT-4	3.2s	$10.00	128k	Maximum quality
Anthropic	Claude 3 Haiku	1.2s	$0.25	200k	Fast + cheap
Anthropic	Claude 3 Opus	4.5s	$15.00	200k	Long texts
Google	Gemini Pro	2.1s	$0.50	32k	Multimodal
Mistral	Mixtral 8x7B	2.3s	$0.24	32k	Low cost

---

🎯 Use Cases by Provider

OpenAI (GPT):

Best for:

• General purpose chat
• Well-documented API
• Maximum adoption (community)

Avoid if:

• Cost is critical
• Context window >16k (use Claude)

---

Anthropic (Claude):

Best for:

• Very long texts (200k context)
• Document analysis
• Low cost (Haiku)

Avoid if:

• You need maximum speed
• Multimodal (images)

---

Google (Gemini):

Best for:

• Multimodal (text + images)
• Google ecosystem integration
• Multilingual

Avoid if:

• Simple text only

---

Mistral (Mixtral):

Best for:

• Ultra-low cost
• Native Spanish
• Open source compatible

Avoid if:

• You need maximum quality (use GPT-4)

---

💰 Cost Comparison (1M tokens)

pricing = {
    "OpenAI GPT-3.5": 0.50,
    "Anthropic Claude Haiku": 0.25,
    "Google Gemini Pro": 0.50,
    "Mistral Mixtral": 0.24,
    "OpenAI GPT-4": 10.00,
    "Anthropic Claude Opus": 15.00
}

# For 1M queries (500 tokens average)
queries = 1_000_000
tokens_per_query = 500
total_tokens = queries * tokens_per_query  # 500M tokens

print("=== Cost for 1M queries (500 tokens avg) ===\n")

for model, price_per_million in pricing.items():
    total_cost = (total_tokens / 1_000_000) * price_per_million
    print(f"{model:30} ${total_cost:,.0f}")

Output:

OpenAI GPT-3.5                 $250
Anthropic Claude Haiku         $125  (50% cheaper!)
Google Gemini Pro              $250
Mistral Mixtral                $120  (52% cheaper!)
OpenAI GPT-4                   $5,000  (20x more expensive!)
Anthropic Claude Opus          $7,500  (30x more expensive!)

---

🔄 Provider Diversity Strategy

class DiversifiedRouter:
    """Router that distributes load across providers."""
    
    def __init__(self):
        self.client = OpenAI(
            base_url="https://openrouter.ai/api/v1",
            api_key=os.getenv("OPENROUTER_API_KEY")
        )
        self.provider_pool = [
            "openai/gpt-3.5-turbo",
            "anthropic/claude-3-haiku",
            "google/gemini-pro"
        ]
        self.current_index = 0
    
    def chat(self, prompt: str) -> str:
        """Round-robin across providers."""
        
        model = self.provider_pool[self.current_index]
        self.current_index = (self.current_index + 1) % len(self.provider_pool)
        
        response = self.client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}]
        )
        
        print(f"[Used: {model}]")
        return response.choices[0].message.content

# Test
router = DiversifiedRouter()
for _ in range(6):
    router.chat("Test")
# Output: Alternates OpenAI → Anthropic → Google → OpenAI...

Advantage: You distribute risk across multiple providers

---

✅ Summary

Provider comparison:

• OpenAI: Standard, high quality
• Anthropic: Long context, cheap (Haiku)
• Google: Multimodal
• Mistral: Ultra-cheap

Strategy: Use multiple providers to diversify risk and optimize cost.

---

Next: 08-project-smart-chatbot.md

Final project: A chatbot that auto-selects the model and has robust fallback.

Time: 60 min