Module 6: The AI API Ecosystem

6. Aggregators and Routers: OpenRouter, Together.ai and Unified Access

Inicia sesion para marcar tu progreso

Description

Problem: Multiple providers (OpenAI, Anthropic, Google, Meta) → multiple APIs, multiple integrations, complexity.

Solution: Aggregators → platforms that unify access to multiple models with a single API.

The main aggregators:

OpenRouter (openrouter.ai)
Together.ai
Replicate

Benefits:

Access to 50+ models (GPT-4, Claude, Llama, Mistral) with one API.
Easy switching between models (changing 1 line of code).
Automatic fallback (if one model fails, use another).

OpenRouter

What it is: An aggregator that unifies access to 100+ LLMs (OpenAI, Anthropic, Google, Meta, Mistral, etc.).

URL: openrouter.ai

Advantages:

✅ 100+ models: GPT-4, Claude, Gemini, Llama, Mistral, etc.
✅ One API: Compatible with the OpenAI SDK (a drop-in replacement).
✅ Transparent pricing: It shows the cost in real time.
✅ Automatic fallback: If model A fails, use model B.

How it works:

import openai

# Change base_url to OpenRouter
client = openai.OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="sk-or-..."  # OpenRouter API key
)

# Use any model (GPT-4, Claude, Llama, etc.)
response = client.chat.completions.create(
    model="anthropic/claude-3-opus",  # Or "openai/gpt-4", "meta-llama/llama-3-70b"
    messages=[{"role": "user", "content": "Hello"}]
)

Changing the model: Just change model → 1 line.

OpenRouter: Use Cases

1. A Multi-Model Strategy

Problem: You want to use GPT-4 for complex tasks, GPT-3.5 for simple ones.

Without OpenRouter:

# Code for OpenAI
response_gpt4 = openai_client.chat.completions.create(...)

# Code for Anthropic
response_claude = anthropic_client.messages.create(...)

With OpenRouter:

# The same code, only the model changes
response_gpt4 = client.chat.completions.create(model="openai/gpt-4", ...)
response_claude = client.chat.completions.create(model="anthropic/claude-3-opus", ...)

2. Automatic Fallback

Problem: If the OpenAI API goes down, your app goes down.

Solution (OpenRouter):

# Configure fallbacks
response = client.chat.completions.create(
    model="openai/gpt-4",
    fallbacks=["anthropic/claude-3-opus", "meta-llama/llama-3-70b"],
    messages=[{"role": "user", "content": "Hello"}]
)

If GPT-4 fails → it tries Claude → if that fails, it tries Llama 3.

3. A/B Testing

Problem: Is GPT-4 or Claude 3 Opus better for my use case?

Solution: Send 50% of requests to GPT-4, 50% to Claude → compare quality.

Together.ai

What it is: A platform for running open-source models (Llama, Mistral, Qwen, etc.) in the cloud.

URL: together.ai

Advantages:

✅ A focus on open-source: Llama 3, Mistral, Qwen, Falcon, etc.
✅ Cheaper than OpenAI: a ~50-80% discount vs GPT-4.
✅ An OpenAI-compatible API.
✅ Fine-tuning: You can fine-tune open-source models easily.

How it works:

import openai

client = openai.OpenAI(
    base_url="https://api.together.xyz/v1",
    api_key="..."
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3-70b-chat-hf",
    messages=[{"role": "user", "content": "Hello"}]
)

Together.ai: Use Cases

1. Open-Source without Self-Hosting

Problem: You want Llama 3, but you don't want to maintain GPUs.

Solution: Together.ai hosts Llama 3 → you access it via API (no infrastructure).

2. Fine-Tuning

Problem: GPT-4 isn't good enough for your specific use case.

Solution: Fine-tune Llama 3 on Together.ai with your data → better quality.

Replicate

What it is: A platform for running ML/AI models (including LLMs, image, audio) in the cloud.

URL: replicate.com

Advantages:

✅ Diverse models: LLMs (Llama, Mistral), image (Stable Diffusion), audio (Whisper).
✅ Serverless: You pay only per second of compute (you don't pay for idle time).
✅ Community models: Anyone can publish a model.

How it works:

import replicate

output = replicate.run(
    "meta/llama-3-70b-instruct",
    input={"prompt": "Hello, how are you?"}
)

Comparison: OpenRouter vs Together.ai vs Replicate

Aspect	OpenRouter	Together.ai	Replicate
Models	100+ (GPT-4, Claude, Llama, Mistral, etc.)	Open-source (Llama, Mistral, Qwen)	Diverse (LLMs, image, audio)
API	Compatible with OpenAI	Compatible with OpenAI	Its own (simple)
Pricing	Pay-per-use (transparent)	Pay-per-use (cheaper than OpenAI)	Pay-per-second (serverless)
Fallback	Yes (automatic)	No (manual)	No (manual)
Fine-tuning	No	Yes (easy)	Yes (manual)
Focus	An aggregator (unified access)	Open-source hosting	Serverless ML

Why Use Aggregators

1. Avoiding Vendor Lock-In

Problem: If you integrate directly with the OpenAI API, switching to Claude requires rewriting code.

Solution (OpenRouter): Changing the model is 1 line → you can change provider without rewriting code.

2. A Multi-Model Strategy

Strategy: Using multiple models depending on the task (GPT-4 for complex, GPT-3.5 for simple).

Without an aggregator: Integrating with OpenAI, Anthropic, Google → 3 SDKs, 3 APIs.

With an aggregator: 1 SDK, 1 API → changing the model is trivial.

3. Automatic Fallback

Problem: APIs fail (downtime, rate limits).

Solution (OpenRouter): Configure fallbacks → if GPT-4 fails, use Claude automatically.

Common Mistakes

1. Assuming that every aggregator has every model

Mistake: Assuming that Together.ai has GPT-4 (it doesn't, only open-source).

Solution: Check which models each aggregator supports.

2. Not considering latency

Mistake: An aggregator adds latency (a proxy between your app and the provider).

Result: 50-200ms of extra latency.

Solution: For ultra-low-latency applications, use the provider directly.

Why this matters for an AI Engineer

1. Flexibility

A multi-provider strategy: You can change model without rewriting code.

2. Resilience

Automatic fallback: If one provider fails, your app keeps working.

3. Fast development

Prototyping: Trying multiple models (GPT-4, Claude, Llama) in minutes (just changing the model parameter).

Summary

Aggregators:

OpenRouter: 100+ models (GPT-4, Claude, Llama), a unified API, automatic fallback.
Together.ai: Open-source (Llama, Mistral), cheaper, easy fine-tuning.
Replicate: Serverless ML (LLMs, image, audio), pay-per-second.

Why use them:

Avoiding vendor lock-in (changing the model = 1 line).
A multi-model strategy (GPT-4 for complex, GPT-3.5 for simple).
Automatic fallback (resilience).

Trade-off: Extra latency (50-200ms) vs flexibility.

Next step: Lesson 07: Exercise: Choosing a Provider — Real use cases → a stack recommendation.

← Previous5. Local vs Cloud: LM Studio, Ollama and When to Use Each One Next →7. Exercise: Choosing the Appropriate Provider and Stack