5. The Model Landscape

Description

The multimodal ecosystem has several main providers: OpenAI (GPT-4 Vision, Whisper, DALL-E, TTS), Anthropic (Claude 3), Google (Gemini). Each one has different strengths, limitations and costs. In this capsule you'll learn to compare them with concrete criteria and pick the right one for each use case. There is no "best model" — there's a "best one for your case".

Why it matters: Choosing the wrong model can mean 10x more cost, worse quality, or missing functionality. Knowing the landscape lets you make informed decisions, design effective fallbacks, and optimize your budget in production.

Connection with the module: Capsules 02-04 showed you the capabilities of vision, audio and combinations. Here you compare the providers to know when to use each one. This information is fundamental for the multimodal Classifier (capsule 08), which has to recommend the optimal model.

The 3 Big Providers

OpenAI

Complete ecosystem: GPT-4o (vision), Whisper (STT), DALL-E 3 (image), TTS (voice).

from openai import OpenAI
client = OpenAI()  # A single library for everything

Strengths:

• ✅ Integrated ecosystem (one API key for everything)
• ✅ GPT-4o is excellent at vision and reasoning
• ✅ Whisper leads in transcription
• ✅ Extensive documentation, large community
• ✅ gpt-4o-mini offers a good quality/price balance

Weaknesses:

• ⚠️ Costs can add up quickly
• ⚠️ Strict rate limits on free/low tiers
• ⚠️ It doesn't offer speaker diarization in the Whisper API

Anthropic

Specialized in reasoning: Claude 3 (Sonnet, Opus, Haiku) with vision.

from anthropic import Anthropic
client = Anthropic()  # Chat/vision only

Strengths:

• ✅ Deep reasoning (especially Opus)
• ✅ Excellent on complex documents and extensive analysis
• ✅ 200K-token context window
• ✅ Good instruction following
• ✅ Fewer "hallucinations" in extraction tasks

Weaknesses:

• ⚠️ It doesn't have Whisper/TTS/DALL-E (vision + text only)
• ⚠️ Image limit: 5MB (base64)
• ⚠️ More expensive than GPT-4o for vision

Google (Gemini)

Massive context: Gemini 1.5 Pro (1M tokens), Flash (fast and cheap).

import google.generativeai as genai
genai.configure(api_key="...")

Strengths:

• ✅ 1M-token context window (the largest)
• ✅ Gemini Flash: very fast and very cheap
• ✅ Natively multimodal (text, image, audio, video in a single model)
• ✅ Generous free tier
• ✅ Processes video natively

Weaknesses:

• ⚠️ Reasoning quality one step behind GPT-4o and Claude 3 Opus
• ⚠️ Less mature API than OpenAI
• ⚠️ Smaller community and fewer examples

Comparison Tables

Vision

Audio (Speech-to-Text)

Text-to-Speech

Image Generation

Decision Tree: Which Model to Use?

For Vision

What do you need?

├── OCR of simple documents
│   └── gpt-4o-mini (cheap, good)
│
├── Deep analysis of complex documents
│   └── Claude 3.5 Sonnet or GPT-4o (better reasoning)
│
├── High volume, low cost
│   └── Gemini 1.5 Flash (very cheap, fast)
│
├── Huge context window (long docs)
│   └── Gemini 1.5 Pro (1M tokens)
│
├── Maximum reasoning quality
│   └── Claude 3 Opus (expensive but superior)
│
└── Integrated ecosystem (vision + audio + image)
    └── OpenAI (GPT-4o + Whisper + DALL-E + TTS)

For Audio

What do you need?

├── Simple transcription of files
│   └── Whisper (simple, cheap, excellent)
│
├── Real-time transcription (streaming)
│   └── Google STT or AssemblyAI
│
├── Speaker diarization (who said what)
│   └── AssemblyAI
│
├── Standard TTS (good quality)
│   └── OpenAI TTS
│
└── Premium TTS or voice cloning
    └── ElevenLabs

Fallback Strategies

Fallback by availability

If the main provider fails (rate limit, timeout, error), use an alternative.

from openai import OpenAI
from anthropic import Anthropic
import base64
import logging

logger = logging.getLogger(__name__)

openai_client = OpenAI()
anthropic_client = Anthropic()

def analyze_image_with_fallback(image_path: str, prompt: str) -> dict:
    """Try OpenAI → Anthropic → Error."""
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    # Attempt 1: OpenAI
    try:
        response = openai_client.chat.completions.create(
            model="gpt-4o",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data}"
                    }}
                ]
            }],
            max_tokens=500
        )
        return {
            "provider": "openai",
            "model": "gpt-4o",
            "result": response.choices[0].message.content
        }
    except Exception as e:
        logger.warning(f"OpenAI failed: {e}")

    # Attempt 2: Anthropic
    try:
        response = anthropic_client.messages.create(
            model="claude-3-5-sonnet-latest",
            max_tokens=500,
            messages=[{
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/jpeg",
                            "data": image_data
                        }
                    },
                    {"type": "text", "text": prompt}
                ]
            }]
        )
        return {
            "provider": "anthropic",
            "model": "claude-3-5-sonnet",
            "result": response.content[0].text
        }
    except Exception as e:
        logger.warning(f"Anthropic failed: {e}")

    raise RuntimeError("All vision providers failed")

Fallback priority table

Fallback by cost

Use a cheap model by default, escalate to premium if the quality isn't enough.

def smart_cost_analysis(image_path: str, prompt: str) -> dict:
    """Start with mini, escalate to pro if needed."""
    # Step 1: Try with the economical model
    result = call_vision("gpt-4o-mini", image_path, prompt)

    # Step 2: Check whether the response is trustworthy
    if needs_escalation(result):
        result = call_vision("gpt-4o", image_path, prompt)
        return {"tier": "premium", "result": result}

    return {"tier": "economic", "result": result}


def needs_escalation(result: str) -> bool:
    """Detect whether the result needs a better model."""
    uncertainty_phrases = [
        "i'm not sure", "i cannot determine",
        "the image is not clear", "it is not possible"
    ]
    return any(phrase in result.lower() for phrase in uncertainty_phrases)

Troubleshooting

Problem 1: Rate limit exceeded

Cause: Too many requests to the same provider.

Solution:

import time

def call_with_rate_limit(func, *args, max_retries=3, **kwargs):
    for attempt in range(max_retries):
        try:
            return func(*args, **kwargs)
        except Exception as e:
            if "rate_limit" in str(e).lower():
                wait = 2 ** attempt
                time.sleep(wait)
            else:
                raise
    raise RuntimeError(f"Rate limit after {max_retries} attempts")

Problem 2: Format differences between providers

Cause: OpenAI, Anthropic and Google have different API formats.

Solution: Create wrappers with a unified interface (as shown in the fallback code above).

Problem 3: Anthropic's API differs from OpenAI's

Cause: The APIs have different interfaces for sending images.

Solution: Note the key differences:

# OpenAI: image as "image_url" with a data URI
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}

# Anthropic: image as "image" with a source object
{"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": "..."}}

# Google: uses PIL.Image directly
model.generate_content([prompt, PIL.Image.open(path)])

Problem 4: Unexpected costs

Cause: Not monitoring usage.

Solution: Implement per-request cost tracking:

COSTS = {
    "gpt-4o": {"input": 2.50, "output": 10.0},      # per 1M tokens
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "whisper-1": {"per_minute": 0.006},
    "tts-1": {"per_million_chars": 15.0},
    "dall-e-3": {"per_image": 0.04},
}

def estimate_cost(model: str, input_tokens: int = 0,
                  output_tokens: int = 0) -> float:
    if model in COSTS:
        cost_info = COSTS[model]
        return (
            (input_tokens / 1_000_000) * cost_info.get("input", 0)
            + (output_tokens / 1_000_000) * cost_info.get("output", 0)
        )
    return 0.0

Exercises

Exercise 1: Custom decision table (Easy)

For an invoice-analysis project in Spanish (10,000 invoices/month), which provider would you choose and why? Consider cost, OCR quality, and volume.

Exercise 2: Multi-provider wrapper (Medium)

Create a vision_analyze function that accepts a provider parameter ("openai", "anthropic", "google") and calls the corresponding API with the same interface.

def vision_analyze(
    image_path: str,
    prompt: str,
    provider: str = "openai"
) -> str:
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    if provider == "openai":
        response = openai_client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data}"
                    }}
                ]
            }],
            max_tokens=500
        )
        return response.choices[0].message.content

    elif provider == "anthropic":
        response = anthropic_client.messages.create(
            model="claude-3-5-sonnet-latest",
            max_tokens=500,
            messages=[{
                "role": "user",
                "content": [
                    {"type": "image", "source": {
                        "type": "base64",
                        "media_type": "image/jpeg",
                        "data": image_data
                    }},
                    {"type": "text", "text": prompt}
                ]
            }]
        )
        return response.content[0].text

    elif provider == "google":
        import google.generativeai as genai
        from PIL import Image
        model = genai.GenerativeModel("gemini-1.5-flash")
        img = Image.open(image_path)
        response = model.generate_content([prompt, img])
        return response.text

    else:
        raise ValueError(f"Unsupported provider: {provider}")

Exercise 3: Provider benchmark (Medium)

Create a function that sends the same image and prompt to the 3 providers and compares results (response, latency).

import time

def benchmark_providers(image_path: str, prompt: str) -> list[dict]:
    providers = ["openai", "anthropic", "google"]
    results = []

    for provider in providers:
        try:
            start = time.time()
            result = vision_analyze(image_path, prompt, provider)
            latency = round(time.time() - start, 2)
            results.append({
                "provider": provider,
                "status": "ok",
                "latency_s": latency,
                "response_length": len(result),
                "response_preview": result[:200]
            })
        except Exception as e:
            results.append({
                "provider": provider,
                "status": "error",
                "error": str(e)
            })

    return results

# Usage
results = benchmark_providers("test_image.jpg", "Describe this image.")
for r in results:
    print(f"{r['provider']}: {r.get('latency_s', 'N/A')}s - {r['status']}")

Exercise 4: Cost calculator (Hard)

Create a function that, given a monthly usage plan (N images with vision, M minutes of audio, K generated images), calculates the cost per provider.

def estimate_monthly_cost(
    vision_images: int = 0,
    audio_minutes: int = 0,
    generated_images: int = 0,
    tts_characters: int = 0
) -> dict:
    avg_tokens_per_image = 800  # average tokens per image
    avg_output_tokens = 300

    costs = {}

    # OpenAI
    vision_cost = (
        (vision_images * avg_tokens_per_image / 1_000_000) * 2.50
        + (vision_images * avg_output_tokens / 1_000_000) * 10.0
    )
    audio_cost = audio_minutes * 0.006
    image_cost = generated_images * 0.04
    tts_cost = (tts_characters / 1_000_000) * 15
    costs["openai"] = round(
        vision_cost + audio_cost + image_cost + tts_cost, 2
    )

    # GPT-4o-mini (cheaper vision)
    vision_mini = (
        (vision_images * avg_tokens_per_image / 1_000_000) * 0.15
        + (vision_images * avg_output_tokens / 1_000_000) * 0.60
    )
    costs["openai_mini"] = round(
        vision_mini + audio_cost + image_cost + tts_cost, 2
    )

    # Gemini Flash
    vision_flash = (
        (vision_images * avg_tokens_per_image / 1_000_000) * 0.075
        + (vision_images * avg_output_tokens / 1_000_000) * 0.30
    )
    costs["gemini_flash"] = round(vision_flash, 2)

    return costs

# Usage
monthly = estimate_monthly_cost(
    vision_images=10000,
    audio_minutes=500,
    generated_images=100,
    tts_characters=500000
)
print(monthly)
# Output: {'openai': 64.5, 'openai_mini': 17.5, 'gemini_flash': 1.5}

Summary

In this capsule you learned:

• 3 main providers: OpenAI (complete ecosystem), Anthropic (deep reasoning), Google (massive context and low cost)
• Every provider has clear trade-offs in cost, quality, latency and functionality
• The decision tree guides you: what you need → which model to use
• Fallback by availability (provider fails → alternative) and by cost (cheap first → premium if needed)
• Gemini Flash is the cheapest for vision at volume
• Claude 3 Opus has the best reasoning but is the most expensive
• GPT-4o-mini offers the best quality/price balance for most cases
• At high volume, the cost difference between models can be 10-40x
• A multi-provider wrapper with a unified interface makes it easy to switch models without rewriting code

Next capsule: Formats and APIs — the technical details of Base64, URLs, sizes and costs per endpoint.

Additional Resources

1. OpenAI Pricing — Up-to-date prices for all OpenAI models
2. Anthropic Pricing — Claude 3 prices
3. Google AI Pricing — Gemini prices (includes the free tier)
4. Replicate Models — Model catalog including Stable Diffusion
5. ElevenLabs Pricing — Premium TTS prices
6. AssemblyAI Pricing — Transcription prices with advanced features