6. Formats and APIs

Description

So far you've built multimodal pipelines: you've sent images, transcribed audio, combined modalities. But every time something failed —a rejected image, a 413 error, a rate limit— the problem wasn't your logic, it was the format. In this capsule you're going to master the layer between your code and the APIs: Base64, URLs, MIME types, size limits, image tokens, real costs and rate limits. It's the most "infrastructure" capsule of the module, but without it, everything else breaks in production.

Why it matters: Format errors are the number-one cause of failures in multimodal integrations. A 6 MB file sent to Anthropic (5 MB limit) returns a cryptic error. A BMP image sent to OpenAI is rejected with no clear explanation. A batch of 50 images that ignores rate limits gets a 429 and loses all its progress. Knowing the exact requirements of each provider saves you hours of debugging and hundreds of dollars in failed requests.

Connection with the module: This capsule is the technical complement to capsules 02 (vision), 03 (audio) and 05 (model landscape). While those capsules taught you what the models can do, this one teaches you how to prepare the data so they'll accept it. The multimodal Classifier (capsule 08) needs to validate formats before sending any file to an API.

Base64: What It Is and How It Works

The concept

Base64 is an encoding scheme that converts binary data (like an image or an audio file) into an ASCII text string. The AI APIs use it because JSON payloads can't contain binary data directly — they need text.

The conversion inflates the file size by ~33%. A 3 MB image becomes ~4 MB of Base64 text. That matters when you're working with size limits.

Binary file (3 MB) → Base64 encoding → Text string (~4 MB)

Basic encoding

import base64
from pathlib import Path


def encode_image_to_base64(image_path: str) -> str:
    """Encode a local image to Base64."""
    path = Path(image_path)
    if not path.exists():
        raise FileNotFoundError(f"Not found: {image_path}")

    with open(path, "rb") as f:
        raw_bytes = f.read()

    encoded = base64.b64encode(raw_bytes).decode("utf-8")
    return encoded


image_b64 = encode_image_to_base64("product_photo.jpg")
print(f"Base64 string length: {len(image_b64):,} characters")
print(f"First 80 characters: {image_b64[:80]}...")

Data URIs: the format OpenAI expects

OpenAI doesn't accept raw Base64. It needs a data URI that includes the MIME type:

def image_to_data_uri(image_path: str) -> str:
    """Convert a local image to a data URI ready for the OpenAI API."""
    path = Path(image_path)
    extension = path.suffix.lower()

    mime_types = {
        ".jpg": "image/jpeg",
        ".jpeg": "image/jpeg",
        ".png": "image/png",
        ".gif": "image/gif",
        ".webp": "image/webp",
    }

    mime = mime_types.get(extension)
    if not mime:
        raise ValueError(
            f"Unsupported format: {extension}. "
            f"Use: {', '.join(mime_types.keys())}"
        )

    encoded = encode_image_to_base64(image_path)
    return f"data:{mime};base64,{encoded}"


data_uri = image_to_data_uri("diagram.png")
# Result: "data:image/png;base64,iVBORw0KGgo..."

Base64 in Anthropic: a different format

Anthropic doesn't use data URIs. It expects the Base64 and the MIME type as separate fields:

import anthropic

client_anthropic = anthropic.Anthropic()

def send_image_to_claude(image_path: str, prompt: str) -> str:
    """Send an image to Claude with Anthropic's Base64 format."""
    path = Path(image_path)
    extension = path.suffix.lower()

    mime_map = {
        ".jpg": "image/jpeg", ".jpeg": "image/jpeg",
        ".png": "image/png", ".gif": "image/gif", ".webp": "image/webp",
    }

    encoded = encode_image_to_base64(image_path)

    response = client_anthropic.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": mime_map[extension],
                        "data": encoded,
                    },
                },
                {"type": "text", "text": prompt},
            ],
        }],
    )
    return response.content[0].text

URLs as Image Input

When to use URLs

If your image is already hosted on a public server (CDN, S3, GitHub), you can send the URL directly. The provider downloads the image for you:

from openai import OpenAI

client = OpenAI()

def analyze_image_from_url(url: str, prompt: str) -> str:
    """Analyze an image using its public URL."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": prompt},
                {"type": "image_url", "image_url": {"url": url}},
            ],
        }],
        max_tokens=500,
    )
    return response.choices[0].message.content


result = analyze_image_from_url(
    "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg",
    "What animal appears in the image and what is it doing?",
)
print(result)

URL requirements

• The URL must be publicly accessible (no authentication)
• OpenAI and Google accept direct HTTP/HTTPS URLs
• Anthropic does not accept direct URLs — only Base64
• Google accepts URLs and also gs:// (Google Cloud Storage)
• The URL must point to a valid image file, not to an HTML page that contains an image

Comparison: Base64 vs URL

Practical rule:

• Local or private files → Base64
• Images already hosted on a CDN/S3 → URL
• Anthropic → always Base64
• Large batch of public images → URL (lighter requests)

Universal Function: Base64 or URL

This helper detects whether the input is a URL or a local path and prepares the right format for OpenAI:

import base64, re
from pathlib import Path

SUPPORTED_FORMATS = {".jpg", ".jpeg", ".png", ".gif", ".webp"}
MIME_TYPES = {
    ".jpg": "image/jpeg", ".jpeg": "image/jpeg",
    ".png": "image/png", ".gif": "image/gif", ".webp": "image/webp",
}

def prepare_image_content(source: str, detail: str = "auto") -> dict:
    is_url = bool(re.match(r"https?://", source))

    if is_url:
        image_url = source
    else:
        path = Path(source)
        if not path.exists():
            raise FileNotFoundError(f"Not found: {source}")
        if path.suffix.lower() not in SUPPORTED_FORMATS:
            raise ValueError(f"Unsupported format: {path.suffix}")

        with open(path, "rb") as f:
            encoded = base64.b64encode(f.read()).decode("utf-8")
        mime = MIME_TYPES[path.suffix.lower()]
        image_url = f"data:{mime};base64,{encoded}"

    return {"type": "image_url", "image_url": {"url": image_url, "detail": detail}}

# Works the same with a URL or a local path
content_url = prepare_image_content("https://example.com/photo.jpg")
content_local = prepare_image_content("./photos/product.png", detail="high")

Size Limits by Provider

Images

Audio (Whisper)

Whisper response formats

# Transcription with timestamps (verbose_json)
with open("meeting.mp3", "rb") as f:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=f,
        language="en",
        response_format="verbose_json",
    )

for segment in transcript.segments:
    start = segment["start"]
    end = segment["end"]
    text = segment["text"]
    print(f"[{start:.1f}s - {end:.1f}s] {text}")

Resize if it exceeds the limit

from PIL import Image
import io


def resize_for_provider(
    image_path: str,
    provider: str = "openai",
) -> bytes:
    """Resize the image if it exceeds the provider's limit."""
    limits_mb = {"openai": 20, "anthropic": 5, "google": 20}
    max_mb = limits_mb.get(provider, 20)

    with open(image_path, "rb") as f:
        raw = f.read()

    current_mb = len(raw) / (1024 * 1024)
    if current_mb <= max_mb:
        return raw

    with Image.open(image_path) as img:
        img = img.convert("RGB")
        buffer = io.BytesIO()
        quality = 95

        while True:
            buffer.seek(0)
            buffer.truncate()
            img.save(buffer, format="JPEG", quality=quality, optimize=True)

            if len(buffer.getvalue()) <= max_mb * 1024 * 1024:
                break

            quality -= 10
            if quality < 30:
                img = img.resize((img.width // 2, img.height // 2))
                quality = 85

        return buffer.getvalue()

Image Tokens: How OpenAI Charges for Images

Images in OpenAI don't have a fixed "price per image". They're converted into tokens based on the resolution, and those tokens are charged just like text tokens.

Detail levels

Tile calculation in high mode

OpenAI scales the image so it fits in 2048×2048, then splits it into 512×512 tiles:

import math


def calculate_image_tokens(width: int, height: int, detail: str = "high") -> int:
    """Calculate image tokens according to OpenAI's documentation.

    Reference: https://platform.openai.com/docs/guides/vision
    """
    if detail == "low":
        return 85

    max_dim = 2048
    if max(width, height) > max_dim:
        scale = max_dim / max(width, height)
        width = int(width * scale)
        height = int(height * scale)

    min_side = 768
    if min(width, height) > min_side:
        scale = min_side / min(width, height)
        width = int(width * scale)
        height = int(height * scale)

    tiles_x = math.ceil(width / 512)
    tiles_y = math.ceil(height / 512)
    total_tiles = tiles_x * tiles_y

    return 85 + (170 * total_tiles)


# Examples
print(calculate_image_tokens(1024, 1024, "high"))   # 85 + 170*4 = 765
print(calculate_image_tokens(1024, 1024, "low"))     # 85
print(calculate_image_tokens(4000, 3000, "high"))    # 85 + 170*4 = 765 (it rescales to 1024x768)
print(calculate_image_tokens(512, 512, "high"))      # 85 + 170*1 = 255

How it affects your bill

def estimate_vision_cost(
    width: int,
    height: int,
    detail: str = "high",
    model: str = "gpt-4o",
    text_tokens: int = 200,
    output_tokens: int = 300,
) -> dict:
    """Estimate the cost of a call with an image."""
    pricing = {
        "gpt-4o": {"input": 2.50, "output": 10.00},
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    }

    if model not in pricing:
        raise ValueError(f"Unsupported model: {model}")

    image_tokens = calculate_image_tokens(width, height, detail)
    total_input = image_tokens + text_tokens
    prices = pricing[model]

    input_cost = (total_input / 1_000_000) * prices["input"]
    output_cost = (output_tokens / 1_000_000) * prices["output"]
    total_cost = input_cost + output_cost

    return {
        "image_tokens": image_tokens,
        "text_tokens": text_tokens,
        "total_input_tokens": total_input,
        "output_tokens": output_tokens,
        "input_cost_usd": round(input_cost, 6),
        "output_cost_usd": round(output_cost, 6),
        "total_cost_usd": round(total_cost, 6),
    }


cost = estimate_vision_cost(1024, 1024, detail="high", model="gpt-4o")
print(f"1024x1024 image with gpt-4o: ${cost['total_cost_usd']:.4f}")
# ~$0.0054

cost_mini = estimate_vision_cost(1024, 1024, detail="high", model="gpt-4o-mini")
print(f"1024x1024 image with gpt-4o-mini: ${cost_mini['total_cost_usd']:.4f}")
# ~$0.0003

Costs by Modality: Real Numbers

Price table (OpenAI, March 2025)

Real cost scenarios

The difference between gpt-4o and gpt-4o-mini is ~17x for vision. Always evaluate whether gpt-4o-mini's quality is enough for your case before scaling up.

Rate Limits: Limits by Provider

Typical limits (varies by tier)

The limits go up as you spend more on the platform. Check the official documentation for your current tier.

Response headers

When you hit a rate limit, the response includes headers that tell you how long to wait:

x-ratelimit-limit-requests: 500
x-ratelimit-remaining-requests: 0
x-ratelimit-reset-requests: 12s
retry-after: 12

Exponential backoff

import time
import random
from openai import OpenAI, RateLimitError, APIError

client = OpenAI()


def call_with_backoff(
    func,
    max_retries: int = 5,
    base_delay: float = 1.0,
    max_delay: float = 60.0,
):
    """Run a function with exponential backoff on rate limits."""
    for attempt in range(max_retries):
        try:
            return func()
        except RateLimitError as e:
            if attempt == max_retries - 1:
                raise

            delay = min(base_delay * (2 ** attempt), max_delay)
            jitter = random.uniform(0, delay * 0.1)
            wait = delay + jitter

            print(f"Rate limit (attempt {attempt + 1}/{max_retries}). "
                  f"Waiting {wait:.1f}s...")
            time.sleep(wait)

        except APIError as e:
            if e.status_code and e.status_code >= 500:
                if attempt == max_retries - 1:
                    raise
                time.sleep(base_delay * (2 ** attempt))
            else:
                raise


result = call_with_backoff(
    lambda: client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Say hello"}],
        max_tokens=10,
    )
)
print(result.choices[0].message.content)

Troubleshooting

Problem 1: Invalid image format or Could not process image

Cause: The file has a .jpg extension but is internally another format (e.g.: a renamed PNG), or it's an unsupported format like BMP or TIFF.

Solution:

from PIL import Image


def validate_and_convert(image_path: str) -> str:
    """Validate an image and convert it to JPEG if necessary."""
    try:
        with Image.open(image_path) as img:
            img.verify()
    except Exception:
        raise ValueError(f"Not a valid image: {image_path}")

    with Image.open(image_path) as img:
        if img.format not in ("JPEG", "PNG", "GIF", "WEBP"):
            converted_path = str(Path(image_path).with_suffix(".jpg"))
            img.convert("RGB").save(converted_path, "JPEG", quality=90)
            return converted_path

    return image_path

Problem 2: Request entity too large (413)

Cause: The Base64-encoded image exceeds the provider's limit. Remember that Base64 inflates it by ~33%.

Solution: Use the resize_for_provider function from the limits section. For Anthropic (5 MB), a 4 MB image on disk already exceeds the limit after Base64 encoding (~5.3 MB).

Problem 3: Rate limit exceeded (429)

Cause: You sent too many requests in a short time.

Solution: Implement exponential backoff (previous section) and reduce requests_per_minute in batch processing. Check your tier in the provider's dashboard — higher tiers have higher limits.

Problem 4: Invalid API key or Authentication error

Cause: The API key isn't configured, it expired, or it was copied wrong (extra spaces, missing characters).

Solution: Verify that your .env has the key with no extra spaces, that it starts with sk- and is longer than 20 characters. Use load_dotenv() before creating the client. If the problem persists, regenerate the key from the provider's dashboard.

Problem 5: An image by URL returns an error but the URL works in the browser

Cause: The URL redirects (301/302), requires cookies, or has a user-agent restriction. The APIs don't follow every redirect.

Solution: Download the image first with httpx.get(url, follow_redirects=True) and send it as Base64. This solves 95% of URL problems.

Exercises

Exercise 1: Base64 encoder with validation (Easy)

Create a safe_encode(path) function that checks the file exists, validates the format (PNG, JPEG, GIF, WebP), and returns a dict with base64, mime, and size_mb.

import base64
from pathlib import Path
from PIL import Image

SUPPORTED = {
    ".jpg": "image/jpeg", ".jpeg": "image/jpeg",
    ".png": "image/png", ".gif": "image/gif", ".webp": "image/webp",
}

def safe_encode(path: str) -> dict:
    p = Path(path)
    if not p.exists():
        return {"success": False, "error": f"Not found: {path}"}

    ext = p.suffix.lower()
    if ext not in SUPPORTED:
        return {"success": False, "error": f"Unsupported format: {ext}"}

    try:
        with Image.open(path) as img:
            img.verify()
    except Exception as e:
        return {"success": False, "error": f"Invalid image: {e}"}

    with open(path, "rb") as f:
        raw = f.read()

    return {
        "success": True,
        "base64": base64.b64encode(raw).decode("utf-8"),
        "mime": SUPPORTED[ext],
        "size_mb": round(len(raw) / (1024 * 1024), 2),
    }

Exercise 2: Batch cost calculator (Easy)

Create a function that calculates the cost of processing N images with a given model.

import math

def calculate_batch_cost(n_images: int, model: str = "gpt-4o-mini") -> dict:
    pricing = {
        "gpt-4o": {"input": 2.50, "output": 10.00},
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    }
    img_tokens = 765  # 1024x1024, detail=high
    prompt_tokens, output_tokens = 50, 100
    p = pricing[model]

    total_input = (img_tokens + prompt_tokens) * n_images
    total_output = output_tokens * n_images
    cost = (total_input / 1e6) * p["input"] + (total_output / 1e6) * p["output"]

    return {"n": n_images, "model": model, "total_cost": round(cost, 4)}

for n in [10, 100, 1000]:
    for m in ["gpt-4o", "gpt-4o-mini"]:
        r = calculate_batch_cost(n, m)
        print(f"{n:>5} imgs × {m:<12} = ${r['total_cost']:.4f}")

Exercise 3: Base64 vs URL auto-selector (Medium)

Create a function that detects whether a source is a URL or a local path, and prepares the right format for the provider (OpenAI uses a direct URL, Anthropic always needs Base64).

import re, httpx

LIMITS_MB = {"openai": 20, "anthropic": 5, "google": 20}

def smart_image_input(source: str, provider: str = "openai") -> dict:
    is_url = bool(re.match(r"https?://", source))

    if is_url and provider in ("openai", "google"):
        return {"method": "url", "content": {"type": "image_url", "image_url": {"url": source}}}

    if is_url:
        resp = httpx.get(source, follow_redirects=True, timeout=30)
        image_bytes = resp.content
        mime = resp.headers.get("content-type", "image/jpeg")
    else:
        with open(source, "rb") as f:
            image_bytes = f.read()
        mime = MIME_TYPES.get(Path(source).suffix.lower(), "image/jpeg")

    encoded = base64.b64encode(image_bytes).decode("utf-8")

    if provider == "anthropic":
        content = {"type": "image", "source": {"type": "base64", "media_type": mime, "data": encoded}}
    else:
        content = {"type": "image_url", "image_url": {"url": f"data:{mime};base64,{encoded}"}}

    return {"method": "base64", "content": content}

Exercise 4: Rate limit handler with backoff (Medium)

Create a call_with_backoff function that retries up to 5 times with exponential backoff and jitter on rate limits.

import time, random
from openai import RateLimitError

def call_with_backoff(func, max_retries=5, base_delay=1.0):
    for attempt in range(max_retries):
        try:
            return func()
        except RateLimitError:
            if attempt == max_retries - 1:
                raise
            delay = min(base_delay * (2 ** attempt), 60)
            jitter = random.uniform(0, delay * 0.25)
            print(f"Rate limit, waiting {delay + jitter:.1f}s...")
            time.sleep(delay + jitter)

Summary

In this capsule you learned:

• Base64 encodes binary data as text so it can be sent in JSON. It inflates the size by ~33%. OpenAI uses data URIs (data:image/png;base64,...), Anthropic uses separate fields (media_type + data)
• URLs are lighter but only work with public images. Anthropic doesn't accept them — you always need Base64 for Claude
• The size limits vary: OpenAI 20 MB, Anthropic 5 MB, Google 20 MB. Resize before sending
• The image tokens in OpenAI depend on the resolution: 85 tokens on low, 85 + 170×tiles on high. A 1024×1024 image = 765 tokens
• The real costs are low for gpt-4o-mini ($0.0002/image) but they add up in large batches with gpt-4o ($0.003/image). Whisper: $0.006/minute
• Rate limits are handled with exponential backoff + jitter. The response headers (retry-after) tell you how long to wait
• API keys go in .env, never in the code. Validate at startup that they exist and have the right format
• MIME types must match the file's real format, not just the extension

Next capsule: Multimodal troubleshooting — common errors in production, systematic diagnosis, and multi-provider fallback strategies.

Additional Resources

1. OpenAI Vision Guide — Official GPT-4 Vision documentation with formats and limits
2. OpenAI Pricing — Up-to-date prices for all the models
3. Anthropic Vision Docs — Base64 format for Claude
4. Google Gemini Vision — Supported formats in Gemini
5. OpenAI Rate Limits — Limits by tier and handling strategies
6. Pillow (PIL) Documentation — Image manipulation and validation in Python
7. Base64 Encoding (MDN) — Technical reference for the encoding scheme
8. httpx Documentation — Modern HTTP client for Python (downloading images by URL)