3. Claude 3 Vision (Anthropic)

Description

Claude 3 is Anthropic's family of multimodal models that competes directly with OpenAI's GPT-4o. It supports images as input alongside text, which lets you carry out the same tasks we saw in the previous capsule: description, OCR, classification and structured extraction.

Why does Anthropic matter as an alternative? Three reasons: a 200K-token context window (vs GPT-4o's 128K), superior performance in OCR and document analysis according to independent benchmarks, and an API with structural differences that every AI Engineer must master in order not to depend on a single provider.

This capsule focuses on the differences from OpenAI. If something works the same, we won't repeat it — check capsule 02 for the fundamentals.

Models with Vision

Anthropic offers three models with vision capabilities, each optimized for a different balance of quality, speed and cost:

Recommendation: Use claude-3-5-sonnet-latest as the default. It has the best quality-price ratio and its vision performance is comparable to Opus on most tasks. Reserve Opus for analysis that requires complex multi-step reasoning. Use Haiku when you need speed and the analysis is simple (binary classification, content detection).

Compared with OpenAI:

Request Structure — Differences from OpenAI

This is the most important difference between the two APIs. Although the concept is the same (sending an array of content blocks), the JSON structure is completely different.

In capsule 02 we saw that OpenAI uses {"type": "image_url", "image_url": {"url": "data:mime;base64,..."}}. Anthropic has a different structure:

Anthropic

import anthropic
import base64

client = anthropic.Anthropic()

with open("image.jpg", "rb") as f:
    image_data = base64.b64encode(f.read()).decode()

response = client.messages.create(
    model="claude-3-5-sonnet-latest",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/jpeg",
                    "data": image_data
                }
            },
            {"type": "text", "text": "Describe this image."}
        ]
    }]
)

result = response.content[0].text

Structure comparison table

The key point: in OpenAI the Base64 goes inside a data URI (data:image/jpeg;base64,XXXX), while in Anthropic the Base64 goes in a separate data field and the media type in its own field. This makes Anthropic more explicit but also stricter — an incorrect media type will produce an error.

Base64 Only — No URLs

This is Anthropic's most impactful limitation vs OpenAI. While OpenAI accepts both Base64 and public URLs, Anthropic only accepts Base64. You can't pass a URL directly.

This means that if your image is on a remote server, you have to download it first and convert it to Base64 before sending it to Claude.

import anthropic
import base64
import httpx
from pathlib import Path

client = anthropic.Anthropic()

def download_and_encode(url: str) -> tuple[str, str]:
    """Download an image from a URL and return (base64_data, media_type)."""
    response = httpx.get(url, follow_redirects=True, timeout=30)
    response.raise_for_status()

    content_type = response.headers.get("content-type", "image/jpeg")
    media_type = content_type.split(";")[0].strip()

    allowed = {"image/jpeg", "image/png", "image/gif", "image/webp"}
    if media_type not in allowed:
        raise ValueError(f"Unsupported format: {media_type}")

    data = response.content
    if len(data) > 5 * 1024 * 1024:
        raise ValueError(f"Image exceeds 5MB: {len(data) / 1024 / 1024:.1f}MB")

    return base64.b64encode(data).decode(), media_type

The 5MB limit

Anthropic imposes a limit of 5MB per image in Base64. In practice, since Base64 increases the size by ~33%, this means your original image must not exceed ~3.75MB. If you need to send larger images, resize with PIL before encoding.

Media Types

Anthropic supports four image formats:

Unlike OpenAI where the media type is embedded in the data URI and can be inferred, in Anthropic it's a separate mandatory field. Sending an incorrect media type produces an error.

Auto-detection function:

from pathlib import Path

MEDIA_TYPES = {
    ".jpg": "image/jpeg",
    ".jpeg": "image/jpeg",
    ".png": "image/png",
    ".gif": "image/gif",
    ".webp": "image/webp",
}

def detect_media_type(path: str) -> str:
    ext = Path(path).suffix.lower()
    if ext not in MEDIA_TYPES:
        raise ValueError(f"Unsupported extension: {ext}. Use: {list(MEDIA_TYPES.keys())}")
    return MEDIA_TYPES[ext]

Example 1: Image Description

A complete example that loads a local image and gets a description:

import anthropic
import base64
from pathlib import Path

client = anthropic.Anthropic()

def describe_image(image_path: str, language: str = "English") -> str:
    ext = Path(image_path).suffix.lower()
    media_types = {".jpg": "image/jpeg", ".jpeg": "image/jpeg",
                   ".png": "image/png", ".gif": "image/gif", ".webp": "image/webp"}
    media_type = media_types.get(ext, "image/jpeg")

    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    response = client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=512,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {"type": "base64", "media_type": media_type, "data": image_data}
                },
                {
                    "type": "text",
                    "text": f"Describe this image in detail in {language}. "
                            f"Include: main elements, colors, composition and context."
                }
            ]
        }]
    )
    return response.content[0].text

description = describe_image("product_photo.jpg")
print(description)

Practical difference from OpenAI: the prompt and the structure are functionally equivalent, but notice how here we have to specify media_type explicitly and max_tokens is mandatory (in OpenAI it's optional).

Example 2: OCR with Claude

Claude particularly stands out in OCR — in independent benchmarks it beats GPT-4o at extracting text from scanned documents, receipts and forms. That makes it the preferred option for document-processing pipelines.

import anthropic
import base64

client = anthropic.Anthropic()

def ocr_claude(image_path: str, structured: bool = False) -> str:
    with open(image_path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()

    if structured:
        prompt = (
            "Extract all the visible text in this image. "
            "Organize the result respecting the visual structure: "
            "headings, paragraphs, lists, tables. "
            "Use Markdown format to represent the structure."
        )
    else:
        prompt = (
            "Extract all the visible text in this image, "
            "line by line, exactly as it appears."
        )

    response = client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=4096,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {"type": "base64", "media_type": "image/png", "data": b64}
                },
                {"type": "text", "text": prompt}
            ]
        }]
    )
    return response.content[0].text

text_raw = ocr_claude("receipt.png")
text_structured = ocr_claude("receipt.png", structured=True)
print(text_structured)

For OCR with Claude, use a high max_tokens (4096+) because scanned documents can generate a lot of text. With OpenAI the max_tokens default is 4096, but in Anthropic you have to specify it explicitly.

Example 3: Document Analysis

Claude excels at analyzing complex documents: contracts, invoices, reports. Its 200K window lets you process long documents with additional context.

import anthropic
import base64

client = anthropic.Anthropic()

def analyze_document(image_path: str, document_type: str, fields: list[str]) -> str:
    with open(image_path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()

    fields_list = "\n".join(f"- {field}" for field in fields)

    response = client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=2048,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {"type": "base64", "media_type": "image/png", "data": b64}
                },
                {
                    "type": "text",
                    "text": f"This is a document of type: {document_type}.\n\n"
                            f"Extract the following fields:\n{fields_list}\n\n"
                            f"For each field, state the value found. "
                            f"If a field isn't visible, state 'Not found'."
                }
            ]
        }]
    )
    return response.content[0].text

result = analyze_document(
    "invoice.png",
    document_type="commercial invoice",
    fields=["Invoice number", "Date", "Vendor", "Total", "Tax", "Payment method"]
)
print(result)

Claude supports multiple images in the same message — just add several {"type": "image", ...} blocks to the content array before the text block.

Example 4: JSON Extraction

In OpenAI you can use response_format={"type": "json_object"} to guarantee JSON output. Anthropic doesn't have this parameter. Instead, you have to use prompt engineering to get valid JSON.

import anthropic
import base64
import json

client = anthropic.Anthropic()

def extract_json_from_image(image_path: str, schema_description: str) -> dict:
    with open(image_path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()

    response = client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=2048,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {"type": "base64", "media_type": "image/jpeg", "data": b64}
                },
                {
                    "type": "text",
                    "text": f"Analyze this image and extract the information as JSON.\n\n"
                            f"Expected schema:\n{schema_description}\n\n"
                            f"Reply ONLY with valid JSON, no additional text, "
                            f"no markdown code blocks, no explanations."
                }
            ]
        }]
    )

    raw = response.content[0].text.strip()
    if raw.startswith("```"):
        raw = raw.split("\n", 1)[1].rsplit("```", 1)[0].strip()

    return json.loads(raw)

product_data = extract_json_from_image(
    "product.jpg",
    '{"name": "string", "price": number, "category": "string", "color": "string"}'
)
print(json.dumps(product_data, indent=2, ensure_ascii=False))

Comparison of JSON strategies:

The cleanup block (if raw.startswith("```")) is necessary because Claude sometimes wraps the JSON in markdown code blocks, even when you ask it not to.

System Prompt in Anthropic

Another important structural difference: in OpenAI the system prompt is a message with "role": "system" inside the messages array. In Anthropic, the system prompt is a separate top-level parameter.

import anthropic
import base64

client = anthropic.Anthropic()

with open("image.jpg", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = client.messages.create(
    model="claude-3-5-sonnet-latest",
    max_tokens=1024,
    system="You are an expert in visual product analysis. "
           "You always reply in English with a structured format.",
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {"type": "base64", "media_type": "image/jpeg", "data": b64}
            },
            {"type": "text", "text": "Analyze this product."}
        ]
    }]
)

If you include {"role": "system", ...} inside messages in Anthropic, you'll get an error. It's a frequent mistake when migrating code from OpenAI to Anthropic.

Tokens and Costs

Anthropic includes usage information in every response. It's essential for monitoring costs in production.

import anthropic
import base64

client = anthropic.Anthropic()

with open("image.jpg", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = client.messages.create(
    model="claude-3-5-sonnet-latest",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {"type": "base64", "media_type": "image/jpeg", "data": b64}
            },
            {"type": "text", "text": "Describe this image."}
        ]
    }]
)

input_tokens = response.usage.input_tokens
output_tokens = response.usage.output_tokens

COST_PER_M_INPUT = 3.00
COST_PER_M_OUTPUT = 15.00

cost_input = (input_tokens / 1_000_000) * COST_PER_M_INPUT
cost_output = (output_tokens / 1_000_000) * COST_PER_M_OUTPUT
total_cost = cost_input + cost_output

print(f"Input:  {input_tokens:,} tokens (${cost_input:.4f})")
print(f"Output: {output_tokens:,} tokens (${cost_output:.4f})")
print(f"Total:  ${total_cost:.4f}")

Images consume input tokens. A typical 1024x1024 image consumes approximately 1,600 tokens. Larger images consume more. Anthropic doesn't document the exact formula, but you can use response.usage.input_tokens to measure the real consumption.

Comparison of the usage object:

Troubleshooting

1. Error: Image exceeds 5MB

anthropic.BadRequestError: Image exceeds maximum size of 5MB

Cause: The Base64-encoded image exceeds 5MB.

Solution: Resize or compress the image with PIL before encoding. Remember the ~33% margin that Base64 adds (see "The 5MB limit" in the "Base64 Only" section).

2. Error: URL not supported

anthropic.BadRequestError: Invalid image source type

Cause: You're trying to pass a direct URL like you did in OpenAI.

Solution: Download the image first and send it as Base64. Use download_and_encode() from the previous section.

3. Error: Incorrect media type

anthropic.BadRequestError: Invalid media type

Cause: The media_type field doesn't match the image's real format, or you used an unsupported format.

Solution: Check that the file extension corresponds to the media_type. Use the detect_media_type() function to automate it.

4. Error: System prompt as a message

anthropic.BadRequestError: Messages must not contain "system" role

Cause: You migrated code from OpenAI without changing the system prompt from a message to a top-level parameter.

Solution: Move the content of {"role": "system", "content": "..."} to the system= parameter of messages.create().

5. Error: max_tokens missing

anthropic.BadRequestError: max_tokens is required

Cause: In OpenAI max_tokens is optional (it has a default). In Anthropic it's mandatory.

Solution: Always include max_tokens in the call. Common values: 512 for short descriptions, 1024 for analysis, 4096 for extensive OCR.

Exercises

Exercise 1 (Easy): Universal image send to Claude

Create a send_image_to_claude(image_path, prompt) function that:

• Automatically detects the media_type from the file extension
• Validates that the file doesn't exceed 5MB
• Sends the image to Claude and returns the response
• Raises ValueError if the format isn't supported or the file is too large

import anthropic
import base64
from pathlib import Path

client = anthropic.Anthropic()

MEDIA_TYPES = {
    ".jpg": "image/jpeg", ".jpeg": "image/jpeg",
    ".png": "image/png", ".gif": "image/gif", ".webp": "image/webp",
}

MAX_SIZE_BYTES = 5 * 1024 * 1024

def send_image_to_claude(image_path: str, prompt: str) -> str:
    path = Path(image_path)

    ext = path.suffix.lower()
    if ext not in MEDIA_TYPES:
        raise ValueError(f"Unsupported format: {ext}")
    media_type = MEDIA_TYPES[ext]

    file_size = path.stat().st_size
    if file_size > MAX_SIZE_BYTES:
        raise ValueError(f"File exceeds 5MB: {file_size / 1024 / 1024:.1f}MB")

    with open(path, "rb") as f:
        b64_data = base64.b64encode(f.read()).decode()

    response = client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {"type": "base64", "media_type": media_type, "data": b64_data}
                },
                {"type": "text", "text": prompt}
            ]
        }]
    )
    return response.content[0].text

try:
    result = send_image_to_claude("photo.jpg", "What do you see in this image?")
    print(result)
except ValueError as e:
    print(f"Validation error: {e}")

Exercise 2 (Medium): Download a URL and send it to Claude

Create a claude_from_url(url, prompt) function that:

• Downloads the image from a URL using httpx
• Detects the media_type from the Content-Type header
• Validates that it's a supported format and doesn't exceed 5MB
• Sends it to Claude and returns the response
• Handles network errors (timeout, 404) with clear messages

import anthropic
import base64
import httpx

client = anthropic.Anthropic()

ALLOWED_TYPES = {"image/jpeg", "image/png", "image/gif", "image/webp"}

def claude_from_url(url: str, prompt: str, timeout: int = 30) -> str:
    try:
        http_response = httpx.get(url, follow_redirects=True, timeout=timeout)
        http_response.raise_for_status()
    except httpx.TimeoutException:
        raise ConnectionError(f"Timeout downloading the image: {url}")
    except httpx.HTTPStatusError as e:
        raise ConnectionError(f"HTTP error {e.response.status_code}: {url}")

    content_type = http_response.headers.get("content-type", "")
    media_type = content_type.split(";")[0].strip()

    if media_type not in ALLOWED_TYPES:
        raise ValueError(f"Unsupported type: {media_type}")

    image_bytes = http_response.content
    if len(image_bytes) > 5 * 1024 * 1024:
        raise ValueError(f"Image exceeds 5MB: {len(image_bytes) / 1024 / 1024:.1f}MB")

    b64_data = base64.b64encode(image_bytes).decode()

    response = client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {"type": "base64", "media_type": media_type, "data": b64_data}
                },
                {"type": "text", "text": prompt}
            ]
        }]
    )
    return response.content[0].text

try:
    result = claude_from_url(
        "https://example.com/product.jpg",
        "Describe this product in English."
    )
    print(result)
except (ConnectionError, ValueError) as e:
    print(f"Error: {e}")

Exercise 3 (Medium): OpenAI vs Claude comparator

Create a compare_vision(image_path, prompt) function that:

• Sends the same image and prompt to GPT-4o AND to Claude 3.5 Sonnet
• Returns a dictionary with both responses and the tokens each one used
• Calculates the cost of each call

You'll need openai and anthropic installed.

from openai import OpenAI
import anthropic
import base64
from pathlib import Path

openai_client = OpenAI()
anthropic_client = anthropic.Anthropic()

MEDIA_TYPES = {
    ".jpg": "image/jpeg", ".jpeg": "image/jpeg",
    ".png": "image/png", ".gif": "image/gif", ".webp": "image/webp",
}

def compare_vision(image_path: str, prompt: str) -> dict:
    ext = Path(image_path).suffix.lower()
    media_type = MEDIA_TYPES.get(ext, "image/jpeg")

    with open(image_path, "rb") as f:
        b64_data = base64.b64encode(f.read()).decode()

    openai_response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": prompt},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:{media_type};base64,{b64_data}"}
                }
            ]
        }],
        max_tokens=1024
    )

    claude_response = anthropic_client.messages.create(
        model="claude-3-5-sonnet-latest",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {"type": "base64", "media_type": media_type, "data": b64_data}
                },
                {"type": "text", "text": prompt}
            ]
        }]
    )

    return {
        "openai": {
            "response": openai_response.choices[0].message.content,
            "input_tokens": openai_response.usage.prompt_tokens,
            "output_tokens": openai_response.usage.completion_tokens,
            "cost": (openai_response.usage.prompt_tokens / 1e6 * 2.50 +
                     openai_response.usage.completion_tokens / 1e6 * 10.00),
        },
        "anthropic": {
            "response": claude_response.content[0].text,
            "input_tokens": claude_response.usage.input_tokens,
            "output_tokens": claude_response.usage.output_tokens,
            "cost": (claude_response.usage.input_tokens / 1e6 * 3.00 +
                     claude_response.usage.output_tokens / 1e6 * 15.00),
        },
    }

results = compare_vision("photo.jpg", "Describe this image in 2 sentences.")
for provider, data in results.items():
    print(f"\n{'='*40}")
    print(f"{provider.upper()}")
    print(f"Response: {data['response']}")
    print(f"Tokens: {data['input_tokens']} in / {data['output_tokens']} out")
    print(f"Cost: ${data['cost']:.4f}")

Summary

• Anthropic uses {"type": "image", "source": {"type": "base64", "media_type": "...", "data": "..."}} — different from OpenAI's image_url
• Models: Sonnet (the recommended default), Opus (maximum quality), Haiku (economical)
• Base64 only — it doesn't accept direct URLs (you need to download first)
• media_type is explicit and mandatory
• max_tokens is mandatory (in OpenAI it's optional)
• The system prompt goes as a top-level parameter, not as a message
• It has no native response_format for JSON — use prompt engineering
• 5MB limit per image
• The response is in response.content[0].text (vs response.choices[0].message.content)

Additional resources

1. Anthropic Vision — Official documentation
2. Anthropic API Reference — Messages
3. Claude 3 Model Card
4. Anthropic Python SDK