2. GPT-4 Vision (OpenAI)

Description

GPT-4 Vision is the ability of OpenAI's gpt-4o and gpt-4o-mini models to take images alongside text in the chat completions API. It isn't a separate model — it's the same language model, but with a visual encoder that turns pixels into tokens the transformer can process next to the text.

In the previous capsule you saw the big picture of LLMs with vision and the three main providers. This capsule focuses exclusively on OpenAI: the exact structure of the request, how images are encoded, how to control quality and cost with the detail parameter, and practical patterns you'll reuse throughout the rest of the module.

If you're coming from Module 1 (capsule 06 — Formats and APIs), you already know Base64, data URIs and MIME types. Here you're going to apply all of that directly against OpenAI's API.

What you're going to build: Five working examples — description, OCR, classification, JSON extraction and using system prompts — plus a function to calculate costs before making the call.

Models with Vision

OpenAI offers two models with visual capability. The main difference is quality vs. cost.

When to choose each one

Use gpt-4o when you need precision: extracting text from scanned documents, analyzing technical diagrams, or generating detailed descriptions where an error has a cost. The quality difference is noticeable in tasks that require reasoning about fine details of the image.

Use gpt-4o-mini when you're processing volume: classifying thousands of images into categories, filtering content, or making a first pass before sending the hard cases to gpt-4o. At $0.15/1M input tokens, you can process ~6,600 simple images for a dollar.

A common strategy in production is a two-stage pipeline: gpt-4o-mini classifies and filters, gpt-4o analyzes in depth only what needs it.

Request Structure

The chat completions API accepts images inside a message's content array. Instead of sending a string as the content, you send an array of objects with type text or image_url.

Request with text and image

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What's in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"
                    }
                }
            ]
        }
    ],
    max_tokens=512
)

print(response.choices[0].message.content)

Request with only an image

You can omit the text block from the content array and send only image_url. The model generates a description by default, but the results are more predictable when you include an explicit prompt. Always prefer including text.

Key parameters

• model: "gpt-4o" or "gpt-4o-mini". Both support vision.
• messages: An array of messages. Each message can have content as a string (text only) or as an array (multimodal).
• max_tokens: The token limit on the response. OpenAI doesn't apply a low default for vision — it's worth setting it explicitly to control costs.
• temperature: Controls randomness. Use 0 for deterministic tasks (OCR, classification, extraction). Use 0.7–1.0 for creative descriptions.

Input: Base64 vs URL

There are two ways to send an image to the API: as a public URL or Base64-encoded inside a data URI.

Base64 with MIME detection

In Module 1 (capsule 06) you saw how to encode files to Base64. Here we apply it with automatic MIME-type detection, which is needed to build the data URI correctly.

import base64
import mimetypes
from pathlib import Path
from openai import OpenAI

client = OpenAI()


def load_image_as_data_uri(image_path: str) -> str:
    path = Path(image_path)
    if not path.exists():
        raise FileNotFoundError(f"Not found: {image_path}")

    mime_type, _ = mimetypes.guess_type(str(path))
    if mime_type is None:
        mime_type = "image/jpeg"

    with open(path, "rb") as f:
        encoded = base64.b64encode(f.read()).decode("utf-8")

    return f"data:{mime_type};base64,{encoded}"


def analyze_local_image(image_path: str, prompt: str) -> str:
    data_uri = load_image_as_data_uri(image_path)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", "image_url": {"url": data_uri}}
                ]
            }
        ],
        max_tokens=1024
    )
    return response.choices[0].message.content


result = analyze_local_image("product.jpg", "Describe this product for an online catalog.")
print(result)

Public URL

For images already accessible on the web, pass the URL directly without encoding:

content = [
    {"type": "text", "text": "What does this image show?"},
    {
        "type": "image_url",
        "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png"}
    }
]

The URL must be public. URLs with authentication, expired temporary tokens or private IPs fail silently.

When to use each one

The practical rule: Base64 for local files and sensitive data, URL for images that are already public on the web.

The detail Parameter: low vs high vs auto

The detail parameter controls how much visual processing the model applies. It affects the quality of the analysis and the cost in tokens.

The three levels

• low: The image is resized to 512×512. The model receives a compressed version. It costs a fixed 85 tokens regardless of the original size.
• high: The image is processed at its original resolution (up to 2048×2048). It's split into 512×512 tiles and each tile costs 170 tokens, plus 85 base tokens. Total: 85 + 170 × num_tiles.
• auto (default): OpenAI decides between low and high based on the image size.

Cost per level

How to configure it

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What text appears in this document?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/document.png",
                        "detail": "high"
                    }
                }
            ]
        }
    ],
    max_tokens=2048,
    temperature=0
)

print(response.choices[0].message.content)

When to use each level

• low: Binary classification (yes/no), general category detection, checking whether an image contains text. Any task where fine details don't matter.
• high: OCR, reading documents, analyzing diagrams, defect inspection, any task where you need to read small text or spot details.
• auto: When you don't control the image type and prefer to let OpenAI decide. Useful in generic applications.

Example 1: Image Description

from openai import OpenAI

client = OpenAI()


def describe_image(image_url: str, max_words: int = 100) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": (
                            f"Describe this image in English, in at most {max_words} words. "
                            "Include: main subject, setting, dominant colors and the general mood."
                        )
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url, "detail": "low"}
                    }
                ]
            }
        ],
        max_tokens=300,
        temperature=0.7
    )
    return response.choices[0].message.content


description = describe_image(
    "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"
)
print(description)

The prompt asks for a language, a maximum length and description axes — that produces consistent results. detail="low" because we don't need high resolution for a general description, and temperature=0.7 for natural text.

Example 2: OCR — Text Extraction

from openai import OpenAI

client = OpenAI()


def extract_text_from_image(image_url: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": (
                            "Extract ALL the visible text in this image. "
                            "Preserve the original structure: if there's a table, "
                            "represent it as a markdown table. If there are paragraphs, "
                            "keep the line breaks. If there are headings, "
                            "use markdown format with #. "
                            "Don't add interpretations, only the text exactly as it appears."
                        )
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url, "detail": "high"}
                    }
                ]
            }
        ],
        max_tokens=4096,
        temperature=0
    )
    return response.choices[0].message.content


text = extract_text_from_image("https://example.com/invoice-scan.png")
print(text)

detail="high" is mandatory for OCR — with low you lose small characters. temperature=0 for exact reproduction. max_tokens=4096 because a one-page document can generate 500–1500 tokens. The prompt asks for markdown format for tables explicitly; without this, the model linearizes tables and loses the column structure.

Example 3: Classification with Fixed Categories

from openai import OpenAI

client = OpenAI()

CATEGORIES = ["product", "person", "landscape", "document", "food", "other"]


def classify_image(image_url: str) -> str:
    categories_str = ", ".join(CATEGORIES)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": (
                            f"Classify this image into exactly ONE of these categories: {categories_str}. "
                            "Reply ONLY with the category name, no punctuation or explanation."
                        )
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url, "detail": "low"}
                    }
                ]
            }
        ],
        max_tokens=20,
        temperature=0
    )

    result = response.choices[0].message.content.strip().lower()

    if result not in CATEGORIES:
        return "other"

    return result


category = classify_image("https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg")
print(f"Category: {category}")

We use gpt-4o-mini because classification is a simple task. detail="low" and max_tokens=20 because the answer is a single word. The final validation ensures we always return a valid category — unexpected responses fall back to "other".

Example 4: Structured Extraction (JSON)

import json
from openai import OpenAI

client = OpenAI()


def extract_product_info(image_url: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": (
                            "Analyze this product image and return a JSON with this exact structure:\n"
                            "{\n"
                            '  "name": "product name",\n'
                            '  "category": "electronics|clothing|food|home|other",\n'
                            '  "main_color": "dominant color",\n'
                            '  "visible_text": ["list", "of", "texts"],\n'
                            '  "condition": "new|used|undetermined"\n'
                            "}\n"
                            "Reply ONLY with valid JSON."
                        )
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url, "detail": "high"}
                    }
                ]
            }
        ],
        response_format={"type": "json_object"},
        max_tokens=500,
        temperature=0
    )

    raw = response.choices[0].message.content

    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        data = {"error": "Invalid JSON", "raw_response": raw}

    return data


info = extract_product_info("https://example.com/product-photo.jpg")
print(json.dumps(info, indent=2, ensure_ascii=False))

response_format={"type": "json_object"} forces valid JSON — without it, the model sometimes wraps the JSON in markdown. The try/except is a safety net: with response_format on it rarely fails, but in production always handle the case.

System Prompt for Vision

The system message (role: "system") establishes context and personality before the model sees the image. It's especially useful for domain-specific tasks.

from openai import OpenAI

client = OpenAI()


def ecommerce_product_analysis(image_url: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are an expert in product photography for e-commerce. "
                    "When you analyze a product image, you evaluate: lighting quality, "
                    "composition, background, angle, and whether the image meets marketplace standards "
                    "(Amazon, MercadoLibre). You reply in English with actionable recommendations."
                )
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Evaluate this product photo and give me recommendations to improve it."
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url, "detail": "high"}
                    }
                ]
            }
        ],
        max_tokens=1024,
        temperature=0.3
    )
    return response.choices[0].message.content


feedback = ecommerce_product_analysis("https://example.com/my-product.jpg")
print(feedback)

Other useful system prompts: a radiologist for medical images ("You describe findings using BIRADS terminology..."), an alt-text generator for web accessibility ("You generate alternative text following WCAG 2.1..."), an industrial quality inspector ("You identify visual defects in manufactured parts...").

The system prompt only consumes text tokens, not image tokens. It's the cheapest way to improve quality without changing models.

Tokens and Costs

Every response includes usage with the exact token count. Use it to monitor real consumption:

usage = response.usage
print(f"Input tokens:  {usage.prompt_tokens}")
print(f"Output tokens: {usage.completion_tokens}")
print(f"Total tokens:  {usage.total_tokens}")

Calculate the cost

def calculate_cost(usage, model: str = "gpt-4o") -> dict:
    pricing = {
        "gpt-4o": {"input": 2.50, "output": 10.00},
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    }

    rates = pricing[model]
    input_cost = (usage.prompt_tokens / 1_000_000) * rates["input"]
    output_cost = (usage.completion_tokens / 1_000_000) * rates["output"]

    return {
        "input_cost": round(input_cost, 6),
        "output_cost": round(output_cost, 6),
        "total_cost": round(input_cost + output_cost, 6),
        "model": model
    }


cost = calculate_cost(response.usage, "gpt-4o")
print(f"Total cost: ${cost['total_cost']:.6f}")

Typical costs by scenario

Troubleshooting

1. Invalid image or the image is rejected

Cause: An unsupported format or corrupt Base64. OpenAI accepts PNG, JPEG, GIF and WebP. Formats like BMP, TIFF or SVG are rejected.

SUPPORTED_FORMATS = {".png", ".jpg", ".jpeg", ".gif", ".webp"}

def validate_image_format(path: str) -> bool:
    ext = Path(path).suffix.lower()
    if ext not in SUPPORTED_FORMATS:
        raise ValueError(f"Format {ext} not supported. Use: {SUPPORTED_FORMATS}")
    return True

2. Request too large (413)

Cause: The Base64-encoded image exceeds the payload limit. OpenAI accepts images up to 20 MB, but the request's total payload has practical limits.

from PIL import Image

def resize_if_needed(image_path: str, max_dimension: int = 2048) -> str:
    img = Image.open(image_path)
    if max(img.size) > max_dimension:
        img.thumbnail((max_dimension, max_dimension))
        resized_path = f"resized_{Path(image_path).name}"
        img.save(resized_path)
        return resized_path
    return image_path

3. Empty or generic response

Cause: A prompt that's too vague or max_tokens that's too low. The model truncates the response and it can end up incomplete.

Solution: Be specific in the prompt (what you want, in what format, what length) and assign enough max_tokens. For document OCR, use at least 2048.

4. Rate limit (429)

Cause: Too many requests per minute. Images consume more capacity than pure text.

import time

def analyze_with_retry(func, *args, max_retries: int = 3):
    for attempt in range(max_retries):
        try:
            return func(*args)
        except Exception as e:
            if "rate_limit" in str(e).lower() and attempt < max_retries - 1:
                time.sleep(2 ** attempt)
            else:
                raise

5. The model "hallucinates" text that isn't in the image

Cause: The model infers text based on visual context instead of reading it. It's more frequent with detail="low".

Solution: Use detail="high", temperature=0, and add to the prompt: "Extract ONLY the text you can read with certainty. If you can't read a fragment, mark it as [illegible].".

Exercises

Exercise 1 (Easy): Image analysis with JSON

Write an analyze_image_json function that takes an image URL and returns a dictionary with two fields: description (a description in English, max 50 words) and detected_objects (a list of objects detected in the image).

import json
from openai import OpenAI

client = OpenAI()


def analyze_image_json(image_url: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": (
                            "Analyze this image and reply in JSON with this structure:\n"
                            "{\n"
                            '  "description": "description in English, max 50 words",\n'
                            '  "detected_objects": ["object1", "object2", "..."]\n'
                            "}\n"
                            "Valid JSON only."
                        )
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url, "detail": "low"}
                    }
                ]
            }
        ],
        response_format={"type": "json_object"},
        max_tokens=300,
        temperature=0
    )

    return json.loads(response.choices[0].message.content)


result = analyze_image_json(
    "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"
)
print(f"Description: {result['description']}")
print(f"Objects: {result['detected_objects']}")

Exercise 2 (Medium): Compare two images

Write a compare_images function that takes two image URLs and returns a dictionary with similarity_score (an integer from 1 to 10) and justification (text explaining the score). Both images go in the same request.

import json
from openai import OpenAI

client = OpenAI()


def compare_images(image_url_1: str, image_url_2: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": (
                            "Compare these two images. Reply in JSON:\n"
                            "{\n"
                            '  "similarity_score": <an integer from 1 to 10, where 1 is completely different and 10 is identical>,\n'
                            '  "justification": "an explanation in English of why you assigned that score"\n'
                            "}\n"
                            "Valid JSON only."
                        )
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url_1, "detail": "low"}
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url_2, "detail": "low"}
                    }
                ]
            }
        ],
        response_format={"type": "json_object"},
        max_tokens=300,
        temperature=0
    )

    return json.loads(response.choices[0].message.content)


result = compare_images(
    "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4d/Cat_November_2010-1a.jpg/1200px-Cat_November_2010-1a.jpg"
)
print(f"Similarity: {result['similarity_score']}/10")
print(f"Justification: {result['justification']}")

Exercise 3 (Medium): Pre-call cost calculator

Write an estimate_vision_cost function that takes the path of a local image, the model ("gpt-4o" or "gpt-4o-mini") and the detail level ("low" or "high"). The function must calculate the estimated image tokens and the approximate input cost without making the API call. For high, calculate the tiles based on the image's real dimensions.

import math
from PIL import Image


def estimate_vision_cost(
    image_path: str,
    model: str = "gpt-4o",
    detail: str = "low"
) -> dict:
    pricing = {
        "gpt-4o": {"input": 2.50},
        "gpt-4o-mini": {"input": 0.15},
    }

    if detail == "low":
        image_tokens = 85
    else:
        img = Image.open(image_path)
        width, height = img.size

        max_dim = 2048
        if max(width, height) > max_dim:
            scale = max_dim / max(width, height)
            width = int(width * scale)
            height = int(height * scale)

        min_side = 768
        if min(width, height) > min_side:
            scale = min_side / min(width, height)
            width = int(width * scale)
            height = int(height * scale)

        tiles_x = math.ceil(width / 512)
        tiles_y = math.ceil(height / 512)
        num_tiles = tiles_x * tiles_y
        image_tokens = 85 + 170 * num_tiles

    rate = pricing[model]["input"]
    estimated_cost = (image_tokens / 1_000_000) * rate

    return {
        "image_tokens": image_tokens,
        "estimated_input_cost": round(estimated_cost, 8),
        "model": model,
        "detail": detail
    }


estimate = estimate_vision_cost("large_photo.jpg", "gpt-4o", "high")
print(f"Image tokens: {estimate['image_tokens']}")
print(f"Estimated input cost: ${estimate['estimated_input_cost']:.8f}")

Summary

• The GPT-4 Vision API uses the content array with text and image_url objects.
• gpt-4o for maximum quality, gpt-4o-mini for volume and low cost.
• Images are sent as Base64 (data URI) or a public URL.
• The detail parameter controls resolution and cost: low (85 tokens), high (85 + 170×tiles).
• temperature=0 and response_format={"type": "json_object"} for deterministic, structured tasks.
• System prompts specialize the model for specific domains at no extra cost in image tokens.
• Always read response.usage to monitor real consumption.

Additional Resources

1. OpenAI Vision Guide
2. API Reference: Chat Completions
3. OpenAI Pricing
4. OpenAI Cookbook: Vision