2. The Vision Modality

Description

The vision modality lets LLMs process images as input. Instead of text only, you can send a photo, a diagram, a scanned document, and the model "sees" the content in order to describe it, extract text (OCR), classify it or answer questions about it. In this capsule you'll learn the capabilities of models with vision, their limitations, and the most common usage patterns with runnable code.

Why it matters: Vision is the most mature and widespread multimodal modality. GPT-4 Vision, Claude 3 and Gemini all support it with high quality. Mastering vision is the first step toward document analysis, image Q&A, and multimodal RAG — topics you'll cover in modules 2, 3 and 6.

Connection with the module: In capsule 01 you saw the modalities as a concept. Here you work with the first of them: vision. What you learn here goes deeper in Module 2 (Vision + LLMs) and gets integrated into the multimodal Classifier in capsule 08.

How Vision Works in LLMs

The concept

An LLM with vision receives two types of input in a single request:

1. Text: Your prompt, instruction or question
2. Image: A photo, screenshot, document, diagram

The model processes both inputs and generates a text response. It does not generate images (that's DALL-E/Stable Diffusion, module 4). Vision is image → text.

Input:  [image of a cat] + "What animal is this?"
Output: "It's a domestic cat, orange in color, probably a tabby."

Image input formats

Models accept images in two ways:

# Format 1: Base64 (local image)
import base64

with open("photo.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

content = {
    "type": "image_url",
    "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}
}

# Format 2: URL (image on the web)
content = {
    "type": "image_url",
    "image_url": {"url": "https://example.com/photo.jpg"}
}

Capabilities of Models with Vision

1. Image description

The model can describe what it sees: objects, people, scenes, visible text, colors, composition.

from openai import OpenAI
import base64

client = OpenAI()

def describe_image(image_path: str, detail: str = "auto") -> str:
    """Describe an image using GPT-4o Vision."""
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe this image in 2-3 sentences. Be specific about what you see."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data}",
                        "detail": detail  # "low", "high", "auto"
                    }
                }
            ]
        }],
        max_tokens=300
    )
    return response.choices[0].message.content

# Usage
result = describe_image("office_photo.jpg")
print(result)
# Expected output: "The image shows a modern office with white desks,
# dual monitors and decorative plants. Three people are working with laptops
# next to a window overlooking the city."

The detail parameter controls the resolution of the analysis:

• "low": Faster and cheaper, enough for a general description
• "high": More detailed, better for OCR and fine-grained analysis
• "auto": The model decides based on the image

2. OCR (text extraction)

Models with vision can "read" text in images: invoices, screenshots, scanned documents, traffic signs, menus.

def extract_text_from_image(image_path: str) -> str:
    """Extract all visible text from an image."""
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        "Extract ALL the visible text in this image. "
                        "Preserve the original structure (paragraphs, lists, tables) "
                        "if possible. If there is text you cannot read with "
                        "certainty, mark it with [illegible]."
                    )
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data}",
                        "detail": "high"  # high for better OCR
                    }
                }
            ]
        }],
        max_tokens=1500
    )
    return response.choices[0].message.content

# Usage
text = extract_text_from_image("scanned_invoice.jpg")
print(text)
# Expected output:
# INVOICE #12345
# Date: 2024-03-15
# Customer: ABC Company
# ─────────────────────
# Consulting services       $2,500.00
# Software license          $1,200.00
# ─────────────────────
# Total:                    $3,700.00

3. Image classification

Classifying images into predefined categories is one of the most direct uses.

def classify_image(
    image_path: str,
    categories: list[str],
    model: str = "gpt-4o-mini"
) -> dict:
    """Classify an image into one of the given categories."""
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    cats = ", ".join(categories)
    response = client.chat.completions.create(
        model=model,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        f"Classify this image into ONE of these categories: {cats}. "
                        f"Reply with JSON: "
                        f'{{"category": "...", "confidence": "high/medium/low", '
                        f'"reason": "..."}}'
                    )
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}
                }
            ]
        }],
        max_tokens=150,
        temperature=0
    )

    import json
    return json.loads(response.choices[0].message.content)

# Usage
categories = ["invoice", "receipt", "contract", "id_document", "other"]
result = classify_image("document.jpg", categories)
print(result)
# Expected output:
# {"category": "invoice", "confidence": "high", "reason": "The document shows
#  an invoice layout with a number, date, items and a total"}

4. Structured extraction

Extracting specific data from an image and returning it in a structured format (JSON).

def extract_structured_data(image_path: str, schema: dict) -> dict:
    """Extract structured data from an image according to a schema."""
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    schema_str = "\n".join(
        f"- {k}: {v}" for k, v in schema.items()
    )

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        f"Extract the following fields from this image:\n{schema_str}\n\n"
                        f"Reply ONLY with valid JSON. "
                        f"If a field is not visible, use null."
                    )
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data}",
                        "detail": "high"
                    }
                }
            ]
        }],
        max_tokens=500,
        temperature=0
    )

    import json
    return json.loads(response.choices[0].message.content)

# Usage: extract data from a business card
schema = {
    "name": "the person's full name",
    "company": "the company name",
    "job_title": "job title or position",
    "email": "email address",
    "phone": "phone number"
}
data = extract_structured_data("business_card.jpg", schema)
print(data)
# Expected output:
# {"name": "María López", "company": "TechCorp",
#  "job_title": "CTO", "email": "maria@techcorp.com",
#  "phone": "+52 55 1234 5678"}

Limitations of Vision

How much does an image cost?

OpenAI charges by image tokens. The cost depends on the resolution:

Tip: For classification and general description, detail: "low" is enough and 10x cheaper. Use detail: "high" only for OCR or fine-detail analysis.

Comparison: Vision API vs Traditional OCR

When to use each one?

• Vision API: Complex documents, tables, forms, you need interpretation (not just raw text), low-medium volume
• Tesseract OCR: High volume of simple documents, zero cost, total privacy, you only need raw text

Common Usage Patterns

Pattern 1: Describe + act

You ask the model to describe the image AND take an action based on what it sees.

prompt = """Analyze this image of a product:
1. Describe the product (name, color, condition)
2. Estimate a price range in USD
3. Suggest 3 keywords to catalog it

Reply in JSON."""

Pattern 2: Extraction with a schema

You define exactly which fields you need. The model extracts only those.

prompt = """Extract from this invoice:
- date (YYYY-MM-DD format)
- total (number with 2 decimals)
- vendor (string)
- currency (ISO code: EUR, USD, MXN)

Reply ONLY with valid JSON."""

Pattern 3: Image comparison

You send 2+ images and ask for a comparison.

prompt = """Compare these two dashboard images.
- What metrics does each one show?
- Which has the better visual design?
- List 3 specific differences."""

Pattern 4: Contextual Q&A

The user asks questions about a specific image.

prompt = """Given this architecture diagram:
1. How many microservices are there?
2. Which database is used?
3. Is there a load balancer? Where?"""

Troubleshooting

Problem 1: "Invalid image format"

Cause: Unsupported format (BMP, TIFF) or a corrupt file.

Solution:

from PIL import Image

def ensure_supported_format(image_path: str) -> str:
    """Convert the image to JPEG if it isn't a supported format."""
    supported = {".jpg", ".jpeg", ".png", ".gif", ".webp"}
    from pathlib import Path
    ext = Path(image_path).suffix.lower()
    if ext in supported:
        return image_path
    output_path = image_path.rsplit(".", 1)[0] + ".jpg"
    Image.open(image_path).convert("RGB").save(output_path, "JPEG")
    return output_path

Problem 2: Image too large (>20MB)

Cause: A high-resolution photo or a document scanned at 600 DPI.

Solution:

from PIL import Image

def resize_if_needed(image_path: str, max_size_mb: float = 15.0) -> str:
    """Resize the image if it exceeds the maximum size."""
    import os
    size_mb = os.path.getsize(image_path) / (1024 * 1024)
    if size_mb <= max_size_mb:
        return image_path

    img = Image.open(image_path)
    # Reduce resolution while keeping the aspect ratio
    factor = (max_size_mb / size_mb) ** 0.5
    new_size = (int(img.width * factor), int(img.height * factor))
    img = img.resize(new_size, Image.LANCZOS)

    output = image_path.rsplit(".", 1)[0] + "_resized.jpg"
    img.save(output, "JPEG", quality=85)
    return output

Problem 3: Inaccurate OCR on documents

Cause: Low-resolution image, small text, noisy background.

Solution:

• Scan at a minimum of 300 DPI
• Use detail: "high" in the API
• Preprocess: increase contrast, convert to grayscale
• For critical documents, verify the results manually

Problem 4: High cost at volume

Cause: Sending many images with detail: "high".

Solution:

• Use gpt-4o-mini for classification and simple tasks
• Use detail: "low" when you don't need precise OCR
• Implement a cache: if the same image is sent twice, reuse the result
• Process in batches: group similar images

Exercises

Exercise 1: Description with constraints (Easy)

Modify the describe_image function so the model replies in a maximum of 50 words and mentions only objects (no people).

def describe_objects_only(image_path: str) -> str:
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        "Describe this image in AT MOST 50 words. "
                        "Mention ONLY visible objects and elements. "
                        "Do NOT describe people. Be concise and specific."
                    )
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}
                }
            ]
        }],
        max_tokens=100
    )
    return response.choices[0].message.content

Exercise 2: Invoice extraction to JSON (Easy)

Write a function that takes an invoice image and returns a dictionary with: date, total, vendor, currency.

import json

def extract_invoice(image_path: str) -> dict:
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        "Extract the following fields from this invoice:\n"
                        "- date (YYYY-MM-DD format)\n"
                        "- total (number with decimals)\n"
                        "- vendor (company name)\n"
                        "- currency (ISO code: EUR, USD, MXN)\n\n"
                        "Reply ONLY with valid JSON."
                    )
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data}",
                        "detail": "high"
                    }
                }
            ]
        }],
        max_tokens=200,
        temperature=0
    )
    return json.loads(response.choices[0].message.content)

Exercise 3: Multi-label classifier (Medium)

Design a function that classifies an image into multiple categories (not just one). It must return the categories that apply with a confidence level.

import json

def classify_multi_label(
    image_path: str,
    categories: list[str]
) -> list[dict]:
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    cats = ", ".join(categories)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        f"Classify this image. It may belong to SEVERAL categories.\n"
                        f"Available categories: {cats}\n\n"
                        f"Reply with JSON: a list of objects with "
                        f'"category" and "confidence" (high/medium/low).\n'
                        f"Only include categories that apply."
                    )
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}
                }
            ]
        }],
        max_tokens=300,
        temperature=0
    )
    return json.loads(response.choices[0].message.content)

# Usage
categories = ["nature", "urban", "indoor", "people",
              "objects", "text", "document", "food"]
result = classify_multi_label("photo.jpg", categories)
# Expected output:
# [{"category": "urban", "confidence": "high"},
#  {"category": "people", "confidence": "medium"}]

Exercise 4: Wrapper with retry and validation (Medium)

Create a safe_vision_call function that: (a) validates that the file exists and is an image, (b) resizes it if it's larger than 15MB, (c) retries up to 3 times if the API fails.

import os
import time
from pathlib import Path
from PIL import Image

SUPPORTED_FORMATS = {".jpg", ".jpeg", ".png", ".gif", ".webp"}

def safe_vision_call(
    image_path: str,
    prompt: str,
    max_retries: int = 3
) -> str:
    path = Path(image_path)
    if not path.exists():
        raise FileNotFoundError(f"Image not found: {image_path}")

    if path.suffix.lower() not in SUPPORTED_FORMATS:
        raise ValueError(
            f"Unsupported format: {path.suffix}. "
            f"Use: {SUPPORTED_FORMATS}"
        )

    # Resize if needed
    size_mb = os.path.getsize(image_path) / (1024 * 1024)
    if size_mb > 15:
        img = Image.open(image_path)
        factor = (15 / size_mb) ** 0.5
        new_size = (int(img.width * factor), int(img.height * factor))
        img = img.resize(new_size, Image.LANCZOS)
        resized_path = str(path.with_suffix("")) + "_resized.jpg"
        img.save(resized_path, "JPEG", quality=85)
        image_path = resized_path

    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{
                    "role": "user",
                    "content": [
                        {"type": "text", "text": prompt},
                        {"type": "image_url", "image_url": {
                            "url": f"data:image/jpeg;base64,{image_data}"
                        }}
                    ]
                }],
                max_tokens=500
            )
            return response.choices[0].message.content
        except Exception as e:
            if attempt < max_retries - 1:
                wait = 2 ** attempt  # exponential backoff
                time.sleep(wait)
            else:
                raise RuntimeError(
                    f"Vision API failed after {max_retries} attempts: {e}"
                )

Exercise 5: Compare two images (Hard)

Create a function that takes two images and returns a comparative analysis in JSON with: similarities, differences, and conclusion.

import json

def compare_images(image_path_1: str, image_path_2: str) -> dict:
    images = []
    for path in [image_path_1, image_path_2]:
        with open(path, "rb") as f:
            b64 = base64.b64encode(f.read()).decode()
        images.append(b64)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        "Compare these two images.\n"
                        "Reply with JSON:\n"
                        '{"similarities": ["..."], '
                        '"differences": ["..."], '
                        '"conclusion": "..."}'
                    )
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{images[0]}"}
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{images[1]}"}
                }
            ]
        }],
        max_tokens=500,
        temperature=0
    )
    return json.loads(response.choices[0].message.content)

Summary

In this capsule you learned:

• The vision modality lets LLMs process images: description, OCR, classification, structured extraction
• The input formats are Base64 (local images) and URL (images on the web)
• The detail parameter (low/high/auto) controls resolution and cost
• The limitations include: maximum size, supported formats, cost per token, OCR accuracy
• Vision API vs Tesseract OCR: vision for complexity and interpretation, Tesseract for volume and zero cost
• The common patterns are: describe+act, extraction with a schema, comparison, contextual Q&A
• For production you need: input validation, resizing, retry, error handling

Next capsule: The audio modality — transcription with Whisper and speech synthesis with TTS.

Additional Resources

1. OpenAI Vision Guide — Official GPT-4 Vision documentation
2. GPT-4o Vision Capabilities — GPT-4o announcement and capabilities
3. Anthropic Vision Docs — Claude 3 with images
4. Tesseract OCR — Open-source OCR as a local alternative
5. Pillow Documentation — Image processing in Python
6. OpenAI Cookbook: Vision — Advanced vision examples