Module 2: Vision + LLMs

1. Introduction: LLMs with Vision

Description

This is the first capsule of Module 2 of the Multimodal AI Guide. In Module 1 you learned the complete landscape: which modalities exist, which models support them, and how to classify inputs. Now you're going to go deep into the most mature and in-demand modality in production: vision.

LLMs with vision can take an image and a prompt, and return textual analysis: descriptions, extracted text (OCR), classifications, answers to questions about the visual content. This turns workflows that used to require specialized computer vision pipelines into API calls.

Why it matters: Vision is the multimodal modality with the widest adoption in production. Three providers dominate the space — OpenAI (GPT-4 Vision), Anthropic (Claude 3) and Google (Gemini) — and each one has different strengths. Mastering all three lets you pick the best one for each case, design fallbacks when one fails, and negotiate costs with technical judgment.

This module isn't a list of APIs. It's a framework for thinking about vision as an engineering tool: when to use it, with which provider, in what format, at what cost, and with what limitations. At the end you'll build a multi-provider Image Analyzer that applies all of this in a working system with automatic fallback.

Where Are We in the Guide?

Context

Phase 1: Multimodal Fundamentals (Modules 1-3)
├── Module 1: Introduction to Multimodal AI    ✅ COMPLETED
├── Module 2: Vision + LLMs                    ← YOU ARE HERE
└── Module 3: Document Understanding

Phase 2: Generation and Audio (Modules 4-5)
├── Module 4: Image Generation
└── Module 5: Audio Processing

Phase 3: RAG, Use Cases and Project (Modules 6-8)
├── Module 6: Multimodal RAG
├── Module 7: Use Cases
└── Module 8: Final Project — Multimodal Document Analyzer

Connection with the previous module

In Module 1 you learned to classify inputs by modality and suggest the optimal model. Now you're going to implement the vision branch of that classifier: given that the input is an image, which provider do I use? How do I send the image? What prompt do I design? How do I handle errors?

Your Classifier from Module 1 detected modalities. Your Image Analyzer from Module 2 executes the vision modality with the three main providers.

Module 2 Roadmap

#	Capsule	What you'll learn	Type
01	Introduction (this one)	Vision in LLMs, providers, use cases, setup	Intro
02	GPT-4 Vision (OpenAI)	The vision API, image formats, visual prompts	Technical
03	Claude 3 Vision (Anthropic)	Claude's API, differences from GPT-4V, media types	Technical
04	Gemini Vision (Google)	Gemini's API, native capabilities, free tier	Technical
05	Provider comparison	Benchmark: cost, quality, latency, when to use each one	Technical
06	Analysis patterns	Description, OCR, classification, structured extraction	Technical
07	Multi-image and context	Multiple images in one prompt, visual context	Technical
08	Project: Image Analyzer	Multi-provider analyzer with automatic fallback	Project

Learning flow

First you'll master each provider individually (capsules 02-04): its API, its formats, its quirks. Then you'll compare them with concrete metrics (capsule 05). Next you'll learn analysis patterns that apply to any provider (capsule 06) and advanced techniques with multiple images (capsule 07). Finally you'll integrate everything into the Image Analyzer (capsule 08).

The progression is: individual provider → comparison → patterns → advanced techniques → project.

Estimated module duration: 50 minutes.

What You'll Learn

Technical objectives

By the end of this module you'll be able to:

Send images to GPT-4 Vision, Claude 3 and Gemini using their respective APIs with working Python code
Compare providers with measurable criteria: cost per image, OCR quality, response latency, supported formats
Apply visual analysis patterns: detailed description, text extraction (OCR), content classification, structured data extraction
Send multiple images in a single request for comparative or contextual analysis
Design prompts for vision that control the detail level, the response format and the focus of the analysis
Implement an Image Analyzer with automatic fallback between providers: if GPT-4V fails, try Claude 3; if Claude fails, try Gemini

Professional objective

When someone tells you "we need the system to analyze product photos to generate automatic descriptions", you'll know: which provider to use based on volume and budget, how to send the images (Base64 vs URL), what prompt to design to get consistent descriptions, and how to implement a fallback so the system doesn't go down if one provider has an outage.

Context: The Vision Era in LLMs

From text-only to vision

Through 2020-2022, LLMs were exclusively textual. GPT-3, the original ChatGPT, Claude 1 — they all processed text as input and generated text as output. If you needed to analyze an image, you used separate pipelines: OpenCV for processing, YOLO for object detection, Tesseract for OCR. Each one required specific setup, different models, and none of them "understood" the semantic context of the image.

2020-2022 (text-only era):
  Analyze an invoice image →
    1. Tesseract for OCR (extract text)
    2. Regex to parse fields (date, total)
    3. LLM to classify the invoice type
  = 3 tools, 3 points of failure, no contextual understanding

2023-2024 (vision era):
  Analyze an invoice image →
    1. GPT-4 Vision: "Extract the date, total and vendor from this invoice"
  = 1 API call, complete contextual understanding

The inflection point: 2023-2024

Three events changed the landscape:

September 2023 — GPT-4 Vision: OpenAI added vision capability to GPT-4. For the first time, a general-purpose LLM could "see" images and reason about them with the same depth it reasoned about text.
March 2024 — Claude 3: Anthropic released Claude 3 (Haiku, Sonnet, Opus) with native vision. Direct competition improved prices and quality for everyone.
December 2023 — Gemini: Google released Gemini with multimodal capabilities native to the design. It wasn't a bolted-on module — the model was trained to understand images from the start.

Why this matters for engineers

Before 2023, integrating vision into an application meant:

Maintaining separate CV models (YOLO, Tesseract, custom classifiers)
Training or fine-tuning for each domain
Handling complex pipelines with multiple points of failure

After 2023, integrating vision means:

One API call with an image + prompt
Zero-shot: it works without training specific to your domain
The same model you already use for text also understands images

The change isn't just technical — it's economic. A custom CV pipeline cost weeks of development and maintenance. An integration with GPT-4 Vision is implemented in hours and the provider maintains the model.

The current state

In 2025-2026, the three main providers have matured:

Aspect	GPT-4 Vision	Claude 3	Gemini
Maturity	High	High	High
OCR	Excellent	Very good	Excellent
Visual reasoning	Excellent	Excellent	Very good
Cost	Medium-high	Medium	Low (free tier)
Context window	128K tokens	200K tokens	1M+ tokens
Multi-image	Yes	Yes	Yes (native)

Competition between providers means prices drop, quality rises, and formats standardize. Knowing how to use all three gives you flexibility to choose per case.

The Three Providers

OpenAI — GPT-4 Vision

GPT-4 Vision (GPT-4V, GPT-4o) was the first general-purpose LLM with vision to reach mass adoption. Its strength is reasoning over complex visual content: it can interpret charts, read text in images, describe scenes in detail, and follow precise instructions about what to analyze.

What it brings:

The best ecosystem of tools and documentation
Two detail levels: low (fast, cheap) and high (precise, expensive)
Native integration with the rest of the OpenAI ecosystem (Assistants, function calling)
The most-used model in production for vision

Main format: Base64 or a direct URL in the message's image_url field.

Anthropic — Claude 3 Vision

Claude 3 stands out in detailed analysis and following complex instructions. Where GPT-4V can be concise, Claude tends to give more thorough analysis. Its 200K-token context window lets you send many images in a single request.

What it brings:

More detailed and structured analysis by default
Better adherence to specific response formats (JSON, tables)
Three tiers: Haiku (fast/cheap), Sonnet (balance), Opus (maximum quality)
Excellent for documents with complex layouts

Main format: Base64 with an explicit media_type in the content's image block.

Google — Gemini Vision

Gemini was designed as multimodal from its architecture. It isn't a text model that had vision added — it processes text and images natively. Its main advantage is the massive context window (1M+ tokens) and a generous free tier.

What it brings:

A free tier for experimentation (Gemini Flash)
A 1M+ token context window for analyzing many images
Native image processing (not as a bolted-on module)
Good quality/price ratio in the paid tier

Main format: Part objects with images as inline_data or file_data.

Real Use Cases

1. Document analysis

The most in-demand case in production. Companies process thousands of invoices, receipts, contracts and forms daily. An LLM with vision can extract structured fields (date, total, vendor, line items) from an image or scan without needing OCR templates specific to each document type.

Real example: A fintech processes 10,000 receipts/day. Before: a Tesseract + regex pipeline with 78% accuracy. After: GPT-4 Vision with a structured prompt and 95% accuracy, with no template maintenance.

2. Product descriptions (e-commerce)

E-commerce platforms generate automatic descriptions from product photos. The model sees the image, identifies the product, its features, colors, materials, and generates an SEO-optimized description.

Real example: A marketplace with 50,000 new products/month. Sellers upload photos and the system generates the title, description, category and tags automatically.

3. Accessibility

LLMs with vision can describe images for people with visual impairments. Unlike generic alt-text, an LLM generates contextual descriptions: not just "a person in a park" but "a young woman reading a book on a park bench, with a golden retriever at her feet, on a sunny autumn day".

4. Quality control

Manufacturing and visual inspection: the model analyzes photos of products on the production line and detects defects, misalignments, or missing components. It doesn't replace specialized industrial CV systems, but it works as a first filter or as a solution for low volumes where a custom system isn't justified.

5. Basic medical analysis

LLMs with vision can describe medical images (X-rays, dermatological photos) as a pre-screening tool. They don't replace professional diagnosis, but they can prioritize cases or generate initial reports that a doctor reviews.

Important note: This use case requires ethical and regulatory considerations that are outside the scope of this guide. We mention it because it's real and growing, not because we'll implement it.

6. Security and surveillance

Analysis of security camera frames: anomaly detection, object identification, scene description. An LLM with vision can answer "is there anyone in this image?" or "describe what's happening in this scene" without needing an object detection model trained specifically for it.

Prerequisites

What you need from Module 1

This module assumes you completed Module 1 and are comfortable with:

The concept of modalities: You know what vision is (image → text) and how it differs from other modalities
Base64: You understand why images are encoded in Base64 to send them over an API and how to do it
API keys: You have keys configured for at least OpenAI; ideally Anthropic and Google too
Cost estimation: You know that calls with images cost more than text-only and how to calculate the approximate cost
Input formats: You know the difference between sending an image as Base64 vs a URL

If you didn't complete Module 1

You can continue if you have prior experience with LLM APIs and Python. The critical concepts you need:

Concept	Why you need it	Where to learn it
Base64 encoding	Every vision API accepts images in Base64	Module 1, Capsule 06
API keys and .env	You're going to use 3 different providers	Module 1, Technical Setup
Cost per token	Images consume a lot of tokens	Module 1, Capsule 06
Prompt design	Vision prompts have a specific structure	Module 1, Capsules 02-03

Technical Setup

Dependencies

This module requires the SDKs of the three main providers:

pip install openai>=1.0.0 anthropic>=0.25.0 google-generativeai>=0.5.0 python-dotenv pillow

Package	What for
`openai`	GPT-4 Vision API
`anthropic`	Claude 3 Vision API
`google-generativeai`	Gemini Vision API
`python-dotenv`	Load API keys from .env
`pillow`	Image manipulation (resize, format)

Environment variables

Create or update your .env file with the three API keys:

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AI...

Verification script

Run this script to confirm that the three providers are configured and responding:

import os
import sys
from dotenv import load_dotenv

load_dotenv()

providers = {
    "OpenAI": os.getenv("OPENAI_API_KEY"),
    "Anthropic": os.getenv("ANTHROPIC_API_KEY"),
    "Google": os.getenv("GOOGLE_API_KEY"),
}

print("=" * 50)
print("Provider verification — Module 2")
print("=" * 50)

results = {}

for name, key in providers.items():
    if not key:
        print(f"\n❌ {name}: API key not found in .env")
        results[name] = False
        continue

    try:
        if name == "OpenAI":
            from openai import OpenAI
            client = OpenAI()
            response = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{"role": "user", "content": "Reply only OK"}],
                max_tokens=5,
            )
            print(f"\n✅ {name}: {response.choices[0].message.content}")
            results[name] = True

        elif name == "Anthropic":
            import anthropic
            client = anthropic.Anthropic()
            response = client.messages.create(
                model="claude-3-haiku-20240307",
                max_tokens=5,
                messages=[{"role": "user", "content": "Reply only OK"}],
            )
            print(f"\n✅ {name}: {response.content[0].text}")
            results[name] = True

        elif name == "Google":
            import google.generativeai as genai
            genai.configure(api_key=key)
            model = genai.GenerativeModel("gemini-1.5-flash")
            response = model.generate_content("Reply only OK")
            print(f"\n✅ {name}: {response.text.strip()}")
            results[name] = True

    except Exception as e:
        print(f"\n❌ {name}: Error — {e}")
        results[name] = False

print("\n" + "=" * 50)
active = sum(1 for v in results.values() if v)
print(f"Result: {active}/3 providers active")

if active == 0:
    print("⚠️  You need at least 1 provider to continue.")
    sys.exit(1)
elif active < 3:
    print("⚠️  You can continue, but some capsules require all 3.")
else:
    print("✅ Setup complete. Ready for Module 2.")

Expected output (with all 3 providers):

==================================================
Provider verification — Module 2
==================================================

✅ OpenAI: OK

✅ Anthropic: OK

✅ Google: OK

==================================================
Result: 3/3 providers active
✅ Setup complete. Ready for Module 2.

Estimated costs for this module

Provider	Model	Cost per image (~)	Estimated for the whole module
OpenAI	gpt-4o (high detail)	$0.01-0.03	$1.00-2.00
OpenAI	gpt-4o-mini	$0.002-0.005	$0.20-0.50
Anthropic	claude-3-haiku	$0.001-0.003	$0.10-0.30
Anthropic	claude-3-sonnet	$0.005-0.015	$0.50-1.50
Google	gemini-1.5-flash	Free (free tier)	$0.00
Google	gemini-1.5-pro	$0.003-0.01	$0.30-1.00

Cost strategy: Use gpt-4o-mini and gemini-1.5-flash for development and experimentation. Reserve gpt-4o (high detail) and claude-3-sonnet for the quality comparisons in capsule 05.

File structure for the module

module_02/
├── 01_verify_setup.py          # Verification script (above)
├── 02_gpt4_vision.py           # GPT-4V exercises
├── 03_claude_vision.py         # Claude 3 exercises
├── 04_gemini_vision.py         # Gemini exercises
├── 05_comparison.py            # Comparative benchmark
├── 06_analysis_patterns.py     # Patterns: OCR, description, etc.
├── 07_multi_image.py           # Multi-image
├── 08_image_analyzer.py        # The module's final project
└── images/
    ├── test_photo.jpg          # General test photo
    ├── test_document.png       # Test document/invoice
    └── test_product.jpg        # Product for description

Limits: What This Module Does NOT Cover

❌ Fine-tuning vision models — We use pre-trained models via API. We don't train or fine-tune custom vision models.
❌ Classic computer vision — We don't cover OpenCV, YOLO, or traditional CV techniques. The focus is LLMs with vision.
❌ In-depth video analysis — We'll mention frame extraction, but full video analysis is covered in later modules.
❌ Open-source vision models — We focus on the three main cloud APIs. We don't deploy LLaVA, Qwen-VL, or other local models.
❌ Image generation — This module is about image analysis (image → text). Generation (text → image) is covered in Module 4.

Evidence of Success

By the end of this module, you'll know you succeeded if:

✅ You can send an image to GPT-4 Vision, Claude 3 and Gemini with working Python code
✅ Given a use case ("I need to extract text from receipts"), you pick the optimal provider and justify why
✅ You know the format differences between the three providers (how each one receives images)
✅ You can apply at least 3 analysis patterns: description, OCR, classification
✅ You know how to send multiple images in a single request and design prompts that compare them
✅ Your Image Analyzer works: it takes an image, analyzes it with one provider, and if that fails, tries the next

Quick self-assessment test

If you can answer these questions, you're on the right track:

What's the difference in how OpenAI, Anthropic and Google receive images in their API?
In which case would you choose Claude 3 over GPT-4V for document analysis?
Why is Gemini a good option for prototyping and experimenting?
What does "high detail" vs "low detail" mean in GPT-4 Vision and how does it affect cost?
How would you implement a fallback if your main provider hits a rate limit error?

Summary

LLMs with vision transformed image analysis: from complex CV pipelines to one API call with an image + prompt.
2023-2024 was the inflection point: GPT-4 Vision, Claude 3 and Gemini took vision to mass production.
The three main providers have different strengths: OpenAI in ecosystem, Anthropic in detailed analysis, Google in cost and context window.
The real use cases include: document analysis, e-commerce, accessibility, quality control, and more.
This module covers the three providers' APIs, analysis patterns, multi-image, and culminates in a multi-provider Image Analyzer with fallback.
You need the three SDKs installed (openai, anthropic, google-generativeai) and the API keys configured.

Additional Resources

OpenAI Vision Guide — Official GPT-4 Vision documentation
Anthropic Vision Docs — Claude 3 with images
Google Gemini Vision — Multimodal Gemini and vision
OpenAI Pricing — Up-to-date GPT-4V prices
Anthropic Pricing — Claude 3 prices by tier
Google AI Pricing — Gemini free and paid tiers

← Previous8. Project: Multimodal Classifier Next →2. GPT-4 Vision (OpenAI)