Multimodal and Reasoning

Capsule overview

Up to now, every example you've seen sends text to the model and gets text back. But modern models can do much more: process images, interpret diagrams, analyze screenshots, and in some cases work with audio and video. All of it without switching frameworks — LangChain uses the same message system you already know, just with a richer content format.

The second half of this capsule covers reasoning models: models that "think" internally before giving a final answer. Instead of generating text token by token from left to right, these models spend tokens reasoning about the problem, weighing alternatives and checking their own logic. The result is more accurate answers on tasks that require math, multi-step analysis or complex logic.

Both capabilities — multimodal and reasoning — plug straight into the flow you already know with init_chat_model and invoke. In the module's project, you'll be able to use multimodal to analyze images inside the chat, and reasoning for questions that need more rigorous answers.

Multimodal content: beyond text

Multimodal models accept images (and other media) as part of the input. In LangChain, this is done through content blocks: instead of passing a plain string as the content, you pass a list of blocks where each block can be text or an image.

Sending images to the model

Up to now you've used HumanMessage with a string:

from langchain_core.messages import HumanMessage

message = HumanMessage(content="What is Python?")

To send images, content goes from being a string to being a list of dictionaries:

message = HumanMessage(
    content=[
        {"type": "text", "text": "What do you see in this image?"},
        {"type": "image_url", "image_url": {"url": "https://..."}},
    ]
)

Each block has a type: "text" for regular text, "image_url" for images (by URL or base64). You can combine as many blocks as you need in a single message.

Image from a URL

The most direct way to send an image is with a public URL:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage

model = init_chat_model("openai:gpt-4.1")

message = HumanMessage(
    content=[
        {"type": "text", "text": "Describe this image in 2-3 sentences."},
        {
            "type": "image_url",
            "image_url": {
                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"
            },
        },
    ]
)

response = model.invoke([message])
print(response.content)
# Expected output: "The image shows a domestic tabby cat..."

Key points:

• invoke takes a list of messages, not a single message
• The URL has to be publicly reachable — the provider downloads it internally
• Use a model with vision support (GPT-4.1, Claude Sonnet 4, Gemini 2.0 Flash)

Image from base64

For local or private images with no public URL, convert them to base64:

from dotenv import load_dotenv
load_dotenv()

import base64
from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage

model = init_chat_model("openai:gpt-4.1")


def encode_image(path: str) -> str:
    """Reads a local image and returns its base64 representation."""
    with open(path, "rb") as f:
        return base64.standard_b64encode(f.read()).decode("utf-8")


image_data = encode_image("photo.jpg")

message = HumanMessage(
    content=[
        {"type": "text", "text": "What object appears in this photo?"},
        {
            "type": "image_url",
            "image_url": {
                "url": f"data:image/jpeg;base64,{image_data}"
            },
        },
    ]
)

response = model.invoke([message])
print(response.content)
# Expected output: a description of the local image's contents

The base64 URL follows the format data:{mime_type};base64,{data}. The common MIME types:

Multiple images in one message

You can send several images in a single message by adding more image_url blocks:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage

model = init_chat_model("openai:gpt-4.1")

message = HumanMessage(
    content=[
        {"type": "text", "text": "Compare these two images. What differences do you see?"},
        {
            "type": "image_url",
            "image_url": {
                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"
            },
        },
        {
            "type": "image_url",
            "image_url": {
                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/YellowLabradorLooking_new.jpg/1200px-YellowLabradorLooking_new.jpg"
            },
        },
    ]
)

response = model.invoke([message])
print(response.content)
# Expected output: "The first image shows a cat, while the second one
# shows a Labrador dog. The main differences are..."

There's no hard limit on how many images you can send, but each one eats tokens from the context window. Bigger images = more tokens = more cost.

Multimodal with Structured Output

You can combine multimodal with with_structured_output from the previous capsule to extract structured data from images:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage
from pydantic import BaseModel, Field


class ImageAnalysis(BaseModel):
    description: str = Field(description="A brief one-sentence description of the image")
    objects: list[str] = Field(description="List of the main objects detected")
    dominant_colors: list[str] = Field(description="Dominant colors in the image")
    mood: str = Field(description="Overall mood or atmosphere")


model = init_chat_model("openai:gpt-4.1")
structured_model = model.with_structured_output(ImageAnalysis)

message = HumanMessage(
    content=[
        {"type": "text", "text": "Analyze this image."},
        {
            "type": "image_url",
            "image_url": {
                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"
            },
        },
    ]
)

result = structured_model.invoke([message])

print(f"Description: {result.description}")
print(f"Objects: {result.objects}")
print(f"Colors: {result.dominant_colors}")
print(f"Mood: {result.mood}")
# Expected output:
# Description: A tabby cat resting in an outdoor setting
# Objects: ['cat', 'grass', 'plants']
# Colors: ['green', 'brown', 'gray']
# Mood: calm

This pattern is very useful for image-processing pipelines. It works with any Pydantic schema from Capsule 05.

Multimodal support by provider

• ✅ GPT-4.1 and Claude Sonnet 4 are the main options for vision
• ✅ Gemini 2.0 Flash is the most versatile: it supports images, audio and video
• ❌ Reasoning models (o3-mini) generally don't support multimodal
• ⚠️ Audio and video support changes often — check the provider's documentation

Audio and video (overview)

Some models accept audio and video as input in addition to images. The pattern is similar — content blocks with specific types:

import base64
from langchain_core.messages import HumanMessage

with open("recording.mp3", "rb") as f:
    audio_data = base64.standard_b64encode(f.read()).decode("utf-8")

# Audio input (GPT-4.1, Gemini 2.0 Flash)
message = HumanMessage(
    content=[
        {"type": "text", "text": "Transcribe this audio."},
        {
            "type": "input_audio",
            "input_audio": {"data": audio_data, "format": "mp3"},
        },
    ]
)

For video, Gemini 2.0 Flash is currently the main model with direct support:

# Video input (Gemini 2.0 Flash)
message = HumanMessage(
    content=[
        {"type": "text", "text": "Describe what happens in this video."},
        {"type": "media", "mime_type": "video/mp4", "data": video_base64},
    ]
)

⚠️ Audio and video are newer capabilities whose APIs are still evolving. The exact content-block format can vary between providers and package versions. Always check the corresponding provider's up-to-date documentation.

Reasoning: models that think step by step

What are reasoning models?

Standard models (GPT-4.1, Claude Sonnet 4) generate text token by token without "thinking" about the problem globally. They work well for most tasks, but they can stumble on problems that require multi-step logic or complex mathematical reasoning.

Reasoning models spend internal tokens reasoning about the problem before generating the final answer: they break the task down, weigh alternatives, verify intermediate steps. More tokens = more time and cost, but higher quality on complex tasks.

There are two main approaches:

The key difference: with OpenAI, the reasoning is internal. With Anthropic, you can see exactly what the model thought.

Reasoning with OpenAI (o3-mini)

o3-mini is a model designed for reasoning. You use it just like any other model:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model

model = init_chat_model("openai:o3-mini")

response = model.invoke(
    "A train leaves Madrid at 8:00 doing 120 km/h. "
    "Another leaves Barcelona (620 km away) at 9:00 doing 150 km/h toward Madrid. "
    "At what time do they cross?"
)
print(response.content)
# Expected output: the working, plus the correct answer (~10:20)

The model reasons internally but doesn't show the steps in content. The reasoning tokens show up in the metadata:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model

model = init_chat_model("openai:o3-mini")
response = model.invoke("How many prime numbers are there between 1 and 50?")

print(response.content)
# Expected output: There are 15 prime numbers between 1 and 50: 2, 3, 5, 7, 11, 13, ...

print(f"Input tokens: {response.usage_metadata['input_tokens']}")
print(f"Output tokens: {response.usage_metadata['output_tokens']}")
print(f"Total: {response.usage_metadata['total_tokens']}")
# Expected output:
# Input tokens: 18
# Output tokens: 245
# Total: 263

The output tokens include both the (internal) reasoning tokens and the visible answer. You pay for both.

Effort levels (OpenAI)

You can control how much effort o3-mini spends reasoning with the reasoning_effort parameter:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model

model_low = init_chat_model("openai:o3-mini", reasoning_effort="low")
model_medium = init_chat_model("openai:o3-mini", reasoning_effort="medium")
model_high = init_chat_model("openai:o3-mini", reasoning_effort="high")

question = "Explain why 0.1 + 0.2 != 0.3 in most programming languages."

for label, m in [("low", model_low), ("medium", model_medium), ("high", model_high)]:
    response = m.invoke(question)
    tokens = response.usage_metadata["output_tokens"]
    print(f"[{label:6s}] Output tokens: {tokens}")
    print(f"[{label:6s}] Response: {response.content[:120]}...")
    print()
# Expected output: each level spends more tokens and gives more detailed answers
# [low   ] ~80 tokens  → a concise answer
# [medium] ~200 tokens → an explanation with context
# [high  ] ~450 tokens → a detailed analysis of the IEEE 754 standard

Important: o3-mini ignores the temperature parameter. Its behavior is controlled exclusively through reasoning_effort. If you need to control randomness, use a standard model.

Extended thinking with Anthropic

Anthropic offers a different approach: Claude models can turn on extended thinking, where the reasoning steps are visible in the response. It requires direct initialization with ChatAnthropic:

from dotenv import load_dotenv
load_dotenv()

from langchain_anthropic import ChatAnthropic

model = ChatAnthropic(
    model="claude-sonnet-4-20250514",
    max_tokens=16000,
    thinking={
        "type": "enabled",
        "budget_tokens": 10000,
    },
)

response = model.invoke(
    "If I have 3 boxes, each box has 2 bags, and each bag has 5 coins, "
    "how many coins do I have in total? Show your reasoning."
)

# With extended thinking, response.content is a LIST of blocks
for block in response.content:
    if block["type"] == "thinking":
        print(f"[The model's reasoning]")
        print(block["thinking"][:500])
        print()
    elif block["type"] == "text":
        print(f"[Final answer]")
        print(block["text"])

# Expected output:
# [The model's reasoning]
# I need to work out the total number of coins.
# 3 boxes × 2 bags = 6 bags. 6 bags × 5 coins = 30. Check: 3×2×5 = 30.
#
# [Final answer]
# You have 30 coins in total.

Requirements for extended thinking:

• ✅ max_tokens is mandatory and has to be greater than budget_tokens
• ✅ budget_tokens controls how many tokens the model can spend thinking
• ⚠️ temperature has to be 1 (or left unset) while extended thinking is on
• ⚠️ Not every Claude model supports it — check Anthropic's documentation

When to use reasoning vs standard models

Reasoning isn't always the better option. More reasoning means more tokens, more latency and more cost. Use this guide to decide:

Rule of thumb: If you can solve the task without thinking for more than 5 seconds, use a standard model. If you need pen and paper to check the answer, use reasoning.

Connection to the project

In the Multi-Provider Chat with Fallback:

• You can add multimodal support so the user can send images into the chat
• Content blocks work the same across every provider with vision support — the fallback can go from GPT-4.1 to Claude Sonnet 4 without changing the message format
• For complex questions, the chat can route to a reasoning model instead of the standard one
• The structured metadata (Capsule 05) can include a field indicating whether reasoning was used

Troubleshooting

Problem 1: "Could not process image" or an error when sending an image

Cause: The model doesn't support multimodal input, or the image's URL isn't reachable. Fix: Check that you're using a model with vision support (see the support table). If you're using a URL, make sure it's public and reachable:

import requests

url = "https://your-image.com/photo.jpg"
r = requests.head(url)
print(f"Status: {r.status_code}")
print(f"Content-Type: {r.headers.get('content-type')}")
# It should be 200 and an image content-type

Problem 2: content as a list doesn't work with the model

Cause: The model doesn't support content blocks (e.g. o3-mini, or local models without vision). Fix: Check the support in the provider table. If you need a model without multimodal support, send text only:

# This works with any model
message = HumanMessage(content="Plain text")

# This requires a model with multimodal support
message = HumanMessage(content=[{"type": "text", "text": "..."}, {"type": "image_url", ...}])

Problem 3: A base64 image causes a size error

Cause: The image is too large. Providers have limits (OpenAI: ~20MB, Anthropic: ~5MB per image). Fix: Resize the image before encoding it:

from PIL import Image
import io
import base64


def encode_image_resized(path: str, max_size: int = 1024) -> str:
    """Resizes the image if it exceeds max_size and returns base64."""
    img = Image.open(path)

    if max(img.size) > max_size:
        img.thumbnail((max_size, max_size))

    buffer = io.BytesIO()
    img.save(buffer, format="JPEG", quality=85)
    return base64.standard_b64encode(buffer.getvalue()).decode("utf-8")

Problem 4: reasoning_effort has no effect

Cause: You're using reasoning_effort with a model that isn't a reasoning model (e.g. gpt-4.1). Fix: That parameter only works with reasoning models like o3-mini. For standard models, use temperature to control behavior.

Problem 5: Anthropic's extended thinking returns a string instead of a list

Cause: The thinking parameter wasn't passed correctly, or max_tokens is missing. Fix: max_tokens is mandatory when you use thinking, and budget_tokens has to be lower than max_tokens:

model = ChatAnthropic(
    model="claude-sonnet-4-20250514",
    max_tokens=16000,
    thinking={"type": "enabled", "budget_tokens": 10000},
)

Exercises

Exercise 1: Describe an image (Easy)

Send an image from a URL to a model with multimodal support and ask it to describe the image. Use gpt-4.1 or claude-sonnet-4-20250514. Print the description.

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage

model = init_chat_model("openai:gpt-4.1")

message = HumanMessage(
    content=[
        {"type": "text", "text": "Describe this image in detail."},
        {
            "type": "image_url",
            "image_url": {
                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/300px-PNG_transparency_demonstration_1.png"
            },
        },
    ]
)

response = model.invoke([message])
print(response.content)
# Expected output: a detailed description of the image
# (two red dice over a checkered background demonstrating PNG transparency)

Exercise 2: Extract data from an image (Medium)

Combine multimodal with Structured Output. Define a Pydantic schema ProductInfo with fields name, price (float), and category (str). Send the URL of a product image and extract the structured data.

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from langchain_core.messages import HumanMessage
from pydantic import BaseModel, Field


class ProductInfo(BaseModel):
    name: str = Field(description="Name of the product visible in the image")
    price: float = Field(description="The product's price if visible, 0.0 if not")
    category: str = Field(description="General product category: electronics, clothing, food, other")


model = init_chat_model("openai:gpt-4.1")
structured_model = model.with_structured_output(ProductInfo)

message = HumanMessage(
    content=[
        {"type": "text", "text": "Extract the product information from this image."},
        {
            "type": "image_url",
            "image_url": {
                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/1/16/AirPods_Pro_%282nd_generation%29.jpg/440px-AirPods_Pro_%282nd_generation%29.jpg"
            },
        },
    ]
)

result = structured_model.invoke([message])
print(f"Name: {result.name}")
print(f"Price: ${result.price}")
print(f"Category: {result.category}")
# Expected output:
# Name: AirPods Pro (2nd generation)
# Price: $0.0  (not visible in the image)
# Category: electronics

Exercise 3: Reasoning with o3-mini (Easy)

Use o3-mini to solve a logic problem. Compare the answer with gpt-4.1-mini's for the same problem. Print both responses and the tokens used.

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model

question = (
    "I have 5 shirts and 3 pairs of pants. "
    "How many different outfits can I put together if I always wear one shirt and one pair of pants?"
)

model_standard = init_chat_model("openai:gpt-4.1-mini")
model_reasoning = init_chat_model("openai:o3-mini")

print("=== GPT-4.1-mini (standard) ===")
r1 = model_standard.invoke(question)
print(f"Response: {r1.content}")
print(f"Tokens: {r1.usage_metadata['total_tokens']}")

print("\n=== o3-mini (reasoning) ===")
r2 = model_reasoning.invoke(question)
print(f"Response: {r2.content}")
print(f"Tokens: {r2.usage_metadata['total_tokens']}")
# Expected output: both answer 15, but o3-mini burns more tokens (internal reasoning).

Exercise 4: Compare effort levels (Medium)

Ask the same complex question to o3-mini at all 3 effort levels (low, medium, high). Print the response, the output tokens, and time each one.

from dotenv import load_dotenv
load_dotenv()

import time
from langchain.chat_models import init_chat_model

question = (
    "A snail climbs 3 meters every day but slides back 2 meters every night. "
    "If the well is 10 meters deep, how many days does it take to get out?"
)

for effort in ["low", "medium", "high"]:
    model = init_chat_model("openai:o3-mini", reasoning_effort=effort)

    start = time.time()
    response = model.invoke(question)
    elapsed = time.time() - start

    print(f"=== Effort: {effort} ===")
    print(f"Response: {response.content[:200]}")
    print(f"Output tokens: {response.usage_metadata['output_tokens']}")
    print(f"Time: {elapsed:.1f}s")
    print()

# Expected output: low is fast with few tokens, high is slow but detailed.
# All of them should answer 8 days (on day 8 it climbs to 10m and gets out).

Exercise 5: Extended thinking with Anthropic (Hard)

Use ChatAnthropic with extended thinking enabled to solve a logic problem. Pull out and display the reasoning and the final answer separately. Work out what percentage of the tokens went into reasoning.

from dotenv import load_dotenv
load_dotenv()

from langchain_anthropic import ChatAnthropic

model = ChatAnthropic(
    model="claude-sonnet-4-20250514",
    max_tokens=8000,
    thinking={"type": "enabled", "budget_tokens": 5000},
)

question = (
    "In a race, you overtake the runner in second place. "
    "What position are you in now?"
)

response = model.invoke(question)

thinking_text = ""
answer_text = ""

for block in response.content:
    if block["type"] == "thinking":
        thinking_text = block["thinking"]
    elif block["type"] == "text":
        answer_text = block["text"]

print("[The model's reasoning]")
print(thinking_text)
print()
print("[Final answer]")
print(answer_text)
print()

input_tokens = response.usage_metadata["input_tokens"]
output_tokens = response.usage_metadata["output_tokens"]
print(f"Input tokens: {input_tokens}")
print(f"Output tokens: {output_tokens}")

# Expected output:
# [The model's reasoning]
# The question is designed to pull an intuitive wrong answer out of you.
# If you overtake the runner in second, you take THEIR position...
#
# [Final answer]
# You're in second place. By overtaking the second-place runner, you take their spot.

Summary

In this capsule you learned:

• Content blocks let you send images (and other media) alongside text in a single message
• For images from a URL, you use {"type": "image_url", "image_url": {"url": "https://..."}}
• For local images, you encode them in base64 with the format data:image/jpeg;base64,...
• Multimodal + Structured Output combine directly — you can extract typed data from images
• Reasoning models (o3-mini) spend internal tokens thinking before answering
• reasoning_effort controls how much o3-mini thinks: "low", "medium", "high"
• Anthropic's extended thinking makes the reasoning visible in blocks separate from the answer
• Not everything needs reasoning — for simple tasks, standard models are faster and cheaper

Next capsule: Local models, caching and rate limiting — how to run models with Ollama, cut costs with prompt caching, and handle rate limits in production.

Additional resources

1. Multimodal — LangChain Docs — The conceptual guide to multimodal in LangChain
2. How to pass multimodal data to models — A step-by-step tutorial with content blocks
3. OpenAI Vision Guide — OpenAI's vision documentation
4. Anthropic Vision Documentation — Anthropic's vision guide with limits and formats
5. OpenAI Reasoning Models — Guide to o3-mini and reasoning effort
6. Anthropic Extended Thinking — Documentation for extended thinking with Claude
7. Gemini Multimodal Capabilities — Images, audio and video with Gemini
8. LangChain ChatAnthropic API Reference — Reference for the thinking parameter

Module 1 — LangChain & LangGraph: From Chains to Agents