4. Multimodal Combinations

Description

So far you've seen vision (image → text) and audio (audio ↔ text) as independent modalities. But the real power of multimodal AI appears when you combine them into pipelines: image + text → analysis, audio → text → LLM → image, document → extraction → summary in audio. In this capsule you'll learn the most-used combination patterns and how to design effective multimodal pipelines.

Why it matters: In production, you rarely use a single modality. A document analysis system combines vision (reading the document), text (processing it with an LLM) and audio (generating a spoken summary). A support assistant combines audio (listening to the question), text (processing it) and vision (showing screenshots). Mastering combinations is what separates a "script with an API" from a "multimodal system".

Connection with the module: This capsule connects capsules 02 (vision) and 03 (audio) into an integrated view. The pipelines you'll see here are the foundation of the multimodal Classifier (capsule 08) and of the final Document Analyzer (module 8).

Types of Combinations

Combination by input

You send multiple modalities as input to the same model or pipeline.

Text + Image → LLM with vision → Text
"What does this chart show?" + [chart.png] → "The chart shows Q4 sales..."

Combination by pipeline

You chain models that process different modalities in sequence.

Audio → Whisper → Text → GPT-4o → Text → TTS → Audio
[meeting.mp3] → "Summary: sales rose 15%" → [summary.mp3]

Combination by output

A system generates outputs in multiple modalities.

PDF document → Vision + LLM →
  ├── Text: extracted data (JSON)
  ├── Image: generated chart (DALL-E)
  └── Audio: spoken summary (TTS)

The 6 Multimodal Pipeline Patterns

Pattern 1: Vision + LLM (the most basic)

You send an image + prompt to a model with vision. It's the simplest and most-used pattern.

from openai import OpenAI
import base64

client = OpenAI()

def vision_plus_llm(image_path: str, question: str) -> str:
    """Basic pattern: image + question → answer."""
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": question},
                {"type": "image_url", "image_url": {
                    "url": f"data:image/jpeg;base64,{image_data}"
                }}
            ]
        }],
        max_tokens=500
    )
    return response.choices[0].message.content

# Usage
answer = vision_plus_llm(
    "architecture_diagram.png",
    "How many microservices does this system have, and what are they?"
)
print(answer)

Use cases: Q&A over images, document analysis, product description.

Pattern 2: Audio → Text → LLM

You transcribe audio and then process it with an LLM. You saw this in capsule 03; here we formalize it.

def audio_to_analysis(audio_path: str, analysis_prompt: str) -> str:
    """Pattern: audio → transcription → analysis with an LLM."""
    # Step 1: Transcribe
    with open(audio_path, "rb") as f:
        transcript = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            language="en"
        ).text

    # Step 2: Analyze
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": analysis_prompt},
            {"role": "user", "content": f"Transcript:\n\n{transcript}"}
        ],
        max_tokens=500
    )
    return response.choices[0].message.content

# Usage: extract action items from a meeting
result = audio_to_analysis(
    "standup.mp3",
    "Extract the action items from this meeting. List each one with an owner and a deadline."
)

Use cases: Meeting summaries, support-call analysis, extracting data from interviews.

Pattern 3: Text → LLM → Image

You generate text with an LLM and then create an image based on that text. Useful for automating visual content.

def text_to_visual(topic: str) -> dict:
    """Pattern: topic → description → generated image."""
    # Step 1: The LLM generates a prompt for the image
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": (
                f"Generate a prompt in English for DALL-E that creates a "
                f"professional illustration about: {topic}\n\n"
                f"The prompt must be descriptive (style, colors, composition). "
                f"Maximum 100 words."
            )
        }],
        max_tokens=150
    )
    image_prompt = response.choices[0].message.content

    # Step 2: Generate the image
    image_response = client.images.generate(
        model="dall-e-3",
        prompt=image_prompt,
        size="1024x1024",
        quality="standard",
        n=1
    )
    image_url = image_response.data[0].url

    return {
        "topic": topic,
        "image_prompt": image_prompt,
        "image_url": image_url
    }

# Usage
result = text_to_visual("cloud computing and microservices")
print(f"Generated prompt: {result['image_prompt']}")
print(f"Image: {result['image_url']}")

Use cases: Automatic thumbnail generation, illustrations for articles, marketing assets.

Pattern 4: Document → Extraction → JSON

You combine vision (to read the document) with an LLM (to structure the data). It's the core pipeline of module 3.

import json

def document_to_structured(image_path: str, fields: dict) -> dict:
    """Pattern: document image → structured data."""
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    fields_str = "\n".join(f"- {k}: {v}" for k, v in fields.items())

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        f"Extract these fields from the document:\n{fields_str}\n\n"
                        f"Reply ONLY with valid JSON. "
                        f"Use null if a field is not visible."
                    )
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data}",
                        "detail": "high"
                    }
                }
            ]
        }],
        max_tokens=500,
        temperature=0
    )
    return json.loads(response.choices[0].message.content)

# Usage: extract invoice data
data = document_to_structured("invoice.jpg", {
    "invoice_number": "invoice number",
    "date": "date in YYYY-MM-DD format",
    "total": "total amount with decimals",
    "company": "issuer name"
})

Use cases: Processing invoices, forms, receipts, contracts.

Pattern 5: Full pipeline (audio + vision + text)

The most complex pipeline: it combines every available modality.

def full_multimodal_pipeline(
    audio_path: str = None,
    image_path: str = None,
    text_input: str = None
) -> dict:
    """Pipeline that accepts any combination of inputs."""
    context_parts = []

    # Process audio if present
    if audio_path:
        with open(audio_path, "rb") as f:
            transcript = client.audio.transcriptions.create(
                model="whisper-1", file=f
            ).text
        context_parts.append(f"[Transcribed audio]: {transcript}")

    # Process the image if present
    if image_path:
        with open(image_path, "rb") as f:
            img_b64 = base64.b64encode(f.read()).decode()

        img_desc = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe this image in detail."},
                    {"type": "image_url", "image_url": {
                        "url": f"data:image/jpeg;base64,{img_b64}"
                    }}
                ]
            }],
            max_tokens=300
        ).choices[0].message.content
        context_parts.append(f"[Analyzed image]: {img_desc}")

    # Add text if present
    if text_input:
        context_parts.append(f"[User text]: {text_input}")

    # Combine and process
    combined = "\n\n".join(context_parts)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "Analyze all the information provided "
                    "(it may include transcribed audio, an image description, "
                    "and text). Give an integrated analysis."
                )
            },
            {"role": "user", "content": combined}
        ],
        max_tokens=500
    )

    return {
        "inputs": {
            "audio": audio_path is not None,
            "image": image_path is not None,
            "text": text_input is not None
        },
        "context": combined,
        "analysis": response.choices[0].message.content
    }

Use cases: Multimodal assistants, support systems, mixed-content analysis.

Pattern 6: Generation → Analysis (closed loop)

You generate content and then analyze it to validate quality.

def generate_and_validate(prompt: str) -> dict:
    """Generate an image and validate it automatically."""
    # Step 1: Generate the image
    image_response = client.images.generate(
        model="dall-e-3",
        prompt=prompt,
        size="1024x1024",
        n=1
    )
    image_url = image_response.data[0].url

    # Step 2: Analyze the generated image
    validation = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        f"The request was to generate: '{prompt}'\n\n"
                        f"Does the image meet what was requested?\n"
                        f"Reply with JSON: "
                        f'{{"meets_request": true/false, "reason": "...", '
                        f'"quality": 1-10}}'
                    )
                },
                {"type": "image_url", "image_url": {"url": image_url}}
            ]
        }],
        max_tokens=200,
        temperature=0
    ).choices[0].message.content

    return {
        "prompt": prompt,
        "image_url": image_url,
        "validation": json.loads(validation)
    }

Use cases: Automatic QA of generated content, generation pipelines with validation.

Designing Pipelines: Principles

1. Every step has a clear purpose

❌ BAD: Audio → Text → LLM → LLM → LLM → Text
   (why 3 LLM steps?)

✅ GOOD: Audio → Whisper(transcribe) → GPT-4o-mini(summarize) → TTS(audio)
   (each step does something different)

2. Use the cheapest model that works

# For simple tasks: gpt-4o-mini ($0.15/1M tokens)
# For complex vision: gpt-4o ($2.50/1M tokens)
# For transcription: whisper-1 ($0.006/min)
# For standard TTS: tts-1 ($15/1M chars)

3. Handle errors at every step

def safe_pipeline(audio_path: str) -> dict:
    result = {"status": "ok", "errors": []}

    # Step 1: Transcribe
    try:
        transcript = transcribe_audio(audio_path)
    except Exception as e:
        result["errors"].append(f"Transcription failed: {e}")
        result["status"] = "partial"
        transcript = None

    # Step 2: Summarize (only if step 1 succeeded)
    summary = None
    if transcript:
        try:
            summary = summarize_text(transcript)
        except Exception as e:
            result["errors"].append(f"Summary failed: {e}")
            result["status"] = "partial"

    result["transcript"] = transcript
    result["summary"] = summary
    return result

4. Measure latency and cost

import time

def timed_pipeline(func, *args, **kwargs):
    start = time.time()
    result = func(*args, **kwargs)
    elapsed = time.time() - start
    return {
        "result": result,
        "latency_seconds": round(elapsed, 2)
    }

Troubleshooting

Problem 1: Slow pipeline (>30 seconds)

Cause: Multiple sequential API calls.

Solution: Parallelize independent steps with asyncio or concurrent.futures.

from concurrent.futures import ThreadPoolExecutor

def parallel_analysis(image_path: str, audio_path: str):
    with ThreadPoolExecutor(max_workers=2) as executor:
        image_future = executor.submit(describe_image, image_path)
        audio_future = executor.submit(transcribe_audio, audio_path)

        image_desc = image_future.result()
        transcript = audio_future.result()

    return {"image": image_desc, "audio": transcript}

Problem 2: High cost in complex pipelines

Cause: Using gpt-4o for everything.

Solution: Use economical models for intermediate steps:

• Classification/routing → gpt-4o-mini
• Deep analysis → gpt-4o
• Transcription → whisper-1

Problem 3: Inconsistent result between runs

Cause: temperature > 0 in extraction steps.

Solution: Use temperature=0 in steps that require determinism (JSON extraction, classification).

Exercises

Exercise 1: Pipeline image → description → audio (Easy)

Create a pipeline that: takes an image → describes it with vision → generates audio of the description.

def image_to_audio_description(image_path: str) -> dict:
    # 1. Describe the image
    description = vision_plus_llm(
        image_path,
        "Describe this image in 3 clear, concise sentences."
    )

    # 2. Generate audio
    response = client.audio.speech.create(
        model="tts-1",
        voice="nova",
        input=description
    )
    audio_path = "image_description.mp3"
    response.stream_to_file(audio_path)

    return {
        "description": description,
        "audio": audio_path
    }

Exercise 2: Modality router (Medium)

Create a function that detects whether a file is an image, audio or text, and applies the right processing.

from pathlib import Path

IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".gif", ".webp"}
AUDIO_EXTENSIONS = {".mp3", ".wav", ".m4a", ".mp4", ".webm"}
TEXT_EXTENSIONS = {".txt", ".md", ".csv"}

def route_and_process(file_path: str, prompt: str = "Analyze this content.") -> dict:
    ext = Path(file_path).suffix.lower()

    if ext in IMAGE_EXTENSIONS:
        result = vision_plus_llm(file_path, prompt)
        return {"modality": "image", "result": result}

    elif ext in AUDIO_EXTENSIONS:
        result = audio_to_analysis(file_path, prompt)
        return {"modality": "audio", "result": result}

    elif ext in TEXT_EXTENSIONS:
        with open(file_path, "r") as f:
            text = f.read()
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": prompt},
                {"role": "user", "content": text}
            ],
            max_tokens=500
        )
        return {"modality": "text", "result": response.choices[0].message.content}

    else:
        return {"modality": "unknown", "result": f"Unsupported format: {ext}"}

Exercise 3: Pipeline with fallback (Medium)

Create an image-analysis pipeline that tries GPT-4o first, and if it fails (error, timeout), uses gpt-4o-mini as a fallback.

import time

def analyze_with_fallback(image_path: str, prompt: str) -> dict:
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    models = [("gpt-4o", "primary"), ("gpt-4o-mini", "fallback")]

    for model, tier in models:
        try:
            start = time.time()
            response = client.chat.completions.create(
                model=model,
                messages=[{
                    "role": "user",
                    "content": [
                        {"type": "text", "text": prompt},
                        {"type": "image_url", "image_url": {
                            "url": f"data:image/jpeg;base64,{image_data}"
                        }}
                    ]
                }],
                max_tokens=500,
                timeout=30
            )
            return {
                "model_used": model,
                "tier": tier,
                "result": response.choices[0].message.content,
                "latency": round(time.time() - start, 2)
            }
        except Exception as e:
            if tier == "fallback":
                raise RuntimeError(f"All models failed: {e}")
            continue

Exercise 4: Multi-input pipeline (Hard)

Create a function that accepts a list of files (mixing images, audio and text) and returns an integrated analysis of all of them.

def analyze_multiple_inputs(file_paths: list[str]) -> dict:
    contexts = []

    for path in file_paths:
        ext = Path(path).suffix.lower()

        if ext in IMAGE_EXTENSIONS:
            desc = vision_plus_llm(path, "Describe this image in 2 sentences.")
            contexts.append(f"[Image: {Path(path).name}] {desc}")

        elif ext in AUDIO_EXTENSIONS:
            with open(path, "rb") as f:
                text = client.audio.transcriptions.create(
                    model="whisper-1", file=f
                ).text
            contexts.append(f"[Audio: {Path(path).name}] {text}")

        elif ext in TEXT_EXTENSIONS:
            with open(path, "r") as f:
                text = f.read()[:2000]  # Limit the length
            contexts.append(f"[Text: {Path(path).name}] {text}")

    combined = "\n\n".join(contexts)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": (
                f"Analyze all this information and generate:\n"
                f"1. Integrated summary\n"
                f"2. Connections between the different inputs\n"
                f"3. Main insights\n\n{combined}"
            )
        }],
        max_tokens=500
    )

    return {
        "num_inputs": len(file_paths),
        "modalities": [Path(p).suffix for p in file_paths],
        "analysis": response.choices[0].message.content
    }

Summary

In this capsule you learned:

• Multimodal combinations can be by input (multiple modalities into the same model), by pipeline (chaining models), or by output (generating in multiple modalities)
• The 6 main patterns: Vision+LLM, Audio→Text→LLM, Text→LLM→Image, Document→Extraction, Full pipeline, Generation→Analysis
• The design principles: every step with a clear purpose, use the cheapest possible model, handle errors per step, measure latency
• Parallelizing independent steps reduces latency significantly
• The modality router (detect type → apply pipeline) is the core of the multimodal Classifier

Next capsule: The model landscape — a detailed comparison of GPT-4V vs Claude 3 vs Gemini.

Additional Resources

1. OpenAI API Reference — Complete reference for all the APIs
2. LangChain Multimodal — Multimodal pipelines with LangChain
3. OpenAI Cookbook — Practical combination examples
4. asyncio Documentation — For parallelizing pipelines
5. concurrent.futures — Threading for I/O-bound tasks
6. Building Multimodal AI Applications — DeepLearning.AI course