Module 7: Use Cases

5. Multi-Modality Combinations

Description

The previous capsules covered individual patterns: Document Q&A, Image Analysis, Video Frames. But the real power of multimodal AI appears when you combine modalities in a single pipeline: meeting audio + PDF slides → integrated minutes; document + voice question → audio answer; image + text + audio → unified analysis.

Why it matters: In production, you rarely use a single modality. A meeting notes system combines audio transcription, presentation analysis, and text generation. A medical assistant combines X-ray images, doctor's notes (audio), and clinical history (text). An educational system combines class video, slides, and transcription to generate study materials. Mastering combinations is what differentiates a script with an API from a real multimodal system.

Connection with the module: This capsule integrates what you saw in capsules 02-04 and connects directly with the Use Case Selector (capsule 08), which needs to detect multi-modal inputs and combine pipelines.

Taxonomy of Combinations

By integration type

Type	Description	Example
Multi-modal input	Multiple modalities as input to a single model	Image + question → GPT-4o → answer
Sequential pipeline	Output of one model as input to another	Audio → Whisper → text → GPT-4o → summary → TTS → audio
Parallel pipeline	Process modalities in parallel and combine	Audio → transcription ∥ PDF → extraction → combine → LLM
Multi-modal output	A pipeline generates outputs in several modalities	Document → text analysis + chart + audio summary

The 5 most-used patterns

1. Meeting + Document  → Integrated minutes
2. Image + Question    → Contextual answer
3. Document → Text + Image + Audio (multi-modal output)
4. Video + Audio → Complete analysis
5. Tri-modal: Text + Image + Audio → Unified answer

Pattern 1: Meeting + Document

A team has a meeting where they discuss a document (contract, proposal, report). The system takes the meeting recording + the document's PDF and generates minutes that connect both.

Visual pipeline

┌─────────────┐                    ┌─────────────┐
│   Meeting   │──▶ Whisper ──▶     │             │
│   audio     │          Transcr.  │   Combine   │
└─────────────┘                    │   contexts  │──▶ LLM ──▶ Integrated
┌─────────────┐                    │             │            minutes
│   Document  │──▶ PyMuPDF ──▶    │             │
│   PDF       │      Text+imgs    └─────────────┘
└─────────────┘

Complete implementation

from openai import OpenAI
import fitz
import base64

client = OpenAI()

def transcribe_audio(audio_path: str) -> str:
    with open(audio_path, "rb") as f:
        response = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="text"
        )
    return response


def extract_text_from_pdf(pdf_path: str) -> str:
    doc = fitz.open(pdf_path)
    text = "\n\n".join(doc[i].get_text() for i in range(len(doc)))
    doc.close()
    return text


def extract_images_from_pdf(pdf_path: str) -> list[str]:
    doc = fitz.open(pdf_path)
    image_descriptions = []

    for page_num in range(len(doc)):
        page = doc[page_num]
        images = page.get_images(full=True)
        for img_info in images:
            xref = img_info[0]
            base_image = doc.extract_image(xref)
            b64 = base64.b64encode(base_image["image"]).decode()

            desc_response = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{
                    "role": "user",
                    "content": [
                        {"type": "text", "text": "Describe this image/diagram in 2-3 sentences."},
                        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
                    ]
                }],
                max_tokens=150
            )
            image_descriptions.append(
                f"[Image p.{page_num + 1}]: {desc_response.choices[0].message.content}"
            )

    doc.close()
    return image_descriptions


def meeting_with_document(audio_path: str, doc_path: str) -> dict:
    transcript = transcribe_audio(audio_path)
    doc_text = extract_text_from_pdf(doc_path)
    doc_images = extract_images_from_pdf(doc_path)

    images_text = "\n".join(doc_images) if doc_images else "No images."

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "Generate professional meeting minutes. "
                    "Connect what was discussed in the meeting with the document's content. "
                    "Structure: Executive summary, Points discussed, Decisions made, "
                    "Pending actions, References to the document."
                )
            },
            {
                "role": "user",
                "content": (
                    f"MEETING TRANSCRIPT:\n{transcript[:6000]}\n\n"
                    f"DOCUMENT CONTENT:\n{doc_text[:4000]}\n\n"
                    f"DOCUMENT IMAGES/DIAGRAMS:\n{images_text[:2000]}"
                )
            }
        ],
        max_tokens=1000,
        temperature=0.3
    )

    return {
        "minutes": response.choices[0].message.content,
        "transcript_length": len(transcript),
        "document_pages": doc_text.count("\n\n"),
        "images_analyzed": len(doc_images)
    }

Usage:

result = meeting_with_document(
    "team_meeting.mp3",
    "project_proposal.pdf"
)
print(result["minutes"])

Pattern 2: Image + Contextual Question

Beyond "describe this image", the user provides textual context that changes how to interpret the image.

def contextual_image_qa(
    image_path: str,
    question: str,
    context: str = ""
) -> dict:
    with open(image_path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()

    prompt = question
    if context:
        prompt = f"Context: {context}\n\nQuestion: {question}"

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": prompt},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
            ]
        }],
        max_tokens=500,
        temperature=0.2
    )

    return {
        "answer": response.choices[0].message.content,
        "context_used": bool(context)
    }

With conversation history

class ImageConversation:
    def __init__(self, image_path: str):
        self.image_path = image_path
        with open(image_path, "rb") as f:
            self.image_b64 = base64.b64encode(f.read()).decode()
        self.messages: list[dict] = []

    def ask(self, question: str) -> str:
        if not self.messages:
            self.messages.append({
                "role": "user",
                "content": [
                    {"type": "text", "text": question},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{self.image_b64}"}}
                ]
            })
        else:
            self.messages.append({"role": "user", "content": question})

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=self.messages,
            max_tokens=500
        )

        answer = response.choices[0].message.content
        self.messages.append({"role": "assistant", "content": answer})
        return answer

Pattern 3: Multi-Modal Output (Document → Text + Image + Audio)

An input document generates three different outputs:

def multimodal_output_pipeline(doc_path: str) -> dict:
    doc_text = extract_text_from_pdf(doc_path)

    summary_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": f"Summarize this document in 3-5 key points:\n\n{doc_text[:6000]}"
        }],
        max_tokens=400
    )
    text_summary = summary_response.choices[0].message.content

    viz_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": (
                f"Based on this summary, generate a prompt for DALL-E "
                f"that illustrates the main concept as a professional infographic:\n\n{text_summary}"
            )
        }],
        max_tokens=200
    )
    image_prompt = viz_response.choices[0].message.content

    image_response = client.images.generate(
        model="dall-e-3",
        prompt=image_prompt,
        size="1024x1024",
        n=1
    )
    image_url = image_response.data[0].url

    audio_response = client.audio.speech.create(
        model="tts-1",
        voice="nova",
        input=text_summary
    )
    audio_path = "/tmp/document_summary.mp3"
    audio_response.stream_to_file(audio_path)

    return {
        "text_summary": text_summary,
        "image_url": image_url,
        "audio_path": audio_path
    }

Pattern 4: Complete Video + Audio

Combines the visual frame analysis (capsule 04) with the transcription of the video's audio:

import subprocess

def extract_audio_from_video(video_path: str, output_path: str = "/tmp/video_audio.mp3") -> str:
    subprocess.run([
        "ffmpeg", "-i", video_path, "-vn", "-acodec", "libmp3lame",
        "-y", output_path
    ], capture_output=True, check=True)
    return output_path


def full_video_analysis(video_path: str, target_frames: int = 15) -> dict:
    audio_path = extract_audio_from_video(video_path)
    transcript = transcribe_audio(audio_path)

    import cv2
    cap = cv2.VideoCapture(video_path)
    fps = cap.get(cv2.CAP_PROP_FPS)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    duration = total_frames / fps
    interval = duration / target_frames

    frames_descriptions = []
    for i in range(target_frames):
        ts = i * interval
        cap.set(cv2.CAP_PROP_POS_FRAMES, int(ts * fps))
        ret, frame = cap.read()
        if not ret:
            break

        frame_path = f"/tmp/full_analysis_frame_{i}.jpg"
        cv2.imwrite(frame_path, frame)

        with open(frame_path, "rb") as f:
            b64 = base64.b64encode(f.read()).decode()

        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe this scene in 1-2 sentences."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
                ]
            }],
            max_tokens=100
        )

        minutes = int(ts // 60)
        seconds = int(ts % 60)
        frames_descriptions.append(f"{minutes}:{seconds:02d} - {response.choices[0].message.content}")

    cap.release()

    visual_timeline = "\n".join(frames_descriptions)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": (
                "Generate a complete analysis of this video integrating the visual with the spoken.\n\n"
                f"VISUAL ANALYSIS (frames):\n{visual_timeline}\n\n"
                f"AUDIO TRANSCRIPT:\n{transcript[:5000]}\n\n"
                "Structure: Summary, Timeline with key moments, Main topics, Conclusions."
            )
        }],
        max_tokens=800
    )

    return {
        "integrated_analysis": response.choices[0].message.content,
        "visual_frames": len(frames_descriptions),
        "transcript_length": len(transcript),
        "duration_seconds": round(duration, 2)
    }

Pattern 5: Tri-Modal (Text + Image + Audio)

The user sends three simultaneous inputs and the system integrates everything:

def tri_modal_analysis(
    text_input: str,
    image_path: str,
    audio_path: str
) -> dict:
    transcript = transcribe_audio(audio_path)

    with open(image_path, "rb") as f:
        image_b64 = base64.b64encode(f.read()).decode()

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        "Analyze these three inputs and generate an integrated answer:\n\n"
                        f"USER'S TEXT:\n{text_input}\n\n"
                        f"TRANSCRIBED AUDIO:\n{transcript}\n\n"
                        "ATTACHED IMAGE (see below):\n"
                        "Connect the information from the three sources into a coherent answer."
                    )
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}
                }
            ]
        }],
        max_tokens=600,
        temperature=0.3
    )

    return {
        "integrated_response": response.choices[0].message.content,
        "inputs": {
            "text_length": len(text_input),
            "transcript_length": len(transcript),
            "image_path": image_path
        }
    }

Generic Multi-Modal Pipeline

To avoid repeating code, a configurable pipeline:

class MultiModalPipeline:
    def __init__(self):
        self.processors = {
            "text": self._process_text,
            "image": self._process_image,
            "audio": self._process_audio,
            "pdf": self._process_pdf
        }

    def process(self, inputs: dict[str, str], task: str = "analyze") -> dict:
        processed = {}
        for input_type, input_path in inputs.items():
            if input_type in self.processors:
                processed[input_type] = self.processors[input_type](input_path)

        return self._combine(processed, task)

    def _process_text(self, text: str) -> str:
        return text

    def _process_image(self, path: str) -> str:
        with open(path, "rb") as f:
            b64 = base64.b64encode(f.read()).decode()
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe this image in detail."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
                ]
            }],
            max_tokens=300
        )
        return response.choices[0].message.content

    def _process_audio(self, path: str) -> str:
        return transcribe_audio(path)

    def _process_pdf(self, path: str) -> str:
        return extract_text_from_pdf(path)

    def _combine(self, processed: dict, task: str) -> dict:
        context_parts = []
        for input_type, content in processed.items():
            context_parts.append(f"[{input_type.upper()}]:\n{content[:3000]}")

        context = "\n\n".join(context_parts)

        task_prompts = {
            "analyze": "Analyze all the information and generate an integrated summary.",
            "compare": "Compare the information from the different sources.",
            "summarize": "Summarize the key points of all the sources.",
            "qa": "Answer any question implied in the inputs."
        }

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{
                "role": "user",
                "content": f"{context}\n\n{task_prompts.get(task, task_prompts['analyze'])}"
            }],
            max_tokens=600
        )

        return {
            "result": response.choices[0].message.content,
            "inputs_processed": list(processed.keys()),
            "task": task
        }

Usage:

pipeline = MultiModalPipeline()

result = pipeline.process(
    inputs={
        "audio": "meeting.mp3",
        "pdf": "proposal.pdf",
        "image": "diagram.png"
    },
    task="summarize"
)
print(result["result"])

Troubleshooting

Problem 1: Token limit exceeded when combining modalities

Symptom: Error maximum context length exceeded when sending transcript + document + image.

Solution:

def truncate_inputs(inputs: dict[str, str], max_total_chars: int = 15000) -> dict[str, str]:
    total = sum(len(v) for v in inputs.values())
    if total <= max_total_chars:
        return inputs

    per_input = max_total_chars // len(inputs)
    return {k: v[:per_input] for k, v in inputs.items()}

Problem 2: Long audio (>25 MB) can't be transcribed

Symptom: Whisper rejects files larger than 25 MB.

Solution: Split the audio into segments:

from pydub import AudioSegment

def transcribe_long_audio(audio_path: str, segment_minutes: int = 10) -> str:
    audio = AudioSegment.from_file(audio_path)
    segment_ms = segment_minutes * 60 * 1000
    segments = [audio[i:i + segment_ms] for i in range(0, len(audio), segment_ms)]

    transcripts = []
    for i, segment in enumerate(segments):
        segment_path = f"/tmp/audio_segment_{i}.mp3"
        segment.export(segment_path, format="mp3")
        transcript = transcribe_audio(segment_path)
        transcripts.append(transcript)

    return "\n\n".join(transcripts)

Problem 3: Incoherent results when combining sources

Symptom: The summary doesn't connect audio with document well.

Solution: Use a more structured prompt:

system_prompt = (
    "Integrate information from multiple sources. For each point, indicate which source it comes from. "
    "If there are contradictions between sources, flag them explicitly. "
    "Structure your answer with: 1) Points in common, 2) Information unique to each source, "
    "3) Contradictions (if any), 4) Integrated conclusion."
)

Problem 4: High latency in multi-step pipelines

Symptom: The pipeline takes 30+ seconds.

Solution: Parallelize independent steps:

import asyncio
from openai import AsyncOpenAI

async def process_inputs_parallel(audio_path: str, doc_path: str, image_path: str) -> dict:
    aclient = AsyncOpenAI()

    async def transcribe_async():
        with open(audio_path, "rb") as f:
            return await aclient.audio.transcriptions.create(model="whisper-1", file=f, response_format="text")

    async def extract_doc():
        return extract_text_from_pdf(doc_path)

    async def describe_image():
        with open(image_path, "rb") as f:
            b64 = base64.b64encode(f.read()).decode()
        response = await aclient.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": "Describe this image."},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
                ]
            }],
            max_tokens=200
        )
        return response.choices[0].message.content

    transcript, doc_text, img_desc = await asyncio.gather(
        transcribe_async(), extract_doc(), describe_image()
    )

    return {"transcript": transcript, "doc_text": doc_text, "image_description": img_desc}

Exercises

Exercise 1: Document → image → analysis pipeline

Extract the first image from a PDF, generate a variation with DALL-E 3, and compare both with Vision.

See solution

def doc_image_variation_pipeline(pdf_path: str) -> dict:
    doc = fitz.open(pdf_path)
    first_image = None

    for page_num in range(len(doc)):
        images = doc[page_num].get_images(full=True)
        if images:
            xref = images[0][0]
            base_image = doc.extract_image(xref)
            first_image = base_image["image"]
            break
    doc.close()

    if not first_image:
        return {"error": "No images found in the PDF"}

    b64_original = base64.b64encode(first_image).decode()

    desc_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image so it can be recreated with DALL-E."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64_original}"}}
            ]
        }],
        max_tokens=200
    )
    description = desc_response.choices[0].message.content

    variation = client.images.generate(
        model="dall-e-3",
        prompt=f"Improved and professional version of: {description}",
        size="1024x1024",
        n=1
    )

    return {
        "original_description": description,
        "variation_url": variation.data[0].url,
        "source_pdf": pdf_path
    }

Exercise 2: Complete meeting notes system

Extend meeting_with_document so it generates: minutes (text), executive summary (short text), and an audio summary (TTS).

See solution

def complete_meeting_notes(audio_path: str, doc_path: str) -> dict:
    base_result = meeting_with_document(audio_path, doc_path)
    minutes = base_result["minutes"]

    exec_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": f"Generate a 3-line executive summary of these minutes:\n\n{minutes}"
        }],
        max_tokens=150
    )
    executive_summary = exec_response.choices[0].message.content

    audio_response = client.audio.speech.create(
        model="tts-1",
        voice="nova",
        input=executive_summary
    )
    audio_output = "/tmp/meeting_summary.mp3"
    audio_response.stream_to_file(audio_output)

    return {
        "full_minutes": minutes,
        "executive_summary": executive_summary,
        "audio_summary_path": audio_output
    }

Exercise 3: Tri-modal with automatic detection

Create a function that takes a list of paths, automatically detects the type of each one (text, image, audio, PDF), and applies the corresponding multi-modal pipeline.

See solution

from pathlib import Path

def auto_multimodal_pipeline(paths: list[str], task: str = "analyze") -> dict:
    type_map = {
        ".pdf": "pdf",
        ".png": "image", ".jpg": "image", ".jpeg": "image", ".webp": "image",
        ".mp3": "audio", ".wav": "audio", ".m4a": "audio",
        ".txt": "text", ".md": "text"
    }

    inputs = {}
    for path in paths:
        ext = Path(path).suffix.lower()
        input_type = type_map.get(ext)
        if input_type:
            if input_type == "text":
                with open(path, "r") as f:
                    inputs[input_type] = f.read()
            else:
                inputs[input_type] = path

    if not inputs:
        return {"error": "No valid inputs detected"}

    pipeline = MultiModalPipeline()
    return pipeline.process(inputs, task)

Additional Resources

OpenAI Audio API — Whisper and TTS
OpenAI Vision — Multi-image in one request
DALL-E 3 API — Image generation
FFmpeg Documentation — Audio extraction from video

← Previous4. Video: Frames + LLM Next →6. Production Patterns