Module 6: Multimodal RAG

1. Introduction: Multimodal RAG

Next →

Description

RAG (Retrieval-Augmented Generation) was a huge leap for LLMs: instead of relying only on their trained knowledge, the model searches a corpus of documents and answers with real context. But classic RAG has a massive blind spot: it only searches text. If your corpus includes diagrams, charts, screenshots, product photos, or PDFs with figures, classic RAG ignores them entirely.

Multimodal RAG solves this. It indexes text and images in the same vector space (or in coordinated spaces) and retrieves relevant chunks of both types when the user asks a question. You can ask "find documents that show a microservices architecture diagram" and the system retrieves both the paragraphs mentioning microservices and the visual diagrams that illustrate them.

Why it matters: Multimodal RAG is the heart of the Document Analyzer's Q&A that you'll build in Module 8. Without it, your project would only extract isolated data. With multimodal RAG, it answers complex questions using text AND figures from the original document. In production, this is the difference between a chatbot that says "I can't find information about the diagram" and one that shows you the relevant diagram alongside the explanation.

Connection with previous modules: Modules 1-5 gave you the individual pieces: vision (M2), document processing (M3), image generation (M4), audio (M5). This module assembles them into a search-and-answer system that combines everything. It's where you move from "I can analyze an image" to "I can search thousands of documents with images and answer questions about them".

Where Are We in the Guide?

Context

This guide has 8 modules organized into 3 phases:

Phase 1: Multimodal Fundamentals (Modules 1-3)
├── Module 1: Introduction to Multimodal AI
├── Module 2: Vision + LLMs
└── Module 3: Document Understanding

Phase 2: Generation and Audio (Modules 4-5)
├── Module 4: Image Generation
└── Module 5: Audio Processing

Phase 3: RAG, Use Cases and Project (Modules 6-8)
├── Module 6: Multimodal RAG    ← YOU ARE HERE
├── Module 7: Use Cases
└── Module 8: Final Project — Multimodal Document Analyzer

Total estimated duration of the guide: 6-7 hours (self-paced).

Where are we headed?

So far you worked with individual modalities: sending an image to an LLM, transcribing audio, generating images. This module connects those skills with intelligent search. Module 7 applies production patterns, and Module 8 integrates everything into the Document Analyzer.

The progression within Phase 3:

  1. Multimodal RAG (this module) — search and answer over a corpus with text + images
  2. Use Cases (module 7) — production patterns: document Q&A, video, monitoring
  3. Document Analyzer (module 8) — the final project that integrates vision + RAG + audio

What Multimodal RAG Is

Classic RAG (text only)

The classic RAG pipeline works like this:

1. INDEX:
   Documents → text chunks → embeddings → vector store

2. QUERY:
   User question → embedding of the question
   → search similar chunks in vector store
   → send chunks to the LLM as context
   → LLM generates answer

Example:
   "What is the returns policy?"
   → retrieves chunks from the returns manual
   → LLM answers with the manual's info

This works well when documents are pure text. But the real world isn't pure text.

The problem: real documents have images

Think about these scenarios:

  • A technical manual with architecture diagrams — the text says "see Figure 3", but classic RAG doesn't know what's in Figure 3
  • A financial report with charts — the trends are in the charts, not the text
  • A product catalog with photos — the user asks "do you have anything like this image?"
  • An academic paper with tables and figures — half the information is in visual elements

If you only index the text, you're losing between 30% and 60% of the information in these documents.

Multimodal RAG: the solution

1. INDEX:
   Documents → text chunks → text embeddings          → vector store
                                                          ↑
             → images → image descriptions/embeddings  → vector store

2. QUERY:
   Question (text or image)
   → embedding of the question
   → search across text chunks AND image chunks
   → merge results (re-ranking)
   → send context (text + image descriptions) to the LLM
   → LLM generates an answer informed by BOTH modalities

The key is that text and images coexist in the same index. When you search, you retrieve whatever is most relevant regardless of whether it's a paragraph or an image.

Two main strategies

There are two ways to do multimodal RAG, and in this module we cover both:

Strategy 1: Textual description of images

Image → Vision API → textual description → text embedding → vector store

You convert each image into a description using a vision-capable model (GPT-4o, Claude 3). Then you treat that description like any other text chunk. Simple, compatible with any vector store, but you lose visual information that the text doesn't capture.

Strategy 2: Native multimodal embeddings

Image → CLIP/multimodal model → image embedding → vector store
Text  → CLIP/multimodal model → text embedding  → vector store

You use a model like CLIP that generates embeddings in a shared space for text and images. An embedding of "dog" and an embedding of a photo of a dog are close in the vector space. More accurate, but it requires specific models.

Why Multimodal RAG Matters

The classic RAG gap

Scenario: 200-page technical manual

Classic RAG:
  Question: "How does module A connect to module B?"
  → Retrieves paragraphs that mention modules A and B
  → Answer: "Module A connects to module B via a REST API"
  → MISSING: The architecture diagram showing exactly the flow

Multimodal RAG:
  Same question
  → Retrieves paragraphs AND the connection diagram
  → Answer: "Module A connects to module B via a REST API.
     The diagram on page 42 shows that communication goes through
     an API Gateway with OAuth 2.0 authentication"
  → COMPLETE: Text + visual context

Production use cases

  1. Q&A over technical documentation: Manuals with diagrams, API documentation with screenshots. The user asks and gets answers that reference both text and figures.

  2. Report analysis: Financial or business reports with charts. "How were Q3 sales?" retrieves the summary paragraph AND the trend chart.

  3. Customer support with catalogs: "Do you have a product similar to this?" The user sends an image and the system searches the catalog by visual similarity.

  4. Academic research: Papers with figures, tables and equations. "What results did they get in the experiment?" retrieves the results section AND the relevant figures.

  5. Legal and compliance: Contracts with signatures, stamps and tables. Multimodal RAG indexes both the textual clauses and the visual elements.

  6. E-commerce: Product search combining textual description ("elegant red dress") with a visual reference (example photo).

Context data

  • 85% of enterprise documents contain visual elements (McKinsey 2024)
  • A RAG that only indexes text loses 30-60% of the information in documents with figures
  • Multimodal Q&A systems show +40% accuracy in answers over technical documents vs text-only RAG
  • ChromaDB reports that >50% of its users index some form of multimodal data

Module Objectives

By the end of this module you'll be able to:

  • ✅ Generate embeddings for images using two strategies: textual description with Vision and native embeddings with CLIP
  • ✅ Build a multimodal index in ChromaDB that contains text chunks and image representations
  • ✅ Implement hybrid retrieval that searches by text and image and merges results with re-ranking
  • ✅ Process mixed documents (PDFs with figures) preserving the text-image association
  • ✅ Use LangChain to build a complete multimodal RAG pipeline
  • ✅ Identify limitations (cost, latency, quality) and apply optimizations (caching, batch, local CLIP)
  • ✅ Build a functional basic multimodal RAG project that indexes, searches and answers

Professional objective

When someone tells you "we need a chatbot that answers questions about our technical manuals with diagrams", you'll know: how to index the documents (text + images), which embedding strategy to use (description vs CLIP), how to implement search (hybrid with re-ranking), how much it will cost (embeddings + Vision + LLM), and what limitations it will have. That level of judgment is what separates a developer who "knows how to use ChatGPT" from one who designs production AI systems.

Module Roadmap

Capsule map

#CapsuleWhat you'll learnType
01Introduction (this one)Multimodal RAG, context, objectives, setupIntro
02Image embeddingsCLIP, Vision+embedding, cosine similarity, ChromaDB introTechnical
03Multimodal indexingChromaDB setup, indexing text+images, metadata, persistenceTechnical
04Hybrid retrievalText+image search, result fusion, RRF, weightingTechnical
05Documents with imagesChunking PDFs with figures, associating text-image, pipelinesTechnical
06Implementation with LangChainDocument loaders, retrievers, multimodal RAG chainsTechnical
07Limitations and optimizationCosts, latency, caching, batch, monitoringTechnical
08Project: basic multimodal RAGComplete system: index, search, answer with text+imagesProject

Learning flow

First you'll understand how to represent images as vectors (capsule 02) — this is the fundamental piece. Then you'll learn to store those vectors alongside text in an index (capsule 03). After that you'll implement the search that combines both (capsule 04). You'll see how to process real documents that mix text and figures (capsule 05). Then you'll use LangChain to simplify the implementation (capsule 06). You'll identify limitations and how to optimize (capsule 07). And finally you'll integrate everything into a functional project (capsule 08).

The progression is: representation → storage → search → real documents → framework → optimization → project.

Estimated module duration: 50-60 minutes.

Connection with the Project

This module's project: Basic Multimodal RAG

The capsule 08 project is a RAG system that:

Input:
  - PDF documents with text and images
  - User question (text)

Process:
  1. Extract text and images from the PDF
  2. Generate embeddings (text directly, images via description)
  3. Store in ChromaDB
  4. Search relevant chunks (text + image)
  5. Generate an answer with the LLM using multimodal context

Output:
  - Textual answer based on text AND images from the document
  - Sources: which chunks and which pages were used

Connection with the final project: Document Analyzer

Module 2: Vision → analyze individual images
Module 3: Documents → extract data from PDFs
Module 6: Multimodal RAG → search and answer over a corpus with images
    ↓
Module 8: Document Analyzer → combines vision + documents + RAG + audio

Your multimodal RAG from module 6 becomes the Document Analyzer's Q&A engine. When a user uploads a PDF and asks questions, it's your multimodal RAG pipeline that finds the answer.

Prerequisites

From previous modules

This module assumes you completed (or at least read) the previous modules:

ModuleWhat you need from it
M1: IntroductionUnderstand modalities, models, formats (Base64, APIs)
M2: VisionSend images to GPT-4o/Claude, get descriptions
M3: DocumentsExtract text and images from PDFs with PyMuPDF
M4: GenerationConcept of embeddings and vector spaces
M5: AudioSequential data processing pipeline

Technical knowledge

  • Intermediate Python: Functions, classes, file handling, basic async/await
  • LLM APIs: Experience with the OpenAI API (chat completions, embeddings)
  • Concept of embeddings: What a vector is, cosine similarity (we go deeper in capsule 02)
  • PDFs: Having worked with PyMuPDF or similar tools

If you don't have the prerequisites

What you're missingResource
Modules 1-5 of this guideComplete at least M1, M2 and M3 before continuing
Concept of embeddingsOpenAI Embeddings Guide
Vector databasesChromaDB Getting Started
PyMuPDFCapsule 05 of Module 3 of this guide

Technical Setup

Dependencies

python --version  # 3.11+ required

pip install openai>=1.0.0 pillow python-dotenv

pip install langchain langchain-openai langchain-community

pip install chromadb

pip install sentence-transformers

# Optional: CLIP for native embeddings
pip install transformers torch

# Optional: PDF processing
pip install pymupdf

Environment variables

# .env
OPENAI_API_KEY=sk-...

# Optional
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=...

Quick verification

from openai import OpenAI
import chromadb
from dotenv import load_dotenv

load_dotenv()

oai = OpenAI()
response = oai.embeddings.create(
    model="text-embedding-3-small",
    input="Embeddings test for multimodal RAG"
)
print(f"Embedding dimension: {len(response.data[0].embedding)}")

chroma = chromadb.Client()
collection = chroma.create_collection("test")
collection.add(
    documents=["Test text"],
    ids=["test_1"]
)
results = collection.query(query_texts=["test"], n_results=1)
print(f"ChromaDB works: {results['documents'][0][0]}")

chroma.delete_collection("test")
print("Setup complete — ready for module 6")

If you see the three output lines with no errors, your environment is ready.

Estimated costs

OperationApproximate cost
Embeddings (text-embedding-3-small)~$0.02 per 1M tokens
Vision (gpt-4o-mini, describing images)~$0.01-0.03 per image
LLM for answers (gpt-4o)~$0.01-0.05 per query
ChromaDBFree (local)
CLIP (Hugging Face)Free (local)

Total estimated to complete the module: $0.50-2.00 USD.

Tip: Use gpt-4o-mini to describe images during development. Only switch to gpt-4o if you need higher description quality. ChromaDB and CLIP are free and local.

Recommended file structure

ai-multimodal-guide/
├── .env
├── module_06/
│   ├── embeddings.py          # Embedding functions (capsule 02)
│   ├── indexer.py             # Multimodal indexing (capsule 03)
│   ├── retriever.py           # Hybrid retrieval (capsule 04)
│   ├── document_processor.py  # Process docs with images (capsule 05)
│   ├── langchain_rag.py       # LangChain implementation (capsule 06)
│   ├── rag_pipeline.py        # Project: complete pipeline (capsule 08)
│   ├── chroma_db/             # ChromaDB persistence
│   └── test_docs/
│       ├── sample.pdf         # Test PDF with images
│       ├── diagram.png        # Test image
│       └── product.jpg        # Another test image
└── ...

Boundaries: What This Module Does NOT Cover

  • ❌ RAG with video — We focus on text + static images. Video is covered in M7.
  • ❌ RAG with audio — Integration with transcriptions exists but isn't the focus.
  • ❌ Fine-tuning embedding models — We use pre-trained models (CLIP, OpenAI).
  • ❌ Vector databases in production (Pinecone, Weaviate, Qdrant) — We use ChromaDB for simplicity. The concepts apply to any vector store.
  • ❌ Advanced RAG (knowledge graphs, iterative RAG) — We cover the fundamental pipeline.
  • ❌ Formal RAG evaluation (RAGAS, benchmarks) — We mention metrics but not evaluation frameworks.

Mental Model: From Classic RAG to Multimodal RAG

Before diving into the technical capsules, be clear about this mental model:

Classic RAG:
  DATA:     [text, text, text, text]
  SEARCH:   text → text
  CONTEXT:  text only

Multimodal RAG:
  DATA:     [text, text, image_desc, text, image_desc, text]
  SEARCH:   text → text + image_desc
            image → text + image_desc
  CONTEXT:  text + image descriptions + visual references

The additional complexity isn't in the concept (it's the same pipeline), but in:

  1. Representation: How do you turn an image into something searchable?
  2. Indexing: How do you store text and images together?
  3. Retrieval: How do you combine results from different modalities?
  4. Documents: How do you preserve the text-image relationship in a PDF?

Each capsule of the module answers one of these questions.

Evidence of Success

By the end of this module, you'll know you succeeded if:

  • ✅ You can generate embeddings for an image (via textual description OR via CLIP) and compare them with text embeddings
  • ✅ Your ChromaDB index contains both text chunks and image representations with correct metadata
  • ✅ Your hybrid search finds relevant results combining text and image, with re-ranking
  • ✅ You can process a PDF with figures and preserve the text-image association in the index
  • ✅ Your LangChain pipeline works end-to-end: document → indexing → query → answer
  • ✅ You know the real costs and how to apply at least 3 optimizations (caching, batch, local CLIP)
  • ✅ Your multimodal RAG project works: index documents, search, and get answers with text and image context

Quick self-assessment

If you can answer these questions before starting, you probably already have the foundation. If not, this module will answer them for you:

  1. What's the difference between generating an embedding of an image with CLIP vs describing the image and generating an embedding of the text?
  2. Why do you need re-ranking when combining text and image search results?
  3. How much does it cost to describe 500 images with GPT-4o-mini, and how would you optimize it?
  4. What happens when a text chunk says "see Figure 3" but the index doesn't have Figure 3?

Summary

  • Multimodal RAG extends classic RAG to index and search across text + images.
  • The pipeline is: extract → represent (embeddings) → index → search → merge → answer.
  • There are two main strategies: textual description of images (simple) and native multimodal embeddings (more accurate).
  • Use cases include: technical documentation, reports, catalogs, papers, contracts.
  • This module covers 7 technical capsules + 1 project that together build a functional multimodal RAG.
  • It's the direct foundation of the Document Analyzer in Module 8.
  • Main tools: OpenAI API (embeddings + vision), ChromaDB (vector store), LangChain (framework), CLIP (optional native embeddings).

Additional Resources

  1. OpenAI Embeddings Guide — Official embeddings documentation
  2. ChromaDB Documentation — Vector store we use in the module
  3. CLIP Paper (Radford et al.) — The model that connects text and images
  4. LangChain RAG — Official RAG tutorial with LangChain
  5. Multimodal RAG (LlamaIndex) — Alternative implementation reference
  6. Sentence Transformers — Open source embedding models
  7. Pinecone: Multimodal Search — Multimodal search concepts
← Previous8. Project: Audio PipelineNext →2. Image Embeddings