1. Introduction: Capstone Project

Description

This is Module 8 of the Multimodal AI Guide: the final capstone project. Modules 1-7 taught the individual pieces — vision, documents, image generation, audio, RAG and usage patterns. This module combines them into a complete, functional system: Multimodal Document Analyzer. A system that receives documents (PDF, scanned images), processes them automatically, extracts structured information with vision, indexes the content for questions and answers, and optionally generates an audio summary.

Why a capstone project: Mastering individual APIs is necessary but not sufficient. The difference between a developer who "knows how to use GPT-4 Vision" and one who "builds multimodal systems" is the integration: how to connect document processing with vision, vision with RAG, RAG with audio, and everything packaged into a production-ready REST API. This module closes that gap.

What the Document Analyzer does: Imagine you work at a company that receives hundreds of documents a day — invoices, contracts, technical manuals, reports. The Document Analyzer automates their full processing:

1. Receives a PDF or image
2. Detects whether it has extractable text or is scanned (needs OCR/vision)
3. Classifies the document type (invoice, contract, manual, other)
4. Extracts structured data according to the type (date, total, vendor for invoices)
5. Generates an executive summary
6. Indexes the content for later Q&A
7. Answers specific questions about the document
8. Optionally generates the summary in audio (TTS)
9. Exposes everything as a REST API with FastAPI

Result: By the end of the module, you'll have a portfolio-worthy, production-ready system that demonstrates real mastery of multimodal AI.

---

Where Are We in the Guide?

Context

This guide has 8 modules organized into 3 phases:

Phase 1: Multimodal Fundamentals (Modules 1-3)
├── Module 1: Introduction to Multimodal AI          ✅ Completed
├── Module 2: Vision + LLMs                          ✅ Completed
└── Module 3: Document Understanding                  ✅ Completed

Phase 2: Generation and Audio (Modules 4-5)
├── Module 4: Image Generation                        ✅ Completed
└── Module 5: Audio Processing                         ✅ Completed

Phase 3: RAG, Use Cases and Project (Modules 6-8)
├── Module 6: Multimodal RAG                           ✅ Completed
├── Module 7: Use Cases                                ✅ Completed
└── Module 8: Final Project — Document Analyzer        ← YOU ARE HERE

Total duration of the guide: 6-7 hours. Duration of this module: ~90 minutes (the longest, since it's the capstone).

How it all connects

Each previous module contributed a capability that you now integrate:

Module 1 (Fundamentals)     → Input classifier: detect file type
Module 2 (Vision)           → Structured extraction: analyze document images
Module 3 (Documents)        → PDF processing: extract text and images
Module 4 (Image Gen.)       → [Not used directly in this project]
Module 5 (Audio)            → TTS: generate spoken summary
Module 6 (RAG)              → Indexing and Q&A: answer questions about documents
Module 7 (Use Cases)        → Patterns: use-case selector, fallbacks, costs

Module 4 (image generation) is not integrated directly — the Document Analyzer consumes images, it doesn't generate them. But the knowledge of formats, resolution and encoding you learned there does apply to processing document images.

---

Document Analyzer Architecture

Full flow diagram

                         ┌──────────────────────────────────┐
                         │     REST API (FastAPI)           │
                         │     POST /analyze                │
                         │     POST /ask                    │
                         │     GET  /health                 │
                         └──────────┬───────────────────────┘
                                    │
                                    ▼
                    ┌───────────────────────────────┐
                    │     Input Validator            │
                    │  - File type (PDF/img)         │
                    │  - Maximum size                │
                    │  - Valid format                │
                    └───────────┬───────────────────┘
                                │
                                ▼
                    ┌───────────────────────────────┐
                    │     DocumentProcessor (M3)     │
                    │  - Extract text from PDF       │
                    │  - Detect scanned pages        │
                    │  - Convert to images           │
                    │  - Chunking for RAG            │
                    └───────────┬───────────────────┘
                                │
                    ┌───────────┴───────────┐
                    ▼                       ▼
        ┌───────────────────┐   ┌───────────────────┐
        │  VisionAnalyzer   │   │   RAGModule (M6)   │
        │  (M2)             │   │  - Index chunks     │
        │  - Classify doc   │   │  - Embeddings       │
        │  - Extract data   │   │  - Retrieval        │
        │  - Multi fallback │   │  - Q&A generation   │
        └────────┬──────────┘   └────────┬──────────┘
                 │                        │
                 ▼                        ▼
        ┌───────────────────┐   ┌───────────────────┐
        │  Summarizer       │   │   AudioModule (M5) │
        │  - Text summary   │   │  - TTS of summary  │
        │  - Key points     │   │  - Configurable voice│
        └───────────────────┘   └───────────────────┘

Simplified data flow

PDF/Image → Validate → Process document → ┬→ Classify type (Vision)
                                          ├→ Extract structured data (Vision)
                                          ├→ Generate summary (LLM)
                                          ├→ Index for Q&A (RAG + ChromaDB)
                                          ├→ Answer question (if sent)
                                          └→ Generate summary audio (TTS, optional)

---

Components and Their Origin

Reuse map

Component	Class/Function	Source module	What it provides
Input Validator	validate_input()	Module 1	Detect file type, validate format and size
DocumentProcessor	DocumentProcessor	Module 3	Extract text/images from PDFs, detect scanned pages
VisionAnalyzer	VisionAnalyzer	Module 2	Classify documents, extract structured data from images
RAGModule	RAGModule	Module 6	Index chunks in ChromaDB, search and generate answers
AudioModule	AudioModule	Module 5	Generate summary audio with OpenAI TTS
UseCaseSelector	classify_document()	Module 7	Determine document type to apply the correct schema
API Router	FastAPI endpoints	Module 7	REST API for external integration

Adaptations from the original modules

It isn't "copy and paste" from previous modules. Each component is adapted:

• DocumentProcessor now detects mixed pages (text + scanned) within the same PDF
• VisionAnalyzer includes multi-provider fallback (OpenAI → Anthropic → Google)
• RAGModule indexes both text and image descriptions (multimodal RAG)
• AudioModule handles long summaries by splitting them into audio chunks
• API integrates everything into a single endpoint with configurable options

---

Module 8 Roadmap

Capsule map

#	Capsule	What you'll build	Duration
01	Introduction (this one)	Context, architecture, setup	10 min
02	Technical specifications	Inputs, outputs, data models, endpoints	10 min
03	Document processing	Complete DocumentProcessor	15 min
04	Vision analysis	VisionAnalyzer with fallback	15 min
05	RAG and Q&A	RAGModule with indexing and answers	15 min
06	Audio integration	AudioModule with TTS	10 min
07	Deployment and optimization	Docker, FastAPI, monitoring, costs	10 min
08	Project: Document Analyzer	Complete integrated system, tests, demo	15 min

Build flow

Capsule 02: You define WHAT the system will do (specifications)
    ↓
Capsules 03-06: You build EACH module individually
    ↓
Capsule 07: You prepare the DEPLOYMENT (Docker, API, monitoring)
    ↓
Capsule 08: You INTEGRATE everything into DocumentAnalyzer and run the full demo

Each capsule is independent: you can build and test each module separately before integrating. This is deliberate — in real systems, you first build and validate components, then connect them.

Estimated total duration: 90 minutes.

---

Module Objectives

By the end of this module you'll be able to:

• ✅ Design the architecture of a multi-component multimodal system
• ✅ Implement document processing with detection of scanned vs text pages
• ✅ Integrate vision APIs with multi-provider fallback for structured extraction
• ✅ Build a RAG pipeline that indexes text and image descriptions
• ✅ Add audio generation (TTS) as an optional output
• ✅ Deploy the system as a REST API with FastAPI and Docker
• ✅ Apply production patterns: logging, validation, error handling, costs
• ✅ Have a portfolio-worthy project that demonstrates mastery of multimodal AI

Professional goal

When someone asks you "can you build a system that processes documents with AI?", your answer won't be "yes, I can call the OpenAI API". It will be "yes, here is the system: it processes PDFs and images, extracts structured data, answers questions, generates audio summaries, has a REST API, per-provider fallbacks, and is Docker-ready". That's the level this module gives you.

---

Prerequisites

Required knowledge (from previous modules)

Module	Key concept	You need to know
Module 1	Input classification	Detect file type, modality
Module 2	Vision APIs	Send images to GPT-4o/Claude, get analysis
Module 3	Document processing	Extract text/images from PDFs with PyMuPDF
Module 5	Audio/TTS	Generate audio from text with OpenAI TTS
Module 6	RAG	Index in ChromaDB, embeddings, retrieval
Module 7	Usage patterns	Use-case selection, fallbacks

If you didn't complete previous modules

You can follow this module, but we recommend you at least read:

• Module 3 (documents) — The foundation of input processing
• Module 2 (vision) — Extraction with vision models
• Module 6 (RAG) — The question-and-answer system

The other modules provide optional functionality or patterns you can review later.

Required tools

• Python 3.11+
• OpenAI API key (required)
• Anthropic API key (optional, for vision fallback)
• Editor with terminal (VS Code, Cursor, PyCharm)
• Docker (optional, for deployment)

---

Technical Setup

Dependencies

python -m venv venv
source venv/bin/activate

pip install openai>=1.0.0 \
            pymupdf>=1.24.0 \
            pillow>=10.0.0 \
            pydantic>=2.0.0 \
            chromadb>=0.5.0 \
            langchain>=0.2.0 \
            langchain-openai>=0.1.0 \
            fastapi>=0.110.0 \
            uvicorn>=0.29.0 \
            python-multipart>=0.0.9 \
            python-dotenv>=1.0.0

Optional dependencies:

pip install anthropic>=0.25.0    # Vision fallback with Claude
pip install pydub>=0.25.0        # Concatenate long audio
pip install slowapi>=0.1.9       # Rate limiting

Environment variables

Create a .env file:

OPENAI_API_KEY=sk-...

# Optional
ANTHROPIC_API_KEY=sk-ant-...
LOG_LEVEL=INFO
MAX_FILE_SIZE_MB=50
REQUEST_TIMEOUT=120

Load it in your code:

from dotenv import load_dotenv
import os

load_dotenv()

assert os.getenv("OPENAI_API_KEY"), "OPENAI_API_KEY is missing in .env"

Setup verification

from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Respond only with 'OK'."}],
    max_tokens=5
)
print(response.choices[0].message.content)

If you see OK, your environment is ready.

Project file structure

document-analyzer/
├── .env                         # API keys (do NOT commit)
├── requirements.txt             # Dependencies
├── Dockerfile                   # Containerization
├── docker-compose.yml           # Orchestration
├── main.py                      # FastAPI app + endpoints
├── modules/
│   ├── __init__.py
│   ├── document_processor.py    # Capsule 03
│   ├── vision_analyzer.py       # Capsule 04
│   ├── rag_module.py            # Capsule 05
│   └── audio_module.py          # Capsule 06
├── models/
│   ├── __init__.py
│   └── schemas.py               # Pydantic models (Capsule 02)
├── tests/
│   ├── test_processor.py
│   ├── test_vision.py
│   └── test_integration.py
└── samples/
    ├── invoice_example.pdf       # Test PDF
    ├── contract_example.pdf      # Test PDF
    └── document_image.png        # Test image

Estimated module costs

Operation	Model	Cost per document
Classification	gpt-4o-mini	~$0.001
Structured extraction (text)	gpt-4o-mini	~$0.003
Structured extraction (image)	gpt-4o	~$0.01-0.03
Summary	gpt-4o-mini	~$0.002
Embeddings (indexing)	text-embedding-3-small	~$0.001
Q&A (generation)	gpt-4o	~$0.01
TTS (audio summary)	tts-1	~$0.015/1K chars
Total per document (with audio)	—	~$0.05-0.10

For development and testing of this module: ~$1-3 USD total.

---

Evidence of Success

By the end of this module, you'll know you succeeded if:

• ✅ Your DocumentProcessor processes PDFs with text and scanned pages correctly
• ✅ Your VisionAnalyzer extracts structured data and has a working fallback
• ✅ Your RAGModule indexes documents and answers questions with sources
• ✅ Your AudioModule generates audio from the summary
• ✅ Your REST API responds to POST /analyze with the complete result
• ✅ You can run docker build and bring up the service in a container
• ✅ The system processes a sample PDF end-to-end without errors

Quick self-assessment test

If you can answer these questions before starting, you're well prepared:

1. How do you extract text from a PDF with PyMuPDF?
2. What format do you need to send an image to GPT-4o Vision?
3. What does ChromaDB do and how do you index a document?
4. How do you generate audio from text with the OpenAI API?
5. What is a POST endpoint in FastAPI and how does it receive files?

If any of them are hard, review the corresponding module before continuing. Or move ahead and consult the relevant module when you reach that capsule.

---

Summary

• Module 8 is the final capstone project of the Multimodal AI Guide.
• You'll build a Multimodal Document Analyzer that processes PDFs/images, extracts data, does Q&A and generates audio.
• The system integrates components from the 7 previous modules: classification (M1), vision (M2), documents (M3), audio (M5), RAG (M6), patterns (M7).
• The architecture has 6 main components: DocumentProcessor, VisionAnalyzer, RAGModule, AudioModule, Summarizer, REST API.
• Each capsule builds an independent component; capsule 08 integrates them.
• Estimated duration: 90 minutes. Estimated cost: $1-3 USD.

---

Additional Resources

1. FastAPI — Framework for the REST API
2. PyMuPDF — PDF processing
3. ChromaDB — Vector database for RAG
4. OpenAI Vision — Image analysis
5. OpenAI TTS — Text-to-speech
6. Docker — Containerization

---