Module 3: Document Understanding
1. Introduction: Documents as Input
Description
This module goes deep into document understanding: how to process PDFs, scanned images and mixed documents to extract text, structured data and answer questions. Documents are the most common input in enterprise systems: invoices, contracts, manuals, reports. It isn't enough to "send an image to the LLM" — you have to master the complete pipeline: text extraction, OCR, conversion to images, structured schemas and long-document handling.
Why it matters: The Document Extractor you'll build here is the core of the Document Analyzer in Module 8. Everything you learn (PyMuPDF, pdf2image, OCR vs vision, Pydantic) gets integrated into the final project.
This module isn't a list of libraries. It's a way of thinking: given an arbitrary document, what is the optimal route to extract what you need? That decision depends on whether the PDF has embedded text, whether it's a scan, whether you need structured data or free text, whether the document has 2 pages or 200. Here you learn to make that decision and to implement it.
Where Are We in the Guide?
Context
This guide has 8 modules organized into 3 phases:
Phase 1: Multimodal Foundations (Modules 1-3)
├── Module 1: Introduction to Multimodal AI
├── Module 2: Vision + LLMs
└── Module 3: Document Understanding ← YOU ARE HERE
Phase 2: Generation and Audio (Modules 4-5)
├── Module 4: Image Generation
└── Module 5: Audio Processing
Phase 3: RAG, Use Cases and Project (Modules 6-8)
├── Module 6: Multimodal RAG
├── Module 7: Use Cases
└── Module 8: Final Project — Multimodal Document Analyzer
Where are we headed?
With the multimodal foundations (Module 1) and vision (Module 2), you now tackle the most in-demand use case in production: documents. Companies don't process random images — they process invoices, contracts, reports, forms. This module turns what you learned about Vision APIs into a complete document pipeline.
The progression is deliberate:
- Module 1 gave you the landscape — which modalities exist, which models support them
- Module 2 gave you vision — sending images to GPT-4V, Claude 3, Gemini
- Module 3 (this one) applies vision to real documents — with additional techniques: text extraction, OCR, Pydantic schemas
- Modules 4-8 build on this foundation — generation, audio, RAG, final project
What Document Understanding Is
Definition
Document understanding is the process of taking documents (PDFs, scanned images, screenshots) and extracting useful information: plain text, structured data (date, total, vendor), or answers to questions. It combines classic techniques (extracting text from PDFs, OCR with Tesseract) with vision models (GPT-4 Vision, Claude 3) for maximum quality.
It isn't a new problem. What changed is how it's solved: before, you needed hardcoded rules per document type; now an LLM with vision understands the document the way a human would.
The three fundamental approaches
There are three paths to extract information from a document. Each one has strengths, limitations and a different range of application:
Approach 1 — Direct text extraction (PyMuPDF)
PDF with embedded text → PyMuPDF extracts text → Plain text ready for the LLM
- When to use it: The PDF was generated digitally (Word, Google Docs, LaTeX). The text is selectable with the cursor.
- Advantages: Fast (milliseconds), free (no API calls), preserves paragraph structure.
- Limitations: Doesn't work with scanned PDFs. Loses visual layout (columns, complex tables). Doesn't capture information in images inside the PDF.
- Typical quality: 95-99% for well-formatted digital PDFs.
Approach 2 — Traditional OCR (Tesseract)
Image/scan → Tesseract detects characters → Plain text (with possible errors)
- When to use it: Scanned documents where you need fast, cheap text without depending on an external API.
- Advantages: Open-source, works offline, supports 100+ languages, no cost per call.
- Limitations: Variable quality with low-resolution documents, handwriting, or complex layouts. It doesn't "understand" the content — it only converts pixels to characters.
- Typical quality: 80-95% depending on the scan quality.
Approach 3 — Vision API (GPT-4V, Claude 3)
Page image → Vision API "sees" the document → Text + semantic understanding
- When to use it: You need understanding of the content (not just plain text). Complex documents with tables, charts, irregular layouts. When you need to answer questions about the document.
- Advantages: Understands visual context, handles complex tables and layouts, can extract structured data directly, works with any type of document.
- Limitations: Cost per call ($0.01-0.05 per page), latency (2-10 seconds per page), token limits, requires an internet connection.
- Typical quality: 90-99% — superior to OCR on complex documents.
Decision matrix
| Scenario | Recommended approach | Reason |
|---|---|---|
| Digital PDF, text only | Direct extraction | Fast, free, high quality |
| Scanned invoice, structured data | Vision API | Understands layout, extracts fields |
| 500 pages of historical archive | OCR + LLM post-processing | Prohibitive cost with Vision API |
| Form with handwriting | Vision API | OCR fails with handwriting |
| Mixed PDF (text + tables + images) | Extraction + Vision API | Combines the best of both |
| Offline/air-gapped processing | OCR (Tesseract) | No dependency on cloud APIs |
Document types
| Type | Characteristics | Main approach |
|---|---|---|
| PDF with text | Selectable, vectorized text | Direct extraction (PyMuPDF) |
| Scanned PDF | Images only per page | OCR or Vision API |
| Document image | JPG/PNG of an invoice, form | Vision API or Tesseract |
| Mixed document | Text + tables + figures | Combine extraction + vision |
| Long document | 50+ pages | Chunking, section-by-section summarization |
Context: From Paper to Pipeline
The evolution of document processing
Document processing wasn't born with LLMs. It has decades of history, and each era brought new tools:
Era 1 — Manual (pre-2000):
Humans read documents and transcribe data by hand
Archivists, data entry operators, literal bureaucracy
Speed: ~5 documents/hour per person
Human error: ~2-5%
Era 2 — Classic OCR (2000-2020):
Tesseract, ABBYY, Adobe Acrobat OCR
Converts images to text with character recognition algorithms
Speed: ~100 documents/hour
Requires post-processing: OCR produces dirty text you need to clean
Hardcoded rules per document type
Era 3 — LLM + Vision (2023-present):
GPT-4 Vision, Claude 3, Gemini
The model "sees" the document like a human
Extracts structured data in a single call
Speed: ~360 documents/hour (10 sec/page with Vision API)
Understands context: it doesn't just read characters, it grasps meaning
Why 2023-2024 changed everything
Before GPT-4 Vision (March 2023), processing a complex document required:
- Detecting the document type (PDF, image, scan)
- Applying OCR with Tesseract
- Cleaning the resulting text with regex and heuristics
- Writing document-type-specific parsers (one parser for invoices, another for contracts)
- Maintaining those parsers when the format changed
That pipeline meant weeks of development per document type and it was fragile — if the vendor changed their invoice layout, the parser broke.
With Vision APIs, the pipeline got simpler:
- Convert the document to an image
- Send the image to the model with a prompt describing what to extract
- Receive structured data
From weeks of development to hours. From fragile parsers to adaptable prompts. From constant maintenance to zero-maintenance (the model adapts to layout changes).
What Vision APIs did NOT replace
Vision APIs didn't eliminate the need for the classic tools:
- PyMuPDF is still the right choice for PDFs with embedded text (faster and cheaper than Vision)
- Tesseract is still useful for massive offline processing where the Vision API cost is prohibitive
- Pydantic became more important, not less — you need to validate that the LLM's extraction has the correct format
The modern developer doesn't pick one tool — they build a pipeline that uses the right tool at each step.
Why Document Understanding Matters
Real use cases
1. Invoice extraction (Finance)
Automate data entry from scanned invoices: date, vendor, total, items, taxes, invoice number. Companies like Stripe, DocuSign and banks process millions of invoices/month with similar pipelines.
- Before: An operator takes ~5 minutes per invoice. 10,000 invoices/month = 833 hours of work.
- After: Vision API + Pydantic extracts data in
10 seconds per invoice. 10,000 invoices = 28 hours of compute ($100-500 in API costs). - Typical ROI: 90%+ reduction in processing time.
2. Q&A over technical manuals (Support)
"How do you configure mode X?" — the system searches the 200-page manual and answers with the relevant section. Used in technical support, onboarding, internal documentation.
- Before: Search the PDF with Ctrl+F, read context, interpret. ~3-5 minutes per question.
- After: Chunking the manual + embeddings + LLM. Answer in ~5 seconds.
3. Contract analysis (Legal)
Extract clauses, dates, involved parties, amounts, penalties. Law firms and compliance departments review hundreds of contracts.
- Impact: From 30 minutes of reading per contract to automatic extraction of key fields in seconds. The lawyer reviews the output instead of reading the whole document.
4. Medical form processing (Healthcare)
Patient forms, prescriptions, lab results — frequently scanned or photographed. Sensitive data that requires precision.
- Special challenge: Handwriting, medical terminology, privacy regulations (HIPAA).
- Solution: Vision API for handwriting + strict validation with Pydantic + local processing when possible.
5. Historical archive indexing (Government/Academia)
Convert legacy documents (scans from decades ago) into searchable, indexable text. Libraries, national archives, universities.
- Scale: Millions of pages. Batch OCR (Tesseract) for the volume, Vision API for problematic documents.
6. Receipt processing (Retail/Expenses)
Photograph a receipt → extract merchant, date, total, category. Expense management apps like Expensify, SAP Concur.
- Challenge: Variable quality (photos with flash, wrinkled, blurry). Vision APIs handle these cases better than classic OCR.
The Document Pipeline in Detail
Complete architecture
This is the pipeline you'll build throughout the module. Each capsule covers one or more stages:
┌─────────────────────────────────────────────────────────────┐
│ DOCUMENT PIPELINE │
│ │
│ ┌──────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ Document │───▶│ Type │───▶│ Text Extraction │ │
│ │ Input │ │ Detection │ │ (PyMuPDF) │ │
│ │ PDF/IMG │ │ │ │ OR │ │
│ └──────────┘ └──────────────┘ │ Image Conversion │ │
│ │ (pdf2image) │ │
│ └────────┬─────────┘ │
│ │ │
│ ┌────────▼─────────┐ │
│ │ OCR (Tesseract) │ │
│ │ OR │ │
│ │ Vision API │ │
│ │ (GPT-4V/Claude) │ │
│ └────────┬─────────┘ │
│ │ │
│ ┌────────▼─────────┐ │
│ │ LLM Processing │ │
│ │ Structured Output│ │
│ │ (Pydantic) │ │
│ └────────┬─────────┘ │
│ │ │
│ ┌────────▼─────────┐ │
│ │ Database / API │ │
│ │ Output │ │
│ └──────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Each stage explained
1. Document Input — The document arrives as a file: .pdf, .jpg, .png, .tiff. It can come from an upload, an email, a file system, or an API.
2. Type Detection — Is it a PDF with embedded text or a scan? Is it a direct image? This decision determines the pipeline's path.
import fitz # PyMuPDF
def detect_document_type(file_path: str) -> str:
if file_path.lower().endswith((".jpg", ".jpeg", ".png", ".tiff", ".bmp")):
return "image"
if file_path.lower().endswith(".pdf"):
doc = fitz.open(file_path)
text = doc[0].get_text().strip()
doc.close()
if len(text) > 50:
return "pdf_with_text"
return "pdf_scanned"
return "unknown"3. Text Extraction (PyMuPDF) — For PDFs with embedded text, PyMuPDF extracts the text directly. Fast, free, high fidelity.
4. Image Conversion (pdf2image) — For scanned PDFs or when you need vision, you convert each page to a PNG/JPG image.
5. OCR (Tesseract) or Vision API — Depending on the case: Tesseract for volume and low cost, Vision API for quality and semantic understanding.
6. LLM Processing + Structured Output — The extracted text (or the image directly) is sent to the LLM with a prompt describing what data to extract. Pydantic validates the output.
7. Database / API Output — The structured data is stored or sent to another system.
Mapping capsules to the pipeline
| Pipeline stage | Capsule |
|---|---|
| Type Detection | 02 (PDF Processing) |
| Text Extraction (PyMuPDF) | 02 (PDF Processing) |
| Image Conversion (pdf2image) | 04 (Document images) |
| OCR (Tesseract) | 03 (OCR + LLM) |
| Vision API | 03 (OCR + LLM), 04 (Document images) |
| Structured Output (Pydantic) | 05 (Structured extraction) |
| Long documents (Chunking) | 06 (Long documents) |
| Complete pipeline | 08 (Project: Document Extractor) |
Module 3 Roadmap
| # | Capsule | What you'll see |
|---|---|---|
| 01 | Introduction (this one) | Documents as input, types, pipeline |
| 02 | PDF Processing | PyMuPDF, pdf2image, text and image extraction |
| 03 | OCR + LLM | Tesseract vs Vision API, when to use each |
| 04 | Document images | Convert pages to images, send to vision |
| 05 | Structured extraction | Pydantic schemas, invoice data |
| 06 | Long documents | Chunking, section-by-section summarization |
| 07 | Troubleshooting documents | Corrupt PDFs, low quality |
| 08 | Project: Document Extractor | Extract structured data from PDFs/images |
Estimated duration: 55 min for the complete module.
Module Objectives
By the end of this module you'll be able to:
- Process PDFs and extract text and images from pages
- Decide when to use traditional OCR (Tesseract) vs a Vision API
- Convert PDF pages to images for vision
- Extract structured data with Pydantic schemas
- Handle long documents (chunking, section-by-section summarization)
- Solve common problems (corrupt PDFs, low quality)
- Build a Document Extractor end-to-end as an integrating project
Professional objective
When someone tells you "we need to automate invoice processing", you'll know: does the PDF have text or is it scanned? Do I need OCR or a Vision API? How do I validate the output with Pydantic? What do I do if the document has 100 pages? How much does it cost to process it? That level of technical judgment is what this module gives you.
Prerequisites from Modules 1-2
What you should already master
This module assumes you completed Modules 1 and 2. Specifically:
| Concept | Where you learned it | Why you need it here |
|---|---|---|
| Base64 encoding | Module 1, Capsule 06 | Sending page images to Vision APIs |
| Vision APIs | Module 2, Capsules 02-05 | Analyzing document pages with GPT-4V/Claude |
| Prompting for vision | Module 2, Capsule 03 | Designing prompts that extract data from documents |
| Cost estimation | Module 1, Capsule 06 | Calculating the cost of processing N pages with Vision |
| API error handling | Module 1, Capsule 07 | Rate limits, timeouts when processing documents |
| Image formats | Module 2, Capsule 04 | Optimal resolution and format for scanned pages |
If any concept raises doubts, review Modules 1-2 before continuing.
Technical Setup
Prerequisites
- Python 3.11+
- OpenAI account (for GPT-4 Vision)
- Optional: Anthropic, Google AI
- Poppler installed (required by pdf2image)
- Tesseract installed (for OCR — optional but recommended)
Installation
pip install openai>=1.0.0 pymupdf pdf2image pillow pydantic python-dotenv
# For traditional OCR
pip install pytesseractSystem dependencies
# macOS
brew install poppler tesseract
# Ubuntu/Debian
sudo apt-get install poppler-utils tesseract-ocr
# Windows: download Poppler and Tesseract from their official releases
# Poppler: https://github.com/oschwartz10612/poppler-windows/releases
# Tesseract: https://github.com/UB-Mannheim/tesseract/wikiEnvironment variables
export OPENAI_API_KEY="sk-..."Complete verification script
Run this script to confirm that all dependencies are installed correctly. It also generates a test PDF you'll use in the following capsules:
import sys
print("=" * 50)
print("SETUP VERIFICATION — MODULE 3")
print("=" * 50)
errors = []
# 1. Python version
print(f"\n[1/6] Python: {sys.version}")
if sys.version_info < (3, 11):
errors.append("Python 3.11+ required")
# 2. PyMuPDF
try:
import fitz
print(f"[2/6] PyMuPDF: v{fitz.version[0]}")
except ImportError:
errors.append("PyMuPDF not installed → pip install pymupdf")
# 3. pdf2image
try:
from pdf2image import convert_from_path
print("[3/6] pdf2image: OK")
except ImportError:
errors.append("pdf2image not installed → pip install pdf2image")
# 4. Tesseract (optional)
try:
import pytesseract
version = pytesseract.get_tesseract_version()
print(f"[4/6] Tesseract: v{version}")
except Exception:
print("[4/6] Tesseract: NOT AVAILABLE (optional)")
# 5. Pydantic
try:
import pydantic
print(f"[5/6] Pydantic: v{pydantic.__version__}")
except ImportError:
errors.append("Pydantic not installed → pip install pydantic")
# 6. OpenAI
try:
from openai import OpenAI
print("[6/6] OpenAI SDK: OK")
except ImportError:
errors.append("OpenAI not installed → pip install openai>=1.0.0")
# Create test PDF
if not errors:
print("\n" + "-" * 50)
print("Creating test PDF...")
doc = fitz.open()
page = doc.new_page()
text_content = (
"INVOICE #2024-001\n\n"
"Date: 2024-03-15\n"
"Vendor: TechCorp Inc.\n"
"Customer: ABC Company\n\n"
"Item Quantity Price\n"
"Software license 1 $500.00\n"
"Technical support 3 months $150.00\n"
"Consulting 10 hours $1,000.00\n\n"
"Subtotal: $1,650.00\n"
"Tax (16%): $264.00\n"
"Total: $1,914.00"
)
page.insert_text((72, 72), text_content, fontsize=12)
doc.save("sample_invoice.pdf")
doc.close()
print("PDF created: sample_invoice.pdf")
# Result
print("\n" + "=" * 50)
if errors:
print(f"ERRORS ({len(errors)}):")
for e in errors:
print(f" ✗ {e}")
print("\nResolve the errors before continuing.")
else:
print("SETUP COMPLETE — Ready for Module 3")
print("=" * 50)Expected output (all OK):
==================================================
SETUP VERIFICATION — MODULE 3
==================================================
[1/6] Python: 3.11.x
[2/6] PyMuPDF: v1.24.x
[3/6] pdf2image: OK
[4/6] Tesseract: v5.x.x
[5/6] Pydantic: v2.x.x
[6/6] OpenAI SDK: OK
--------------------------------------------------
Creating test PDF...
PDF created: sample_invoice.pdf
==================================================
SETUP COMPLETE — Ready for Module 3
==================================================
Estimated module costs
| Operation | Approximate cost |
|---|---|
| Text extraction (PyMuPDF) | $0.00 (local) |
| OCR with Tesseract | $0.00 (local) |
| Vision API per page (gpt-4o-mini) | ~$0.01-0.02 |
| Vision API per page (gpt-4o) | ~$0.03-0.05 |
| Complete module (exercises + project) | ~$0.50-2.00 |
Tip: Use PyMuPDF and Tesseract for development and testing. Reserve the Vision API for when you need maximum quality or semantic understanding.
Connection with the Guide's Project
The Document Extractor (capsule 08) is a system that receives a PDF or document image and returns structured data (e.g. an invoice with date, total, items). This component gets integrated into the Document Analyzer of Module 8, which adds RAG for Q&A and optional TTS.
Module 3: Document Extractor → extracts data from PDFs/images
↓
Module 8: Document Analyzer → uses extraction + RAG + TTS
Limits: What This Module Does NOT Cover
- ❌ PDF generation — This module extracts data FROM documents, it doesn't generate new documents. To create PDFs, use ReportLab or WeasyPrint.
- ❌ Digital signatures and cryptography — We don't cover signature verification, PDF encryption or digital certificates.
- ❌ Form builders — We don't build interactive forms. We process existing forms.
- ❌ Enterprise DMS (Document Management Systems) — We don't integrate with SharePoint, Alfresco or enterprise document management systems. We focus on the extraction pipeline.
- ❌ PDF editing — We don't modify existing PDFs (adding annotations, merging pages). We use PyMuPDF for reading only.
- ❌ Advanced handwriting recognition — We cover the basic case with Vision API, but we don't train specialized handwriting recognition models.
Evidence of Success
By the end of this module, you'll know you succeeded if:
- ✅ Given a PDF, you can determine whether it has embedded text or is a scan, and choose the correct processing route
- ✅ You can extract text from a digital PDF with PyMuPDF in fewer than 5 lines of code
- ✅ You can convert a PDF to images with pdf2image and send them to a Vision API
- ✅ You know when to use Tesseract vs a Vision API and can justify the decision with cost, quality and latency criteria
- ✅ You can define a Pydantic schema for an invoice and validate that the LLM's extraction is correct
- ✅ You can process a 50+ page document with chunking without exceeding token limits
- ✅ Your Document Extractor works end-to-end: it receives a PDF, detects the type, extracts structured data
Quick self-assessment test
If you can answer these questions by the end of the module, you're on the right track:
- How do you detect whether a PDF has embedded text or is a scan?
- How much does it cost to process 100 pages with gpt-4o-mini vs Tesseract?
- What happens if the LLM returns a JSON with missing fields and you have a Pydantic schema?
- How do you handle a 200-page document that exceeds the model's context limit?
Summary
- Document understanding combines three approaches: direct extraction (PyMuPDF), OCR (Tesseract) and Vision APIs (GPT-4V, Claude 3).
- The evolution of document processing went from manual → classic OCR → LLM + Vision, and 2023-2024 was the inflection point.
- Document types: PDF with text, scanned, images, mixed, long — each one has its optimal route.
- The complete pipeline: document → type detection → extraction/OCR/vision → LLM → Pydantic → structured output.
- Real use cases: invoices, contracts, manuals, medical forms, historical archives — with measurable impact on time and cost.
- This module prepares you for the Document Analyzer of Module 8.
- Setup: PyMuPDF, pdf2image, Poppler, Tesseract, OpenAI, Pydantic.
Additional Resources
- PyMuPDF Documentation — PDF extraction
- pdf2image — PDF to images
- Tesseract OCR — Traditional OCR
- OpenAI Vision — Analysis with LLMs
- Pydantic — Schemas and validation
- Poppler — PDF tools (required by pdf2image)
- pytesseract — Python wrapper for Tesseract