3. OCR + LLM

Description

When a document has no selectable text (scanned, photograph, image), you need OCR (Optical Character Recognition). There are two main approaches: traditional OCR (Tesseract) and Vision APIs (GPT-4 Vision, Claude 3). In this capsule you'll learn when to use each, how to preprocess images to maximize quality, and how to combine OCR with LLMs for intelligent extraction.

Why it matters: The OCR vs Vision decision affects the cost, quality and latency of your document pipeline. Tesseract is free, offline and fast for high volume. Vision APIs offer better quality on complex documents but have a per-use cost. The hybrid approach (Tesseract + LLM for cleanup) is the most efficient option in many real-world scenarios.

Connection with the module: This capsule connects with capsule 02 (PDFs) for scanned documents, with 04 (document images) for photographs, and with 05 (structured extraction) where the extracted text becomes schema-validated data. The Document Extractor (capsule 08) uses these techniques as its input layer.

Key Concepts

Traditional OCR (Tesseract)

Tesseract is the most widely used open-source OCR engine. Google has maintained it since 2006, and version 4+ uses LSTM networks.

• Pros: Free, local, no API limits, fast, works offline
• Cons: Variable quality on tables, forms, handwriting; requires preprocessing
• When: Clean printed text, high volume, limited budget, privacy requirements

Vision APIs (GPT-4V, Claude 3)

Vision models "read" the entire image with semantic understanding — they grasp structure, layout and context.

• Pros: High quality on tables, forms, complex layouts; native multilingual
• Cons: Cost per token, API latency, rate limits, require a connection
• When: Documents with tables/forms, structured extraction, critical quality

Hybrid pipeline

Tesseract for fast extraction + a text LLM to clean and structure. Cheaper than Vision, better than Tesseract alone.

• Pros: Reduced cost vs pure Vision, better quality than Tesseract alone
• Cons: Two steps, added LLM latency
• When: Medium-high volume where Vision is too expensive but Tesseract falls short

OCR with Tesseract

Installation and language packages

# Installing the Tesseract engine
# macOS:   brew install tesseract
# Ubuntu:  sudo apt-get install tesseract-ocr
# Windows: https://github.com/UB-Mannheim/tesseract/wiki

# Language packages (English)
# macOS:   brew install tesseract-lang
# Ubuntu:  sudo apt-get install tesseract-ocr-eng

# Python wrapper
# pip install pytesseract Pillow

import pytesseract

available_langs = pytesseract.get_languages()
print(f"Installed languages: {available_langs}")

Basic usage

import pytesseract
from PIL import Image
import io


def ocr_with_tesseract(image_path: str, lang: str = "eng") -> str:
    """Extracts text from an image using Tesseract."""
    img = Image.open(image_path)
    text = pytesseract.image_to_string(img, lang=lang)
    return text.strip()


def ocr_from_bytes(image_bytes: bytes, lang: str = "eng") -> str:
    """Extracts text from image bytes in memory."""
    img = Image.open(io.BytesIO(image_bytes))
    return pytesseract.image_to_string(img, lang=lang).strip()

PSM modes (Page Segmentation Mode)

The PSM parameter controls how Tesseract segments the page. Choosing the right mode dramatically improves quality.

#  3 = Full automatic segmentation (DEFAULT)
#  4 = Assume a single column of text
#  6 = Assume a uniform block of text
#  7 = Treat the image as a single line of text
# 11 = Sparse text with no particular order
# 13 = Raw line — no internal preprocessing

def ocr_single_column(image_path: str, lang: str = "eng") -> str:
    """OCR optimized for single-column documents."""
    img = Image.open(image_path)
    return pytesseract.image_to_string(img, lang=lang, config="--oem 3 --psm 4").strip()


def ocr_text_block(image_path: str, lang: str = "eng") -> str:
    """OCR optimized for a uniform block of text."""
    img = Image.open(image_path)
    return pytesseract.image_to_string(img, lang=lang, config="--oem 3 --psm 6").strip()


def ocr_single_line(image_path: str, lang: str = "eng") -> str:
    """OCR for a single line (invoice number, date)."""
    img = Image.open(image_path)
    return pytesseract.image_to_string(img, lang=lang, config="--oem 3 --psm 7").strip()

HOCR output and coordinates

HOCR is a format that includes the coordinates of each word. Useful for reconstructing layout or marking regions.

def ocr_word_boxes(image_path: str, lang: str = "eng") -> list[dict]:
    """Extracts each word with its bounding box and confidence."""
    img = Image.open(image_path)
    data = pytesseract.image_to_data(img, lang=lang, output_type=pytesseract.Output.DICT)

    words = []
    for i in range(len(data["text"])):
        if data["text"][i].strip():
            words.append({
                "text": data["text"][i],
                "x": data["left"][i],
                "y": data["top"][i],
                "w": data["width"][i],
                "h": data["height"][i],
                "confidence": data["conf"][i],
            })
    return words

Preprocessing for Better OCR

The quality of Tesseract OCR depends directly on the quality of the image. A preprocessing pipeline transforms the image to maximize accuracy.

Complete pipeline with Pillow

from PIL import Image, ImageEnhance, ImageFilter


def preprocess_for_ocr(
    image_path: str,
    binarize: bool = True,
    denoise: bool = True,
) -> Image.Image:
    """
    Pipeline: grayscale → resize → contrast → binarization → denoising.
    """
    img = Image.open(image_path)

    img = img.convert("L")

    width, height = img.size
    if width < 1000:
        scale = 2.0
        img = img.resize((int(width * scale), int(height * scale)), Image.LANCZOS)

    enhancer = ImageEnhance.Contrast(img)
    img = enhancer.enhance(2.0)

    if binarize:
        img = img.point(lambda p: 255 if p > 128 else 0, "1")
        img = img.convert("L")

    if denoise:
        img = img.filter(ImageFilter.MedianFilter(size=3))

    return img

Compare quality before and after

def compare_preprocessing(image_path: str) -> dict:
    """Compares OCR with and without preprocessing."""
    img_raw = Image.open(image_path)
    img_processed = preprocess_for_ocr(image_path)

    text_raw = pytesseract.image_to_string(img_raw, lang="eng").strip()
    text_processed = pytesseract.image_to_string(img_processed, lang="eng").strip()

    return {
        "raw_chars": len(text_raw),
        "processed_chars": len(text_processed),
        "raw_preview": text_raw[:200],
        "processed_preview": text_processed[:200],
    }


result = compare_preprocessing("low_quality_invoice.jpg")
print(f"Without preprocessing: {result['raw_chars']} characters")
print(f"With preprocessing: {result['processed_chars']} characters")

OCR with Vision API

OpenAI (GPT-4o)

from openai import OpenAI
import base64

client = OpenAI()


def ocr_with_vision(image_path: str, prompt: str = None) -> str:
    """Extracts text from an image using GPT-4 Vision."""
    with open(image_path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()

    default_prompt = """Extract ALL the visible text in this document image.
Keep the original structure: paragraphs, lists, tables.
If there are tables, represent them in markdown format.
Do not invent or interpret — just transcribe what you see.
If something is illegible, mark it [illegible]."""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": prompt or default_prompt},
                {"type": "image_url", "image_url": {
                    "url": f"data:image/jpeg;base64,{b64}",
                    "detail": "high",
                }}
            ]
        }],
        max_tokens=4000,
    )
    return response.choices[0].message.content

Specialized prompts per document type

OCR_PROMPTS = {
    "invoice": """Extract the text from this invoice. Identify and preserve:
- Header (company, tax ID, address)
- Recipient data
- Line-item table (description, quantity, unit price, amount)
- Totals (subtotal, tax, total)
- Number and date
Use markdown format for tables.""",

    "form": """Extract the fields of this form.
For each field: field_name: value
Checkboxes: [X] checked or [ ] empty.
Empty fields: field: [empty].""",

    "table": """Extract the table in strict markdown format with |.
Preserve all exact numeric values.
Do not round or modify figures.""",
}


def ocr_vision_typed(image_path: str, doc_type: str) -> str:
    """OCR with the Vision API using a specialized prompt per type."""
    prompt = OCR_PROMPTS.get(doc_type)
    if not prompt:
        raise ValueError(f"Unsupported type. Use: {list(OCR_PROMPTS.keys())}")
    return ocr_with_vision(image_path, prompt=prompt)

Anthropic (Claude 3)

import anthropic


def ocr_with_claude(image_path: str, prompt: str = None) -> str:
    """Extracts text using Claude 3 Vision."""
    import mimetypes
    mime_type = mimetypes.guess_type(image_path)[0] or "image/jpeg"

    with open(image_path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()

    ac = anthropic.Anthropic()
    response = ac.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=4000,
        messages=[{
            "role": "user",
            "content": [
                {"type": "image", "source": {
                    "type": "base64", "media_type": mime_type, "data": b64,
                }},
                {"type": "text", "text": prompt or "Transcribe all the visible text. Use markdown for tables."},
            ]
        }],
    )
    return response.content[0].text

Detailed Comparison: Tesseract vs Vision API

Practical summary:

• Tesseract → high volume, low budget, clean printed text, privacy, offline.
• Vision API → critical quality, tables, forms, handwriting, complex documents.
• Hybrid → medium volume, optimize cost vs quality.

Hybrid Pipeline: Tesseract + LLM

Tesseract extracts raw text (free and fast), then a text LLM cleans and structures the result. Significantly cheaper than Vision because text tokens cost much less than image tokens.

def hybrid_ocr(image_path: str, lang: str = "eng") -> str:
    """Tesseract extracts raw text, the LLM cleans and corrects it."""
    img = preprocess_for_ocr(image_path)
    raw_text = pytesseract.image_to_string(img, lang=lang).strip()

    if len(raw_text) < 10:
        return ocr_with_vision(image_path)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": """You are an OCR text corrector.
Correct obvious errors (confused letters, misplaced spaces).
Preserve the structure. Do NOT invent content.
If there is unrecoverable text, mark it [illegible]."""},
            {"role": "user", "content": f"OCR text to correct:\n\n{raw_text}"},
        ],
        temperature=0,
    )
    return response.choices[0].message.content

Structured extraction with the hybrid pipeline

import json


def hybrid_structured_extraction(
    image_path: str,
    schema: dict[str, str],
    lang: str = "eng",
) -> dict:
    """Tesseract extracts text → LLM structures it per schema. ~10x cheaper than Vision."""
    img = preprocess_for_ocr(image_path)
    raw_text = pytesseract.image_to_string(img, lang=lang).strip()
    schema_desc = "\n".join(f"- {k}: {v}" for k, v in schema.items())

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You extract structured data from OCR text."},
            {"role": "user", "content": f"""Extract these fields from the OCR text:
{schema_desc}

OCR text:
{raw_text}

Respond ONLY with valid JSON. Use null if you can't find a field."""},
        ],
        response_format={"type": "json_object"},
        temperature=0,
    )
    return json.loads(response.choices[0].message.content)

Cost comparison

Estimated cost per document:
┌───────────────────────┬────────────────┬───────────────┐
│ Method                │ Approx. cost   │ Quality       │
├───────────────────────┼────────────────┼───────────────┤
│ Tesseract only        │ $0.00          │ ★★★☆☆        │
│ Hybrid (Tess + Mini)  │ $0.001–0.005   │ ★★★★☆        │
│ Vision API direct     │ $0.01–0.04     │ ★★★★★        │
└───────────────────────┴────────────────┴───────────────┘

For 10,000 documents/month:
  Tesseract only:   $0
  Hybrid:           $10–50
  Vision direct:    $100–400

Decision Tree: Which Method to Use?

Does the document have selectable text (digital PDF)?
├── YES → Use PyMuPDF (capsule 02). You don't need OCR.
│
└── NO → It's an image or scanned PDF.
         │
         ├── Do you have a budget for a Vision API?
         │   ├── NO → Tesseract
         │   │        ├── Printed and clean text? → Direct Tesseract (PSM 3/6)
         │   │        ├── Low-quality image? → Preprocessing + Tesseract
         │   │        └── Need structured data? → Hybrid (Tess + LLM)
         │   │
         │   └── YES → How many documents/day?
         │       ├── < 100     → Direct Vision API (maximum quality)
         │       ├── 100–1000  → Hybrid for most + Vision for complex docs
         │       └── > 1000    → Tesseract + LLM + Vision as fallback
         │
         └── Does it have tables, forms or handwriting?
             ├── YES → Vision API or hybrid with a specialized prompt
             └── NO → Tesseract with preprocessing usually suffices

OCR of Complex Documents

Tables

Tesseract loses column structure. Use image_to_data for coordinates and reconstruct by position.

import pandas as pd


def ocr_table_from_image(image_path: str, lang: str = "eng") -> pd.DataFrame:
    """Reconstructs a table from OCR with coordinates (grouping by row Y and column X)."""
    img = preprocess_for_ocr(image_path, binarize=True)
    data = pytesseract.image_to_data(img, lang=lang, output_type=pytesseract.Output.DICT)

    words = []
    for i in range(len(data["text"])):
        text = data["text"][i].strip()
        if text and int(data["conf"][i]) > 30:
            words.append({"text": text, "x": data["left"][i], "y": data["top"][i]})

    if not words:
        return pd.DataFrame()

    rows, current_row = [], [words[0]]
    for word in words[1:]:
        if abs(word["y"] - current_row[0]["y"]) < 15:
            current_row.append(word)
        else:
            rows.append(sorted(current_row, key=lambda w: w["x"]))
            current_row = [word]
    rows.append(sorted(current_row, key=lambda w: w["x"]))

    table_data = [[w["text"] for w in row] for row in rows]
    max_cols = max(len(r) for r in table_data)
    for row in table_data:
        row.extend([""] * (max_cols - len(row)))

    return pd.DataFrame(table_data[1:], columns=table_data[0] if table_data else None)

Multi-column layouts

Split the image into columns and apply OCR separately to preserve reading order.

def ocr_multicolumn(image_path: str, num_columns: int = 2) -> list[str]:
    """OCR column by column for newspaper/magazine-style documents."""
    img = Image.open(image_path)
    width, height = img.size
    col_width = width // num_columns

    columns_text = []
    for i in range(num_columns):
        left = i * col_width
        right = (i + 1) * col_width if i < num_columns - 1 else width
        col_img = img.crop((left, 0, right, height))
        text = pytesseract.image_to_string(col_img, lang="eng", config="--psm 4")
        columns_text.append(text.strip())
    return columns_text

Forms and handwritten text

def ocr_form_with_vision(image_path: str) -> str:
    """For forms with checkboxes, the Vision API is the most reliable option."""
    return ocr_with_vision(image_path, prompt=OCR_PROMPTS["form"])


def ocr_handwritten(image_path: str) -> dict:
    """For handwriting, always the Vision API. Returns text + confidence."""
    prompt = """This image contains handwritten text.
Transcribe everything you can read. If something is illegible, mark it [illegible].
At the end, indicate your confidence: HIGH, MEDIUM or LOW."""

    result = ocr_with_vision(image_path, prompt=prompt)
    confidence = "LOW"
    for level in ["HIGH", "MEDIUM", "LOW"]:
        if level in result.upper():
            confidence = level
            break
    return {"text": result, "confidence": confidence}

Exercises

Exercise 1: Configurable preprocessing pipeline

Create a function preprocess_pipeline that takes an image and a list of steps (["grayscale", "contrast", "binarize", "denoise", "resize"]). Each step is optional. Then apply OCR with Tesseract and return the text.

def preprocess_pipeline(
    image_path: str,
    steps: list[str],
    contrast_factor: float = 2.0,
    threshold: int = 128,
) -> str:
    """Configurable preprocessing + OCR pipeline."""
    img = Image.open(image_path)

    for step in steps:
        if step == "grayscale":
            img = img.convert("L")
        elif step == "contrast":
            if img.mode != "L":
                img = img.convert("L")
            img = ImageEnhance.Contrast(img).enhance(contrast_factor)
        elif step == "binarize":
            if img.mode != "L":
                img = img.convert("L")
            img = img.point(lambda p: 255 if p > threshold else 0, "1").convert("L")
        elif step == "denoise":
            img = img.filter(ImageFilter.MedianFilter(size=3))
        elif step == "resize":
            w, h = img.size
            img = img.resize((w * 2, h * 2), Image.LANCZOS)

    return pytesseract.image_to_string(img, lang="eng").strip()


text = preprocess_pipeline(
    "blurry_invoice.jpg",
    steps=["grayscale", "resize", "contrast", "binarize", "denoise"],
)

Exercise 2: Hybrid OCR with intelligent fallback

Implement a function that: (1) tries Tesseract, (2) evaluates the quality of the result (ratio of alphabetic characters vs garbage), (3) if quality is low, retries with the Vision API. Returns the result and the method used.

def evaluate_ocr_quality(text: str) -> float:
    """Scores OCR text quality between 0.0 and 1.0."""
    if not text:
        return 0.0
    alpha_ratio = sum(c.isalpha() or c.isspace() for c in text) / len(text)
    has_words = len(text.split()) > 3
    no_garbage = not any(c in text for c in ["\x00", "□", "■"])
    return min(alpha_ratio * 0.6 + (0.2 if has_words else 0) + (0.2 if no_garbage else 0), 1.0)


def ocr_smart_fallback(image_path: str, quality_threshold: float = 0.5) -> dict:
    """OCR with intelligent fallback based on quality."""
    img = preprocess_for_ocr(image_path)
    tesseract_text = pytesseract.image_to_string(img, lang="eng").strip()
    quality = evaluate_ocr_quality(tesseract_text)

    if quality >= quality_threshold:
        return {"text": tesseract_text, "method": "tesseract", "quality": quality}

    vision_text = ocr_with_vision(image_path)
    return {"text": vision_text, "method": "vision_fallback", "quality": quality}

Exercise 3: Batch OCR with cost report

Process a list of images using the hybrid pipeline. Keep count of how many used Tesseract only vs how many needed the LLM/Vision. Generate an estimated cost report.

def batch_ocr_with_report(
    image_paths: list[str],
    quality_threshold: float = 0.5,
    cost_per_vision: float = 0.02,
    cost_per_llm: float = 0.003,
) -> dict:
    """Batch OCR with a cost report."""
    results = []
    stats = {"tesseract_only": 0, "hybrid": 0, "vision_fallback": 0}

    for path in image_paths:
        img = preprocess_for_ocr(path)
        raw = pytesseract.image_to_string(img, lang="eng").strip()
        quality = evaluate_ocr_quality(raw)

        if quality >= quality_threshold:
            results.append({"path": path, "text": raw, "method": "tesseract"})
            stats["tesseract_only"] += 1
        elif quality >= 0.3:
            cleaned = hybrid_ocr(path)
            results.append({"path": path, "text": cleaned, "method": "hybrid"})
            stats["hybrid"] += 1
        else:
            vision = ocr_with_vision(path)
            results.append({"path": path, "text": vision, "method": "vision"})
            stats["vision_fallback"] += 1

    total_cost = stats["hybrid"] * cost_per_llm + stats["vision_fallback"] * cost_per_vision
    return {
        "results": results,
        "stats": stats,
        "total_cost_usd": round(total_cost, 4),
        "cost_if_all_vision": round(len(image_paths) * cost_per_vision, 4),
    }

Exercise 4: Table extractor with Vision + validation

Use the Vision API to extract a table as CSV. Convert it to a pandas DataFrame. Validate that the numeric columns contain real numbers.

import io


def extract_table_vision(image_path: str) -> pd.DataFrame:
    """Extracts a table from an image with the Vision API → DataFrame with numeric validation."""
    prompt = """Extract the table in CSV format.
First line = headers. Comma as separator.
Do not add text before or after. Preserve exact numeric values."""

    csv_text = ocr_with_vision(image_path, prompt=prompt).strip()
    if csv_text.startswith("```"):
        lines = csv_text.split("\n")
        csv_text = "\n".join(lines[1:-1])

    df = pd.read_csv(io.StringIO(csv_text))

    for col in df.columns:
        try:
            df[col] = pd.to_numeric(df[col])
        except (ValueError, TypeError):
            pass

    numeric_cols = df.select_dtypes(include=["number"]).columns
    print(f"Table: {len(df)} rows, {len(df.columns)} cols ({len(numeric_cols)} numeric)")
    return df

Troubleshooting

Problem: Tesseract can't find the "spa" language

Cause: Language data not installed.

Solution: brew install tesseract-lang (macOS) or sudo apt-get install tesseract-ocr-spa (Ubuntu). Verify with tesseract --list-langs. Manual alternative: download spa.traineddata from tesseract-ocr/tessdata.

Problem: The Vision API returns made-up text

Cause: A prompt that isn't restrictive enough or an ambiguous image.

Solution: Use a defensive prompt with temperature 0:

defensive_prompt = """Transcribe ONLY the visible text.
Do NOT interpret, do NOT complete, do NOT invent.
If something is illegible: [illegible]. If there is no text: NO VISIBLE TEXT."""

Problem: Tesseract returns garbage on low-quality images

Cause: Low contrast, noise, insufficient resolution.

Solution: Apply the preprocessing pipeline. Ensure width > 1000px (~300 DPI for a letter-size document). If preprocessing doesn't help, use the hybrid pipeline.

Problem: Table OCR loses column structure

Cause: Tesseract treats the table as continuous text.

Solution: Use --psm 6 for blocks, or split the image into individual cells. For complex tables, use the Vision API with OCR_PROMPTS["table"].

Problem: A scanned PDF with many pages is expensive

Cause: Each page with Vision scales linearly in cost.

Solution: Selective pipeline — Tesseract first, Vision only for pages that fail:

def smart_pdf_ocr(page_images: list[Image.Image]) -> list[str]:
    """Tesseract first, Vision as a per-page fallback."""
    results = []
    for i, img in enumerate(page_images):
        text = pytesseract.image_to_string(img, lang="eng").strip()
        if evaluate_ocr_quality(text) < 0.4:
            img.save(f"/tmp/page_{i}.jpg")
            text = ocr_with_vision(f"/tmp/page_{i}.jpg")
        results.append(text)
    return results

Problem: Timeout or rate limit with the Vision API in batch

Cause: Too many simultaneous requests.

Solution: Retry with exponential backoff:

import time


def ocr_with_retry(image_path: str, max_retries: int = 3) -> str:
    """Vision OCR with retry and exponential backoff."""
    for attempt in range(max_retries):
        try:
            return ocr_with_vision(image_path)
        except Exception as e:
            if "rate_limit" in str(e).lower() or "429" in str(e):
                wait = 2 ** attempt
                time.sleep(wait)
            else:
                raise
    raise RuntimeError(f"Failed after {max_retries} attempts")

Additional Resources

1. Tesseract Documentation
2. pytesseract — Python wrapper
3. OpenAI Vision Guide
4. Anthropic Vision (Claude)
5. Pillow ImageEnhance
6. Tesseract — Improve Quality