6. Long Documents

Description

Documents of 50+ pages (technical manuals, financial reports, lengthy contracts, academic papers) don't fit in a single request to an LLM — or if they do, the cost and latency are prohibitive. In this capsule you'll learn chunking (dividing documents into processable fragments), token estimation, hierarchical summarization with map-reduce, and strategies for Q&A over long documents. These techniques are the direct foundation for document RAG in Module 6.

Why it matters: Without chunking, you can't process 200-page manuals or answer questions about lengthy reports. Sending a complete document to GPT-4o can cost $5-15 per request. The techniques in this capsule let you process any document efficiently, controlling cost and quality. They apply directly in the Document Analyzer of Module 8.

Connection with the module: This capsule connects text extraction (capsule 05) with intelligent processing. The chunking you learn here is exactly what you'll use in RAG (Module 6) to index documents in vector databases.

Key Concepts

Chunking: divide and conquer

Chunking is the process of dividing a long document into smaller fragments that an LLM can process individually. The choice of strategy directly impacts the quality of the results.

Comparison of chunking strategies

Context limits (2024-2026)

Even with 1M-token contexts, sending 100 full pages is expensive and slow. Chunking + selective processing is almost always more efficient.

Chunking by Pages

The simplest strategy: group N consecutive pages into each chunk.

def chunk_by_pages(text_by_page: list[dict], chunk_size: int = 3) -> list[dict]:
    """
    Groups consecutive pages into chunks.
    text_by_page: [{"page": 1, "text": "..."}, ...]
    """
    chunks = []
    for i in range(0, len(text_by_page), chunk_size):
        group = text_by_page[i:i + chunk_size]
        combined = "\n\n".join(p["text"] for p in group)
        chunks.append({
            "chunk_id": len(chunks) + 1,
            "pages": [p["page"] for p in group],
            "text": combined,
            "token_estimate": len(combined) // 4
        })
    return chunks


pages = [{"page": i, "text": f"Page {i} content..."} for i in range(1, 51)]
chunks = chunk_by_pages(pages, chunk_size=5)
print(f"50 pages → {len(chunks)} chunks of 5 pages each")

Chunking by Paragraphs

Respects the natural structure of the text. Groups consecutive paragraphs until reaching a token limit, without ever cutting a paragraph in half.

def chunk_by_paragraphs(text: str, max_tokens: int = 1500) -> list[dict]:
    """Groups consecutive paragraphs up to max_tokens per chunk."""
    max_chars = max_tokens * 4
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks = []
    current_paragraphs = []
    current_len = 0

    for para in paragraphs:
        para_len = len(para)
        if current_len + para_len > max_chars and current_paragraphs:
            chunks.append({
                "chunk_id": len(chunks) + 1,
                "text": "\n\n".join(current_paragraphs),
                "paragraph_count": len(current_paragraphs),
                "token_estimate": current_len // 4
            })
            current_paragraphs = []
            current_len = 0
        current_paragraphs.append(para)
        current_len += para_len + 2

    if current_paragraphs:
        chunks.append({
            "chunk_id": len(chunks) + 1,
            "text": "\n\n".join(current_paragraphs),
            "paragraph_count": len(current_paragraphs),
            "token_estimate": current_len // 4
        })
    return chunks

Chunking by Tokens with tiktoken

For precise control, use tiktoken — OpenAI's official library for counting exact tokens instead of approximations.

import tiktoken


def chunk_by_tokens(text: str, max_tokens: int = 1000, model: str = "gpt-4o") -> list[dict]:
    """Splits text into chunks of exactly max_tokens using tiktoken."""
    enc = tiktoken.encoding_for_model(model)
    tokens = enc.encode(text)
    chunks = []

    for i in range(0, len(tokens), max_tokens):
        chunk_tokens = tokens[i:i + max_tokens]
        chunk_text = enc.decode(chunk_tokens)
        chunks.append({
            "chunk_id": len(chunks) + 1,
            "text": chunk_text,
            "token_count": len(chunk_tokens)
        })
    return chunks


text = "This is a sample text to compare token-counting methods."
enc = tiktoken.encoding_for_model("gpt-4o")
real_tokens = len(enc.encode(text))
approx_tokens = len(text) // 4
print(f"Real tokens: {real_tokens}, Approximation (chars/4): {approx_tokens}")

Chunking by Sections (Headers)

Ideal for structured documents. Each section becomes a semantically coherent chunk.

import re


def chunk_by_sections(text: str) -> list[dict]:
    """Detects sections by headers (##, 1., 1.1, etc.) and creates chunks."""
    header_pattern = re.compile(
        r'^(#{1,6}\s+.+|\d+\.\s+.+|\d+\.\d+\s+.+)$', re.MULTILINE
    )
    sections = []
    current_header = "Introduction"
    current_content = []

    for line in text.split("\n"):
        if header_pattern.match(line.strip()):
            if current_content:
                content = "\n".join(current_content).strip()
                sections.append({
                    "header": current_header,
                    "content": content,
                    "token_estimate": len(content) // 4
                })
            current_header = line.strip()
            current_content = []
        else:
            current_content.append(line)

    if current_content:
        content = "\n".join(current_content).strip()
        sections.append({
            "header": current_header,
            "content": content,
            "token_estimate": len(content) // 4
        })
    return sections

Chunking with Overlap

Why overlap matters

When you split a document into chunks without overlap, context is lost at the edges. If an important sentence ends up at the end of chunk 3 and its continuation at the start of chunk 4, neither has the complete idea.

Without overlap:
  Chunk 1: [A B C D E]
  Chunk 2: [F G H I J]
  → Context is lost between E and F

With overlap (2 elements):
  Chunk 1: [A B C D E]
  Chunk 2: [D E F G H]
  → D and E appear in both, preserving continuity

Typical overlap is 10-20% of the chunk size. More overlap = more context preserved, but also more chunks and more cost.

Implementation

import tiktoken


def chunk_with_overlap(
    text: str,
    max_tokens: int = 1000,
    overlap_tokens: int = 150,
    model: str = "gpt-4o"
) -> list[dict]:
    """Token chunking with configurable overlap."""
    enc = tiktoken.encoding_for_model(model)
    tokens = enc.encode(text)
    total = len(tokens)
    step = max_tokens - overlap_tokens

    if step <= 0:
        raise ValueError("overlap_tokens must be less than max_tokens")

    chunks = []
    for start in range(0, total, step):
        end = min(start + max_tokens, total)
        chunk_tokens = tokens[start:end]
        chunks.append({
            "chunk_id": len(chunks) + 1,
            "text": enc.decode(chunk_tokens),
            "token_count": len(chunk_tokens),
            "has_overlap": start > 0
        })
        if end == total:
            break
    return chunks


long_text = "word " * 5000
chunks = chunk_with_overlap(long_text, max_tokens=1000, overlap_tokens=150)
print(f"Total chunks: {len(chunks)}, Chunk 2 overlap: {chunks[1]['has_overlap']}")

Token Estimation

Before processing a document, you need to know how many tokens it has to choose the right strategy and estimate costs.

import tiktoken


def estimate_document_tokens(text_by_page: list[dict], model: str = "gpt-4o") -> dict:
    """Counts tokens, checks context and recommends a strategy."""
    enc = tiktoken.encoding_for_model(model)
    context_limits = {
        "gpt-4o": 128_000, "gpt-4o-mini": 128_000,
        "claude-3-5-sonnet": 200_000, "gemini-1.5-pro": 1_000_000}

    page_stats = []
    total_tokens = 0
    for page in text_by_page:
        page_tokens = len(enc.encode(page["text"]))
        total_tokens += page_tokens
        page_stats.append({"page": page["page"], "tokens": page_tokens})

    avg_tokens_per_page = total_tokens // max(len(text_by_page), 1)

    fits_in = {
        name: total_tokens < (limit * 0.8)
        for name, limit in context_limits.items()
    }

    if total_tokens < 4_000:
        strategy, reason = "direct", "Short document, send it whole"
    elif total_tokens < 50_000:
        strategy, reason = "single_pass", "Fits in context, but consider the cost"
    else:
        strategy, reason = "chunking", "Long document, use chunking"

    return {
        "total_tokens": total_tokens,
        "total_pages": len(text_by_page),
        "avg_tokens_per_page": avg_tokens_per_page,
        "fits_in_context": fits_in,
        "recommended_strategy": strategy,
        "reason": reason,
        "page_stats": page_stats
    }


import fitz

doc = fitz.open("annual_report.pdf")
pages = [{"page": i + 1, "text": doc[i].get_text()} for i in range(len(doc))]
doc.close()

analysis = estimate_document_tokens(pages)
print(f"Tokens: {analysis['total_tokens']:,} | Strategy: {analysis['recommended_strategy']}")
for model_name, fits in analysis["fits_in_context"].items():
    print(f"  {model_name}: {'✓ Fits' if fits else '✗ Does not fit'}")

Hierarchical Summarization (Map-Reduce)

For very long documents, the map-reduce pattern is the most robust strategy: summarize each chunk individually (map), then combine the summaries into a final summary (reduce).

Document (200 pages)
    ↓ chunking
[Chunk 1] [Chunk 2] [Chunk 3] ... [Chunk 20]
    ↓ map (summarize each one)
[Summary 1] [Summary 2] [Summary 3] ... [Summary 20]
    ↓ reduce (combine summaries)
[Final Summary]

from openai import OpenAI
client = OpenAI()

def summarize_chunk(chunk_text: str, chunk_id: int, context: str = "") -> str:
    """Map: summarizes an individual chunk."""
    prompt = f"""Summarize the following fragment in 3-5 sentences.
Keep specific data (figures, names, dates).
{f"Document context: {context}" if context else ""}

Fragment {chunk_id}:
{chunk_text[:6000]}"""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=300
    )
    return response.choices[0].message.content


def combine_summaries(summaries: list[str], doc_type: str = "document") -> str:
    """Reduce: combines partial summaries into a final summary."""
    combined = "\n\n".join(
        f"[Section {i+1}]: {s}" for i, s in enumerate(summaries)
    )
    prompt = f"""From these partial summaries of a {doc_type}, generate:
1. **Executive summary** (1 paragraph, maximum 100 words)
2. **Key points** (5-7 bullets)
3. **Relevant data** (figures, dates, names mentioned)

Partial summaries:
{combined}"""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=800
    )
    return response.choices[0].message.content


def hierarchical_summary(text: str, chunk_tokens: int = 1500, doc_type: str = "document") -> dict:
    """Complete map-reduce hierarchical summary pipeline."""
    chunks = chunk_by_tokens(text, max_tokens=chunk_tokens)
    print(f"Processing {len(chunks)} chunks...")

    chunk_summaries = []
    for i, chunk in enumerate(chunks):
        text_content = chunk["text"] if isinstance(chunk, dict) else chunk
        summary = summarize_chunk(text_content, i + 1)
        chunk_summaries.append(summary)
        print(f"  Chunk {i+1}/{len(chunks)} summarized")

    print("Generating final summary...")
    final_summary = combine_summaries(chunk_summaries, doc_type)
    return {
        "chunk_count": len(chunks),
        "chunk_summaries": chunk_summaries,
        "final_summary": final_summary
    }

Q&A over Long Documents

Two approaches depending on the document size:

Approach 1: Stuff (whole document in context)

If the document fits in the context window, send it whole. Simple and effective.

def qa_stuff(document_text: str, question: str) -> str:
    """Q&A sending the whole document. Works if it fits in context."""
    enc = tiktoken.encoding_for_model("gpt-4o")
    doc_tokens = len(enc.encode(document_text))

    if doc_tokens > 100_000:
        raise ValueError(
            f"Document has {doc_tokens:,} tokens. Use qa_chunk_retrieve."
        )

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Answer based ONLY on the document. If the answer isn't there, say so."},
            {"role": "user", "content": f"Document:\n{document_text}\n\nQuestion: {question}"}
        ],
        max_tokens=500
    )
    return response.choices[0].message.content

Approach 2: Chunk + Retrieve (Proto-RAG)

For documents that don't fit in context: split into chunks, find the most relevant ones, and answer using only those.

def qa_chunk_retrieve(chunks: list[dict], question: str, top_k: int = 3) -> dict:
    """Proto-RAG: scores chunks by relevance and answers with the best ones."""
    scored_chunks = []
    for chunk in chunks:
        text = chunk["text"] if isinstance(chunk, dict) else chunk
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": (
                f"How relevant is this fragment to the question?\n"
                f"Respond with ONLY a number from 0 to 10.\n\n"
                f"Question: {question}\n\nFragment:\n{text[:2000]}"
            )}],
            max_tokens=5
        )
        try:
            score = int(response.choices[0].message.content.strip())
        except ValueError:
            score = 0
        scored_chunks.append({"chunk": chunk, "score": score})

    top_chunks = sorted(scored_chunks, key=lambda x: x["score"], reverse=True)[:top_k]
    context = "\n\n---\n\n".join(
        c["chunk"]["text"] if isinstance(c["chunk"], dict) else c["chunk"]
        for c in top_chunks
    )

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Answer based ONLY on the fragments."},
            {"role": "user", "content": f"Fragments:\n{context}\n\nQuestion: {question}"}
        ],
        max_tokens=500
    )
    return {
        "answer": response.choices[0].message.content,
        "chunks_used": [c["chunk"].get("chunk_id", "?") for c in top_chunks],
        "relevance_scores": [c["score"] for c in top_chunks]
    }

Note: This "proto-RAG" approach uses the LLM to evaluate relevance, which is slow and expensive. In Module 6 you'll learn to use embeddings + vector search to do this in milliseconds.

Pipeline: Long Document → Chunks → Processing → Combination

An end-to-end pipeline that integrates extraction, analysis, chunking and processing.

import fitz
from openai import OpenAI
client = OpenAI()

def process_long_document(pdf_path: str, task: str = "summary") -> dict:
    """
    Complete pipeline for long documents.
    task: "summary" for hierarchical summary, "extract" for raw chunks
    """
    doc = fitz.open(pdf_path)
    text_by_page = []
    for i in range(len(doc)):
        page_text = doc[i].get_text()
        if page_text.strip():
            text_by_page.append({"page": i + 1, "text": page_text})
    doc.close()

    if not text_by_page:
        return {"error": "No text extracted. The PDF may be scanned (use OCR)."}

    full_text = "\n\n".join(p["text"] for p in text_by_page)
    analysis = estimate_document_tokens(text_by_page)
    print(f"Pages: {analysis['total_pages']} | Tokens: {analysis['total_tokens']:,}")

    if analysis["recommended_strategy"] == "direct":
        chunks = [{"chunk_id": 1, "text": full_text}]
    else:
        sections = chunk_by_sections(full_text)
        if len(sections) >= 3:
            chunks = [
                {"chunk_id": i + 1, "text": f"{s['header']}\n{s['content']}", "header": s["header"]}
                for i, s in enumerate(sections)
            ]
        else:
            chunks = chunk_with_overlap(full_text, max_tokens=1500, overlap_tokens=200)

    if task == "summary":
        result = hierarchical_summary(full_text, doc_type="PDF document")
        return {
            "task": "summary", "pages": analysis["total_pages"],
            "tokens": analysis["total_tokens"],
            "chunks_processed": result["chunk_count"],
            "summary": result["final_summary"]
        }
    if task == "extract":
        return {
            "task": "extract", "pages": analysis["total_pages"],
            "tokens": analysis["total_tokens"],
            "chunks": chunks, "analysis": analysis
        }
    return {"error": f"Unrecognized task: {task}"}


result = process_long_document("technical_manual.pdf", task="summary")
print(result["summary"])

Cost Optimization for Long Documents

Three strategies to reduce costs without sacrificing quality.

1. Estimate cost before processing

def estimate_processing_cost(total_tokens: int, task: str = "summary") -> dict:
    """Estimates the processing cost. Approx. prices per 1M tokens (2025)."""
    prices = {
        "gpt-4o": {"input": 2.50, "output": 10.00},
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    }
    if task == "summary":
        input_tokens, output_tokens, model = total_tokens * 1.1, total_tokens * 0.15, "gpt-4o-mini"
    elif task == "qa":
        input_tokens, output_tokens, model = total_tokens * 0.3, 500, "gpt-4o"
    else:
        input_tokens, output_tokens, model = total_tokens, total_tokens * 0.1, "gpt-4o-mini"

    cost_input = (input_tokens / 1_000_000) * prices[model]["input"]
    cost_output = (output_tokens / 1_000_000) * prices[model]["output"]
    total_cost = cost_input + cost_output
    return {
        "model": model,
        "estimated_input_tokens": int(input_tokens),
        "estimated_output_tokens": int(output_tokens),
        "total_cost": round(total_cost, 4),
        "cost_display": f"${total_cost:.4f}"
    }


cost = estimate_processing_cost(150_000, task="summary")
print(f"Estimated cost: {cost['cost_display']} with {cost['model']}")

2. Selective processing

You don't always need to process the entire document. Filter relevant pages first.

def selective_processing(text_by_page: list[dict], topic: str, max_pages: int=20) -> list[dict]:
    """Selects only the pages relevant to the topic via keyword matching."""
    keywords = topic.lower().split()
    scored_pages = []
    for page in text_by_page:
        text_lower = page["text"].lower()
        score = sum(text_lower.count(kw) for kw in keywords)
        scored_pages.append({**page, "relevance": score})

    relevant = sorted(scored_pages, key=lambda x: x["relevance"], reverse=True)
    selected = [p for p in relevant[:max_pages] if p["relevance"] > 0]
    if not selected:
        selected = relevant[:5]
    selected.sort(key=lambda x: x["page"])
    return selected

3. Progressive detail

Quick summary first; go deeper only where the user asks.

def progressive_detail(text_by_page: list[dict], question: str = None) -> dict:
    """Level 1: quick summary. Level 2: detail on a specific question."""
    sample_text = "\n\n".join(p["text"] for p in text_by_page[:10])
    quick_summary = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"Summarize in 5 bullets:\n{sample_text[:8000]}"}],
        max_tokens=300
    ).choices[0].message.content

    result = {"level_1_summary": quick_summary}
    if question:
        relevant_pages = selective_processing(text_by_page, question, max_pages=10)
        context = "\n\n".join(p["text"] for p in relevant_pages)
        detailed = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "Answer in detail based on the document."},
                {"role": "user", "content": f"Document:\n{context[:20000]}\n\nQuestion: {question}"}
            ],
            max_tokens=800
        ).choices[0].message.content
        result["level_2_detail"] = detailed
        result["pages_used"] = [p["page"] for p in relevant_pages]
    return result

Troubleshooting

Problem: Context length exceeded

Symptom: maximum context length exceeded error when sending text to the LLM. Solution: Use estimate_document_tokens() before sending. If it exceeds 80% of the context, apply chunking. Always leave room for the system prompt and the response.

Problem: Information lost at chunk edges

Symptom: Incomplete answers that ignore data at the boundary between two chunks. Solution: Use chunk_with_overlap() with 10-20% overlap. Increase the overlap or use section-based chunking if a critical piece of data gets cut.

Problem: Inconsistent summaries across chunks

Symptom: The final summary contains contradictions or repeats information. Solution: In the map step, include document context (title, type, topic). In reduce, instruct the model to resolve contradictions and remove redundancies.

Problem: High cost when processing many documents

Symptom: An unexpectedly high bill when processing batches of long documents. Solution: Use gpt-4o-mini for summaries and scoring; reserve gpt-4o only for the final answer. Implement estimate_processing_cost() before each run.

Problem: Slow processing on large documents

Symptom: The pipeline takes minutes for 100+ page documents. Solution: Process chunks in parallel with asyncio or concurrent.futures. Use larger chunks (2000-3000 tokens) to reduce the total.

Exercises

Exercise 1: Document analyzer with recommendation

Create a function that takes a text, counts tokens with tiktoken, and returns: total tokens, whether it fits in GPT-4o/Claude, and the recommended strategy.

import tiktoken

def analyze_document(text: str) -> dict:
    enc = tiktoken.encoding_for_model("gpt-4o")
    total_tokens = len(enc.encode(text))
    fits_gpt4o = total_tokens < 100_000
    fits_claude = total_tokens < 160_000

    if total_tokens < 4_000:
        strategy = "direct"
    elif total_tokens < 50_000:
        strategy = "single_pass"
    else:
        strategy = "chunking"

    return {
        "total_tokens": total_tokens,
        "fits_gpt4o": fits_gpt4o,
        "fits_claude": fits_claude,
        "strategy": strategy
    }

result = analyze_document("Your long text here..." * 1000)
print(f"Tokens: {result['total_tokens']:,}, Strategy: {result['strategy']}")

Exercise 2: Intelligent chunking with fallback

Try section-based chunking first. If there are fewer than 3 sections, fall back to chunking with 100-token overlap.

def smart_chunking(text: str, max_tokens: int = 1500, overlap_tokens: int = 100) -> list[dict]:
    sections = chunk_by_sections(text)

    if len(sections) >= 3:
        return [
            {"chunk_id": i + 1, "text": f"{s['header']}\n{s['content']}",
             "strategy": "sections", "header": s["header"]}
            for i, s in enumerate(sections)
        ]

    chunks = chunk_with_overlap(text, max_tokens=max_tokens, overlap_tokens=overlap_tokens)
    for c in chunks:
        c["strategy"] = "paragraphs_with_overlap"
    return chunks

chunks = smart_chunking("# Intro\nText...\n\n# Method\nText...\n\n# Results\nText...")
print(f"Strategy: {chunks[0]['strategy']}, Total: {len(chunks)}")

Exercise 3: Q&A with automatic approach selection

If the document has fewer than 80K tokens, use "stuff". If it has more, use "chunk + retrieve" with the 3 most relevant chunks.

def smart_qa(document_text: str, question: str) -> dict:
    enc = tiktoken.encoding_for_model("gpt-4o")
    total_tokens = len(enc.encode(document_text))

    if total_tokens < 80_000:
        answer = qa_stuff(document_text, question)
        return {"approach": "stuff", "tokens": total_tokens, "answer": answer}

    chunks = chunk_with_overlap(document_text, max_tokens=1500, overlap_tokens=200)
    result = qa_chunk_retrieve(chunks, question, top_k=3)
    return {"approach": "chunk_retrieve", "tokens": total_tokens, "chunks_total": len(chunks), **result}

result = smart_qa("Extended document here...", "What is the main conclusion?")
print(f"Approach: {result['approach']}, Answer: {result['answer']}")

Exercise 4: Cost estimator for a batch of PDFs

Create a function that takes a list of PDF paths and generates a report with tokens per document, individual cost and total batch cost.

import fitz
import tiktoken

def batch_cost_report(pdf_paths: list[str]) -> dict:
    enc = tiktoken.encoding_for_model("gpt-4o")
    documents = []
    total_cost = 0

    for path in pdf_paths:
        doc = fitz.open(path)
        text = "\n".join(doc[i].get_text() for i in range(len(doc)))
        page_count = len(doc)
        doc.close()

        tokens = len(enc.encode(text))
        cost = estimate_processing_cost(tokens, task="summary")
        total_cost += cost["total_cost"]
        documents.append({
            "path": path, "pages": page_count,
            "tokens": tokens, "cost": cost["cost_display"]
        })

    return {
        "documents": documents, "total_documents": len(documents),
        "total_cost": f"${total_cost:.4f}", "model": "gpt-4o-mini"
    }

report = batch_cost_report(["doc1.pdf", "doc2.pdf", "doc3.pdf"])
for d in report["documents"]:
    print(f"  {d['path']}: {d['pages']} pages, {d['tokens']:,} tokens → {d['cost']}")
print(f"Total cost: {report['total_cost']}")

Additional Resources

1. tiktoken — OpenAI's official tokenizer
2. LangChain Text Splitters
3. OpenAI Context length per model
4. Chunking strategies for RAG