Capsule 10: Document chunking — the whole-document problem

Capsule description

You have a 50-page technical manual. Your first intuition: "I embed it whole, ChromaDB searches it, the LLM reads it". It sounds logical — and it almost always fails.

The problem isn't technical capacity. OpenAI text-embedding-3-small accepts up to 8191 tokens per input (more than 30 pages). The problem is search resolution: a 1536-dimensional vector that represents 50 pages is a fuzzy average of everything. When the user asks "how do I configure HNSW?", that average vector doesn't point to the specific section that answers — it points to the "general topic of the manual".

The solution is called chunking: splitting each long document into smaller pieces, embedding each piece separately, and letting retrieval find the exact piece. It sounds simple. The decisions (what size, what overlap, split by characters or by structure) are what separate a mediocre RAG from one that returns precise answers.

This capsule gives you the mental model, the default decisions that work in 80% of cases, and the criteria to adjust them when your domain demands it.

By the end of this capsule, you'll be able to:

• ✅ Explain why embedding a whole long document degrades retrieval
• ✅ Apply chunking with RecursiveCharacterTextSplitter, justifying the parameters
• ✅ Decide chunk_size and chunk_overlap for a specific domain
• ✅ Compare three chunking strategies (fixed, recursive, semantic) and choose the right one
• ✅ Configure chunk metadata to preserve traceability to the original document
• ✅ Anticipate the most subtle mistake: chunks that split sentences in half and break the meaning

Estimated time: 35-45 minutes

Why a whole document fails

Imagine two documents in your collection:

Document A (1500 words): a technical manual that covers ChromaDB installation in section 1, HNSW configuration in section 2, basic queries in section 3, and troubleshooting in section 4.

Document B (1500 words): another manual about Pinecone that covers the same four topics in their respective sections.

Both are embedded as a single vector each. What does that vector represent?

Geometrically, the embedding of a long text is something like the semantic "center of gravity" of all the content. If the document touches four different topics, the resulting vector ends up at some central point that doesn't precisely represent any of the four.

The experiment that makes it evident

Load these two documents into a collection and run a very specific query:

# experiment_whole_document.py
import chromadb
from chromadb.utils import embedding_functions
import os

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small"
)

client = chromadb.PersistentClient(path="./chroma_chunking_test")
collection = client.get_or_create_collection(
    name="whole_docs",
    embedding_function=openai_ef
)

doc_a = """
Chapter 1: Installing ChromaDB.
ChromaDB is installed with pip install chromadb. It requires Python 3.8+.

Chapter 2: HNSW Configuration.
HNSW has two main parameters: M (16 default, controls graph connections)
and construction_ef (100 default, controls build quality). For production with
high accuracy, use M=32 and construction_ef=200.

Chapter 3: Basic queries.
collection.query(query_texts=["..."], n_results=10) returns the k most similar vectors.

Chapter 4: Troubleshooting.
If query_embeddings don't match documents, verify that the embedding_function is consistent.
"""

doc_b = """
Chapter 1: Installing Pinecone.
Pinecone requires registration at pinecone.io and getting an API key. pip install pinecone-client.

Chapter 2: Pod configuration.
Pinecone pods have types s1 (storage), p1 (performance), p2 (high performance).

Chapter 3: Queries with Pinecone.
index.query(vector=[...], top_k=10) returns results with metadata.

Chapter 4: Troubleshooting.
If latency is high, verify the pod type and the cluster region.
"""

collection.add(
    documents=[doc_a, doc_b],
    ids=["doc_a", "doc_b"],
    metadatas=[{"product": "chromadb"}, {"product": "pinecone"}]
)

# Very specific query about HNSW (it's only in doc_a, section 2)
query = "What are the recommended HNSW parameters for production with high accuracy?"
results = collection.query(query_texts=[query], n_results=2)

for doc, dist in zip(results['documents'][0], results['distances'][0]):
    first_line = doc.strip().split('\n')[0]
    print(f"Distance: {dist:.3f} | Starts with: {first_line}")

Expected output:

Distance: 0.413 | Starts with: Chapter 1: Installing ChromaDB.
Distance: 0.587 | Starts with: Chapter 1: Installing Pinecone.

What happened? The query was 100% specific to section 2 of doc_a, but ChromaDB returned both whole documents. The score of the relevant one (0.413) isn't bad, but what the LLM will read is all 4 ChromaDB chapters whole, not just the section about HNSW. The context window fills with irrelevant information (installation, queries, troubleshooting) that dilutes the section that really matters.

And worse: the difference between the relevant doc (0.413) and the irrelevant one (0.587) is narrow. A poorly calibrated threshold lets Pinecone in too — and the LLM ends up reading two manuals in its context window when it should be reading two paragraphs.

The pedagogical insight

The embedding averages the content. If your document covers four topics, the vector points to the "centroid of the four topics", not to any one of them. For retrieval to work well, each vector must represent a coherent, compact idea — ideally, a single question that that chunk answers.

That's why chunking isn't an implementation detail. It's the architectural decision that defines your retrieval resolution.

The three chunking strategies

1. Fixed-size chunking (fixed chunking by characters)

The simplest strategy: split the text every N characters. Without understanding structure, without respecting words.

def fixed_chunk(text: str, size: int = 500) -> list[str]:
    return [text[i:i+size] for i in range(0, len(text), size)]

text = "ChromaDB uses HNSW by default. The parameters are M=16 and construction_ef=100..."
chunks = fixed_chunk(text, size=30)
# ['ChromaDB uses HNSW by default.',
#  ' The parameters are M=16 and c',  ← splits a word
#  'onstruction_ef=100...']

Advantage: trivial to implement, predictable.

Fatal problem: it splits words, sentences, and concepts in half. The middle chunk of the example (The parameters are M=16 and c) embedded in isolation is semantic noise — it doesn't mean what it says.

When to use it: practically never. It exists only as a contrast.

2. Recursive character chunking (what you'll use 80% of the time)

RecursiveCharacterTextSplitter (from the LangChain library) tries to split while respecting the natural structure of the text: first by double line breaks (paragraph separators), then by single line breaks (sentence separators), then by spaces, and only as a last resort does it split words.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,        # Target size in characters
    chunk_overlap=50,      # Overlap between consecutive chunks
    separators=["\n\n", "\n", ". ", " ", ""],  # Preference order
    length_function=len
)

chunks = splitter.split_text(long_document)

How the algorithm operates (mental model):

1. It tries to split the text at \n\n (paragraphs). If the pieces come out ≤500 chars, done.
2. If any piece is still >500, it splits it again with \n (lines).
3. If it's still >500, it tries ". " (sentences).
4. If it's still >500, it splits by spaces (words).
5. If it's still >500 (a super rare word), it splits by character.

The result: chunks that respect the natural structure of the text when possible. A sentence isn't split in half except in extreme cases.

When to use it: the first default for almost everything. It covers 80% of cases without much thought.

3. Semantic chunking (when conceptual coherence matters)

Instead of a fixed size, it uses embeddings to find points where the meaning changes. When two consecutive sentences have very different embeddings, it splits there.

# Pseudo-code of the algorithm
def semantic_chunk(text: str, threshold: float = 0.7):
    sentences = split_into_sentences(text)
    embeddings = [embed(s) for s in sentences]

    chunks = []
    current_chunk = [sentences[0]]
    for i in range(1, len(sentences)):
        similarity = cosine_similarity(embeddings[i-1], embeddings[i])
        if similarity < threshold:
            # Topic change → new chunk
            chunks.append(' '.join(current_chunk))
            current_chunk = [sentences[i]]
        else:
            current_chunk.append(sentences[i])
    chunks.append(' '.join(current_chunk))
    return chunks

Advantage: each chunk is semantically coherent.

Cost: you have to embed each sentence just to decide where to split. It triples the ingestion cost. Processing time rises noticeably.

When to use it: domains where the quality difference justifies the cost (legal, medical, academic papers where a badly cut chunk leads to dangerously wrong answers).

When not to use it: most cases. Start with recursive, measure, and only change if the quality doesn't reach.

The two critical decisions: chunk_size and chunk_overlap

chunk_size: how many characters per chunk

The right size depends on the information density of your domain:

Reasonable default if you don't know: chunk_size=500. It works well for general text in Spanish or English. If your tokens are ~4 characters on average (an approximate rule for English), 500 chars ≈ 125 tokens — well below the embedding model's limit.

chunk_overlap: how many characters consecutive chunks share

Overlap exists for a specific reason: to prevent an answer from falling split between two chunks with neither capturing it whole.

Without overlap (chunk_size=100):
Chunk 1: "...to configure HNSW with high accuracy, M=32 and"
Chunk 2: "construction_ef=200 are recommended, values that improve recall in production..."
                    ↑
        The answer to "which M to use?" falls split.
        No individual chunk contains "M=32 and construction_ef=200" whole.

With overlap=50:
Chunk 1: "...to configure HNSW with high accuracy, M=32 and"
Chunk 2: "M=32 and construction_ef=200 are recommended, values that improve..."
              ↑                ↑
        The last 50 chars of chunk 1 reappear at the start of chunk 2.
        Now chunk 2 contains "M=32 and construction_ef=200" whole.

Practical rule:

• chunk_overlap = chunk_size * 0.1 (10% is a good default)
• For chunk_size=500, use chunk_overlap=50
• For chunk_size=1000, use chunk_overlap=100

Trade-off: more overlap = more partially duplicated chunks = more embedding and storage cost. 10% is the point where coverage improves noticeably without inflating the cost.

Don't do this: chunk_overlap=0. You'll lose answers at the boundaries. You'll see it as "the RAG didn't find the answer and I swear it's in the documents".

Also don't do this: chunk_overlap > chunk_size / 2. The second chunk repeats more than 50% of the first. You're duplicating almost the whole dataset and the retrieval results fill with similar chunks.

Practical implementation with ChromaDB

Complete pipeline: document → chunks with metadata → ChromaDB

# chunking_pipeline.py
import os
from langchain.text_splitter import RecursiveCharacterTextSplitter
import chromadb
from chromadb.utils import embedding_functions

# Setup
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small"
)

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""]
)

client = chromadb.PersistentClient(path="./chroma_with_chunks")
collection = client.get_or_create_collection(
    name="documents_chunked",
    embedding_function=openai_ef
)


def ingest_document(doc_id: str, content: str, source_path: str, category: str):
    """
    Process a document: chunking + metadata per chunk + insert into ChromaDB.

    Each chunk preserves traceability to the original document (doc_id, chunk_index)
    so the RAG system can show the exact source of each answer.
    """
    chunks = splitter.split_text(content)

    chunk_ids = [f"{doc_id}_chunk_{i:03d}" for i in range(len(chunks))]
    chunk_metadatas = [
        {
            "doc_id": doc_id,
            "chunk_index": i,
            "total_chunks": len(chunks),
            "source": source_path,
            "category": category,
            "chunk_size_chars": len(chunk),
        }
        for i, chunk in enumerate(chunks)
    ]

    collection.add(
        documents=chunks,
        ids=chunk_ids,
        metadatas=chunk_metadatas
    )

    print(f"Ingested {doc_id}: {len(chunks)} chunks (avg {sum(len(c) for c in chunks)//len(chunks)} chars)")


# Usage
long_document = """
Chapter 1: Installing ChromaDB.
ChromaDB is installed with pip install chromadb. It requires Python 3.8 or higher.
For production use, we recommend pinning a specific version in requirements.txt.

Chapter 2: HNSW Configuration.
HNSW has two main parameters that affect the balance between speed and accuracy.
M (default 16) controls how many connections each node has in the HNSW graph.
For production with high accuracy, use M=32 and construction_ef=200.
Increasing M improves recall but also increases memory usage proportionally.

Chapter 3: Basic queries.
A basic query uses collection.query(query_texts=["..."], n_results=10).
ChromaDB returns the n_results most similar according to the configured distance metric.
For OpenAI text embeddings, cosine similarity is the recommended metric.
"""

ingest_document(
    doc_id="manual_chromadb_v1",
    content=long_document,
    source_path="docs/chromadb/manual.md",
    category="documentation"
)

Expected output:

Ingested manual_chromadb_v1: 3 chunks (avg 263 chars)

Retrieve and reconstruct the original document's context

When a query returns chunks, the metadata lets you locate the source:

# Search and show the full context
query = "What are the recommended HNSW parameters for production?"
results = collection.query(query_texts=[query], n_results=3)

print(f"Query: {query}\n")
for doc, meta, dist in zip(
    results['documents'][0],
    results['metadatas'][0],
    results['distances'][0]
):
    print(f"--- Distance: {dist:.3f} ---")
    print(f"Source: {meta['source']}")
    print(f"Doc: {meta['doc_id']} (chunk {meta['chunk_index']+1}/{meta['total_chunks']})")
    print(f"Content: {doc}")
    print()

Expected output:

Query: What are the recommended HNSW parameters for production?

--- Distance: 0.198 ---
Source: docs/chromadb/manual.md
Doc: manual_chromadb_v1 (chunk 2/3)
Content: Chapter 2: HNSW Configuration. HNSW has two main parameters that affect
the balance between speed and accuracy. M (default 16) controls how many
connections each node has in the HNSW graph. For production with high accuracy,
use M=32 and construction_ef=200. Increasing M...

--- Distance: 0.412 ---
Source: docs/chromadb/manual.md
Doc: manual_chromadb_v1 (chunk 3/3)
Content: Chapter 3: Basic queries. A basic query uses collection.query...

Compare with the experiment at the start: now the relevant chunk has distance 0.198 (vs 0.413 with the whole document). Retrieval went from "it found the right document among two options" to "it found the exact paragraph that answers the question".

Traps and common mistakes

Trap 1: chunk_overlap at zero "to save"

The mistake: you decide overlap is a waste because it duplicates data. You set it to 0.

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)

Symptom: specific queries that should match specific chunks don't return the right chunks. The LLM's answer says "I don't have information about that" when the document clearly has it.

Why it happens: the answer falls split at the boundary between two chunks. Neither the previous chunk nor the next one contains it whole. Without overlap, that information is invisible to retrieval.

How to fix it: use at least 10% overlap. The "savings" of removing overlap is trivial (~10% more chunks) compared to the damage to recall.

Trap 2: chunk_size too small

The mistake: someone tells you "smaller is more precise", so you use chunk_size=100.

Symptom: the returned chunks don't have enough context for the LLM to generate a coherent answer. The LLM's answer is fragmentary, missing surrounding information.

Why it happens: a 100-character chunk is ~25 tokens. The answer to a technical question rarely fits in 25 tokens. The LLM reads the chunk and says "it says X here, but I don't know the full context".

How to detect it: queries that should have a complete answer return truncated or vague answers, even when the right chunks are retrieved.

How to fix it: raise chunk_size to 400-800 depending on your domain. Measure recall AND answer quality, not just recall.

Trap 3: chunk_size too large

The mistake: "if large is bad, even larger is worse but at least it covers more". You use chunk_size=3000.

Symptom: retrieval works, but the results rank badly — irrelevant chunks appear high and the relevant ones sometimes don't.

Why it happens: you're back to the whole-document problem at a smaller scale. The embedding of a 3000-char chunk averages multiple topics. You lose resolution.

How to fix it: keep chunks ≤1500 chars except in domains with very long arguments (academic papers).

Trap 4: splitting by the wrong delimiters for your domain

The mistake: you use the default separators ["\n\n", "\n", ". ", " ", ""] to process source code. Code doesn't use "\n\n" to separate logical blocks — it uses indentation and braces.

Symptom: the chunks split functions in half, leave braces open, break the structure.

How to fix it: adjust the separators to the domain. For code:

splitter_code = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=80,
    separators=["\nclass ", "\ndef ", "\n\n", "\n", " ", ""]
)

For Markdown:

splitter_md = RecursiveCharacterTextSplitter(
    chunk_size=600,
    chunk_overlap=60,
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " ", ""]
)

Trap 5: forgetting the doc_id in metadata

The mistake: you chunk, insert, everything works. But you only save the chunk content without metadata connecting it to the original document.

# ❌ Insert chunks without traceability to the original doc
collection.add(
    documents=chunks,
    ids=[f"chunk_{i}" for i in range(len(chunks))]
    # No metadata
)

Symptom: when a query returns a useful chunk, you can't show the user "this answer comes from manual.md, section 2". The chunks are orphaned. If the user reports an error in an answer, you can't find the original document to fix it.

How to fix it: always include doc_id, chunk_index, and source in metadata. This is non-negotiable for production RAG.

Trap 6: re-chunking without re-inserting

The mistake: you decide to change from chunk_size=300 to chunk_size=600 to improve context. You modify the code that processes new documents. But the old documents keep their 300-char chunks.

Symptom: queries that depend on old documents keep failing as before; queries about new documents improve. Inconsistent results depending on which document answers.

How to fix it: when you change chunking parameters, you must re-process all existing documents. The migration is:

1. Create a new collection
2. Re-read each original document
3. Re-chunk with the new parameters
4. Insert into the new collection
5. Verify that count() is consistent
6. Switch the app to use the new collection
7. Delete the old collection

There's no shortcut.

Applied exercise

Scenario: You work with a legal services company. They ask you to build a RAG over 200 PDF documents of case law. Each PDF has between 5 and 30 pages. Characteristics:

• Language: legal Spanish (formal, long sentences)
• Typical structure: case header → facts → legal grounds → resolution
• The lawyers' queries are specific: "what did the judge say about the statute of limitations in cases of moral damage?"
• SLA: the answers must literally cite the fragment of the document
• It's critical not to lose context between sections (a legal ground may reference facts described earlier)

Question: Configure RecursiveCharacterTextSplitter for this case. Justify each parameter and add any additional strategy you'd recommend.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter_legal = RecursiveCharacterTextSplitter(
    chunk_size=1200,       # Larger than default due to long legal sentences
    chunk_overlap=200,     # ~17% overlap for references between sections
    length_function=len,
    separators=[
        "\nRESOLUCIÓN",     # Known structure of the domain
        "\nFUNDAMENTOS",
        "\nHECHOS",
        "\n\n",              # Paragraph separation
        "\n",                # Line
        ". ",                # Sentence (careful: it can split legal abbreviations)
        " ",
        ""
    ]
)

metadata = {
    "doc_id": "case_2024_001",
    "chunk_index": i,
    "section": detect_section(chunk),  # "facts" / "grounds" / "resolution"
    "court": "Supreme Court",
    "year": 2024,
    "case_topic": "moral_damage_statute_of_limitations",
    "source_page": detect_pdf_page(chunk),  # To cite the exact page
}

def clean_legal_pdf_text(raw: str) -> str:
    raw = remove_page_headers_footers(raw)
    raw = normalize_section_titles(raw)  # "Fundamentos jurídicos" → "FUNDAMENTOS"
    return raw

Summary and next step

What you learned:

• Embedding a whole document degrades retrieval — the resulting vector is a fuzzy average that doesn't point to any specific section.
• Chunking splits long documents into semantically coherent pieces. Each chunk is embedded separately and competes individually in retrieval.
• Three strategies: fixed-size (don't use), recursive character (default for 80% of cases), semantic (when the domain justifies the cost).
• RecursiveCharacterTextSplitter with chunk_size=500 and chunk_overlap=50 is the reasonable default. Adjust according to the information density of your domain.
• Overlap (~10% of chunk_size) prevents answers from falling split between two chunks.
• Per-chunk metadata (doc_id, chunk_index, source) is critical for traceability in production.

Checkpoint: before moving on, you should be able to:

• Explain in your own words why embedding long documents as a single vector worsens retrieval.
• Configure RecursiveCharacterTextSplitter with justified parameters for a given domain.
• Identify the three most expensive chunking traps (overlap=0, chunk_size too small, chunks without metadata).

Next capsule: 11 — Basic end-to-end RAG pipeline.

You have quality embeddings (M4/09) and chunking that preserves resolution (M4/10). Now you'll put it all together: complete ingestion of a dataset → chunking → OpenAI embeddings → ChromaDB with metadata → retrieval → generation with GPT.

It will be the first time in the guide that you see a functional end-to-end RAG system. It's the minimal version that proves the pipeline works. Module 8 later scales it to 1000+ documents and adds a REST API, Docker, and testing — but you build the core logic in M4/11.

Resources

1. LangChain RecursiveCharacterTextSplitter — Official documentation
2. Chunking Strategies for LLM Applications (Pinecone) — Comparison of strategies
3. The Five Levels of Chunking (Greg Kamradt) — From fixed to agentic chunking
4. Semantic Chunking with LangChain — Official implementation of semantic chunking
5. Tokenizer Playground (OpenAI) — Verify how many tokens your chunks are
6. LlamaIndex Node Parsers — Alternative to LangChain with finer sentence splitters

Estimated time: 35-45 minutes Next: 11-pipeline-rag-end-to-end.md