Capsule 06: Chunk overlap — the insurance against answers cut in half

Capsule overview

There's a silent failure in RAG that no log ever captures: the correct answer exists in your documents, but it lands split exactly between two chunks, and neither one contains it whole. The query doesn't find it, the LLM answers "I don't have that information", and you'd swear the fact is right there — and you're right. The problem is that the chunker cut the sentence exactly where it shouldn't have.

Chunk overlap is the insurance against that scenario. You duplicate the last N characters of the previous chunk at the start of the next one, guaranteeing that any idea crossing the boundary between two chunks appears complete in at least one of them. It costs space (partially duplicated chunks) and a bit of redundancy in the results, but it buys a measurable recall improvement that is almost always worth it.

This capsule explains why overlap matters pedagogically, how to choose the right percentage for your domain, and which traps turn a useful insurance policy into bloated redundancy that doubles your cost without improving quality.

By the end of this capsule you'll be able to:

• ✅ Explain why a chunk without overlap can lose information that falls on the boundary
• ✅ Choose a justified overlap percentage (10% by default, adjusted for cross-section references)
• ✅ Calculate the impact on storage and embedding cost of raising the overlap
• ✅ Implement consistent overlap in RecursiveCharacterTextSplitter and verify it
• ✅ Anticipate the "diminishing returns" mistake: when raising overlap stops improving recall
• ✅ Diagnose missing answers as a possible insufficient-overlap problem

Estimated time: 25-30 minutes

The concrete problem: the information that falls on the boundary

Picture a technical document with this sentence:

"To configure HNSW in production with high accuracy, the recommended values are M=32 and construction_ef=200, adjustable based on the dataset's specific requirements and the available memory."

And your chunker splits it like this, with no overlap:

Chunk N (ends here):
  "...To configure HNSW in production with high accuracy, the recommended values are M=32 and"

Chunk N+1 (starts here):
  "construction_ef=200, adjustable based on the dataset's specific requirements..."

When a user asks "what values of M and construction_ef do you recommend for production?", what happens?

• Chunk N mentions M=32 but not construction_ef. A partial match.
• Chunk N+1 mentions construction_ef=200 but not M. A partial match.

Each chunk's embedding represents an incomplete fragment of the concept. The query "M and construction_ef together" doesn't find any chunk as a strong match. The system fails silently — it returns some chunk related to HNSW, the LLM reads half the answer, and it responds with something vague or wrong.

With overlap, the problem disappears

With chunk_overlap=50, the last 50 characters of chunk N repeat at the start of chunk N+1:

Chunk N:
  "...To configure HNSW in production with high accuracy, the recommended values are M=32 and"

Chunk N+1:
  "the recommended values are M=32 and construction_ef=200, adjustable based on the..."
       ↑ overlap (50 chars that repeat)

Now chunk N+1 contains the complete sentence: "M=32 and construction_ef=200". The query finds the match. The LLM has the whole context. The answer is correct.

Overlap is the pedagogical insurance policy: the information can fall anywhere in the document, but it gets captured by at least one chunk that represents it whole.

Why 10% is the reasonable default

chunk_overlap is measured in characters (or tokens, depending on the splitter). The practical rule is:

chunk_overlap ≈ chunk_size × 0.10

For the typical values:

Why 10%:

1. It covers most sentences that cross boundaries. Technical sentences run 15-30 words (~80-150 characters) in English or Spanish. An overlap of 50-80 characters catches the whole sentence when it lands on the boundary.

2. The cost is acceptable. A 10% overlap means 10% more chunks than without it. If your dataset is 1M chunks, that's 100K extra chunks — an extra embedding cost of ~$2 with OpenAI. Trivial.

3. It doesn't bloat retrieval. With moderate overlap, the chunks returned by a query are distinct from each other. With a large overlap (>30%), you start receiving several near-identical chunks in the top-K, wasting the LLM's context.

When to raise the overlap

If you're unsure, start at 10%, measure it, then adjust.

When NOT to use overlap=0

# ❌ DON'T do this in production
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=0,  # "to save storage"
)

The "saving" from removing overlap is trivial (~10% fewer chunks). The damage is that you'll lose information at the boundaries and nobody will understand why some queries return incomplete answers. It's the optimization that breaks the system without warning you.

When NOT to raise overlap above 30%

# ❌ Don't do this either
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=300,  # 60%! "so we don't lose anything"
)

With 60% overlap, every chunk repeats more than half of the previous one. Your collection bloats (5 chunks for every 1 effective chunk), your retrieval results come back with 3-4 near-identical chunks instead of varied ones, and the LLM receives the same context three times over. You gain ~3-5% recall, you pay 3x more in costs, and you degrade generation quality with redundant context.

The correct implementation with LangChain

# chunking_with_overlap.py
from langchain_text_splitters import RecursiveCharacterTextSplitter

# A reasonable default configuration for technical text
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,        # 10% of chunk_size
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""]
)

document = """
To configure HNSW in production with high accuracy, the recommended values are
M=32 and construction_ef=200, adjustable based on the dataset's specific requirements
and the available memory. Raising M improves recall but it also raises memory
usage. Raising construction_ef only affects the initial build time, not the
queries at runtime.
"""

chunks = splitter.split_text(document)

for i, chunk in enumerate(chunks):
    print(f"\n--- Chunk {i+1} ({len(chunk)} chars) ---")
    print(chunk)

Expected output:

--- Chunk 1 (487 chars) ---
To configure HNSW in production with high accuracy, the recommended values are
M=32 and construction_ef=200, adjustable based on the dataset's specific requirements
and the available memory. Raising M improves recall but it also raises memory
usage.

--- Chunk 2 (213 chars) ---
but it also raises memory usage. Raising construction_ef only affects the initial
build time, not the queries at runtime.

Notice the last ~50 characters of chunk 1 ("but it also raises memory usage.") reappearing at the start of chunk 2. That's the overlap doing its job: the sentence that closes chunk 1 also opens chunk 2, guaranteeing the context.

Verifying the overlap empirically

def verify_overlap(chunks: list[str], expected_overlap: int):
    """Checks that each consecutive chunk shares ~expected_overlap chars with the next one."""
    for i in range(len(chunks) - 1):
        chunk_end = chunks[i][-expected_overlap * 2:]  # the last N*2 chars
        chunk_start = chunks[i + 1][:expected_overlap * 2]  # the first N*2 chars

        # Look for the common substring
        max_overlap = 0
        for length in range(expected_overlap * 2, 10, -1):
            if chunk_end[-length:] in chunk_start:
                max_overlap = length
                break

        print(f"Chunk {i} → {i+1}: overlap detected = {max_overlap} chars")

verify_overlap(chunks, expected_overlap=50)

Benchmarking the real impact

Let's measure the effect of overlap on recall empirically.

# benchmark_overlap.py
import chromadb
from chromadb.utils import embedding_functions
from langchain_text_splitters import RecursiveCharacterTextSplitter
import os

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.getenv("OPENAI_API_KEY"),
    model_name="text-embedding-3-small"
)

# The eval set: queries whose answers we know are in the document
eval_set = [
    {"query": "what values of M and construction_ef do you recommend for production?",
     "expected_keywords": ["M=32", "construction_ef=200"]},
    {"query": "how does M affect memory usage?",
     "expected_keywords": ["raises memory usage", "M"]},
    # ... more queries
]


def benchmark_overlap_pct(overlap_pct: int, document: str, queries: list) -> dict:
    """Ingests the document with a given overlap, then measures recall."""
    chunk_size = 500
    overlap = chunk_size * overlap_pct // 100

    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=overlap,
        separators=["\n\n", "\n", ". ", " ", ""]
    )
    chunks = splitter.split_text(document)

    client = chromadb.PersistentClient(path=f"./chroma_overlap_{overlap_pct}")
    name = f"overlap_{overlap_pct}"
    try:
        client.delete_collection(name)
    except Exception:
        pass

    collection = client.create_collection(name, embedding_function=openai_ef)
    collection.add(
        documents=chunks,
        ids=[f"chunk_{i}" for i in range(len(chunks))]
    )

    # Recall: does the top-1 chunk contain the expected keywords?
    hits = 0
    for item in queries:
        result = collection.query(query_texts=[item["query"]], n_results=1)
        retrieved_text = result['documents'][0][0]
        if all(kw in retrieved_text for kw in item["expected_keywords"]):
            hits += 1

    return {
        "overlap_pct": overlap_pct,
        "n_chunks": len(chunks),
        "recall": hits / len(queries),
        "storage_increase_pct": (len(chunks) / len(splitter.split_text(document)) - 1) * 100 if overlap_pct == 0 else None,
    }


# Compare the overlaps
for pct in [0, 5, 10, 20, 40, 60]:
    result = benchmark_overlap_pct(pct, long_document, eval_set)
    print(f"overlap={pct}%: chunks={result['n_chunks']}, recall={result['recall']:.0%}")

Typical output (over a real technical dataset):

overlap=0%:  chunks=120, recall=72%   ← information lost at the boundaries
overlap=5%:  chunks=126, recall=85%
overlap=10%: chunks=132, recall=92%   ← the sweet spot
overlap=20%: chunks=145, recall=94%
overlap=40%: chunks=180, recall=96%   ← diminishing returns
overlap=60%: chunks=240, recall=96%   ← spending with no benefit

The lessons from the benchmark:

1. From 0% to 10%, recall jumps 20 points. That's the highest-return move there is.
2. From 10% to 20%, recall improves 2% — still worth it in sensitive domains.
3. From 20% onward, the returns fall off a cliff. You're duplicating data without gaining real precision.
4. The sweet spot is almost always 10-15%.

Traps and common mistakes

Trap 1: overlap=0 "to save money"

The mistake: someone decides the overlap is waste. They remove it.

The symptom: specific queries combining two nearby concepts in the text start failing. Recall drops with no visible cause. The team blames the embedding model or the LLM, when the problem is in the chunking.

How to prevent it: stay at 10% minimum. The quick audit: if recall is low and the concepts from the failing queries sit close together in the document, suspect the overlap.

Trap 2: overlap > 50% of chunk_size

The mistake: "more overlap is safer".

The symptom: the collection bloats, retrieval returns near-identical chunks in the top-K, embedding costs rise, and generation quality can get worse (redundant context).

How to prevent it: stay at ≤25%. If you need more coverage, look at advanced techniques (parent-child chunking, contextual retrieval) instead of inflating the overlap.

Trap 3: changing the overlap between rebuilds without re-indexing

The mistake: you ingested everything with overlap=50. Then you read a blog recommending overlap=100. You change the code that processes NEW documents. The old ones keep overlap=50.

The symptom: queries whose relevant chunks are old keep failing exactly as before; queries about new docs improve. Inconsistent results across parts of the dataset.

How to prevent it: when you change a chunking parameter, re-process the whole dataset. It isn't optional. Covered in M03/M02/05 too.

Trap 4: overlap measured in characters vs tokens

The mistake: you assume chunk_overlap=50 means 50 tokens. In RecursiveCharacterTextSplitter it means 50 characters (~12-15 tokens in English/Spanish).

The symptom: you sized the overlap thinking in tokens, but you're actually covering only a quarter of what you expected.

How to prevent it: always check which unit your splitter measures in. For overlap in tokens, use TokenTextSplitter or pass length_function=len_in_tokens:

import tiktoken

enc = tiktoken.encoding_for_model("text-embedding-3-small")

def len_in_tokens(text: str) -> int:
    return len(enc.encode(text))

splitter = RecursiveCharacterTextSplitter(
    chunk_size=200,           # 200 tokens
    chunk_overlap=20,         # 20 tokens
    length_function=len_in_tokens,
)

Trap 5: forgetting overlap when chunking hierarchical structure

The mistake: you use a structural chunker (by markdown heading, for example). It passes whole sections through as individual chunks. You skip the overlap because "each section is a complete chunk".

The symptom: a section that closes by referring to the next one ends up split — the reader has to read both to understand it. The chunker never connects them.

How to prevent it: even in structural chunking, add 1-2 sentences of overlap between consecutive sections if cross-references are common.

Trap 6: assuming overlap fixes chunks that are too small

The mistake: your chunk_size=200 (far too small). The answers don't fit in a single chunk. You raise overlap=100 (50%), thinking it "covers the difference".

The symptom: a massive overlap doesn't compensate for the fact that no individual chunk has enough context. Queries that need 400 characters of context still fail, overlap and all.

How to prevent it: first tune chunk_size to the right size for your domain, then set the overlap to 10%. Raising overlap never compensates for a badly chosen chunk_size.

Applied exercise

The scenario: you're an AI Engineer at a financial services company. This complaint lands on your desk:

"The chatbot can't find information that's clearly in the documents. For example, I asked it 'what's the maximum refund window for premium users with an account older than 2 years?' and it said it didn't know. But I opened the policy document and the information is right there in section 4.3."

You investigate and you find:

• Section 4.3 says: "For premium users with an account active for more than 2 years, the maximum refund window extends to 90 days, subject to approval from the compliance team."
• Your chunker is configured with chunk_size=400, chunk_overlap=0.
• Inspecting the chunks, you see that section 4.3 got split like this:
  • Chunk N ends with: "...the maximum refund window extends to 90 days,"
  • Chunk N+1 starts with: "subject to approval from the compliance team. For users..."

Your job:

1. Diagnose the problem in terms of chunk overlap.
2. Propose a fix with concrete chunk_size and chunk_overlap values.
3. Explain what risk the team runs if they decide to raise the overlap to 50% instead of 10%.

splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,
    chunk_overlap=80,        # 20% — legal policies have long sentences full of conditions
    separators=["\n\n", "\n", ". ", " ", ""]
)

Summary and next step

What you learned:

• Without overlap, information that falls on the boundary between two chunks gets split and retrieval fails silently.
• Overlap is the pedagogical insurance policy: it guarantees that any idea near a boundary gets captured by at least one complete chunk.
• A reasonable default: chunk_overlap = chunk_size × 0.10 (10%).
• Domains with long sentences (legal, medical, papers) usually need 15-20%.
• Raising the overlap above 25% causes severe diminishing returns: the collection bloats, retrieval returns duplicates, generation gets worse.
• Changing the overlap means re-processing the entire dataset. It isn't just a parameter.
• Overlap is measured in characters in RecursiveCharacterTextSplitter, not tokens — watch the unit.

Checkpoint: before moving on, you should be able to:

• Explain to a colleague why chunk_overlap=0 can cause invisible failures.
• Calculate the impact of an overlap change on the chunk count and the embedding cost.
• Diagnose a "the bot can't find the information" complaint as a possible overlap problem.

Next capsule: 07 — Comparing the chunking strategies.

We covered four strategies in this module: fixed-size, recursive, semantic, structural. Plus overlap as a complementary technique. Capsule 07 puts them side by side with real benchmarks and a decision framework — what to choose based on your domain, your budget and your target latency. It's the capsule you'll come back to when you start a new project and have to decide the chunking on day one.

Resources

1. LangChain — RecursiveCharacterTextSplitter — The official documentation, with the parameters
2. Chunking Strategies for LLM Applications (Pinecone) — Overlap compared by domain
3. Anthropic — Contextual Retrieval — The advanced technique for when overlap isn't enough
4. Lost in the Middle (paper) — Why excessive overlap makes generation worse
5. LlamaIndex — Sentence Window Retrieval — Dynamic overlap at runtime
6. Tokenizer Playground (OpenAI) — Verify the chars↔tokens conversion

Estimated time: 25-30 minutes Next: 07-strategy-comparison-1.md