Module 2: Chunking Strategies

Introduction to Chunking Strategies

Next →

Capsule overview

Chunking is the first technical decision you make in RAG: how do you split long documents into retrievable chunks? That decision has a direct impact on quality: badly formed chunks (cut mid-sentence, stripped of semantic context) produce poor retrieval, confuse the LLM, and drop precision by 15-20%.

In Module 1 you used fixed-size chunking (naive: cut every 500 characters). It works for a baseline, but it's suboptimal. This module teaches you 4 advanced strategies that respect the document's structure, preserve semantic context, and improve precision by +10-20% with zero changes to any other component.

This capsule gives you the module's complete map: what you'll learn, why chunking matters, the roadmap through its 8 capsules, the technical setup, and the connection with the evolving project where you'll integrate optimized chunking.

🎯 Learning objectives

By the end of this module, you'll be able to:

  1. ✅ Explain why fixed-size chunking is suboptimal (it loses context, it cuts arbitrarily)
  2. ✅ Implement recursive chunking with LangChain (it respects paragraphs/sentences)
  3. ✅ Implement embedding-based semantic chunking (it groups by topic)
  4. ✅ Implement structural chunking for code/HTML/Markdown
  5. ✅ Use chunk overlap to preserve context between chunks
  6. ✅ Compare the 4 strategies with quantitative benchmarks
  7. ✅ Select the optimal strategy for a given document type
  8. ✅ Improve the baseline RAG by +10-15% precision with optimized chunking

📐 Why chunking matters

The problem: long documents don't fit in embeddings or an LLM context

# A long document
document = """
FastAPI is a modern, fast (high-performance) web framework for building APIs with Python 3.8+ based on standard Python type hints.

Key features:
- Fast: Very high performance, on par with NodeJS and Go (thanks to Starlette and Pydantic).
- Fast to code: Increases development speed by roughly 200% to 300%.
- Fewer bugs: Reduces human errors by roughly 40%.
- Intuitive: Great editor support, with completion everywhere.
- Easy: Designed to be easy to use and learn. Less time reading documentation.
- Short: Minimizes code duplication. Multiple features from each parameter declaration.
- Robust: Get production-ready code with automatic, interactive documentation.
- Standards-based: Based on (and fully compatible with) the open standards for APIs: OpenAPI and JSON Schema.

FastAPI was created by Sebastián Ramírez and released in 2018. Since then it's been adopted by companies like Microsoft, Netflix and Uber to build production APIs.

History:
FastAPI was born out of the need to build fast APIs with automatic data validation...
"""

# The problem: 
len(document)  # ~1,250 characters
# - Embedding models: max 8,191 tokens (OpenAI ada-002)
# - LLM context: max 4,096-128K tokens (GPT-3.5 to GPT-4)
# - But real documents run 10K-100K+ characters

The solution: split it into smaller chunks.

Chunking's impact on the RAG pipeline:

┌─────────────────────────────────────────┐
│          INDEXING (Offline)             │
├─────────────────────────────────────────┤
│ 1. Chunking: [Fixed | Recursive |      │ ← THIS MODULE
│               Semantic | Structural]    │
│ 2. Embeddings: 1 embedding per chunk    │ ← Affected by chunk quality
│ 3. Storage: Store chunks + embeddings   │ ← Affected by chunk count/size
└─────────────────────────────────────────┘
              ↓
┌─────────────────────────────────────────┐
│         RETRIEVAL (Query time)          │
├─────────────────────────────────────────┤
│ 1. Search: Find top-K chunks            │ ← Bad chunks = poor retrieval
│ 2. Retrieved chunks → LLM context       │ ← Context-free chunks = poor generation
└─────────────────────────────────────────┘

Bad chunking:

  • Chunks cut mid-sentence → incoherent → poor embeddings → poor retrieval
  • Chunks with no context → the LLM can't generate a correct answer

Good chunking:

  • Coherent chunks → quality embeddings → good retrieval
  • Chunks with context → the LLM generates grounded answers

🗺️ The module's roadmap

Capsule 01 (this one): Introduction

  • Why chunking matters
  • An overview of the 4 strategies
  • Technical setup

Capsule 02: The problems with fixed-size chunking

  • Problem 1: arbitrary cuts
  • Problem 2: lost context
  • The impact on your metrics (-15-20% precision)

Capsule 03: Recursive chunking

  • How it works (separators)
  • Implementing it with LangChain
  • Trade-offs and tuning

Capsule 04: Semantic chunking

  • How it works (embeddings + clustering)
  • Implementing it with SemanticChunker
  • Trade-offs (cost vs quality)

Capsule 05: Structural chunking

  • Code: chunking by function (ast)
  • HTML: chunking by section (BeautifulSoup)
  • Markdown: chunking by header

Capsule 06: Chunk overlap

  • Why overlap preserves context
  • Fixed vs semantic overlap
  • Trade-offs (storage vs recall)

Capsule 07: The strategies compared

  • A benchmark of all 4 strategies
  • The decision matrix
  • Hybrid cases

Capsule 08: Project - Chunking Optimizer

  • Implement all 4 strategies
  • Compare them with benchmarks
  • Integrate the winner into the baseline RAG

📊 An overview of the chunking strategies

Strategy 1: Fixed-size (the Module 1 baseline)

# The naive baseline
chunks = [document[i:i+500] for i in range(0, len(document), 500)]

# The problem: it cuts at arbitrary points
# Chunk 1: "FastAPI is a modern, fast (high-perform..."  ← Cut mid-word
# Chunk 2: "...ance) web framework for building AP..." ← No context

Pros: ✅ Simple, fast
Cons: ❌ Loses context, cuts arbitrarily
Precision: Baseline (68% in Module 1)

Strategy 2: Recursive (LangChain)

from langchain.text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]  # It tries paragraphs first
)

chunks = splitter.split_text(document)

# Output: chunks that respect paragraphs and sentences
# Chunk 1: "FastAPI is a modern, fast web framework..."  ← A complete sentence
# Chunk 2: "Key features:\n- Fast: ..."  ← A complete section

Pros: ✅ Respects structure, and the overlap preserves context
Cons: ⚠️ Slightly more complex than fixed-size
Precision: +10-15% vs baseline
Capsule: 03

Strategy 3: Semantic (embeddings-based)

from langchain.text_splitters import SemanticChunker
from langchain_openai import OpenAIEmbeddings

splitter = SemanticChunker(
    embeddings=OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile"  # It splits when similarity drops
)

chunks = splitter.split_text(document)

# Output: chunks grouped by semantic topic
# Chunk 1: everything about FastAPI's features (topic: features)
# Chunk 2: everything about FastAPI's history (topic: background)

Pros: ✅ Maximum semantic coherence
Cons: ❌ Slow (it generates embeddings), API cost
Precision: +15-20% vs baseline
Capsule: 04

Strategy 4: Structural (code/HTML/Markdown)

import ast

def chunk_by_functions(python_code: str) -> list[str]:
    """Splits code by function/class"""
    tree = ast.parse(python_code)
    chunks = []
    
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
            chunk = ast.get_source_segment(python_code, node)
            chunks.append(chunk)
    
    return chunks

# Output: every chunk is a complete function
# Chunk 1: def create_user(...): ...
# Chunk 2: def update_user(...): ...

Pros: ✅ Respects the document's structure (code, HTML, Markdown)
Cons: ⚠️ Needs a format-specific parser
Precision: +20-25% vs baseline (for code/HTML)
Capsule: 05

🛠️ Technical setup

Step 1: Install the new dependencies

# Activate the virtual environment
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install the new libraries
pip install langchain-text-splitters==0.0.1
pip install sentence-transformers==2.3.1
pip install beautifulsoup4==4.12.3
pip install markdown==3.5.2

# Update requirements.txt
pip freeze > requirements.txt

Step 2: Verify the installation

# test_module2_setup.py
from langchain.text_splitters import RecursiveCharacterTextSplitter, SemanticChunker
from langchain_openai import OpenAIEmbeddings
from sentence_transformers import SentenceTransformer
from bs4 import BeautifulSoup
import markdown
import ast

print("✅ LangChain text splitters")
print("✅ Sentence-Transformers")
print("✅ BeautifulSoup4")
print("✅ Markdown")
print("✅ ast (built-in)")

# Test RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)
test_text = "Hello world. " * 20
chunks = splitter.split_text(test_text)
print(f"\n✅ RecursiveCharacterTextSplitter works ({len(chunks)} chunks)")

print("\n🎉 Module 2 setup complete!")

Run it:

python test_module2_setup.py

Expected output:

✅ LangChain text splitters
✅ Sentence-Transformers
✅ BeautifulSoup4
✅ Markdown
✅ ast (built-in)

✅ RecursiveCharacterTextSplitter works (4 chunks)

🎉 Module 2 setup complete!

📋 The project structure (Module 2)

rag_baseline_project/  (from Module 1)
├── src/
│   ├── indexing.py              # Baseline (Module 1)
│   ├── retrieval.py             # Unchanged
│   ├── generation.py            # Unchanged
│   ├── evaluation.py            # Unchanged
│   └── chunking/                # NEW (Module 2)
│       ├── __init__.py
│       ├── fixed_size.py        # Baseline
│       ├── recursive.py         # Strategy 2
│       ├── semantic.py          # Strategy 3
│       ├── structural.py        # Strategy 4
│       └── comparator.py        # The strategy benchmark
├── notebooks/
│   └── chunking_comparison.ipynb  # An interactive demo
└── README.md                    # Update it with your chunking decisions

🎯 The project's objectives (Capsule 08)

By the end of the module, you'll implement:

# chunking_optimizer.py (Preview)

class ChunkingOptimizer:
    """A comparator for chunking strategies"""
    
    def __init__(self):
        self.strategies = {
            "fixed": FixedSizeChunker(chunk_size=500),
            "recursive": RecursiveChunker(chunk_size=500, overlap=50),
            "semantic": SemanticChunker(embeddings=OpenAIEmbeddings()),
            "structural": StructuralChunker()  # For code
        }
    
    def compare_strategies(self, document: str) -> dict:
        """Compares the 4 strategies with benchmarks"""
        results = {}
        
        for name, chunker in self.strategies.items():
            chunks = chunker.chunk(document)
            
            # Measure the metrics
            results[name] = {
                "num_chunks": len(chunks),
                "avg_chunk_size": statistics.mean([len(c) for c in chunks]),
                "precision": self.measure_precision(chunks),  # With a golden dataset
                "recall": self.measure_recall(chunks),
                "latency": self.measure_latency(chunks)
            }
        
        return results

# Usage
optimizer = ChunkingOptimizer()
results = optimizer.compare_strategies(document)

# Output:
# {
#   "fixed": {"precision": 0.68, "recall": 0.52, ...},
#   "recursive": {"precision": 0.78, "recall": 0.57, ...},  ← +10% precision
#   "semantic": {"precision": 0.83, "recall": 0.62, ...},   ← +15% precision
#   "structural": {"precision": 0.75, "recall": 0.55, ...}
# }

📊 The improvement you should expect from each strategy

Baseline (fixed-size):

# The Module 1 baseline
chunks = [document[i:i+500] for i in range(0, len(document), 500)]

# Baseline metrics:
# - Precision@5: 68%
# - Recall@50: 52%
# - Latency: 0ms (instant)

Recursive (+10-15%):

# Recursive with overlap
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_text(document)

# Expected improvement:
# - Precision@5: 78% (+10% vs baseline)
# - Recall@50: 57% (+5% vs baseline)
# - Latency: 0ms (instant)
# - Trade-off: +10-15% storage (the overlap)

Semantic (+15-20%):

# Semantic with embeddings
splitter = SemanticChunker(OpenAIEmbeddings())
chunks = splitter.split_text(document)

# Expected improvement:
# - Precision@5: 83% (+15% vs baseline)
# - Recall@50: 62% (+10% vs baseline)
# - Latency: +slow indexing (embeddings)
# - Trade-off: +the cost of the embedding API

Structural (+20-25% for code):

# Structural, for code
chunks = chunk_by_functions(python_code)

# Expected improvement (code):
# - Precision@5: 88% (+20% vs baseline)
# - Recall@50: 67% (+15% vs baseline)
# - Latency: 0ms (instant)
# - Trade-off: needs a format-specific parser

🔗 Connection with the evolving project

Module 1 (baseline):

# indexing.py (Module 1)
def chunk_document(document: str) -> list[str]:
    """Fixed-size chunking (naive)"""
    return [document[i:i+500] for i in range(0, len(document), 500)]

Baseline metrics: precision 68%, recall 52%

Module 2 (optimized):

# indexing.py (Module 2 - updated)
from langchain.text_splitters import RecursiveCharacterTextSplitter

def chunk_document(document: str) -> list[str]:
    """Recursive chunking (optimized)"""
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=50,
        separators=["\n\n", "\n", ". ", " ", ""]
    )
    return splitter.split_text(document)

Optimized metrics: precision 78% (+10%), recall 57% (+5%)

Module 8 (final):

# advanced_rag_system.py (Module 8 - final)

class AdvancedRAGSystem:
    def __init__(self, document_type: str):
        # Pick the chunking strategy based on the document type
        if document_type == "code":
            self.chunker = StructuralChunker()  # +20-25% precision
        elif document_type == "narrative":
            self.chunker = SemanticChunker()    # +15-20% precision
        else:
            self.chunker = RecursiveChunker()   # +10-15% precision (default)

Final metrics: precision 93% (accumulated across modules 1-8)

🎯 Summary

Key concepts:

  • ✅ Chunking is critical: it's the first technical decision in RAG, with a direct impact on precision
  • ✅ Fixed-size is suboptimal: it cuts arbitrarily and loses context (-15-20% precision)
  • ✅ 4 advanced strategies: recursive (balance), semantic (quality), structural (format-specific), overlap (context)
  • ✅ Expected gains: +10-20% precision depending on the strategy
  • ✅ Trade-offs: speed vs quality, cost vs precision, storage vs recall
  • ✅ The decision depends on the doc type: code → structural, narrative → semantic, general → recursive

What's next:

Capsule 02 breaks down fixed-size chunking's specific problems with concrete examples and impact metrics.

📚 Additional resources

  1. LangChain Text Splitters Guide - The official documentation
  2. Chunking Strategies for RAG - Pinecone's complete guide
  3. Optimal Chunk Size Research - The academic paper
  4. LlamaIndex Node Parsers - Alternative implementations
  5. Semantic Chunking Deep Dive - The LlamaIndex blog
  6. RAG Chunking Best Practices - A community discussion

Created: February 6, 2026
Version: 1.0

← PreviousProject: Baseline RAG SystemNext →The Problems with Fixed-Size Chunking