Semantic Chunking: Respecting Document Structure

Capsule overview

Semantic chunking splits text at natural boundaries (sentences, paragraphs, sections) instead of every N tokens. This preserves coherence and context, producing chunks that are more understandable for humans and LLMs.

In this capsule you'll learn sentence-based chunking, paragraph-based chunking, how to use LangChain's RecursiveCharacterTextSplitter, and the trade-offs vs fixed-size. You'll also implement production-ready semantic chunking.

By the end, you'll be able to choose between fixed-size and semantic depending on your use case.

Sentence-Based Chunking

Concept:

# Split by sentences
# Accumulate sentences until reaching ~N tokens

text = """
Python is a programming language. It was created in 1991. 
Python is known for simplicity. It supports multiple paradigms.
"""

# Fixed-size (cuts mid-sentence):
# Chunk 1: "Python is a programming language. It wa"
# Chunk 2: "s created in 1991. Python is known for"
# ❌ Cuts badly

# Sentence-based (respects sentences):
# Chunk 1: "Python is a programming language. It was created in 1991."
# Chunk 2: "Python is known for simplicity. It supports multiple paradigms."
# ✅ Coherent

Implementation with a simple split:

import tiktoken

class SentenceChunker:
    """Sentence-based chunker"""
    
    def __init__(self, max_tokens=500, model="gpt-4"):
        self.max_tokens = max_tokens
        self.encoding = tiktoken.encoding_for_model(model)
    
    def chunk(self, text):
        """Split into chunks respecting sentences"""
        # Split by periods (simple sentence boundary)
        sentences = text.split('. ')
        
        chunks = []
        current_chunk = []
        current_tokens = 0
        
        for sentence in sentences:
            sentence = sentence.strip() + '.'
            sentence_tokens = len(self.encoding.encode(sentence))
            
            # If adding this sentence exceeds the limit, close the current chunk
            if current_tokens + sentence_tokens > self.max_tokens and current_chunk:
                chunks.append(' '.join(current_chunk))
                current_chunk = []
                current_tokens = 0
            
            current_chunk.append(sentence)
            current_tokens += sentence_tokens
        
        # Add the last chunk
        if current_chunk:
            chunks.append(' '.join(current_chunk))
        
        return chunks

# Usage
chunker = SentenceChunker(max_tokens=30)

text = """
Python is a high-level programming language. It was created by Guido van Rossum. 
Python emphasizes code readability. It supports multiple programming paradigms. 
Python has a large standard library. It is widely used in data science and AI.
"""

chunks = chunker.chunk(text)

for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}:")
    print(chunk)
    print(f"Tokens: {len(chunker.encoding.encode(chunk))}\n")

Output:

Chunk 1:
Python is a high-level programming language. It was created by Guido van Rossum. Python emphasizes code readability.
Tokens: 23

Chunk 2:
It supports multiple programming paradigms. Python has a large standard library. It is widely used in data science and AI..
Tokens: 25

✅ Each chunk ends at a natural boundary (end of sentence).

Paragraph-Based Chunking

Concept:

# Split by paragraphs (double newline: \n\n)
# More context than sentences, respects the author's structure

text = """
Python is a programming language created in 1991.
It emphasizes code readability and simplicity.

Python supports multiple paradigms including OOP.
It has a large ecosystem of libraries and frameworks.

Python is widely used in data science and AI.
Popular libraries include NumPy, Pandas, and TensorFlow.
"""

# Paragraph-based:
# Chunk 1: "Python is a programming language... simplicity."
# Chunk 2: "Python supports multiple paradigms... frameworks."
# Chunk 3: "Python is widely used... TensorFlow."
# ✅ Preserves the author's logical structure

Implementation:

class ParagraphChunker:
    """Paragraph-based chunker"""
    
    def __init__(self, max_tokens=500, model="gpt-4"):
        self.max_tokens = max_tokens
        self.encoding = tiktoken.encoding_for_model(model)
    
    def chunk(self, text):
        """Split by paragraphs"""
        # Split by double newline
        paragraphs = text.split('\n\n')
        
        chunks = []
        current_chunk = []
        current_tokens = 0
        
        for paragraph in paragraphs:
            paragraph = paragraph.strip()
            if not paragraph:
                continue
            
            para_tokens = len(self.encoding.encode(paragraph))
            
            # If the paragraph alone exceeds the limit, split by sentences
            if para_tokens > self.max_tokens:
                # Fall back to sentence-based for this paragraph
                sentence_chunker = SentenceChunker(self.max_tokens)
                para_chunks = sentence_chunker.chunk(paragraph)
                chunks.extend(para_chunks)
                continue
            
            # If adding this paragraph exceeds the limit
            if current_tokens + para_tokens > self.max_tokens and current_chunk:
                chunks.append('\n\n'.join(current_chunk))
                current_chunk = []
                current_tokens = 0
            
            current_chunk.append(paragraph)
            current_tokens += para_tokens
        
        # Add the last chunk
        if current_chunk:
            chunks.append('\n\n'.join(current_chunk))
        
        return chunks

LangChain RecursiveCharacterTextSplitter

What it is:

LangChain's recursive chunker: It tries to split by hierarchical separators (\n\n → \n → space → character).

Advantage: A balance between semantic and fixed-size.

Code with LangChain:

from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken

# Create the splitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=200,          # Maximum size in characters
    chunk_overlap=20,        # Overlap
    length_function=len,     # Function to measure size
    separators=["\n\n", "\n", " ", ""]  # Hierarchical separators
)

text = """
# Python Programming

Python is a high-level programming language.
It was created by Guido van Rossum in 1991.

## Features

Python emphasizes code readability and simplicity.
It supports multiple programming paradigms.

Python has a large standard library.
The ecosystem includes frameworks like Django and Flask.

## Use Cases

Python is widely used in:
- Data Science
- Machine Learning
- Web Development
- Automation
"""

# Split
chunks = text_splitter.split_text(text)

print(f"Total chunks: {len(chunks)}\n")

for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}:")
    print(chunk)
    print(f"Length: {len(chunk)} chars\n")

Output:

Total chunks: 4

Chunk 1:
# Python Programming

Python is a high-level programming language.
It was created by Guido van Rossum in 1991.

## Features
Length: 123 chars

Chunk 2:
## Features

Python emphasizes code readability and simplicity.
It supports multiple programming paradigms.
Length: 107 chars

Chunk 3:
Python has a large standard library.
The ecosystem includes frameworks like Django and Flask.

## Use Cases
Length: 107 chars

Chunk 4:
## Use Cases

Python is widely used in:
- Data Science
- Machine Learning
- Web Development
- Automation
Length: 104 chars

✅ It respects headers, paragraphs, and lists (and the overlap repeats ## Features and ## Use Cases between chunks).

Token-Aware Recursive Splitter

The problem with RecursiveCharacterTextSplitter:

# LangChain uses characters, not tokens
# chunk_size=500 characters ≠ 500 tokens

# Example:
text = "Python" * 100  # 600 characters
# Characters: 600
# Tokens (tiktoken): ~100 tokens

# If you set chunk_size=500 characters,
# but the real limit is 500 tokens,
# the chunk can exceed the token limit!

Solution: Token-aware splitter

from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken

# Token counting function
def tiktoken_len(text):
    encoding = tiktoken.encoding_for_model("gpt-4")
    return len(encoding.encode(text))

# Token-aware splitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,          # Now in TOKENS
    chunk_overlap=50,
    length_function=tiktoken_len,  # ← Use tiktoken
    separators=["\n\n", "\n", " ", ""]
)

text = "Python is great. " * 200  # ~600 tokens

chunks = text_splitter.split_text(text)

print(f"Total chunks: {len(chunks)}")

for i, chunk in enumerate(chunks):
    tokens = tiktoken_len(chunk)
    print(f"Chunk {i+1}: {tokens} tokens")

✅ It guarantees that each chunk ≤ 500 tokens.

Trade-Offs: Fixed vs Semantic

Comparison:

When to use each one:

Fixed-Size:

✅ Critical speed (latency <5ms)
✅ Documents without clear structure
✅ Simplicity required
✅ Predictable size is important

Semantic:

✅ Critical coherence (legal, medical)
✅ Structured documents (headers, paragraphs)
✅ Context is important
✅ OK with variable-size chunks

Exercises

Exercise 1: Sentence-based chunker

Implement a chunker that splits by sentences:

import tiktoken

def sentence_chunker(text, max_tokens=100):
    """Split by sentences up to max_tokens"""
    # Implement here
    pass

# Test
text = "Python is great. JavaScript is popular. Go is fast. Rust is safe."
chunks = sentence_chunker(text, max_tokens=50)
print(chunks)

import tiktoken

def sentence_chunker(text, max_tokens=100):
    """Split by sentences"""
    encoding = tiktoken.encoding_for_model("gpt-4")
    sentences = [s.strip() + '.' for s in text.split('.') if s.strip()]
    
    chunks = []
    current = []
    current_tokens = 0
    
    for sentence in sentences:
        tokens = len(encoding.encode(sentence))
        
        if current_tokens + tokens > max_tokens and current:
            chunks.append(' '.join(current))
            current = []
            current_tokens = 0
        
        current.append(sentence)
        current_tokens += tokens
    
    if current:
        chunks.append(' '.join(current))
    
    return chunks

# Test
text = "Python is great. JavaScript is popular. Go is fast. Rust is safe."
chunks = sentence_chunker(text, max_tokens=20)

for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}: {chunk}")

Exercise 2: LangChain token-aware

Use RecursiveCharacterTextSplitter with tiktoken:

from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken

def tiktoken_len(text):
    encoding = tiktoken.encoding_for_model("gpt-4")
    return len(encoding.encode(text))

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=100,
    chunk_overlap=10,
    length_function=tiktoken_len,
    separators=["\n\n", "\n", " ", ""]
)

text = """
Python is a programming language.
It is very popular for AI and data science.

JavaScript is used for web development.
It runs in the browser and on servers with Node.js.
"""

chunks = text_splitter.split_text(text)

for i, chunk in enumerate(chunks):
    tokens = tiktoken_len(chunk)
    print(f"Chunk {i+1} ({tokens} tokens):")
    print(chunk)
    print()

Summary

What you learned:

• ✅ Sentence-based: Respects sentences, more coherent
• ✅ Paragraph-based: Respects paragraphs, preserves structure
• ✅ LangChain Recursive: Balance of fixed/semantic
• ✅ Token-aware: Guarantees chunks ≤ max_tokens
• ✅ Trade-offs: Fixed (fast) vs Semantic (coherent)

Key concepts:

1. Semantic chunking preserves coherence
2. RecursiveCharacterTextSplitter = best of both worlds
3. Token-aware is critical for API limits

Additional resources

1. LangChain Text Splitters - Official docs
2. RecursiveCharacterTextSplitter - API reference
3. Chunking Strategies - Pinecone guide

In the next capsule

Capsule 04: Retrieval Metrics

You'll learn:

• nDCG (Normalized Discounted Cumulative Gain)
• MRR (Mean Reciprocal Rank)
• Recall@K, Precision@K
• Implementation in Python

From chunking to quantitative evaluation.

Module 4 - Embeddings Deep Dive Guide Semantic chunking: respecting the document structure