Fixed-Size Chunking: Practical Implementation

Capsule overview

Fixed-size chunking is the simplest strategy: splitting text every N characters or tokens with optional overlap. Although it doesn't respect semantic boundaries, it's deterministic, fast, and good enough for many use cases.

In this capsule you'll learn the difference between character-based and token-based chunking, implement chunking with tiktoken (precise token counting), overlap strategies with a sliding window, and the trade-offs of fixed-size. You'll also see reusable production-ready code.

By the end, you'll be able to implement robust fixed-size chunking for your RAG projects.

Character-Based vs Token-Based

Character-based chunking:

# Split every N characters
text = "Python is a programming language. It is very popular."

def char_chunking(text, chunk_size=20, overlap=5):
    """Character-based chunking"""
    chunks = []
    start = 0
    
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start += (chunk_size - overlap)
    
    return chunks

chunks = char_chunking(text, chunk_size=30, overlap=5)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}: '{chunk}'")

Output:

Chunk 1: 'Python is a programming langua'
Chunk 2: 'anguage. It is very popular.'
Chunk 3: 'ar.'

❌ Problem: It cuts words (language is split between Chunk 1 and Chunk 2).

Token-based chunking (better):

import tiktoken

def token_chunking(text, chunk_size=10, overlap=2):
    """Token-based chunking with tiktoken"""
    encoding = tiktoken.encoding_for_model("gpt-4")
    
    # Tokenize the full text
    tokens = encoding.encode(text)
    
    chunks = []
    start = 0
    
    while start < len(tokens):
        end = start + chunk_size
        chunk_tokens = tokens[start:end]
        
        # Decode tokens back to text
        chunk_text = encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
        
        start += (chunk_size - overlap)
    
    return chunks

text = "Python is a programming language. It is very popular for AI."
chunks = token_chunking(text, chunk_size=10, overlap=2)

for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}: '{chunk}'")

Output:

Chunk 1: 'Python is a programming language. It is very popular'
Chunk 2: ' very popular for AI.'

✅ Better: It doesn't cut words, it respects the model's tokenization.

Production-Ready Implementation

The TokenChunker class:

import tiktoken
from typing import List, Dict

class TokenChunker:
    """
    Token-based chunker with overlap
    
    Features:
    - Precise token counting (tiktoken)
    - Configurable overlap
    - Metadata tracking
    """
    
    def __init__(
        self,
        chunk_size: int = 500,
        overlap: int = 50,
        model: str = "gpt-4"
    ):
        """
        Args:
            chunk_size: Chunk size in tokens
            overlap: Overlap tokens between chunks
            model: Model for tokenization (gpt-4, text-embedding-3-small)
        """
        self.chunk_size = chunk_size
        self.overlap = overlap
        self.encoding = tiktoken.encoding_for_model(model)
    
    def chunk(self, text: str) -> List[Dict]:
        """
        Split text into chunks
        
        Returns:
            List of dicts with keys: text, start_token, end_token, chunk_id
        """
        # Tokenize
        tokens = self.encoding.encode(text)
        
        chunks = []
        chunk_id = 0
        start = 0
        
        while start < len(tokens):
            end = min(start + self.chunk_size, len(tokens))
            chunk_tokens = tokens[start:end]
            
            # Decode
            chunk_text = self.encoding.decode(chunk_tokens)
            
            chunks.append({
                'chunk_id': chunk_id,
                'text': chunk_text,
                'start_token': start,
                'end_token': end,
                'n_tokens': len(chunk_tokens)
            })
            
            chunk_id += 1
            start += (self.chunk_size - self.overlap)
        
        return chunks
    
    def get_stats(self, chunks: List[Dict]) -> Dict:
        """Chunking statistics"""
        return {
            'n_chunks': len(chunks),
            'avg_tokens': sum(c['n_tokens'] for c in chunks) / len(chunks) if chunks else 0,
            'total_tokens': sum(c['n_tokens'] for c in chunks)
        }

# Usage
chunker = TokenChunker(chunk_size=500, overlap=50)

text = """
Python is a high-level programming language known for its simplicity
and readability. It was created by Guido van Rossum and first released
in 1991. Python supports multiple programming paradigms including
procedural, object-oriented, and functional programming.
""" * 10  # Repeat to make it longer

chunks = chunker.chunk(text)

print(f"Total chunks: {len(chunks)}")
print(f"\nFirst chunk:")
print(chunks[0]['text'][:100] + "...")
print(f"Tokens: {chunks[0]['n_tokens']}")

stats = chunker.get_stats(chunks)
print(f"\nStats:")
print(f"  Chunks: {stats['n_chunks']}")
print(f"  Avg tokens/chunk: {stats['avg_tokens']:.0f}")

Output:

Total chunks: 2

First chunk:
Python is a high-level programming language known for its simplicity
and readability. It was...
Tokens: 500

Stats:
  Chunks: 2
  Avg tokens/chunk: 290

Overlap Strategies

Why overlap:

# Without overlap:
Chunk 1: [tokens 0-500]
Chunk 2: [tokens 500-1000]

# Problem:
# If important info is at the boundary (tokens 495-505),
# it's split between 2 chunks:
# Chunk 1: "...Python supports multiple programming par"
# Chunk 2: "adigms including procedural..."
# ❌ "paradigms" was cut

# With overlap (50 tokens):
Chunk 1: [tokens 0-500]
Chunk 2: [tokens 450-950]  ← 50-token overlap

# Now the boundary info appears complete in both chunks:
# Chunk 1: "...Python supports multiple programming paradigms"
# Chunk 2: "Python supports multiple programming paradigms including..."
# ✅ "paradigms" complete in both

Optimal overlap:

# Rule of thumb: 10-20% of the chunk size

chunk_size = 500
overlap = int(chunk_size * 0.15)  # 15% = 75 tokens

# Trade-off:
# - Overlap 0%: Risk of losing context at boundaries
# - Overlap 50%: Excessive redundancy (wasted storage)
# - Overlap 10-20%: Optimal balance

Fixed-Size Trade-Offs

Advantages:

✅ 1. Simple and deterministic

# Same inputs → same chunks (reproducible)
# Doesn't depend on NLP models (spaCy, NLTK)

✅ 2. Fast

# Just tokenize + slice
# No complex semantic processing
# ~1ms for a 10K-token document

✅ 3. Predictable size

# All chunks ~500 tokens (±overlap)
# Easy to estimate storage, latency, costs

Disadvantages:

❌ 1. Doesn't respect semantic boundaries

# Can cut in the middle of a:
# - Sentence
# - Paragraph
# - List
# - Table

# Example:
# Chunk 1: "...The three main types are: 1. Type A, 2. Type B,"
# Chunk 2: "3. Type C. Each type has different..."
# ❌ Split list (chunk 1 incomplete)

❌ 2. Loses structural context

# Headers, sections, hierarchy are lost
# A chunk may not have enough context to be understood

Optimize Chunk Size

Experiment: Different sizes

import tiktoken
from openai import OpenAI
import numpy as np
from dotenv import load_dotenv
import os

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# Long document (sample)
document = """
[Your technical documentation here - 5000 tokens]
"""

# Test different chunk sizes
chunk_sizes = [200, 400, 600, 800, 1000]

for size in chunk_sizes:
    chunker = TokenChunker(chunk_size=size, overlap=int(size*0.1))
    chunks = chunker.chunk(document)
    
    # Generate embeddings (to calculate storage)
    n_embeddings = len(chunks)
    storage_gb = n_embeddings * 1536 * 4 / (1024**3)  # 1536 dims, 4 bytes/float
    
    print(f"Chunk size {size}:")
    print(f"  Chunks: {len(chunks)}")
    print(f"  Storage: {storage_gb*1000:.2f} MB (per 1M docs)")
    print()

Typical output:

Chunk size 200:
  Chunks: 27
  Storage: 158.20 MB

Chunk size 400:
  Chunks: 14
  Storage: 82.03 MB

Chunk size 600:
  Chunks: 9
  Storage: 52.73 MB

Chunk size 800:
  Chunks: 7
  Storage: 41.01 MB

Chunk size 1000:
  Chunks: 6
  Storage: 35.16 MB

Trade-off: Larger chunks → less storage, but less retrieval precision.

Exercises

Exercise 1: Implement a basic chunker

Implement a function that splits text into N-token chunks:

import tiktoken

def simple_chunker(text, chunk_size=500):
    """Chunking without overlap"""
    # Implement here
    pass

# Test
text = "Python is great" * 200
chunks = simple_chunker(text, chunk_size=100)
print(f"Total chunks: {len(chunks)}")

import tiktoken

def simple_chunker(text, chunk_size=500):
    """Chunking without overlap"""
    encoding = tiktoken.encoding_for_model("gpt-4")
    tokens = encoding.encode(text)
    
    chunks = []
    for i in range(0, len(tokens), chunk_size):
        chunk_tokens = tokens[i:i+chunk_size]
        chunk_text = encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
    
    return chunks

# Test
text = "Python is great" * 200  # ~600 tokens
chunks = simple_chunker(text, chunk_size=100)

print(f"Total chunks: {len(chunks)}")  # ~6 chunks
print(f"First chunk: {chunks[0][:50]}...")

Exercise 2: Add overlap

Modify the previous function to add 10% overlap:

def chunker_with_overlap(text, chunk_size=500, overlap_pct=0.1):
    """Chunking with percentage overlap"""
    encoding = tiktoken.encoding_for_model("gpt-4")
    tokens = encoding.encode(text)
    
    overlap = int(chunk_size * overlap_pct)
    step = chunk_size - overlap
    
    chunks = []
    start = 0
    
    while start < len(tokens):
        end = start + chunk_size
        chunk_tokens = tokens[start:end]
        chunk_text = encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
        start += step
    
    return chunks

# Test
chunks = chunker_with_overlap("Python is great" * 200, chunk_size=100, overlap_pct=0.1)
print(f"Total chunks: {len(chunks)}")  # ~7 chunks (more than without overlap)

Summary

What you learned:

• ✅ Character vs Token: Token-based is better (doesn't cut words)
• ✅ Implementation: TokenChunker with tiktoken
• ✅ Overlap: 10-20% is optimal (context/storage balance)
• ✅ Trade-offs: Simple/fast but not semantic
• ✅ Chunk size: 400-800 tokens is typical

Key concepts:

1. Token-based chunking preserves words
2. Overlap is critical for boundaries
3. Fixed-size = simple but not perfect

Additional resources

1. tiktoken GitHub - Token counting
2. LangChain CharacterTextSplitter - Alternative implementation
3. Chunking Strategies - Pinecone guide

In the next capsule

Capsule 03: Semantic Chunking

You'll learn:

• Sentence-based chunking
• Paragraph-based chunking
• Topic segmentation
• Trade-offs vs fixed-size

From fixed-size to semantic boundaries.

Module 4 - Embeddings Deep Dive Guide Fixed-size chunking: simple, fast, and effective