Fixed-Size Chunking: Implementación Práctica

Descripción de la cápsula

Fixed-size chunking es la estrategia más simple: dividir texto cada N caracteres o tokens con overlap opcional. Aunque no respeta boundaries semánticas, es determinístico, rápido, y suficiente para muchos casos de uso.

En esta cápsula aprenderás la diferencia entre character-based y token-based chunking, implementarás chunking con tiktoken (token counting preciso), overlap strategies con sliding window, y trade-offs de fixed-size. También verás código production-ready reutilizable.

Al final, podrás implementar fixed-size chunking robusto para tus proyectos RAG.

Character-Based vs Token-Based

Character-based chunking:

# Dividir cada N caracteres
text = "Python is a programming language. It is very popular."

def char_chunking(text, chunk_size=20, overlap=5):
    """Character-based chunking"""
    chunks = []
    start = 0
    
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start += (chunk_size - overlap)
    
    return chunks

chunks = char_chunking(text, chunk_size=30, overlap=5)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}: '{chunk}'")

Output:

Chunk 1: 'Python is a programming la'
Chunk 2: 'ming language. It is very '
Chunk 3: ' very popular.'

❌ Problema: Corta palabras ("language", "ming").

Token-based chunking (mejor):

import tiktoken

def token_chunking(text, chunk_size=10, overlap=2):
    """Token-based chunking con tiktoken"""
    encoding = tiktoken.encoding_for_model("gpt-4")
    
    # Tokenizar texto completo
    tokens = encoding.encode(text)
    
    chunks = []
    start = 0
    
    while start < len(tokens):
        end = start + chunk_size
        chunk_tokens = tokens[start:end]
        
        # Decodificar tokens a texto
        chunk_text = encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
        
        start += (chunk_size - overlap)
    
    return chunks

text = "Python is a programming language. It is very popular for AI."
chunks = token_chunking(text, chunk_size=10, overlap=2)

for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}: '{chunk}'")

Output:

Chunk 1: 'Python is a programming language.'
Chunk 2: 'language. It is very popular'
Chunk 3: 'popular for AI.'

✅ Mejor: No corta palabras, respeta tokenización del modelo.

Implementación Production-Ready

Clase TokenChunker:

import tiktoken
from typing import List, Dict

class TokenChunker:
    """
    Token-based chunker con overlap
    
    Características:
    - Token counting preciso (tiktoken)
    - Overlap configurable
    - Metadata tracking
    """
    
    def __init__(
        self,
        chunk_size: int = 500,
        overlap: int = 50,
        model: str = "gpt-4"
    ):
        """
        Args:
            chunk_size: Tamaño de chunk en tokens
            overlap: Tokens de overlap entre chunks
            model: Modelo para tokenization (gpt-4, text-embedding-3-small)
        """
        self.chunk_size = chunk_size
        self.overlap = overlap
        self.encoding = tiktoken.encoding_for_model(model)
    
    def chunk(self, text: str) -> List[Dict]:
        """
        Dividir texto en chunks
        
        Returns:
            Lista de dicts con keys: text, start_token, end_token, chunk_id
        """
        # Tokenizar
        tokens = self.encoding.encode(text)
        
        chunks = []
        chunk_id = 0
        start = 0
        
        while start < len(tokens):
            end = min(start + self.chunk_size, len(tokens))
            chunk_tokens = tokens[start:end]
            
            # Decodificar
            chunk_text = self.encoding.decode(chunk_tokens)
            
            chunks.append({
                'chunk_id': chunk_id,
                'text': chunk_text,
                'start_token': start,
                'end_token': end,
                'n_tokens': len(chunk_tokens)
            })
            
            chunk_id += 1
            start += (self.chunk_size - self.overlap)
        
        return chunks
    
    def get_stats(self, chunks: List[Dict]) -> Dict:
        """Estadísticas de chunking"""
        return {
            'n_chunks': len(chunks),
            'avg_tokens': sum(c['n_tokens'] for c in chunks) / len(chunks) if chunks else 0,
            'total_tokens': sum(c['n_tokens'] for c in chunks)
        }

# Uso
chunker = TokenChunker(chunk_size=500, overlap=50)

text = """
Python is a high-level programming language known for its simplicity
and readability. It was created by Guido van Rossum and first released
in 1991. Python supports multiple programming paradigms including
procedural, object-oriented, and functional programming.
""" * 10  # Repetir para hacer más largo

chunks = chunker.chunk(text)

print(f"Total chunks: {len(chunks)}")
print(f"\nFirst chunk:")
print(chunks[0]['text'][:100] + "...")
print(f"Tokens: {chunks[0]['n_tokens']}")

stats = chunker.get_stats(chunks)
print(f"\nStats:")
print(f"  Chunks: {stats['n_chunks']}")
print(f"  Avg tokens/chunk: {stats['avg_tokens']:.0f}")

Output:

Total chunks: 3

First chunk:
Python is a high-level programming language known for its simplicity
and readability. It was...
Tokens: 500

Stats:
  Chunks: 3
  Avg tokens/chunk: 483

Overlap Strategies

Por qué overlap:

# Sin overlap:
Chunk 1: [tokens 0-500]
Chunk 2: [tokens 500-1000]

# Problema:
# Si info importante está en boundary (tokens 495-505),
# se divide entre 2 chunks:
# Chunk 1: "...Python supports multiple programming par"
# Chunk 2: "adigms including procedural..."
# ❌ "paradigms" se cortó

# Con overlap (50 tokens):
Chunk 1: [tokens 0-500]
Chunk 2: [tokens 450-950]  ← overlap de 50 tokens

# Ahora boundary info aparece completa en ambos chunks:
# Chunk 1: "...Python supports multiple programming paradigms"
# Chunk 2: "Python supports multiple programming paradigms including..."
# ✅ "paradigms" completo en ambos

Optimal overlap:

# Rule of thumb: 10-20% del chunk size

chunk_size = 500
overlap = int(chunk_size * 0.15)  # 15% = 75 tokens

# Trade-off:
# - Overlap 0%: Riesgo de perder contexto en boundaries
# - Overlap 50%: Redundancia excesiva (storage desperdiciado)
# - Overlap 10-20%: Balance óptimo

Trade-Offs de Fixed-Size

Ventajas:

✅ 1. Simple y determinístico

# Mismos inputs → mismos chunks (reproducible)
# No depende de modelos NLP (spaCy, NLTK)

✅ 2. Rápido

# Solo tokenizar + slice
# No procesamiento semántico complejo
# ~1ms para document de 10K tokens

✅ 3. Tamaño predecible

# Todos los chunks ~500 tokens (±overlap)
# Fácil estimar storage, latencia, costos

Desventajas:

❌ 1. No respeta boundaries semánticas

# Puede cortar en medio de:
# - Frase
# - Párrafo
# - Lista
# - Tabla

# Ejemplo:
# Chunk 1: "...The three main types are: 1. Type A, 2. Type B,"
# Chunk 2: "3. Type C. Each type has different..."
# ❌ Lista dividida (chunk 1 incompleto)

❌ 2. Pierde contexto estructural

# Headers, secciones, jerarquía se pierden
# Chunk puede no tener suficiente contexto para entender

Optimizar Chunk Size

Experimento: Diferentes tamaños

import tiktoken
from openai import OpenAI
import numpy as np
from dotenv import load_dotenv
import os

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# Documento largo (sample)
document = """
[Tu documentación técnica aquí - 5000 tokens]
"""

# Test diferentes chunk sizes
chunk_sizes = [200, 400, 600, 800, 1000]

for size in chunk_sizes:
    chunker = TokenChunker(chunk_size=size, overlap=int(size*0.1))
    chunks = chunker.chunk(document)
    
    # Generar embeddings (para calcular storage)
    n_embeddings = len(chunks)
    storage_gb = n_embeddings * 1536 * 4 / (1024**3)  # 1536 dims, 4 bytes/float
    
    print(f"Chunk size {size}:")
    print(f"  Chunks: {len(chunks)}")
    print(f"  Storage: {storage_gb*1000:.2f} MB (por 1M docs)")
    print()

Output típico:

Chunk size 200:
  Chunks: 27
  Storage: 158.20 MB

Chunk size 400:
  Chunks: 14
  Storage: 82.03 MB

Chunk size 600:
  Chunks: 9
  Storage: 52.73 MB

Chunk size 800:
  Chunks: 7
  Storage: 41.01 MB

Chunk size 1000:
  Chunks: 6
  Storage: 35.16 MB

Trade-off: Chunks más grandes → menos storage, pero menos precision en retrieval.

Ejercicios

Ejercicio 1: Implementar chunker básico

Implementa función que divide texto en chunks de N tokens:

import tiktoken

def simple_chunker(text, chunk_size=500):
    """Chunking sin overlap"""
    # Implementa aquí
    pass

# Test
text = "Python is great" * 200
chunks = simple_chunker(text, chunk_size=100)
print(f"Total chunks: {len(chunks)}")

import tiktoken

def simple_chunker(text, chunk_size=500):
    """Chunking sin overlap"""
    encoding = tiktoken.encoding_for_model("gpt-4")
    tokens = encoding.encode(text)
    
    chunks = []
    for i in range(0, len(tokens), chunk_size):
        chunk_tokens = tokens[i:i+chunk_size]
        chunk_text = encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
    
    return chunks

# Test
text = "Python is great. " * 200  # ~600 tokens
chunks = simple_chunker(text, chunk_size=100)

print(f"Total chunks: {len(chunks)}")  # ~6 chunks
print(f"First chunk: {chunks[0][:50]}...")

Ejercicio 2: Agregar overlap

Modifica la función anterior para agregar overlap de 10%:

def chunker_with_overlap(text, chunk_size=500, overlap_pct=0.1):
    """Chunking con overlap porcentual"""
    encoding = tiktoken.encoding_for_model("gpt-4")
    tokens = encoding.encode(text)
    
    overlap = int(chunk_size * overlap_pct)
    step = chunk_size - overlap
    
    chunks = []
    start = 0
    
    while start < len(tokens):
        end = start + chunk_size
        chunk_tokens = tokens[start:end]
        chunk_text = encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
        start += step
    
    return chunks

# Test
chunks = chunker_with_overlap("Python is great. " * 200, chunk_size=100, overlap_pct=0.1)
print(f"Total chunks: {len(chunks)}")  # ~7 chunks (más que sin overlap)

Resumen

Qué aprendiste:

• ✅ Character vs Token: Token-based mejor (no corta palabras)
• ✅ Implementación: TokenChunker con tiktoken
• ✅ Overlap: 10-20% óptimo (balance contexto/storage)
• ✅ Trade-offs: Simple/rápido pero no semántico
• ✅ Chunk size: 400-800 tokens típico

Conceptos clave:

1. Token-based chunking preserva palabras
2. Overlap crítico para boundaries
3. Fixed-size = simple pero no perfecto

Recursos adicionales

1. tiktoken GitHub - Token counting
2. LangChain CharacterTextSplitter - Implementación alternativa
3. Chunking Strategies - Pinecone guide

En la siguiente cápsula

Cápsula 03: Semantic Chunking

Aprenderás:

• Sentence-based chunking
• Paragraph-based chunking
• Topic segmentation
• Trade-offs vs fixed-size

De fixed-size a semantic boundaries.

Módulo 4 - Embeddings Deep Dive Guide Fixed-size chunking: simple, rápido, y efectivo