Semantic Chunking: Respetando Estructura del Documento

Descripción de la cápsula

Semantic chunking divide texto en boundaries naturales (frases, párrafos, secciones) en lugar de cada N tokens. Esto preserva coherencia y contexto, generando chunks más comprensibles para humanos y LLMs.

En esta cápsula aprenderás sentence-based chunking, paragraph-based chunking, cómo usar LangChain RecursiveCharacterTextSplitter, y trade-offs vs fixed-size. También implementarás semantic chunking production-ready.

Al final, podrás elegir entre fixed-size y semantic según tu caso de uso.

Sentence-Based Chunking

Concepto:

# Dividir por frases (sentences)
# Acumular frases hasta alcanzar ~N tokens

text = """
Python is a programming language. It was created in 1991. 
Python is known for simplicity. It supports multiple paradigms.
"""

# Fixed-size (corta mid-sentence):
# Chunk 1: "Python is a programming language. It wa"
# Chunk 2: "s created in 1991. Python is known for"
# ❌ Corta mal

# Sentence-based (respeta frases):
# Chunk 1: "Python is a programming language. It was created in 1991."
# Chunk 2: "Python is known for simplicity. It supports multiple paradigms."
# ✅ Coherente

Implementación con split simple:

import tiktoken

class SentenceChunker:
    """Sentence-based chunker"""
    
    def __init__(self, max_tokens=500, model="gpt-4"):
        self.max_tokens = max_tokens
        self.encoding = tiktoken.encoding_for_model(model)
    
    def chunk(self, text):
        """Dividir en chunks respetando frases"""
        # Split por puntos (simple sentence boundary)
        sentences = text.split('. ')
        
        chunks = []
        current_chunk = []
        current_tokens = 0
        
        for sentence in sentences:
            sentence = sentence.strip() + '.'
            sentence_tokens = len(self.encoding.encode(sentence))
            
            # Si agregar esta frase excede límite, cerrar chunk actual
            if current_tokens + sentence_tokens > self.max_tokens and current_chunk:
                chunks.append(' '.join(current_chunk))
                current_chunk = []
                current_tokens = 0
            
            current_chunk.append(sentence)
            current_tokens += sentence_tokens
        
        # Agregar último chunk
        if current_chunk:
            chunks.append(' '.join(current_chunk))
        
        return chunks

# Uso
chunker = SentenceChunker(max_tokens=100)

text = """
Python is a high-level programming language. It was created by Guido van Rossum. 
Python emphasizes code readability. It supports multiple programming paradigms. 
Python has a large standard library. It is widely used in data science and AI.
"""

chunks = chunker.chunk(text)

for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}:")
    print(chunk)
    print(f"Tokens: {len(chunker.encoding.encode(chunk))}\n")

Output:

Chunk 1:
Python is a high-level programming language. It was created by Guido van Rossum. Python emphasizes code readability.
Tokens: 28

Chunk 2:
It supports multiple programming paradigms. Python has a large standard library. It is widely used in data science and AI.
Tokens: 26

✅ Cada chunk termina en boundary natural (fin de frase).

Paragraph-Based Chunking

Concepto:

# Dividir por párrafos (double newline: \n\n)
# Más contexto que sentences, respeta estructura del autor

text = """
Python is a programming language created in 1991.
It emphasizes code readability and simplicity.

Python supports multiple paradigms including OOP.
It has a large ecosystem of libraries and frameworks.

Python is widely used in data science and AI.
Popular libraries include NumPy, Pandas, and TensorFlow.
"""

# Paragraph-based:
# Chunk 1: "Python is a programming language... simplicity."
# Chunk 2: "Python supports multiple paradigms... frameworks."
# Chunk 3: "Python is widely used... TensorFlow."
# ✅ Preserva estructura lógica del autor

Implementación:

class ParagraphChunker:
    """Paragraph-based chunker"""
    
    def __init__(self, max_tokens=500, model="gpt-4"):
        self.max_tokens = max_tokens
        self.encoding = tiktoken.encoding_for_model(model)
    
    def chunk(self, text):
        """Dividir por párrafos"""
        # Split por double newline
        paragraphs = text.split('\n\n')
        
        chunks = []
        current_chunk = []
        current_tokens = 0
        
        for paragraph in paragraphs:
            paragraph = paragraph.strip()
            if not paragraph:
                continue
            
            para_tokens = len(self.encoding.encode(paragraph))
            
            # Si párrafo solo excede límite, dividir por frases
            if para_tokens > self.max_tokens:
                # Fallback a sentence-based para este párrafo
                sentence_chunker = SentenceChunker(self.max_tokens)
                para_chunks = sentence_chunker.chunk(paragraph)
                chunks.extend(para_chunks)
                continue
            
            # Si agregar este párrafo excede límite
            if current_tokens + para_tokens > self.max_tokens and current_chunk:
                chunks.append('\n\n'.join(current_chunk))
                current_chunk = []
                current_tokens = 0
            
            current_chunk.append(paragraph)
            current_tokens += para_tokens
        
        # Agregar último chunk
        if current_chunk:
            chunks.append('\n\n'.join(current_chunk))
        
        return chunks

LangChain RecursiveCharacterTextSplitter

Qué es:

Chunker recursivo de LangChain: Intenta dividir por separadores jerárquicos (\n\n → \n → espacio → carácter).

Ventaja: Balance entre semantic y fixed-size.

Código con LangChain:

from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken

# Crear splitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,          # Tamaño máximo en caracteres
    chunk_overlap=50,        # Overlap
    length_function=len,     # Función para medir tamaño
    separators=["\n\n", "\n", " ", ""]  # Separadores jerárquicos
)

text = """
# Python Programming

Python is a high-level programming language.
It was created by Guido van Rossum in 1991.

## Features

Python emphasizes code readability and simplicity.
It supports multiple programming paradigms.

Python has a large standard library.
The ecosystem includes frameworks like Django and Flask.

## Use Cases

Python is widely used in:
- Data Science
- Machine Learning
- Web Development
- Automation
"""

# Dividir
chunks = text_splitter.split_text(text)

print(f"Total chunks: {len(chunks)}\n")

for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}:")
    print(chunk)
    print(f"Length: {len(chunk)} chars\n")

Output:

Total chunks: 3

Chunk 1:
# Python Programming

Python is a high-level programming language.
It was created by Guido van Rossum in 1991.

## Features

Python emphasizes code readability and simplicity.
Length: 168 chars

Chunk 2:
It supports multiple programming paradigms.

Python has a large standard library.
The ecosystem includes frameworks like Django and Flask.

## Use Cases
Length: 154 chars

Chunk 3:
Python is widely used in:
- Data Science
- Machine Learning
- Web Development
- Automation
Length: 98 chars

✅ Respeta headers, párrafos, y listas.

Token-Aware Recursive Splitter

Problema con RecursiveCharacterTextSplitter:

# LangChain usa caracteres, no tokens
# chunk_size=500 caracteres ≠ 500 tokens

# Ejemplo:
text = "Python" * 100  # 600 caracteres
# Caracteres: 600
# Tokens (tiktoken): ~100 tokens

# Si configuraste chunk_size=500 caracteres,
# pero el límite real es 500 tokens,
# ¡chunk puede exceder límite de tokens!

Solución: Token-aware splitter

from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken

# Token counting function
def tiktoken_len(text):
    encoding = tiktoken.encoding_for_model("gpt-4")
    return len(encoding.encode(text))

# Token-aware splitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,          # Ahora en TOKENS
    chunk_overlap=50,
    length_function=tiktoken_len,  # ← Usar tiktoken
    separators=["\n\n", "\n", " ", ""]
)

text = "Python is great. " * 200  # ~600 tokens

chunks = text_splitter.split_text(text)

print(f"Total chunks: {len(chunks)}")

for i, chunk in enumerate(chunks):
    tokens = tiktoken_len(chunk)
    print(f"Chunk {i+1}: {tokens} tokens")

✅ Garantiza que cada chunk ≤ 500 tokens.

Trade-Offs: Fixed vs Semantic

Comparación:

Cuándo usar cada uno:

Fixed-Size:

✅ Velocidad crítica (latencia <5ms)
✅ Documentos sin estructura clara
✅ Simplicidad requerida
✅ Tamaño predecible importante

Semantic:

✅ Coherencia crítica (legal, medical)
✅ Documentos estructurados (headers, párrafos)
✅ Contexto importante
✅ OK con chunks de tamaño variable

Ejercicios

Ejercicio 1: Sentence-based chunker

Implementa chunker que divide por frases:

import tiktoken

def sentence_chunker(text, max_tokens=100):
    """Dividir por frases hasta max_tokens"""
    # Implementa aquí
    pass

# Test
text = "Python is great. JavaScript is popular. Go is fast. Rust is safe."
chunks = sentence_chunker(text, max_tokens=50)
print(chunks)

import tiktoken

def sentence_chunker(text, max_tokens=100):
    """Dividir por frases"""
    encoding = tiktoken.encoding_for_model("gpt-4")
    sentences = [s.strip() + '.' for s in text.split('.') if s.strip()]
    
    chunks = []
    current = []
    current_tokens = 0
    
    for sentence in sentences:
        tokens = len(encoding.encode(sentence))
        
        if current_tokens + tokens > max_tokens and current:
            chunks.append(' '.join(current))
            current = []
            current_tokens = 0
        
        current.append(sentence)
        current_tokens += tokens
    
    if current:
        chunks.append(' '.join(current))
    
    return chunks

# Test
text = "Python is great. JavaScript is popular. Go is fast. Rust is safe."
chunks = sentence_chunker(text, max_tokens=20)

for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}: {chunk}")

Ejercicio 2: LangChain token-aware

Usa RecursiveCharacterTextSplitter con tiktoken:

from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken

def tiktoken_len(text):
    encoding = tiktoken.encoding_for_model("gpt-4")
    return len(encoding.encode(text))

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=100,
    chunk_overlap=10,
    length_function=tiktoken_len,
    separators=["\n\n", "\n", " ", ""]
)

text = """
Python is a programming language.
It is very popular for AI and data science.

JavaScript is used for web development.
It runs in the browser and on servers with Node.js.
"""

chunks = text_splitter.split_text(text)

for i, chunk in enumerate(chunks):
    tokens = tiktoken_len(chunk)
    print(f"Chunk {i+1} ({tokens} tokens):")
    print(chunk)
    print()

Resumen

Qué aprendiste:

• ✅ Sentence-based: Respeta frases, más coherente
• ✅ Paragraph-based: Respeta párrafos, preserva estructura
• ✅ LangChain Recursive: Balance fixed/semantic
• ✅ Token-aware: Garantiza chunks ≤ max_tokens
• ✅ Trade-offs: Fixed (rápido) vs Semantic (coherente)

Conceptos clave:

1. Semantic chunking preserva coherencia
2. RecursiveCharacterTextSplitter = best of both worlds
3. Token-aware crítico para límites de API

Recursos adicionales

1. LangChain Text Splitters - Docs oficiales
2. RecursiveCharacterTextSplitter - API reference
3. Chunking Strategies - Pinecone guide

En la siguiente cápsula

Cápsula 04: Métricas de Retrieval

Aprenderás:

• nDCG (Normalized Discounted Cumulative Gain)
• MRR (Mean Reciprocal Rank)
• Recall@K, Precision@K
• Implementación en Python

De chunking a evaluación cuantitativa.

Módulo 4 - Embeddings Deep Dive Guide Semantic chunking: respetando la estructura del documento