Module 4: Evaluation and Chunking Strategies

Introduction to Module 4: Evaluation and Chunking Strategies

Next →

Module welcome

You already know what embeddings are (M1), how they work internally (M2), and how to choose the right model (M3). Now comes the critical step for implementing RAG: chunking (splitting long documents into fragments) and evaluation (measuring whether your retrieval system works well).

In this module you'll learn chunking strategies (fixed-size, semantic, recursive), how to evaluate retrieval quality (metrics like nDCG, MRR, Recall@K), how to optimize chunk size and overlap, and how to build a RAG system with intelligent chunking.

By the end, you'll be able to implement production-ready RAG with optimal chunking and robust evaluation metrics.

Module objectives

By completing this module, you'll be able to:

  1. ✅ Implement chunking: Fixed-size, semantic, recursive
  2. ✅ Optimize chunks: Optimal size, overlap, boundaries
  3. ✅ Evaluate retrieval: nDCG, MRR, Recall@K, Precision@K
  4. ✅ Build datasets: Evaluation sets for RAG
  5. ✅ A/B testing: Compare chunking strategies
  6. ✅ Context window: Handle token limits (8K)
  7. ✅ Metadata enrichment: Add context to chunks
  8. ✅ Project: A RAG system with intelligent chunking and evaluation

Module roadmap

Phase 1: Chunking fundamentals (Capsules 01-03)

Capsule 01: Introduction (this capsule)

  • Objectives, roadmap, context

Capsule 02: Fixed-Size Chunking

  • Character-based, token-based
  • Overlap strategies
  • Implementation code

Capsule 03: Semantic Chunking

  • Sentence boundaries
  • Paragraph-based
  • Topic-based segmentation
  • Trade-offs

Phase 2: Retrieval evaluation (Capsules 04-06)

Capsule 04: Retrieval Metrics

  • nDCG (Normalized Discounted Cumulative Gain)
  • MRR (Mean Reciprocal Rank)
  • Recall@K, Precision@K, F1@K

Capsule 05: Create Evaluation Datasets

  • Formats (queries + relevant docs)
  • Synthetic data generation
  • Human annotation
  • Quality assurance

Capsule 06: A/B Testing Chunking Strategies

  • Compare fixed vs semantic
  • Statistical significance
  • Optimization loops

Phase 3: Production patterns (Capsules 07-08)

Capsule 07: Advanced Chunking Patterns

  • Recursive chunking (LangChain-style)
  • Metadata enrichment
  • Hierarchical chunks
  • Context preservation

Capsule 08: Mini-Project - RAG System with Intelligent Chunking

  • Automatic chunking
  • Embedding + vector storage (in-memory)
  • Retrieval with evaluation
  • End-to-end metrics

Connection with the AI Engineering Path

Prerequisites (completed):

✅ Module 1: What are embeddings? ✅ Module 2: Embedding architecture ✅ Module 3: Model comparison

This module in the path:

Embeddings Deep Dive - M3 (Model comparison)
          ↓
Embeddings Deep Dive - M4 (Chunking + Evaluation) ← YOU ARE HERE
          ↓
Embeddings Deep Dive - M5 (Distance Metrics)
          ↓
Vector Databases Guide
          ↓
RAG Production Patterns

Why chunking is critical for RAG

The problem:

# Long document (10,000 tokens):
document = """
[10,000 words of technical documentation]
"""

# Problem 1: Token limit
# OpenAI embeddings: Max 8,191 tokens
# This document does NOT fit in 1 embedding

# Problem 2: Granularity
# An embedding of the whole document = too general
# The user asks about a specific feature
# The embedding doesn't capture enough detail

# Problem 3: Retrieval precision
# If you retrieve the whole document, the LLM receives 10K irrelevant tokens
# The context window is wasted

Solution: Chunking (splitting into fragments).

Chunking example:

# Original (10,000 tokens):
document = """
Chapter 1: Introduction to Python
Python is a high-level programming language...
[8,000 more words]

Chapter 2: Variables and Data Types
Variables store data values...
[2,000 more words]
"""

# After chunking (500-token chunks):
chunks = [
    "Chapter 1: Introduction to Python. Python is a high-level...",
    "...programming language created by Guido van Rossum...",
    "Chapter 2: Variables and Data Types. Variables store...",
    "...data values. Python has several data types..."
]

# Now:
# - Each chunk < 8,191 tokens ✅
# - Specific granularity ✅
# - Precise retrieval (only relevant chunks) ✅

Chunking strategies (overview)

1. Fixed-Size Chunking:

# Split every N characters/tokens
chunk_size = 500  # tokens
overlap = 50      # overlap tokens

# Pros:
# - Simple, deterministic
# - Fast

# Cons:
# - Can cut in the middle of a sentence/paragraph
# - Doesn't respect semantics

2. Semantic Chunking:

# Split at natural boundaries (sentences, paragraphs, sections)

# Pros:
# - Respects the document structure
# - More coherent chunks

# Cons:
# - Variable-size chunks (some very large/small)
# - More complex

3. Recursive Chunking:

# Split hierarchically:
# 1. Try to split by paragraphs
# 2. If a paragraph is too large → split by sentences
# 3. If a sentence is too large → split by characters

# Pros:
# - Balance between fixed and semantic
# - Guarantees a maximum size

# Cons:
# - More complex to implement

Chunking trade-offs

Chunk size:

# Small chunks (200 tokens):
✅ Very precise retrieval (only relevant info)
✅ Less noise for the LLM
❌ May lose context (fragments too small)
❌ More chunks = more embeddings = more storage

# Large chunks (1000 tokens):
✅ More context preserved
✅ Fewer chunks = less storage
❌ Less precise retrieval (more noise)
❌ Wastes the LLM's context window

Optimal range: 400-800 tokens (depends on the case).

Overlap:

# Without overlap:
Chunk 1: [tokens 0-500]
Chunk 2: [tokens 500-1000]
# Problem: If important info is at the boundary (token 499-501),
# it's split between 2 chunks → loses coherence

# With overlap (50 tokens):
Chunk 1: [tokens 0-500]
Chunk 2: [tokens 450-950]  ← 50-token overlap
# Advantage: Info at the boundary appears complete in both chunks

# Optimal overlap: 10-20% of the chunk size

Retrieval evaluation

Why evaluate:

# Without evaluation:
# "The system works... I think?"
# You don't know if chunking is optimal, if the model is right, etc.

# With evaluation:
# "500-token chunking has Recall@5 = 0.85"
# "Changing to 800 tokens improves it to 0.90"
# → Data-driven decisions

Main metrics:

1. Recall@K:

# Of the relevant docs, how many are in the top-K?
# Example:
# - Query: "How to install Python?"
# - Relevant docs: [doc_3, doc_7]
# - Top-5 retrieved: [doc_1, doc_3, doc_5, doc_7, doc_9]
# Recall@5 = 2/2 = 1.0 (both relevant docs in the top-5)

2. Precision@K:

# Of the top-K retrieved, how many are relevant?
# Precision@5 = 2/5 = 0.4 (2 relevant out of 5 retrieved)

3. nDCG@K:

# Like Recall but it considers ranking (a better score for docs higher up)
# nDCG@5 = 0.85 (doc_3 in position 2, doc_7 in position 4)

4. MRR (Mean Reciprocal Rank):

# Position of the FIRST relevant doc
# doc_3 is in position 2 → RR = 1/2 = 0.5

What you'll learn (in detail)

1. Chunking strategies:

# Fixed-size (implementation):
def fixed_size_chunking(text, chunk_size=500, overlap=50):
    """Split text into fixed-size chunks"""
    # You'll implement this function with tiktoken
    pass

# Semantic (implementation):
def semantic_chunking(text, model="sentence-transformers"):
    """Split by semantic boundaries"""
    # You'll implement it with spaCy or LangChain
    pass

# Recursive (implementation):
def recursive_chunking(text, max_chunk_size=800):
    """Split hierarchically"""
    # You'll implement a simplified version of LangChain
    pass

2. Systematic evaluation:

# Create an evaluation dataset:
eval_dataset = [
    {
        "query": "How to install Python?",
        "relevant_doc_ids": [3, 7, 12],
        "corpus": [...1000 documents...]
    },
    # ... more queries
]

# Evaluate retrieval:
metrics = evaluate_retrieval(
    queries=eval_dataset,
    retrieval_function=my_rag_system,
    k=5
)

print(f"Recall@5: {metrics['recall@5']:.2f}")
print(f"nDCG@5: {metrics['ndcg@5']:.2f}")

3. Optimization:

# A/B test different chunk sizes:
chunk_sizes = [200, 400, 600, 800, 1000]

for size in chunk_sizes:
    chunks = fixed_size_chunking(docs, chunk_size=size)
    metrics = evaluate_retrieval(chunks)
    print(f"Size {size}: Recall@5 = {metrics['recall@5']:.2f}")

# Output:
# Size 200: Recall@5 = 0.75
# Size 400: Recall@5 = 0.85
# Size 600: Recall@5 = 0.90  ← Optimal
# Size 800: Recall@5 = 0.88
# Size 1000: Recall@5 = 0.82

Theory/practice balance (40/60)

Theory (40%):

  • Chunking strategies (concepts)
  • Evaluation metrics (formulas)
  • Trade-offs (size, overlap)

Practice (60%):

  • Implement chunking (code)
  • Calculate metrics (code)
  • A/B testing (code)
  • Complete RAG project

What you will NOT learn (out of scope)

❌ Vector databases (Pinecone, Weaviate):

Reason: A dedicated module comes next (Vector Databases Guide).
Coverage: In-memory storage (numpy) is enough to learn.

❌ LLM generation (GPT-4 responses):

Reason: Focus on retrieval (the R of RAG), not generation (the G).
Coverage: Only up to retrieving relevant chunks.

❌ Advanced NLP (NER, POS tagging):

Reason: Not critical for basic chunking.
Coverage: Only sentence/paragraph boundaries.

Module tools

Python libraries:

# Chunking
import tiktoken  # Token counting (OpenAI)
from langchain.text_splitter import RecursiveCharacterTextSplitter

# NLP (sentence boundaries)
import spacy  # Optional for semantic chunking

# Evaluation
import numpy as np  # Custom metrics

# Embeddings (already seen)
from openai import OpenAI
from sentence_transformers import SentenceTransformer

Required setup:

# Install LangChain (chunking utilities)
pip install langchain

# Install spaCy (optional - sentence boundaries)
pip install spacy
python -m spacy download en_core_web_sm

Module structure

module-04-chunking-evaluation/
└── en/
    ├── 01-module-introduction-4.md      ← You are here
    ├── 02-fixed-size-chunking.md
    ├── 03-semantic-chunking.md
    ├── 04-retrieval-metrics.md
    ├── 05-evaluation-datasets.md
    ├── 06-ab-testing-chunking.md
    ├── 07-advanced-chunking.md
    └── 08-project-smart-rag-chunking.md

Professional skills

By completing this module, you'll demonstrate:

1. RAG Implementation:

  • Implement production-ready chunking
  • Optimize chunk size (data-driven)
  • Preserve context

2. Evaluation mindset:

  • Create evaluation datasets
  • Calculate metrics (nDCG, MRR)
  • Systematic A/B testing

3. Trade-off analysis:

  • Balance size vs context
  • Optimal overlap
  • Fixed vs semantic chunking

Real use cases

1. Documentation search (1000 technical docs):

Requirements:

  • Chunks must respect sections (semantic)
  • Recall@5 > 0.85
  • Chunk size: 400-600 tokens

Decision:

Strategy: Semantic chunking (by section)
Average size: 500 tokens
Overlap: 50 tokens
Result: Recall@5 = 0.88 ✅

2. Legal document Q&A (contracts):

Requirements:

  • Chunks must preserve complete clauses
  • Critical context (don't cut mid-clause)
  • Recall@10 > 0.90

Decision:

Strategy: Semantic chunking (by clause)
Variable size: 300-1200 tokens
Overlap: 0 (clauses don't overlap)
Result: Recall@10 = 0.92 ✅

3. Customer support chatbot (FAQs):

Requirements:

  • Small chunks (concise answers)
  • Latency <100ms
  • Precision@3 > 0.80

Decision:

Strategy: Fixed-size chunking (simple, fast)
Size: 200 tokens
Overlap: 20 tokens
Result: Precision@3 = 0.83 ✅

Module methodology

Iterative learning:

Step 1: Implement basic chunking
         ↓
Step 2: Evaluate with metrics
         ↓
Step 3: Optimize (size, overlap, strategy)
         ↓
Step 4: Re-evaluate (loop)
         ↓
Step 5: Complete RAG project

Initial exercise (reflection)

Scenario:

You have 500 technical documents (Python tutorials), each averaging 2,000 tokens. A user asks: "How to use list comprehensions?"

Questions:

  1. Which chunking strategy would you use? Why?
  2. What chunk size would be optimal?
  3. Do you need overlap? How much?
  4. How would you evaluate whether your chunking works well?

There's no single answer (it depends on trade-offs).

By the end of the module, you'll be able to answer with an implementation and data.

Module resources

Papers and references:

  1. LangChain Text Splitters - Chunking strategies
  2. Retrieval Metrics - nDCG, MRR, etc.
  3. RAG Best Practices - Pinecone guide

Tools:

  1. LangChain - Chunking utilities
  2. tiktoken - Token counting

In the next capsule

Capsule 02: Fixed-Size Chunking

You'll learn:

  • Character-based vs token-based chunking
  • Implement chunking with tiktoken
  • Overlap strategies (sliding window)
  • Fixed-size trade-offs
  • Production-ready code

From introduction to practical implementation.

Module 4 - Embeddings Deep Dive Guide Chunking and evaluation: from long documents to production-ready RAG

← PreviousMini-Project: Benchmark Framework for Embedding ModelsNext →Fixed-Size Chunking: Practical Implementation