Capsule 04: Semantic chunking — chunks that respect the change of topic

Capsule overview

Recursive chunking (capsule 03) splits the document by structural separators — line breaks, periods, spaces. It works well when the document has a consistent format, but it has a subtle problem: it respects structure, not semantics. If your 2000-character document talks about FastAPI in its first 1200 chars and about Python in general in the last 800, recursive can hand you a 500-char chunk that mixes FastAPI's last sentences with Python's first ones.

Semantic chunking solves that: it splits when the content changes topic, regardless of where the structural separator happens to be. It uses embeddings to detect where semantic coherence breaks — where a sentence has an embedding very different from the one before it. That transition is the natural cut point.

This capsule teaches you how the algorithm works conceptually, how to implement it with LangChain's SemanticChunker, how to tune the breakpoint threshold, and when the extra cost (embeddings during indexing + 10x latency) is justified over recursive.

By the end of this capsule you'll be able to:

• ✅ Explain the semantic chunking algorithm in terms of cosine similarity between sentences
• ✅ Implement semantic chunking with SemanticChunker and OpenAI embeddings
• ✅ Tune the breakpoint_threshold for your dataset (percentile, standard deviation, interquartile)
• ✅ Calculate the extra cost of semantic chunking vs recursive for a given volume
• ✅ Decide when semantic chunking beats recursive, with data
• ✅ Anticipate the traps: chunks that are too big or too small depending on the threshold

Estimated time: 25-30 minutes

The insight: detecting topic transitions with embeddings

Think about a real technical document:

"FastAPI is a modern web framework for Python. It was created in 2018 by Sebastián Ramírez.
It is built on Pydantic for validation and Starlette for the underlying ASGI.

Its main features include performance comparable to Node.js, automatic type validation
with type hints, interactive documentation with Swagger UI, and native async/await
support.

Python is a high-level programming language. It was created by Guido van Rossum
in 1991. It is known for its readable syntax and its 'batteries included' philosophy.

The popularity of Python grew enormously in the 2010s thanks to its use in data
science and machine learning."

There are three topics here: an introduction to FastAPI, FastAPI's features, and Python in general. Recursive chunking with chunk_size=500 can split the document right between "FastAPI's features" and "Python as a language" mid-sentence — and you lose context in both chunks.

Semantic chunking sees the transition:

Sentence 1: "FastAPI is a web framework..."           embedding ──┐
Sentence 2: "It was created in 2018..."               embedding ──┼─ similarity 0.85 (same topic)
Sentence 3: "It is built on Pydantic..."              embedding ──┘

Sentence 4: "Its main features include..."            embedding ──┐
Sentence 5: "...performance comparable to Node.js..." embedding ──┼─ similarity 0.78 (still FastAPI)
Sentence 6: "...native async/await support..."        embedding ──┘

Sentence 7: "Python is a language..."                 embedding ←─ similarity with sentence 6: 0.32 ← TRANSITION
                                                                    (topic change)
Sentence 8: "It was created by Guido van Rossum..."   embedding ──┐
Sentence 9: "...readable syntax..."                   embedding ──┼─ similarity 0.81 (still Python)
                                                                  └

The algorithm detects the similarity drop from 0.78 to 0.32 between sentences 6 and 7, and inserts a breakpoint exactly there. The result: two coherent chunks, one entirely about FastAPI, the other entirely about Python.

The algorithm, step by step

def semantic_chunk_conceptual(document: str, threshold: float = 0.50):
    """
    Pseudo-code for the semantic chunking algorithm.
    """
    # 1. Split the doc into sentences
    sentences = split_into_sentences(document)

    # 2. Generate an embedding for each sentence
    embeddings = [embed(s) for s in sentences]

    # 3. Compute the cosine similarity between consecutive sentences
    similarities = []
    for i in range(len(sentences) - 1):
        sim = cosine_similarity(embeddings[i], embeddings[i + 1])
        similarities.append(sim)

    # 4. Find the breakpoints (similarity < threshold)
    breakpoints = [i for i, sim in enumerate(similarities) if sim < threshold]

    # 5. Build the chunks using the breakpoints
    chunks = []
    start = 0
    for bp in breakpoints:
        chunk = " ".join(sentences[start:bp + 1])
        chunks.append(chunk)
        start = bp + 1
    chunks.append(" ".join(sentences[start:]))

    return chunks

The algorithm's cost: generating an embedding for every sentence in the document. For 1M docs × an average of 30 sentences × $0.02 per 1M tokens, that's ~$1-3 of extra embeddings vs recursive (which is free). Plus the latency: ~50-100ms per document during indexing (vs ~5-10ms for recursive).

Implementing it with LangChain

# semantic_chunking.py
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings
import os


# Configure the embeddings
embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",
    api_key=os.getenv("OPENAI_API_KEY"),
)

# Configure the splitter
splitter = SemanticChunker(
    embeddings=embeddings,
    breakpoint_threshold_type="percentile",  # see the options below
    breakpoint_threshold_amount=75,  # split where similarity < P75
)

document = """
FastAPI is a modern web framework for Python. It was created in 2018 by Sebastián Ramírez.
It is built on Pydantic for validation and Starlette for the underlying ASGI.

FastAPI's main features include performance comparable to Node.js,
automatic type validation with type hints, interactive documentation with Swagger UI,
and native async/await support.

Python is a high-level programming language. It was created by Guido van Rossum
in 1991. It is known for its readable syntax and its 'batteries included' philosophy.

The popularity of Python grew enormously in the 2010s thanks to its use in data
science and machine learning.
"""

chunks = splitter.split_text(document)

print(f"Total chunks: {len(chunks)}")
for i, chunk in enumerate(chunks, 1):
    print(f"\n--- Chunk {i} ({len(chunk)} chars) ---")
    print(chunk)

Expected output:

Total chunks: 2

--- Chunk 1 (354 chars) ---
FastAPI is a modern web framework for Python. It was created in 2018 by Sebastián Ramírez.
It is built on Pydantic for validation and Starlette for the underlying ASGI.

FastAPI's main features include performance comparable to Node.js,
automatic type validation with type hints, interactive documentation with Swagger UI,
and native async/await support.

--- Chunk 2 (276 chars) ---
Python is a high-level programming language. It was created by Guido van Rossum
in 1991. It is known for its readable syntax and its 'batteries included' philosophy.

The popularity of Python grew enormously in the 2010s thanks to its use in data
science and machine learning.

Notice that the chunks have different sizes (354 and 276 chars). That's expected and desirable — each chunk fits the topic's "natural span", not some arbitrary fixed size.

Tuning the breakpoint threshold

SemanticChunker offers three threshold types:

Type 1: percentile (recommended)

splitter = SemanticChunker(
    embeddings=embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=75,
)

How it works: it computes every similarity in the document, then finds the 75th percentile. Transitions where the similarity falls below that percentile become breakpoints.

The advantage: it adapts to the document. Documents with many topic transitions get many breakpoints; coherent documents get few.

Tuning:

• amount=75 (default): medium-large chunks, ~20% of sentences flagged as transitions
• amount=85: larger chunks, fewer transitions detected (more conservative)
• amount=65: smaller chunks, more transitions (more aggressive)

Type 2: standard_deviation

splitter = SemanticChunker(
    embeddings=embeddings,
    breakpoint_threshold_type="standard_deviation",
    breakpoint_threshold_amount=2,  # split where similarity < mean - 2*std
)

How it works: it flags as breakpoints the sentences where similarity falls more than N standard deviations below the mean.

When to use it: when your dataset is heterogeneous (some docs with many transitions, others with few) and you want a threshold that's robust to the distribution.

Type 3: interquartile

splitter = SemanticChunker(
    embeddings=embeddings,
    breakpoint_threshold_type="interquartile",
)

How it works: it uses the interquartile range (IQR) of the similarities. Outlier transitions (lower than Q1 - 1.5×IQR) become breakpoints.

When to use it: datasets with outliers you want to catch — very strong but rare topic transitions.

Compared with recursive: when each one wins

# benchmark_semantic_vs_recursive.py
from langchain.text_splitter import RecursiveCharacterTextSplitter
import time

# Same document, two splitters
recursive_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
)
semantic_splitter = SemanticChunker(
    embeddings=embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=75,
)


def benchmark(splitter, name, document):
    start = time.perf_counter()
    chunks = splitter.split_text(document)
    elapsed = (time.perf_counter() - start) * 1000

    avg_size = sum(len(c) for c in chunks) / len(chunks)
    print(f"\n{name}:")
    print(f"  Chunks generated: {len(chunks)}")
    print(f"  Average size:     {avg_size:.0f} chars")
    print(f"  Latency:          {elapsed:.0f} ms")


# Try it on a 5000-char document
benchmark(recursive_splitter, "Recursive", document)
benchmark(semantic_splitter, "Semantic", document)

Typical output:

Recursive:
  Chunks generated: 11
  Average size:     500 chars
  Latency:          8 ms

Semantic:
  Chunks generated: 6
  Average size:     833 chars (range: 280-1240 chars)
  Latency:          420 ms (it generates 30 embeddings)

How to read it:

• Recursive: fast (~50x faster), free, uniformly sized chunks.
• Semantic: slow, costs money (~$0.0001 per 5K-char document), but the chunks are coherent.

When the extra cost is justified

The general rule: semantic chunking is for continuous narrative text where the topic transitions are subtle. For structured text, structural or recursive are better.

Traps and common mistakes

Trap 1: a threshold that's too low → chunks that are too small

The mistake: breakpoint_threshold_amount=50 (the 50th percentile, a very low threshold).

The symptom: the splitter detects a transition between almost every pair of sentences. Every chunk has 1-2 sentences. Recall collapses because the chunks don't carry enough context.

How to prevent it: start with amount=75 (the default) and only adjust if the result is clearly not working. If you need smaller chunks, you're better off using recursive with a small chunk_size.

Trap 2: a threshold that's too high → chunks that are too big

The mistake: breakpoint_threshold_amount=95.

The symptom: the splitter only detects the obvious transitions. Documents with several sub-topics end up as one giant chunk. You lose the whole advantage of semantic chunking.

How to prevent it: amount=70-80 is typically the optimal range. Validate it empirically on your own dataset.

Trap 3: using semantic chunking when you don't need to

The mistake: "semantic is the newest thing, I'll apply it to my whole corpus".

The symptom: indexing takes 10x longer, embedding costs shoot up, and the improvement over recursive is <3%.

How to prevent it: measure it against recursive on your eval set. If the improvement is <5% precision, recursive is enough.

Trap 4: ignoring overlap in semantic chunking

The mistake: you assume that because the chunks are "coherent", you don't need overlap.

The symptom: queries that need information mentioned in the transition between topics (e.g. a sentence that connects two sub-topics) get lost.

How to prevent it: SemanticChunker has no native overlap, but you can add it in post-processing:

def add_overlap_to_semantic_chunks(chunks: list[str], overlap_sentences: int = 1):
    """Adds a manual overlap of N sentences between consecutive chunks."""
    if not chunks:
        return chunks

    overlapped = [chunks[0]]
    for i in range(1, len(chunks)):
        prev_sentences = chunks[i-1].split('. ')[-overlap_sentences:]
        new_chunk = '. '.join(prev_sentences) + '. ' + chunks[i]
        overlapped.append(new_chunk)
    return overlapped

Trap 5: changing the embedding model invalidates your chunks

The mistake: you indexed with semantic chunking using text-embedding-3-small. You migrated to text-embedding-3-large. The old chunks keep the cuts that were based on the previous model.

The symptom: inconsistency — old docs chunked "according to the old model", new docs chunked "according to the new model". Retrieval quality becomes inconsistent.

How to prevent it: when you change the embedding model, re-process the whole corpus with the new model's semantic chunking. It isn't optional.

Trap 6: forgetting the cost of re-indexing

The mistake: you decide to change the threshold (from 75 to 80). You re-process the entire corpus.

The symptom: you generate every embedding all over again. For 100K docs, that's ~$30-100 depending on the average size.

How to prevent it: cache the sentence embeddings (not just the final chunks'). If you're only changing the threshold, the similarities don't change — only where you cut does.

Applied exercise

The scenario: you're an AI Engineer at a company that indexes technical podcast transcripts to run RAG over the content.

The data:

• 5K episodes, ~45 minutes each
• Average transcript: 8K words (~50K characters)
• The transcripts have NO structural separators (one giant paragraph)
• The queries are technical: "how to handle authentication according to [the podcast host]"

The metrics with the current recursive chunking (chunk_size=500, overlap=50):

• Precision@5: 71%
• Recall@5: 58%
• The complaints: "the bot mixes different topics into a single answer"

Your job:

1. Decide whether semantic chunking applies to this case.
2. Design the implementation, with threshold tuning.
3. Calculate the cost of re-indexing the whole corpus with semantic chunking.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings


embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Threshold 75 by default, adjust empirically
splitter = SemanticChunker(
    embeddings=embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=75,  # the starting point
)


def chunk_podcast_transcript(transcript: str) -> list[str]:
    """Semantic chunking with post-processing."""
    chunks = splitter.split_text(transcript)

    # Validation: discard chunks that are too short (<150 chars)
    # Those are usually noise (interruptions, "uh", "right")
    chunks = [c for c in chunks if len(c) > 150]

    # Add a manual 1-sentence overlap between chunks
    if len(chunks) > 1:
        for i in range(1, len(chunks)):
            last_sentence = chunks[i-1].rsplit('. ', 1)[-1]
            chunks[i] = last_sentence + '. ' + chunks[i]

    return chunks


# Validate on a few episodes and adjust the threshold
sample_episode = transcripts[0]
chunks = chunk_podcast_transcript(sample_episode)
avg_size = sum(len(c) for c in chunks) / len(chunks)
print(f"Chunks: {len(chunks)}, Avg size: {avg_size:.0f} chars")
# If avg_size > 1500: raise the threshold to 80 (chunks that big are a problem)
# If avg_size < 300: lower the threshold to 65 (chunks are too small)
# If it lands between 400-1200: the threshold is reasonable

TOTAL_EPISODES = 5000
AVG_WORDS_PER_EPISODE = 8000
AVG_TOKENS_PER_EPISODE = 8000 * 1.3  # ~10K tokens

# Semantic chunking has to generate an embedding for every sentence
# Assuming an average of 50 sentences per episode
SENTENCES_PER_EPISODE = 50
TOKENS_PER_SENTENCE = 200

total_tokens_for_chunking = TOTAL_EPISODES * SENTENCES_PER_EPISODE * TOKENS_PER_SENTENCE
# = 50M tokens

# Embedding cost for text-embedding-3-small: $0.02 / 1M tokens
chunking_embedding_cost = (total_tokens_for_chunking / 1_000_000) * 0.02
# = $1.00 — the cost of the SENTENCE embeddings (to detect the transitions)

# Then, the embeddings of the final chunks, to index them in ChromaDB
# Assuming ~10 final chunks per episode, ~1000 tokens each
total_tokens_for_indexing = TOTAL_EPISODES * 10 * 1000
indexing_cost = (total_tokens_for_indexing / 1_000_000) * 0.02
# = $1.00 — the cost of the actual indexing

total_cost = chunking_embedding_cost + indexing_cost
print(f"Total re-indexing cost: ${total_cost:.2f}")

Summary and next step

What you learned:

• Semantic chunking detects topic changes using cosine similarity between consecutive sentences.
• The typical gain over recursive: +3-7% precision on narrative text, up to +10% on unstructured transcripts.
• The cost: 10x more indexing latency + the cost of the extra embeddings (~$1-2 USD per 100K docs).
• Threshold tuning: percentile 75 by default. Lower it for smaller chunks, raise it for bigger ones.
• It wins on: continuous narrative text, transcripts, conversations, academic papers.
• It loses on: documentation with clear headers, source code, MVPs with zero budget.
• Changing the embedding model or the threshold means re-indexing the whole corpus.

Checkpoint: before moving on, you should be able to:

• Explain the semantic chunking algorithm with cosine similarity between sentences.
• Implement SemanticChunker with threshold tuning.
• Decide when semantic chunking beats recursive, with data.

Next capsule: 05 — Structural chunking.

For text with formal structure (code, HTML, markdown), neither recursive nor semantic is optimal. Structural chunking respects the natural syntactic units: functions in code, headings in markdown, sections in HTML. Capsule 05 teaches you to implement it with specialized parsers.

Resources

1. LangChain — SemanticChunker — The official documentation
2. Pinecone — Chunking Strategies — A comparison with benchmarks
3. Greg Kamradt — 5 Levels of Text Splitting — A visual tutorial
4. Anthropic — Contextual Retrieval — A technique that complements chunking
5. Lost in the Middle Paper — Why coherent chunks matter
6. LlamaIndex — Semantic Splitter — An alternative implementation

Estimated time: 25-30 minutes Next: 05-structural-chunking.md