Capsule 07: The decision framework — which chunking strategy to choose

Capsule overview

We covered four chunking strategies in this module: fixed-size, recursive, semantic, structural. Each with its trade-offs, its use cases, its failure modes. The operational question that closes the module: which one do you pick for a new project?

This capsule consolidates everything you learned into a reproducible decision framework. You'll learn to make the call in under 10 minutes based on five questions, and to justify it with quantitative data when a Tech Lead asks. It's the capsule you'll come back to every time you start a new RAG project.

By the end of this capsule you'll be able to:

• ✅ Compare the four strategies across six dimensions: precision, recall, latency, cost, coherence, complexity
• ✅ Apply a 5-question flowchart to pick a strategy for a new project
• ✅ Tell when a hybrid strategy is worth it (mixing two for different parts of the corpus)
• ✅ Calculate the TCO (total cost of ownership) of each strategy for a given volume
• ✅ Anticipate when to migrate from one strategy to another as the product evolves
• ✅ Spot the expensive mistake: choosing semantic chunking "because it's the newest thing" without justifying it

Estimated time: 25-30 minutes

The consolidated benchmark of all four strategies

Quality and performance

The key reading:

• Fixed-size is the baseline. Almost always suboptimal, only justifiable for prototypes.
• Recursive is the quality/cost sweet spot for narrative text. A reasonable default.
• Semantic wins on coherence but it costs money (extra LLM or embedding calls) and time.
• Structural wins when the document has clear structure (code, HTML, markdown).

Cost and operations

The decision framework: the 5 questions

                    ┌──────────────────────────────────┐
                    │ 1. Is this a quick prototype or  │
                    │    an MVP where "it works" is    │
                    │    good enough?                   │
                    └──────────────┬───────────────────┘
                                   │
                  ┌────────────────┴────────────────┐
                  │ YES                             │ NO
                  ▼                                 ▼
         ┌──────────────────────┐         ┌──────────────────────────┐
         │ Fixed-size.          │         │ 2. Does your corpus have │
         │ It's suboptimal, but │         │    formal structure      │
         │ you'll iterate soon. │         │    (code, HTML, MD)?     │
         └──────────────────────┘         └──────────┬───────────────┘
                                                     │
                                    ┌────────────────┴────────────┐
                                    │ YES                         │ NO
                                    ▼                             ▼
                        ┌──────────────────────┐      ┌────────────────────────┐
                        │ Structural chunking. │      │ 3. Is it long-form     │
                        │ It respects          │      │    narrative text      │
                        │ functions, headers,  │      │    where coherence     │
                        │ sections.            │      │    matters? (legal,    │
                        └──────────────────────┘      │    medical, papers)    │
                                                      └──────────┬─────────────┘
                                                                 │
                                            ┌────────────────────┴───────────┐
                                            │ YES                            │ NO
                                            ▼                                ▼
                                 ┌──────────────────────┐         ┌──────────────────────────┐
                                 │ 4. Do you have the   │         │ Recursive chunking.      │
                                 │    budget for it and │         │ The default for 80% of   │
                                 │    can you tolerate  │         │ cases. Start here.       │
                                 │    re-index latency? │         └──────────────────────────┘
                                 └──────────┬───────────┘
                                            │
                              ┌─────────────┴────────────┐
                              │ YES                      │ NO
                              ▼                          ▼
                   ┌──────────────────────┐    ┌──────────────────────┐
                   │ Semantic chunking.   │    │ Recursive chunking.  │
                   │ Better coherence,    │    │ Good coherence, no   │
                   │ worth the cost.      │    │ extra cost.          │
                   └──────────────────────┘    └──────────────────────┘

Applied to real cases

Hybrid strategies: when one chunker isn't enough

Sometimes your corpus contains wildly different document types. In that case, a single strategy isn't optimal — use differentiated chunking by document type.

# hybrid_chunking.py
from langchain_text_splitters import RecursiveCharacterTextSplitter, MarkdownHeaderTextSplitter
import re


def chunk_by_doc_type(document: str, doc_type: str) -> list[str]:
    """
    Applies a different strategy depending on the document type.
    """
    if doc_type == "code":
        return chunk_code_structurally(document)

    elif doc_type == "markdown":
        # Respect the markdown headers
        splitter = MarkdownHeaderTextSplitter(
            headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
        )
        return [section.page_content for section in splitter.split_text(document)]

    elif doc_type == "html":
        return chunk_html_structurally(document)

    elif doc_type == "legal":
        # Long-form narrative text where coherence is critical → semantic
        return semantic_chunk(document, threshold=0.7)

    else:  # general narrative
        # Recursive is the reasonable default
        splitter = RecursiveCharacterTextSplitter(
            chunk_size=500,
            chunk_overlap=50,
            separators=["\n\n", "\n", ". ", " ", ""]
        )
        return splitter.split_text(document)


def chunk_code_structurally(code: str) -> list[str]:
    """Splits code by function and class (parser-based)."""
    # A simplified implementation using AST
    import ast
    chunks = []
    try:
        tree = ast.parse(code)
        for node in ast.walk(tree):
            if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
                chunk = ast.get_source_segment(code, node)
                if chunk:
                    chunks.append(chunk)
    except SyntaxError:
        # Fall back to recursive if it doesn't parse
        return RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50).split_text(code)
    return chunks


def detect_doc_type(content: str, filename: str = "") -> str:
    """A simple heuristic for detecting the document type."""
    if filename.endswith(('.py', '.js', '.ts', '.java', '.go')):
        return "code"
    if filename.endswith(('.md', '.markdown')):
        return "markdown"
    if filename.endswith(('.html', '.htm')):
        return "html"
    if re.search(r'^(SECTION|ARTICLE|WHEREAS)', content, re.MULTILINE):
        return "legal"
    return "narrative"

When hybrid chunking is worth it

• A significantly mixed corpus: if >20% of your docs are a different type from the dominant one, it's worth the effort.
• Quality is critical for one type: if the docs of one type (e.g. code) are the most queried, optimizing them pays off.
• A team with bandwidth: maintaining multiple chunkers has an engineering cost.

When it isn't

• A corpus that's 90%+ one type (a single chunker is enough).
• An MVP where "it works" is good enough.
• A small team with no bandwidth to maintain multiple parsers.

Calculating the TCO (total cost of ownership)

Before you choose, run the concrete numbers. Assume 100K documents to index, with the dataset growing by 10K docs/month:

# tco_calculator.py
DOCS_INITIAL = 100_000
DOCS_PER_MONTH_NEW = 10_000
AVG_TOKENS_PER_DOC = 800

# The costs per strategy
def calculate_tco(strategy: str, months: int = 12) -> dict:
    """Calculates the 12-month TCO for a specific strategy."""

    # The initial indexing costs
    if strategy == "fixed":
        initial_cost = 0  # free
        recurring_monthly = 0
    elif strategy == "recursive":
        initial_cost = 0
        recurring_monthly = 0
    elif strategy == "semantic":
        # Semantic has to embed every sentence to detect the boundaries
        # ~3x the document's tokens
        sentences_factor = 3
        initial_cost = (DOCS_INITIAL * AVG_TOKENS_PER_DOC * sentences_factor / 1_000_000) * 0.02
        recurring_monthly = (DOCS_PER_MONTH_NEW * AVG_TOKENS_PER_DOC * sentences_factor / 1_000_000) * 0.02
    elif strategy == "structural":
        initial_cost = 0
        recurring_monthly = 0
    elif strategy == "hybrid_recursive_structural":
        initial_cost = 0
        recurring_monthly = 0  # the only cost is the engineering at setup

    # The engineering cost at setup (estimated)
    if strategy == "fixed":
        eng_setup_hours = 1
    elif strategy == "recursive":
        eng_setup_hours = 4
    elif strategy == "semantic":
        eng_setup_hours = 16
    elif strategy == "structural":
        eng_setup_hours = 24  # a parser per doc type
    elif strategy == "hybrid_recursive_structural":
        eng_setup_hours = 32

    eng_cost = eng_setup_hours * 80  # $80/hr

    total_recurring = recurring_monthly * months
    total_cost = initial_cost + total_recurring + eng_cost

    return {
        "strategy": strategy,
        "initial_cost_usd": initial_cost,
        "monthly_recurring_usd": recurring_monthly,
        "engineering_setup_usd": eng_cost,
        "total_12_months_usd": total_cost,
    }


# Compare them
strategies = ["fixed", "recursive", "semantic", "structural", "hybrid_recursive_structural"]
for s in strategies:
    tco = calculate_tco(s, months=12)
    print(f"\n{s.upper()}:")
    for k, v in tco.items():
        if isinstance(v, (int, float)):
            print(f"  {k}: ${v:.2f}")
        else:
            print(f"  {k}: {v}")

Typical output:

FIXED:
  initial_cost_usd: $0.00
  monthly_recurring_usd: $0.00
  engineering_setup_usd: $80.00
  total_12_months_usd: $80.00

RECURSIVE:
  initial_cost_usd: $0.00
  monthly_recurring_usd: $0.00
  engineering_setup_usd: $320.00
  total_12_months_usd: $320.00

SEMANTIC:
  initial_cost_usd: $4.80
  monthly_recurring_usd: $0.48
  engineering_setup_usd: $1280.00
  total_12_months_usd: $1290.56

STRUCTURAL:
  initial_cost_usd: $0.00
  monthly_recurring_usd: $0.00
  engineering_setup_usd: $1920.00
  total_12_months_usd: $1920.00

HYBRID_RECURSIVE_STRUCTURAL:
  initial_cost_usd: $0.00
  monthly_recurring_usd: $0.00
  engineering_setup_usd: $2560.00
  total_12_months_usd: $2560.00

How to read this:

• The dominant cost is NOT the embedding cost — it's the engineering at setup.
• Semantic, structural and hybrid are all noticeably more expensive to implement.
• If the quality gain over recursive is <5%, it probably doesn't justify the investment.
• For semantic chunking to be worth $1290 over recursive's $320, the improvement has to be substantial.

When to migrate between strategies

Your first chunking isn't necessarily your last. The signals to migrate:

From fixed-size to recursive:

• You started with fixed for the MVP. The system works, but the support tickets mention "answers cut in half".
• The investment: ~4 hours of engineering + re-indexing the whole corpus.
• The typical gain: +10 points of precision.

From recursive to structural:

• Your corpus is 30%+ code or structured HTML.
• Recursive is splitting functions in half or ignoring markdown headers.
• The investment: ~16-24 hours + a parser per type + re-indexing.
• The typical gain: +5-10 points on the queries that touch the structured types.

From recursive to semantic:

• Your corpus is long-form narrative text with extended arguments (legal, medical, papers).
• Recursive is splitting arguments right at their critical transitions.
• The investment: ~16 hours + the recurring cost of the extra embeddings.
• The typical gain: +5-8 points, but only if your domain genuinely needs narrative coherence.

From any of them to hybrid:

• You've identified that specific doc types underperform relative to the rest.
• Consider optimizing only those types instead of migrating everything.

The expensive mistake: choosing the newest instead of the appropriate

The antipattern: you read a blog post about "semantic chunking is the new state-of-the-art" and pick it without ever measuring it against recursive.

Why it's a trap:

1. Implementation cost: semantic takes 4x more effort than recursive.
2. Recurring cost: every re-index costs money (the extra embeddings).
3. A marginal gain in many cases: if your corpus has no long-form arguments, semantic improves ~2-3% over recursive — that doesn't justify the cost.

How to avoid it:

• Start with recursive. It works in 80% of cases.
• Build a solid eval set.
• If the system doesn't hit your quality target, diagnose the cause before you change the chunking.
• Sometimes the problem isn't chunking at all (it's the embeddings, the rerank, the queries).

# The diagnosis: is chunking really the problem?
def diagnose_chunking_impact(eval_set, current_strategy="recursive"):
    """
    Measures whether the quality problem comes from the chunking or from another component.
    """
    issues = []

    for item in eval_set:
        # Retrieve with the current strategy
        chunks_retrieved = retrieve(item["query"], strategy=current_strategy)

        # Does the correct info exist in any chunk?
        relevant_in_top_k = any(item["expected_text"] in c for c in chunks_retrieved)

        # Does the correct info exist in any chunk of the corpus, at any rank?
        relevant_in_corpus = check_full_corpus(item["expected_text"])

        if not relevant_in_corpus:
            issues.append(("missing_data", item["query"]))  # not a chunking problem
        elif not relevant_in_top_k:
            # The info is there but doesn't rank — could be chunking, but also retrieval
            issues.append(("retrieval_or_chunking", item["query"]))
        # If it's in the top-K, chunking isn't the problem

    return issues

Traps and common mistakes

Trap 1: changing the strategy without re-indexing

The mistake: you modify the chunker's code and deploy. The old chunks are still the old chunks.

The symptom: queries about old docs keep failing exactly as before; queries about new docs improve. Inconsistent results.

How to prevent it: when you change the chunking, re-process the whole corpus. It isn't optional.

Trap 2: migrating to semantic without measuring the ROI

The mistake: "the blog says semantic chunking improves things 15%". You migrate. Your real gain is 3%, but you spent $1500 in engineering.

How to prevent it: measure the expected gain on your eval set before you invest.

Trap 3: structural with no fallback

The mistake: the structural chunker fails when a document has broken formatting. There's no fallback. Those docs never get chunked — they stay out of the indexed corpus entirely.

The symptom: some docs never show up in any query.

How to prevent it: structural with a fallback to recursive whenever the parser fails.

Trap 4: a chunk_size copied without understanding it

The mistake: you copy chunk_size=500 from a tutorial. Your corpus is code (where 500 chars is rarely a complete function).

The symptom: functions split in half. Chunks that end in the middle of a loop.

How to prevent it: tune chunk_size to the content type. For code, chunk_size=800-1500. For conversations, chunk_size=200-400.

Trap 5: ignoring overlap when picking a strategy

The mistake: you choose structural chunking. The chunks are functions (self-contained). You assume you don't need overlap.

The symptom: references between functions (a class whose methods depend on each other) get lost.

How to prevent it: even in structural, add 1-2 sentences of overlap between consecutive chunks if there are cross-references.

Trap 6: not monitoring chunking quality in production

The mistake: you pick a strategy at setup. You never check it again.

The symptom: six months later, the corpus has changed (a new doc type got added), but the chunker is still the same. Quality degrades gradually.

How to prevent it: a chunking quality dashboard — average chars per chunk, size distribution, % of chunks that end mid-sentence. Alert when the metrics drift.

Applied exercise

The scenario: you're an AI Engineer at an online education startup. Your corpus is course material:

• 40% narrative text (explanatory lessons)
• 30% Python source code with inline explanations
• 20% video lecture transcripts
• 10% diagrams (OCR text from images)

The volume: 50K documents. Growing 5K/month. The team: 2 engineers, no dedicated MLE.

Your current pipeline uses fixed-size with chunk_size=500. The metrics:

• Precision@5: 65%
• Recall@5: 58%
• Frequent complaints: "the bot cuts the code in half", "the explanations have no context"

Your job:

1. Apply the 5-question framework. Which strategy (or strategies) do you choose?
2. Justify it, taking the mixed corpus into account.
3. Estimate the migration cost and the expected impact.

# The migration plan
phases = {
    "phase_1_recursive_for_text": {
        "effort": "8 hours (1 day)",
        "covers": "70% of the corpus (narrative + transcripts + OCR)",
        "expected_improvement": "+10-12 points of precision on narrative queries",
        "eng_cost": "$640",
    },
    "phase_2_structural_for_code": {
        "effort": "24 hours (3 days — Python parser + tests)",
        "covers": "30% of the corpus (code)",
        "expected_improvement": "+15-20 points of precision on code queries",
        "eng_cost": "$1920",
    },
    "phase_3_validation": {
        "effort": "8 hours",
        "covers": "building an eval set of 100 queries (mixed code/narrative) + the A/B measurement",
        "eng_cost": "$640",
    },
}

total_cost = sum(p["eng_cost"].lstrip("$").replace("$", "") for p in phases.values())
print(f"Total engineering: $3,200")
print(f"Recurring costs: $0 (there's no extra LLM)")
print(f"Total time: 5 working days, spread out")

Summary and next step

What you learned:

• Fixed-size is a baseline for prototypes only. Recursive is the default for 80% of cases.
• Structural is optimal when the corpus has formal structure (code, HTML, markdown).
• Semantic wins on coherence but costs money and time. Only justifiable in critical narrative domains.
• Hybrid strategies are valid for significantly mixed corpora. Extra engineering cost, but better quality per document type.
• Chunking TCO includes more than the embedding cost — the dominant cost is usually the engineering at setup.
• Migrating between strategies means re-indexing the whole corpus. It isn't just a code change.
• The key antipattern: choosing semantic "because it's the newest thing" without measuring the ROI on your eval set.

Checkpoint: before moving on, you should be able to:

• Apply the 5-question framework to a new project and pick a strategy in <10 min.
• Calculate an approximate TCO for the different strategies at your volume.
• Design a hybrid chunking setup when the corpus is mixed.

Next capsule: 08 — The Chunking Optimizer project.

The module's closer: you'll build a system that takes a corpus, tries multiple strategies against the same eval set, and produces a comparison report that lets you make the decision on data. It's the tool you'd use on day 1 of any new RAG project.

Resources

1. LangChain — Text Splitters Comparison — The splitters compared
2. LlamaIndex — Node Parsers — Alternative implementations
3. Greg Kamradt — 5 Levels of Chunking — A visual tutorial of the strategies
4. Pinecone — Chunking Strategies — A comparison with benchmarks
5. Anthropic — Contextual Retrieval — A technique that complements chunking
6. LangChain — Markdown Header Splitter — For structural chunking in MD

Estimated time: 25-30 minutes Next: 08-project-chunking-optimizer.md