A/B Testing Chunking Strategies

Quick overview

Systematic A/B testing lets you compare strategies (fixed vs semantic, chunk size 400 vs 800) with data. Without A/B testing, you choose arbitrarily. With it, you optimize with statistical confidence.

You'll learn experiment setup, statistical significance, and optimization loops.

---

A/B experiment setup

# Strategies to compare:
strategies = {
    'fixed_400': TokenChunker(chunk_size=400, overlap=40),
    'fixed_800': TokenChunker(chunk_size=800, overlap=80),
    'semantic': SemanticChunker(max_tokens=600)
}

# Evaluate each strategy:
results = {}
for name, chunker in strategies.items():
    chunks = chunker.chunk(documents)
    # Embed + evaluate
    metrics = evaluate_retrieval(chunks, eval_dataset)
    results[name] = metrics

# Compare:
print("Strategy    | Recall@5 | nDCG@5")
print("------------|----------|--------")
for name, metrics in results.items():
    print(f"{name:12}| {metrics['recall@5']:.3f}   | {metrics['ndcg@5']:.3f}")

Example output:

Strategy    | Recall@5 | nDCG@5
------------|----------|--------
fixed_400   | 0.820    | 0.780
fixed_800   | 0.880    | 0.850
semantic    | 0.900    | 0.870  ← Best

---

Statistical Significance

T-test to compare 2 strategies:

from scipy import stats

# Metrics from both strategies (100 queries each)
recall_strategy_a = [0.8, 0.9, 0.7, ...]  # 100 values
recall_strategy_b = [0.85, 0.92, 0.75, ...]

# T-test
t_stat, p_value = stats.ttest_ind(recall_strategy_a, recall_strategy_b)

print(f"P-value: {p_value:.4f}")

if p_value < 0.05:
    print("✅ Difference is statistically significant")
else:
    print("❌ Difference is NOT significant (could be noise)")

---

Summary

What you learned:

• ✅ A/B testing setup
• ✅ Comparing multiple strategies
• ✅ Statistical significance (p < 0.05)

Key insight: Semantic chunking is typically better, but more complex.

---

In the next capsule

Capsule 07: Advanced Chunking Patterns

You'll learn recursive chunking, metadata enrichment, hierarchical chunks.

---

Module 4 - Embeddings Deep Dive Guide A/B testing: data-driven optimization