A/B Testing Chunking Strategies

Descripción breve

A/B testing sistemático te permite comparar strategies (fixed vs semantic, chunk size 400 vs 800) con datos. Sin A/B testing, eliges arbitrariamente. Con él, optimizas con confianza estadística.

Aprenderás setup de experimentos, statistical significance, y optimization loops.

---

Setup de experimento A/B

# Estrategias a comparar:
strategies = {
    'fixed_400': TokenChunker(chunk_size=400, overlap=40),
    'fixed_800': TokenChunker(chunk_size=800, overlap=80),
    'semantic': SemanticChunker(max_tokens=600)
}

# Evaluar cada estrategia:
results = {}
for name, chunker in strategies.items():
    chunks = chunker.chunk(documents)
    # Embed + evaluate
    metrics = evaluate_retrieval(chunks, eval_dataset)
    results[name] = metrics

# Comparar:
print("Strategy    | Recall@5 | nDCG@5")
print("------------|----------|--------")
for name, metrics in results.items():
    print(f"{name:12}| {metrics['recall@5']:.3f}   | {metrics['ndcg@5']:.3f}")

Output ejemplo:

Strategy    | Recall@5 | nDCG@5
------------|----------|--------
fixed_400   | 0.820    | 0.780
fixed_800   | 0.880    | 0.850
semantic    | 0.900    | 0.870  ← Best

---

Statistical Significance

T-test para comparar 2 strategies:

from scipy import stats

# Métricas de ambas strategies (100 queries cada una)
recall_strategy_a = [0.8, 0.9, 0.7, ...]  # 100 valores
recall_strategy_b = [0.85, 0.92, 0.75, ...]

# T-test
t_stat, p_value = stats.ttest_ind(recall_strategy_a, recall_strategy_b)

print(f"P-value: {p_value:.4f}")

if p_value < 0.05:
    print("✅ Difference is statistically significant")
else:
    print("❌ Difference is NOT significant (puede ser ruido)")

---

Resumen

Qué aprendiste:

• ✅ Setup de A/B testing
• ✅ Comparar múltiples strategies
• ✅ Statistical significance (p < 0.05)

Key insight: Semantic chunking típicamente mejor, pero más complejo.

---

En la siguiente cápsula

Cápsula 07: Advanced Chunking Patterns

Aprenderás recursive chunking, metadata enrichment, hierarchical chunks.

---

Módulo 4 - Embeddings Deep Dive Guide A/B testing: optimización data-driven