4. Stage 3: Detailed Technical Decisions

Overview

In this stage you justify the specific technical decisions: chunking parameters, top-K, RRF k, MMR λ, the ANN index, etc. Every decision must be grounded in a requirement or a trade-off.

---

Decision 1: Chunk size and overlap

The decision:

• Chunk size: 600 tokens
• Overlap: 100 tokens (16.7%)
• Strategy: Hierarchical (by paper section)

Justification:

Why 600 tokens?

Academic papers have typical sections:
- Abstract: ~200 tokens
- Introduction: ~800 tokens (split into 2 chunks)
- Methods: ~1000 tokens (split into 2 chunks)
- Results: ~1200 tokens (split into 2 chunks)
- Conclusion: ~300 tokens

600 tokens = A balance between:
- Enough context (a complete section or subsection)
- Precision (not mixing different topics)

Why a 100-token overlap?

The transitions between sections:
"...the conclusion of the previous section. The next section..."
→ The overlap captures the transitional context
→ 100 tokens = ~2-3 sentences

The alternatives considered:

• 400 tokens (less context, more precision) → Rejected: the sections would be fragmented
• 800 tokens (more context) → Rejected: it would mix topics, lowering precision

---

Decision 2: The embedding model

The decision:

• Model: Sentence-BERT all-mpnet-base-v2
• Dimensions: 768D
• Inference: A local GPU (NVIDIA T4)

Justification:

Why Sentence-BERT?

Benchmark (the MS MARCO dataset, recall@10):
- all-mpnet-base-v2: 0.69
- OpenAI ada-002: 0.72 (+3%)
- OpenAI text-embedding-3-small: 0.75 (+6%)

The trade-off:
- 6% lower recall vs OpenAI
- But: $0 cost + private data ✅

Why all-mpnet-base-v2 over the other variants?

all-mpnet-base-v2: 768D, 420M params
all-MiniLM-L6-v2: 384D, 22M params (faster but -10% quality)

→ all-mpnet-base-v2 has the better quality/speed balance

Expected throughput:

GPU: NVIDIA T4 (16GB)
Batch size: 32
Latency: 50ms/batch
→ ~640 chunks/second (enough for 50K queries/month)

---

Decision 3: The vector database and the ANN index

The decision:

• Vector DB: Weaviate
• Index: HNSW
• Parameters:
  • efConstruction: 256 (indexing)
  • maxConnections: 64
  • ef: 128 (query time)

Justification:

Why HNSW over IVF?

Dataset: 500K chunks

HNSW:
- Recall@10: 0.95
- Latency: 50-100ms
- Memory: ~4GB (768D × 500K × 8 bytes + the graph)

IVF (nlist=1000):
- Recall@10: 0.90 (lower)
- Latency: 30-60ms (faster)
- Memory: ~2GB

→ HNSW: Better recall (+5%), with acceptable latency

Why ef=128?

ef = the number of candidates evaluated during the search

Internal benchmark (expected):
- ef=64: recall@10 = 0.92, latency = 40ms
- ef=128: recall@10 = 0.95, latency = 80ms
- ef=256: recall@10 = 0.96, latency = 150ms

→ ef=128: The optimal balance (high recall, latency < 100ms)

---

Decision 4: Top-K and reranking

The decision:

• Initial top-K (kNN): 100
• Reranking: NO (to keep the latency low)
• Final top-K (after MMR): 20

Justification:

Why top-K=100?

MMR needs enough candidates to diversify:
- top-K=50 → MMR has little room for diversity
- top-K=100 → MMR can pick 20 diverse ones out of 100
- top-K=200 → Latency increases with no meaningful gain

→ 100 is the sweet spot

Why NO reranking?

Reranking (a cross-encoder) adds 200-300ms
→ Total latency: 230ms + 300ms = 530ms (search)
→ Total RAG: 1440ms + 300ms = 1740ms ✅ (still < 2s)

But: Reranking isn't critical if HNSW already has recall@10 = 0.95

The decision: Skip reranking initially
→ If precision@10 < 0.75 in production → add reranking

---

Decision 5: Hybrid search (the RRF k)

The decision:

• Method: RRF (Reciprocal Rank Fusion)
• Parameter k: 60

Justification:

Why RRF over a weighted sum?

A weighted sum requires tuning α:
score = α × semantic + (1-α) × keyword

→ α is hard to tune (it depends on the query)

RRF:
RRF(doc) = 1/(k + rank_semantic) + 1/(k + rank_keyword)
→ No α to tune ✅
→ Documents in both rankings are boosted automatically

Why k=60?

The typical benchmark (from the literature):
- k=10: It favors the high ranks (top-5) too heavily
- k=60: The standard balance (used in Weaviate, Elasticsearch)
- k=100: It flattens the scores (less differentiation)

→ k=60 is the proven standard value

---

Decision 6: MMR (diversity)

The decision:

• λ (lambda): 0.6
• Diversify by: Department (CS, Physics, Biology, etc.)

Justification:

Why λ=0.6?

λ = the balance between relevance and diversity

λ=0.0: Diversity only (not recommended)
λ=0.5: A 50/50 balance
λ=0.6: 60% relevance, 40% diversity ← Recommended
λ=1.0: Relevance only (no diversity)

→ λ=0.6: It prioritizes relevance while still guaranteeing diversity

Why diversify by department?

FR4: Suggest related papers from multiple areas

The user searches for: "optimization algorithms"
Without MMR:
- 20 papers all from CS (the same department)

With MMR (diversity by department):
- 12 papers from CS (optimization)
- 5 papers from Math (theory)
- 3 papers from Physics (applications)
→ Broader coverage ✅

---

Decision 7: The metadata boost

The decision:

• Papers < 2 years old: a 1.2x boost
• Papers 2-5 years old: 1.0x (no boost)
• Papers > 5 years old: 0.9x (a slight penalty)

Justification:

Why favor recent papers?

Users (especially students) prefer recent papers:
- More current methods
- Larger datasets
- The current state of the art

But: Don't eliminate old papers (they can be foundational)
→ A moderate boost (1.2x, not 2x)

The formula:

if year >= 2022:
    boost = 1.2
elif year >= 2019:
    boost = 1.0
else:
    boost = 0.9

final_score = base_score × boost

---

Decision 8: The LLM for Q&A

The decision:

• Model: GPT-3.5 Turbo
• Temperature: 0.1 (low)
• Max output tokens: 500
• Streaming: Enabled

Justification:

Why GPT-3.5 over GPT-4?

Cost/query:
- GPT-3.5: $0.0011 (2500 input + 300 output)
- GPT-4: $0.022 (20x more expensive)

Latency:
- GPT-3.5: 1-2s
- GPT-4: 2-5s

The trade-off:
- GPT-4 is better, but GPT-3.5 is enough for academic Q&A
- A limited budget → GPT-3.5

Why temperature=0.1?

Temperature controls "creativity":
- 0.0: Deterministic (always the same answer)
- 0.1: Nearly deterministic, minimal variation
- 0.7: The default (more creative)

For academic Q&A:
→ We want consistent, precise answers
→ temperature=0.1 ✅

Why streaming?

Without streaming:
- The user stares at a blank screen for 1.5s → then the full answer appears

With streaming:
- The user sees the answer as it's generated (word by word)
- The perceived latency is lower
- A better UX ✅

---

Decision 9: The evaluation metrics

The decision:

• Precision@10: Target > 0.75
• NDCG@20: Target > 0.70
• Diversity (departments in the top-20): Target ≥ 3
• Latency (p95): Target < 2000ms

Justification:

Why Precision@10 (not @5)?

Users typically review the top-10 results (the first "page")
→ Precision@10 measures what % of those 10 are relevant

Why NDCG@20?

NDCG penalizes relevant results in low positions
→ NDCG@20 measures the quality of the full ranking (the first page)

Why diversity ≥ 3 departments?

FR4: Suggest papers from multiple areas
→ At least 3 different departments in the top-20

---

A summary of the technical decisions

Component	Decision	Justification
Chunk size	600 tokens	A balance of context/precision
Overlap	100 tokens	It captures the transitions
Embedding	Sentence-BERT (768D)	Private data, $0 cost
Vector DB	Weaviate HNSW	High recall, low latency
Top-K	100 → MMR → 20	Enough room for diversity
Hybrid	RRF (k=60)	A balance of keyword+semantic
MMR	λ=0.6	60% relevance, 40% diversity
Boost	< 2 years: 1.2x	Favoring recent papers
LLM	GPT-3.5 (temp=0.1)	Acceptable cost/latency
Metrics	P@10, NDCG@20, diversity	Measuring quality and diversity

---

Next stage: 05-cost-estimation.md — Calculating the complete monthly cost.