Module 8: The Capstone Final Project

6. The Final Stage: The Technical Design Document

Overview

This is the project's final stage. Here you consolidate everything (architecture, decisions, costs) into a professional technical design document that you can present to stakeholders or use as an implementation blueprint.

Technical Design Document: An Academic RAG System

1. Executive Summary

Project: A RAG system for searching and Q&A over academic papers
Client: A university (20K students, 1K professors)
Dataset: 100K documents (papers, theses, guides)

Objective:
An intelligent chatbot that answers both conceptual and exact questions about papers, with latency < 2s and a cost < $2000/month.

The solution:
A hybrid RAG system (semantic + keyword search) with local embeddings (Sentence-BERT), a vector DB (Weaviate), and an LLM (GPT-3.5 Turbo).

Expected results:

  • Precision@10: > 0.75
  • Latency (p95): < 2000ms
  • Monthly cost: $570/month (71% under budget)
  • Diversity: ≥ 3 departments in the top-20

2. Requirements

Functional:

  1. Conceptual semantic search ("papers about deep learning")
  2. Exact search by metadata (author, year, department)
  3. Q&A with an LLM ("What methodology does Smith's paper use?")
  4. Suggesting related papers with diversity (MMR)
  5. Advanced filters (year, department, type)

Non-functional:

  1. Latency < 2s (p95)
  2. Cost < $2000/month
  3. Scalable to 200K documents
  4. 99.5% uptime (SLA)

Constraints:

  1. Private data (the embeddings must be local)
  2. A limited budget ($2000/month)
  3. A small team (2 engineers)

3. Architecture

The main components:

The Ingestion Pipeline (offline):

PDFs/LaTeX/DOCX → Text Extraction → Chunking (600 tokens, overlap 100)
→ Embeddings (local Sentence-BERT, 768D) → Weaviate (HNSW)

Query Processing (online):

User Query → Query Analysis → Hybrid Search (Keyword BM25 + Semantic cosine)
→ RRF Fusion (k=60) → MMR (λ=0.6) → Metadata Boost → Top-20 Results

The RAG Pipeline (Q&A):

User Question → Retrieval (top-5 chunks) → Augmentation (build the prompt)
→ LLM Generation (GPT-3.5, temp=0.1) → Response + Sources

The technology stack:

ComponentTechnology
Text extractionPyPDF2, pandoc, python-docx
ChunkingLangChain RecursiveCharacterTextSplitter
EmbeddingsSentence-BERT (all-mpnet-base-v2, 768D)
Vector DBWeaviate (self-hosted, HNSW)
LLMOpenAI GPT-3.5 Turbo
APIFastAPI (AWS Lambda)
OrchestrationAirflow (periodic re-indexing)
InfrastructureAWS (a GPU server + a Weaviate server)

4. Key Technical Decisions

Embeddings: Local Sentence-BERT

Justification:

  • ✅ The privacy constraint (the data can't leave)
  • ✅ $0 cost after setup
  • ✅ Acceptable quality (recall@10: 0.69 vs OpenAI's 0.75 = -6%)

Trade-off: 6% lower quality than OpenAI, but it satisfies the constraint and saves cost.

Vector DB: Weaviate HNSW

Justification:

  • ✅ Open-source (no license)
  • ✅ HNSW (recall@10: 0.95, latency: 50-100ms)
  • ✅ Native hybrid search
  • ✅ Metadata filtering

Parameters:

  • efConstruction: 256, maxConnections: 64, ef: 128 (query time)

LLM: GPT-3.5 Turbo

Justification:

  • ✅ Cost: 13x cheaper than GPT-4 per query ($0.0017 vs $0.022/query)
  • ✅ Latency: 2x faster (1-2s vs 2-5s)
  • ⚠️ Trade-off: Lower quality, but enough for academic Q&A

The alternative: GPT-4 is available for +$609/month (still within budget).

Hybrid search: RRF (k=60) + MMR (λ=0.6)

Justification:

  • ✅ RRF: It fuses keyword + semantic without tuning α by hand
  • ✅ MMR: It diversifies the top-20 by department (FR4)
  • ✅ k=60: The proven standard value
  • ✅ λ=0.6: 60% relevance, 40% diversity

Chunking: Hierarchical (600 tokens, overlap 100)

Justification:

  • ✅ 600 tokens: Complete sections of papers (Abstract, Methods, etc.)
  • ✅ 100-token overlap: It captures the transitions between sections
  • ✅ Hierarchical: It respects the semantic structure (it doesn't cut sentences)

5. Data Flows and Latencies

The initial indexing (once):

100K docs → Text extraction (24h) → Chunking (2h) → Embeddings (8h)
→ Weaviate upsert (2h)
Total: ~36 hours (1.5 days)
Cost: $3.48 (negligible)

A search query (typical):

1. Embed the query (Sentence-BERT): 50ms
2. Weaviate hybrid search (keyword + semantic): 80ms
3. RRF + MMR (Python): 100ms
Total: 230ms ✅ (< the 2s target)

A Q&A query (RAG):

1. Retrieval (same as the search): 230ms
2. Build the prompt (Python): 10ms
3. LLM generation (GPT-3.5 streaming): 1200ms
Total: 1440ms ✅ (< the 2s target)

6. Costs

Recurring monthly:

CategoryCost/month
GPU server (embeddings)$255
Weaviate server$248
API server (Lambda)$5
LLM (GPT-3.5)$51
Storage + Monitoring$11
Total$570/month

Budget: $2000/month
Headroom: $1430/month (71% under budget) ✅

Year 5 projection:

Docs: 125K (+25%)
Queries: 100K/month (2x the initial)
Cost: $869/month (still < the budget) ✅

7. Evaluation

Metrics:

  1. Precision@10: > 0.75 (75% of the top-10 are relevant)
  2. NDCG@20: > 0.70 (the quality of the ranking)
  3. Diversity: ≥ 3 departments in the top-20
  4. Latency (p95): < 2000ms

The evaluation set:

100 queries labeled by hand by professors:
- 40 conceptual queries ("papers about X")
- 30 exact queries ("Y's thesis")
- 30 Q&A queries ("What does paper Z say?")

Department coverage: CS, Physics, Biology, Math

8. Limitations and Mitigations

Limitation 1: Embedding quality (vs OpenAI)

Impact: Sentence-BERT has -6% recall vs OpenAI

Mitigation:

  • Hybrid search (keyword compensates)
  • Reranking (if precision < 0.75 in production)
  • Fine-tune Sentence-BERT on an academic corpus (future)

Limitation 2: Latency at peak

Impact: The GPU can saturate with 100+ concurrent queries

Mitigation:

  • Horizontal scaling: Add a second GPU server (+$255/month)
  • Cache frequent queries (30% of traffic)
  • Rate limiting (100 req/min per user)

Limitation 3: Residual hallucinations

Impact: The LLM can "make things up" if the context is ambiguous

Mitigation:

  • Strict prompt engineering ("Answer ONLY if the context contains the answer")
  • Citation forcing (require source citations)
  • Temperature=0.1 (more deterministic answers)

9. Implementation Roadmap

Phase 1: MVP (4 weeks)

Week 1: Set up the infrastructure (the GPU + Weaviate servers)
Week 2: Index 10K docs (a sample)
Week 3: A basic API (semantic search only)
Week 4: Testing with 20 alpha users

Phase 2: Beta (4 weeks)

Weeks 5-6: Full indexing (100K docs)
Week 7: Hybrid search + MMR
Week 8: The RAG pipeline (Q&A with GPT-3.5)
Beta launch: 200 users

Phase 3: Production (4 weeks)

Week 9: The evaluation set (100 labeled queries)
Week 10: Parameter tuning (ef, k, λ)
Week 11: Monitoring + alerts (CloudWatch)
Week 12: Public launch (20K students)

10. Conclusion

Technical viability: ✅ High

  • A proven architecture (RAG + hybrid search)
  • Mature technologies (Weaviate, Sentence-BERT, GPT-3.5)
  • It meets every functional and non-functional requirement

Economic viability: ✅ High

  • Cost: $570/month (71% under budget)
  • $1430/month of headroom for growth and upgrades
  • Scalable to 10x the queries without exceeding the budget

The main risks:

  1. The quality of the local embeddings (-6% vs OpenAI) → Mitigated with hybrid search + possible reranking
  2. Latency at peak (GPU saturation) → Mitigated with horizontal scaling + caching

Recommendation: Proceed with implementation in 3 phases (MVP → Beta → Production).

Module 8 Summary

Congratulations on completing Module 8: The Final Project. 🎉

What you achieved:

  1. ✅ You analyzed the requirements and constraints (functional, non-functional)
  2. ✅ You designed a complete architecture (components, flows, stack)
  3. ✅ You made justified technical decisions (chunking, embeddings, ranking)
  4. ✅ You estimated the costs ($570/month, 71% under budget)
  5. ✅ You identified the trade-offs (quality vs latency vs cost)
  6. ✅ You created a professional technical design document

The skill, consolidated:

"Designing a complete RAG system requires integrating EVERYTHING you've learned: vectors, embeddings, similarity, ANN, ranking, and trade-offs. There's no single solution; every decision depends on specific requirements (latency, budget, privacy) and must be justified with data."

Conclusion of the Complete Guide

Congratulations on completing AI Semantics: Vectors, Embeddings and Semantic Search. 🎉🎉🎉

Your journey:

  1. Module 1: Vectors (concepts, visualization, operations)
  2. Module 2: Vector spaces (bases, dimensions, normalization)
  3. Module 3: Similarity and distance (Euclidean, cosine)
  4. Module 4: Proximity search (kNN, ANN, HNSW, IVF)
  5. Module 5: Keyword vs semantic search (comparison, hybrid)
  6. Module 6: Designing search systems (architecture, ranking)
  7. Module 7: RAG and semantic search (the problem, the architecture, the limitations)
  8. Module 8: The final project (a complete end-to-end design)

The master intuition:

"Semantic search is the practical application of linear algebra (vectors, cosine) + data structures (HNSW, IVF) + ML (embeddings) to search by meaning instead of by words. RAG connects semantic search with LLMs to eliminate hallucinations. Every production RAG system is a balance of decisions: precision vs latency vs cost, guided by specific requirements."

Next steps:

  1. Implement a small RAG system (10 documents) to validate the concepts
  2. Take the Vector Databases guide (technical implementation with code)
  3. Take the Embeddings Deep Dive guide (fine-tuning, evaluation)

Thank you for your time and dedication! 🚀

← Previous5. Stage 4: Cost Estimation