Module 8: The Capstone Final Project
3. Stage 2: Architecture Design
Overview
In this stage you design the RAG system's complete architecture: components, data flows, technologies, and diagrams.
The main components
Based on the requirements and constraints, your architecture should include:
1. The Ingestion Pipeline (offline)
PDFs/LaTeX/DOCX → Text extraction → Chunking → Embeddings → Vector DB
2. Query Processing (online)
User Query → Type detection → Search (semantic/keyword/hybrid) → Results
3. The RAG Pipeline (online, for Q&A)
Query → Retrieval → Augmentation → LLM → Response
Architecture diagram (the suggested solution)
┌─────────────────────────────────────────────────────────┐
│ INGESTION (Offline) │
├─────────────────────────────────────────────────────────┤
│ │
│ [100K Docs] (PDF, LaTeX, DOCX) │
│ ↓ │
│ [Text Extraction] │
│ - PyPDF2 (PDFs) │
│ - pandoc (LaTeX → text) │
│ - python-docx (DOCX) │
│ ↓ │
│ [Chunking] │
│ Strategy: Hierarchical (by section) │
│ Size: 600 tokens, overlap 100 │
│ ↓ │
│ [Embeddings] │
│ Model: Local Sentence-BERT (all-mpnet-base-v2) │
│ Dimensions: 768D │
│ ↓ │
│ [Metadata Extraction] │
│ - Author, year, department, type │
│ ↓ │
│ [Vector Database] │
│ - Weaviate (self-hosted) │
│ - HNSW index │
│ - ~500K chunks (100K docs × 5 chunks avg) │
│ │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ QUERY PROCESSING (Online) │
├─────────────────────────────────────────────────────────┤
│ │
│ [User Query] │
│ ↓ │
│ [Query Analysis] │
│ Does it contain exact metadata? (author, year, ID) │
│ └─ YES: Exact match (keyword + filter) │
│ └─ NO: Semantic search │
│ ↓ │
│ ┌──────────────┬──────────────┐ │
│ │ Keyword │ Semantic │ │
│ │ (BM25) │ (cosine) │ │
│ └──────┬───────┴──────┬───────┘ │
│ │ │ │
│ └──────┬───────┘ │
│ │ │
│ v │
│ [RRF Fusion] (k=60) │
│ ↓ │
│ [MMR] (λ=0.6, diversity) │
│ ↓ │
│ [Metadata Boost] │
│ - Papers < 2 years old: 1.2x │
│ ↓ │
│ [Top-20 Results] │
│ │
└─────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────┐
│ RAG PIPELINE (Q&A) │
├─────────────────────────────────────────────────────────┤
│ │
│ [User Question] │
│ "What methodology does Smith's paper use?" │
│ ↓ │
│ [Retrieval] (same as Query Processing) │
│ Top-5 chunks │
│ ↓ │
│ [Augmentation] │
│ Build the prompt: │
│ "Context: [5 chunks] │
│ Question: [user question]" │
│ ↓ │
│ [LLM Generation] │
│ Model: GPT-3.5 Turbo │
│ Temperature: 0.1 │
│ Max tokens: 500 │
│ ↓ │
│ [Response + Sources] │
│ "According to Smith's paper (2023): │
│ The methodology consists of..." │
│ │
└─────────────────────────────────────────────────────────┘
Architecture decisions
Decision 1: Local embeddings (Sentence-BERT)
Justification:
- ✅ The constraint: private data (it can't leave)
- ✅ Cost: $0 after setup (a local GPU)
- ✅ Quality: Sentence-BERT (all-mpnet-base-v2) is competitive for academic texts
- ❌ Trade-off: 768D (vs OpenAI's 1536D), lower quality but acceptable
The rejected alternative: OpenAI embeddings (they violate the privacy constraint)
Decision 2: Self-hosted Weaviate (the vector DB)
Justification:
- ✅ Open-source (no license cost)
- ✅ Native HNSW (fast for 500K chunks)
- ✅ Native hybrid search (keyword + semantic)
- ✅ Metadata filtering (author, year, department)
- ✅ Self-hosted (private data)
The rejected alternative: Pinecone (managed, but it costs $70-200/month, which eats into the budget)
Decision 3: GPT-3.5 Turbo as the LLM
Justification:
- ✅ Cost: 20x cheaper than GPT-4 ($0.0005 vs $0.01/1K tokens)
- ✅ Latency: 2x faster (1-2s vs 2-5s)
- ⚠️ Trade-off: Lower quality than GPT-4, but enough for academic Q&A
The alternative considered: Llama 3 (self-hosted) → it saves cost but requires an additional dedicated GPU
Decision 4: Hybrid search (RRF) + MMR
Justification:
- ✅ FR2 (exact search): Keyword guarantees exact matches (author, year)
- ✅ FR1 (conceptual search): Semantic understands synonyms
- ✅ FR4 (diversity): MMR avoids 20 similar papers
Implementation:
1. Run keyword (BM25) and semantic (cosine) in parallel
2. Fuse them with RRF (k=60)
3. Apply MMR (λ=0.6) over the top-100 → diversify down to the top-20
4. Time boost (recent papers)
Decision 5: Hierarchical chunking (by section)
Justification:
- ✅ Papers have structure (Abstract, Methods, Results, Conclusion)
- ✅ Semantically coherent chunks (a complete section)
- ✅ Metadata per chunk (e.g. section="Methods")
Parameters:
- Size: 600 tokens (a balance of context/precision)
- Overlap: 100 tokens (it captures the transitions)
- Strategy: Split on Markdown headers (##, ###) or detect the sections in the PDFs
The complete technology stack
Ingestion:
- Text extraction: PyPDF2, pandoc, python-docx
- Chunking: LangChain's RecursiveCharacterTextSplitter
- Embeddings: Sentence-BERT (all-mpnet-base-v2)
- Orchestration: Python scripts + Airflow (for periodic re-indexing)
Storage:
- Vector DB: Weaviate (self-hosted on the GPU server)
- Metadata DB (optional): PostgreSQL (for complex queries over the metadata)
Query Processing:
- API: FastAPI
- Search: Weaviate hybrid search (BM25 + cosine)
- Ranking: Custom (RRF + MMR in Python)
RAG:
- LLM: OpenAI GPT-3.5 Turbo (API)
- Prompting: Custom prompt templates
- Streaming: The OpenAI streaming API (a better UX)
Infrastructure:
- Embedding server: A GPU server (NVIDIA T4, $0.35/hour = $250/month)
- Weaviate server: CPU + SSD (an AWS c5.2xlarge, $0.34/hour = $245/month)
- API server: Serverless (AWS Lambda + API Gateway, ~$50/month)
Data flows
Flow 1: The initial indexing (once)
1. Extract the text from 100K docs (in parallel, 24 hours)
2. Hierarchical chunking → 500K chunks
3. Generate the embeddings (local Sentence-BERT):
- Batch size: 32
- Throughput: ~1000 chunks/minute
- Time: 500K / 1000 = 500 minutes (~8 hours)
4. Upsert to Weaviate (batches of 100):
- Time: ~2 hours
Total: ~34 hours (1.5 days)
Cost: $0 (the hardware already exists)
Flow 2: A search query (typical)
User: "papers about machine learning after 2020"
1. Parse the query:
- Topic: "machine learning"
- Filter: year >= 2020
2. Embed the query (local Sentence-BERT): 50ms
3. Weaviate hybrid search:
- Keyword: "machine learning" (BM25)
- Semantic: cosine on the embedding
- Filter: year >= 2020
- Top-K: 100
- Latency: 80ms
4. RRF fusion + MMR (Python): 100ms
5. Return the top-20 → the user
Total latency: 230ms ✅ (< the 2s target)
Flow 3: A Q&A query (RAG)
User: "What does Smith's paper say about NLP?"
1. Retrieval (same as Flow 2):
- Filter: author = "Smith" AND topic ~ "NLP"
- Top-5 chunks
- Latency: 230ms
2. Build the prompt (Python): 10ms
3. LLM generation (GPT-3.5 Turbo):
- Input: 2500 tokens (the query + 5 chunks)
- Output: 300 tokens
- Latency: 1200ms (streaming)
4. Return the response + the sources
Total latency: 1440ms ✅ (< the 2s target)
Architecture summary
Components:
- Sentence-BERT (local embeddings)
- Weaviate (a self-hosted vector DB)
- GPT-3.5 Turbo (the LLM)
- FastAPI (the API)
- Airflow (orchestration)
Flows:
- Indexing: 34 hours (once)
- Search: 230ms
- Q&A (RAG): 1440ms
Meeting the requirements:
- ✅ Latency < 2s
- ✅ Private data (local embeddings)
- ✅ Semantic + exact search
- ✅ Q&A with an LLM
- ✅ Diversity (MMR)
Next stage: 04-technical-decisions.md — Justifying the specific decisions (chunking, top-K, parameters).