Module 8: Final Capstone Project - Complete RAG System

Module 8: Final Capstone Project - Complete RAG System

Next →

Introduction

This is the final module that integrates EVERYTHING you learned across the previous 7 modules. You'll build a production-ready RAG system end-to-end with:

  • M1-M2: Embeddings (OpenAI + SBERT fallback)
  • M3: Model comparison and selection
  • M4: Smart chunking + evaluation
  • M5: Optimized distance metrics (FAISS)
  • M6: Embedding operations (query expansion)
  • M7: Production patterns (caching, monitoring, fault tolerance)

Final project objectives

  1. ✅ Complete system: Ingest → Chunk → Embed → Index → Retrieve → Evaluate
  2. ✅ Production-ready: Caching, monitoring, error handling
  3. ✅ Evaluation: Metrics (nDCG, Recall@K)
  4. ✅ Optimization: FAISS, batch processing
  5. ✅ Interactive CLI: Testing tool
  6. ✅ Documentation: Setup, API, deployment

Module roadmap

Phase 1: Architecture (Capsules 01-03)

  • Capsule 01: Introduction and architecture
  • Capsule 02: Document ingestion pipeline
  • Capsule 03: Chunking & embedding pipeline

Phase 2: Retrieval & Evaluation (Capsules 04-06)

  • Capsule 04: Vector search with FAISS
  • Capsule 05: Evaluation framework
  • Capsule 06: Query expansion & reranking

Phase 3: Production & Deploy (Capsules 07-08)

  • Capsule 07: Production deployment
  • Capsule 08: Conclusions and next steps

System architecture

┌─────────────────────────────────────────────┐
│           DOCUMENT INGESTION                │
│  - Load docs (PDF, MD, TXT)                 │
│  - Metadata extraction                      │
└──────────────────┬──────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────┐
│            CHUNKING PIPELINE                │
│  - Recursive chunking (LangChain)           │
│  - Token-aware (tiktoken)                   │
│  - Metadata enrichment                      │
└──────────────────┬──────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────┐
│          EMBEDDING GENERATION               │
│  - OpenAI API (primary)                     │
│  - SBERT (fallback)                         │
│  - Redis caching                            │
│  - Batch processing                         │
└──────────────────┬──────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────┐
│           VECTOR INDEXING                   │
│  - FAISS (HNSW index)                       │
│  - Normalized embeddings                    │
│  - Metadata storage                         │
└──────────────────┬──────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────┐
│            RETRIEVAL SYSTEM                 │
│  - Query embedding                          │
│  - FAISS search (top-K)                     │
│  - Optional: Query expansion                │
│  - Optional: Reranking                      │
└──────────────────┬──────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────┐
│             EVALUATION                      │
│  - nDCG@K, Recall@K                         │
│  - A/B testing chunking strategies          │
│  - Performance monitoring                   │
└─────────────────────────────────────────────┘

Tech Stack

# Core
- Python 3.10+
- NumPy
- OpenAI API

# Embeddings
- openai==1.54.0
- sentence-transformers==2.3.1

# Chunking
- langchain==0.1.0
- tiktoken==0.5.2

# Vector search
- faiss-cpu==1.7.4

# Production
- redis==5.0.1
- prometheus-client==0.19.0

# Evaluation
- scikit-learn==1.4.0

Project structure

rag-system-final/
├── src/
│   ├── ingestion/
│   │   ├── __init__.py
│   │   ├── document_loader.py
│   │   └── metadata_extractor.py
│   ├── chunking/
│   │   ├── __init__.py
│   │   └── smart_chunker.py
│   ├── embeddings/
│   │   ├── __init__.py
│   │   ├── embedding_client.py
│   │   └── cache.py
│   ├── indexing/
│   │   ├── __init__.py
│   │   └── faiss_index.py
│   ├── retrieval/
│   │   ├── __init__.py
│   │   ├── retriever.py
│   │   └── reranker.py
│   ├── evaluation/
│   │   ├── __init__.py
│   │   └── metrics.py
│   └── monitoring/
│       ├── __init__.py
│       └── metrics_collector.py
├── data/
│   ├── documents/          # Input docs
│   ├── eval_dataset.json   # Evaluation set
│   └── cache/              # Redis alternative
├── indexes/                # FAISS indexes
├── config/
│   └── config.yaml         # Configuration
├── tests/
│   └── test_*.py
├── main.py                 # CLI
├── api.py                  # REST API (FastAPI)
├── requirements.txt
├── README.md
└── docker-compose.yml

Key features

1. Intelligent Chunking

# Recursive chunking with token awareness
# - Respects boundaries (paragraphs, sentences)
# - Guarantees max_tokens
# - Metadata enrichment (source, headers)

2. Multi-provider embeddings

# Primary: OpenAI (high quality)
# Fallback: local SBERT (fault tolerance)
# Caching: Redis (80% hit rate)

3. Optimized search

# FAISS HNSW (100x faster)
# Normalized embeddings (dot product)
# Query expansion (better recall)

4. Production patterns

# - Retry logic (exponential backoff)
# - Circuit breaker
# - Health checks
# - Monitoring (Prometheus)
# - Cost tracking

5. Comprehensive evaluation

# - nDCG@K, Recall@K, Precision@K
# - A/B testing
# - Performance benchmarks

In the next capsules

You'll implement each component step by step, culminating in a complete production-ready RAG system.

Module 8 - Capsule 01 Final Project: integrating 7 modules of knowledge

← PreviousMini-Project: Production-Ready RAG SystemNext →Document Ingestion Pipeline