Module 1: The Complete RAG Pipeline (Architecture Overview)

RAG Architecture Decisions

Capsule overview

Building advanced RAG isn't just wiring components together. It's making technical decisions at every layer: which chunking strategy to use, which embeddings to choose, which vector DB to select, when to apply re-ranking. Every decision has real trade-offs: performance vs quality, cost vs precision, simplicity vs features.

This capsule gives you a complete decision framework: comparison tables, selection criteria, typical use cases, and examples of real decisions. There are no one-size-fits-all answers ("always use X"), just the context you need to decide intelligently for your specific use case: dataset size, latency requirements, budget constraints, quality targets.

By the end of this capsule, you'll be able to justify your architecture decisions with data and clear trade-offs. You won't say "I use Pinecone because it's popular", you'll say "I use Pinecone because my dataset has 5M documents, I need <100ms latency, and I have a $200/month budget for a managed service".

🧩 Decision 1: Chunking strategy

The problem:

Long documents (5,000+ tokens) don't fit in a vector DB or an LLM context window. You need to split them into chunks.

Your options:

Option A: Fixed-size chunking

How it works:

def fixed_size_chunking(document: str, chunk_size: int = 500) -> list[str]:
    """Splits a document into fixed-size chunks"""
    return [document[i:i+chunk_size] for i in range(0, len(document), chunk_size)]

# Example
doc = "Python is a language. " * 100  # 2400 characters
chunks = fixed_size_chunking(doc, chunk_size=500)

print(f"Chunks: {len(chunks)}")  # 5 chunks
print(f"First chunk: {chunks[0]}")

Pros:

✅ Simple to implement
✅ Fast
✅ Predictable chunk sizes

Cons:

❌ Can cut in the middle of a sentence
❌ Loses semantic context
❌ Doesn't respect the document's structure

When to use it:

Structured documents (tables, lists)
Quick prototypes
Small datasets (<1,000 docs)

Option B: Semantic chunking

How it works:

from langchain.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

def semantic_chunking(document: str) -> list[str]:
    """Splits a document while preserving semantic coherence"""
    
    embeddings = OpenAIEmbeddings()
    splitter = SemanticChunker(embeddings)
    
    chunks = splitter.split_text(document)
    return chunks

# Example
doc = """
Python is an interpreted programming language.
It was created by Guido van Rossum in 1991.

FastAPI is a web framework for Python.
It was created by Sebastián Ramírez in 2018.
"""

chunks = semantic_chunking(doc)

# Output: 2 chunks (Python vs FastAPI)
# Chunk 1: "Python is... 1991."
# Chunk 2: "FastAPI is... 2018."

Pros:

✅ Preserves semantic coherence
✅ Doesn't cut at arbitrary points
✅ Better retrieval quality

Cons:

❌ Slower (it generates embeddings)
❌ Variable chunk sizes
❌ Requires API calls (cost)

When to use it:

Narrative documents (articles, books)
Quality > speed
The budget allows for extra embeddings

Option C: Recursive chunking

How it works:

from langchain.text_splitter import RecursiveCharacterTextSplitter

def recursive_chunking(
    document: str,
    chunk_size: int = 500,
    chunk_overlap: int = 50
) -> list[str]:
    """
    Splits a document recursively using separators.
    
    It tries to split by:
    1. Paragraphs (\n\n)
    2. Sentences (. )
    3. Words ( )
    4. Characters (if everything else fails)
    """
    
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separators=["\n\n", "\n", ". ", " ", ""]
    )
    
    chunks = splitter.split_text(document)
    return chunks

# Example
chunks = recursive_chunking(doc, chunk_size=200, chunk_overlap=30)

# Output: chunks that respect paragraphs/sentences
# The 30-char overlap preserves context between chunks

Pros:

✅ Balanced: respects structure while keeping size under control
✅ Overlap preserves context between chunks
✅ Doesn't need extra embeddings

Cons:

❌ More complex than fixed-size
❌ Custom separators need tuning
❌ Overlap increases storage (duplication)

When to use it:

The default for production (best balance)
Mixed documents (narrative + code + lists)
You need to preserve context without semantic's cost

Chunking strategies compared:

Strategy	Speed	Retrieval quality	Cost	Chunk size	When to use
Fixed-size	Very fast	Medium	Zero	Fixed (500)	Prototype, structured docs
Semantic	Slow	High	High	Variable (200-800)	Narrative, quality-critical
Recursive	Fast	Medium-High	Zero	Controlled (500±50)	Production default

Recommended for the baseline (Module 1): Fixed-size (simple)
Recommended for production (Module 7): Recursive (balanced)
Going deeper: Module 2 covers chunking strategies in detail

🧠 Decision 2: Embedding model

The problem:

You need to turn text into vectors. Which model do you use?

Your options:

Option A: OpenAI text-embedding-ada-002

from openai import OpenAI

client = OpenAI()

embedding = client.embeddings.create(
    model="text-embedding-ada-002",
    input="Your text here"
).data[0].embedding

# Dimensions: 1536
# Cost: $0.0001 per 1K tokens

Pros:

✅ High quality (state-of-the-art)
✅ Managed (no deployment)
✅ 1536 dimensions (good resolution)

Cons:

❌ Cost per call
❌ Requires an API key
❌ Network latency (~50ms)

When to use it:

Production with a budget
Quality-critical work
English or multilingual

Option B: Local (Sentence-Transformers)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')  # 22MB model

embedding = model.encode("Your text here")

# Dimensions: 384
# Cost: Zero (local)
# Latency: ~20ms (local)

Pros:

✅ Free (zero cost)
✅ Total privacy (local)
✅ Fast (~20ms, no network)

Cons:

❌ Lower quality than OpenAI
❌ 384 dimensions (less resolution)
❌ Requires deploying the model

When to use it:

Zero budget
Privacy-critical work
A prototype with no external dependencies

Option C: Cohere embed-multilingual-v3

import cohere

co = cohere.Client("api_key")

embedding = co.embed(
    texts=["Your text here"],
    model="embed-multilingual-v3.0",
    input_type="search_document"  # Or "search_query"
).embeddings[0]

# Dimensions: 1024
# Cost: $0.0001 per 1K tokens

Pros:

✅ Excellent multilingual support (100+ languages)
✅ Different input types (document vs query)
✅ High quality

Cons:

❌ Cost similar to OpenAI
❌ Less adoption than OpenAI
❌ Requires a different API key

When to use it:

Multilingual is critical (Spanish, French, etc.)
You need to separate document from query embeddings

Embedding models compared:

Model	Dimensions	Cost (1M tokens)	Quality	Multilingual	Latency	When to use
OpenAI ada-002	1536	$0.10	⭐⭐⭐⭐⭐	Good	50ms	English production
Cohere multilingual	1024	$0.10	⭐⭐⭐⭐⭐	Excellent	60ms	Multilingual production
Local MiniLM	384	Free	⭐⭐⭐	Limited	20ms	Dev, prototype, privacy

Recommended decision:

Baseline (Module 1): OpenAI ada-002 (simple, good quality)
Production: OpenAI if English, Cohere if multilingual
Local dev: Sentence-Transformers (zero cost)

🗄️ Decision 3: Vector database

The problem:

You need to store millions of vectors and search them fast (<100ms).

Your options:

Option A: ChromaDB (local)

import chromadb

# Local client (in memory)
client = chromadb.Client()

# Persistent client (on disk)
client = chromadb.PersistentClient(path="./chroma_db")

collection = client.create_collection("docs")

# Add documents
collection.add(
    documents=["text1", "text2"],
    embeddings=[[0.1, ...], [0.2, ...]],
    metadatas=[{"source": "doc1"}, {"source": "doc2"}],
    ids=["1", "2"]
)

# Query
results = collection.query(
    query_embeddings=[[0.15, ...]],
    n_results=5
)

Pros:

✅ Zero setup (pip install)
✅ Free (local)
✅ Persistence on disk
✅ Perfect for development

Cons:

❌ Doesn't scale beyond 100K docs (slow)
❌ Single-machine (not distributed)
❌ Not managed (you administer it)

Performance:

10K docs: <50ms query ✅
100K docs: ~200-500ms ⚠️
1M+ docs: Not recommended ❌

When to use it:

Development and prototypes (modules 1-6)
Datasets under 50K documents
You don't need a managed service

Option B: Pinecone (managed cloud)

from pinecone import Pinecone

# Initialize
pc = Pinecone(api_key="your-api-key")

# Create an index
index = pc.create_index(
    name="my-index",
    dimension=1536,
    metric="cosine"
)

# Add documents
index.upsert(
    vectors=[
        ("id1", [0.1, ...], {"text": "text1"}),
        ("id2", [0.2, ...], {"text": "text2"})
    ]
)

# Query
results = index.query(
    vector=[0.15, ...],
    top_k=5,
    include_metadata=True
)

Pros:

✅ Infinite scalability (millions of docs)
✅ Consistent performance (<50ms, always)
✅ Managed (you don't administer it)
✅ Advanced features (namespaces, filtering)

Cons:

❌ Cost ($70/month minimum for serverless)
❌ Requires an API key and an account
❌ Vendor lock-in

Performance:

10K docs: <50ms ✅
1M docs: <50ms ✅
100M docs: <50ms ✅ (with the right pods)

When to use it:

Production (module 7)
Datasets over 100K documents
You need a guaranteed <100ms latency
You have the budget for a managed service

Option C: Weaviate (self-hosted or cloud)

import weaviate

# Cloud client
client = weaviate.Client(
    url="https://your-cluster.weaviate.network",
    auth_client_secret=weaviate.AuthApiKey("api_key")
)

# Create a schema
client.schema.create_class({
    "class": "Document",
    "vectorizer": "text2vec-openai"
})

# Add documents
client.data_object.create(
    data_object={"text": "text1"},
    class_name="Document"
)

# Query
results = client.query.get(
    "Document", ["text"]
).with_near_text({"concepts": ["query"]}).with_limit(5).do()

Pros:

✅ Self-hosting is possible (total control)
✅ GraphQL queries (flexible)
✅ Built-in vectorizers

Cons:

❌ More complex than ChromaDB
❌ Self-hosting requires DevOps
❌ Cloud pricing similar to Pinecone

When to use it:

You need self-hosting (privacy/compliance)
GraphQL queries are an advantage for you
Your company already uses Weaviate

Vector databases compared:

Vector DB	Setup	Cost	Performance	Max docs	Managed	When to use
ChromaDB	`pip install`	Free	10K: fast, 100K+: slow	~100K	❌	Dev, prototypes
Pinecone	API key	$70+/mo	Always <50ms	Millions	✅	Production
Weaviate	Docker/Cloud	Free (self) or $70+/mo	Good	Millions	✅/❌	Self-hosting, GraphQL
Qdrant	Docker	Free (self)	Very good	Millions	❌	Self-hosting, Rust
Milvus	Docker/Cloud	Free (self)	Excellent	Millions	❌	Enterprise, scale

Decision matrix:

Dataset <10K docs + Dev → ChromaDB ✅
Dataset 10K-100K + Dev → ChromaDB ⚠️
Dataset >100K + Production → Pinecone ✅
Privacy/Compliance → Weaviate/Qdrant (self-hosted) ✅
Zero budget + <100K docs → ChromaDB ✅
Budget available + >100K docs → Pinecone ✅

🎯 Decision 4: Re-ranking

The problem:

The top-K documents from similarity search include irrelevant ones (false positives).

When do you apply re-ranking?

Without re-ranking (baseline):

# Similarity search returns the top-5
results = collection.query(query_embeddings=[...], n_results=5)
top_k_docs = results['documents'][0]

# The problem: some docs are irrelevant
# Doc 1: Relevant (cosine: 0.85)
# Doc 2: Irrelevant (cosine: 0.83) ← False positive
# Doc 3: Relevant (cosine: 0.81)

With re-ranking (Module 4):

# 1. Wide retrieval (top-20)
results = collection.query(query_embeddings=[...], n_results=20)

# 2. Re-ranking with a cross-encoder
from sentence_transformers import CrossEncoder
model = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

scores = model.predict([
    (user_query, doc) for doc in results['documents'][0]
])

# 3. Select the re-ranked top-5
top_5_indices = np.argsort(scores)[::-1][:5]
top_k_docs = [results['documents'][0][i] for i in top_5_indices]

# Result: only genuinely relevant documents

Re-ranking trade-offs:

Aspect	Without re-ranking	With re-ranking	Delta
Latency	200ms	400ms	+200ms
Precision@5	65%	85%	+20%
Compute cost	Low	Medium	+an extra model
Complexity	Simple	Medium	+a component

The decision:

# Decision tree for re-ranking
if precision_requirements > 0.80:
    if latency_budget > 500ms:
        use_reranking = True  # ✅ Worth it
    else:
        use_reranking = False  # ❌ Latency is critical
else:
    use_reranking = False  # ❌ 65% precision is enough

Typical use cases:

✅ Use re-ranking when:

Precision is critical (medical, legal, financial)
Your latency budget is >500ms
False positives are expensive

❌ DON'T use re-ranking when:

<300ms latency is critical
60-70% precision is good enough
You're building simple prototypes

🔗 Decision 5: Optional components

Hybrid search (Module 5):

When do you add it?

✅ Queries contain exact keywords (names, codes, IDs)
✅ Semantic search fails in specific cases
❌ Purely narrative content (semantic is enough)

Trade-off: +complexity, +better coverage

Metadata filtering (Module 6):

When do you add it?

✅ You need to filter by context (date, author, category)
✅ A large corpus with logical subsets
❌ A small, homogeneous corpus

Trade-off: +metadata storage, +contextual relevance

Query optimization (Module 3):

When do you add it?

✅ User queries are ambiguous
✅ You need to boost recall (find more relevant docs)
❌ Queries are already clear and specific

Trade-off: +LLM calls, +better retrieval

📋 The complete decision framework

Step 1: Define your requirements

My use case:
- Dataset size: [X documents]
- Query volume: [Y queries/day]
- Latency target: [Z ms]
- Precision target: [W%]
- Budget: [$X/month]
- Deployment: [Cloud/On-prem]

Step 2: Select your components

# Chunking
if document_type == "narrative":
    chunking = "semantic"  # Module 2
elif document_type == "technical":
    chunking = "recursive"  # Default
else:
    chunking = "fixed"  # Simple

# Embeddings
if budget == "zero":
    embeddings = "sentence-transformers"  # Local
elif multilingual:
    embeddings = "cohere"  # Better multilingual
else:
    embeddings = "openai"  # Default

# Vector DB
if dataset_size > 100_000:
    vector_db = "pinecone"  # Production
else:
    vector_db = "chromadb"  # Dev

# Re-ranking
if precision_target > 0.80 and latency_budget > 500:
    use_reranking = True  # Module 4
else:
    use_reranking = False

# Hybrid search
if queries_contain_keywords:
    use_hybrid = True  # Module 5
else:
    use_hybrid = False

# Metadata filtering
if need_contextual_filtering:
    use_metadata = True  # Module 6
else:
    use_metadata = False

Step 3: Estimate cost and performance

# Monthly cost estimate
embeddings_cost = (docs * avg_tokens_per_doc / 1000) * 0.0001
queries_cost = (queries_per_month * avg_tokens_per_query / 1000) * 0.0001
vector_db_cost = 70 if use_pinecone else 0
reranking_cost = queries_per_month * 0.001 if use_reranking else 0

total_monthly_cost = embeddings_cost + queries_cost + vector_db_cost + reranking_cost

# Performance estimate
baseline_latency = 200  # Retrieval + generation
reranking_latency = 200 if use_reranking else 0
total_latency = baseline_latency + reranking_latency

print(f"Estimated monthly cost: ${total_monthly_cost:.2f}")
print(f"Expected latency: {total_latency}ms")

🎯 Summary

Key concepts:

✅ Chunking: Fixed (simple) vs Semantic (quality) vs Recursive (balance) - Module 2 goes deeper
✅ Embeddings: OpenAI (quality) vs Local (free) vs Cohere (multilingual) - Module 1 uses OpenAI
✅ Vector DB: ChromaDB (dev) vs Pinecone (production) - Modules 1-6 use Chroma, Module 7 migrates to Pinecone
✅ Re-ranking: Add it if >80% precision is critical - Module 4 goes deeper
✅ Decision framework: Requirements → Components → Cost/Performance → Decision
✅ Trade-offs: Everything has one (cost vs quality, latency vs precision, simplicity vs features)

What's next:

Capsule 04 teaches you success metrics: how to define what "good" means in RAG (latency targets, accuracy targets, cost budgets), and how to measure your system with objective metrics.

📚 Additional resources

Vector Database Comparison - A detailed comparison of vector DBs
Embedding Models Benchmark - MTEB leaderboard (the official benchmark)
ChromaDB vs Pinecone - When to migrate to a managed service
Chunking Strategies Overview - An analysis of chunk size
OpenAI vs Cohere Embeddings - A practical comparison
RAG Architecture Patterns - LangChain patterns

Created: February 6, 2026
Version: 1.0

← PreviousRAG Pipeline Components Next →Success Metrics in RAG