Module 1: The Complete RAG Pipeline (Architecture Overview)

Success Metrics in RAG

Capsule overview

"Is my RAG any good?" That isn't a philosophical question. It's quantifiable with objective metrics: latency (speed), accuracy (precision), and cost (operational cost). This capsule teaches you how to define what "good" means for your use case and how to measure your system with real numbers.

Without metrics, you're optimizing blind. You add re-ranking and "it seems better", but how much better? Is the extra latency worth it? Does it justify the cost? With metrics you can answer: "Re-ranking improves precision from 65% to 85% (+20%), adds 180ms of latency, and costs $0.002 extra per query. For my use case (legal document search), precision >80% is critical, so it's worth it."

This capsule gives you: (1) industry-standard metrics, (2) typical targets by application type, (3) how to measure each metric, (4) how to interpret the results and make data-driven decisions.

⏱️ Metric 1: Latency (response speed)

What is latency?

The time from when a user submits a query until they receive the complete answer.

The components of latency in RAG:

import time

start = time.time()

# 1. Query embedding (~50ms)
query_embedding = create_embedding(user_query)  
t1 = time.time()

# 2. Vector search (~30ms for 10K docs)
docs = collection.query(query_embeddings=[query_embedding], n_results=5)
t2 = time.time()

# 3. LLM generation (~500-1500ms depending on tokens)
answer = llm.invoke(context + query)
t3 = time.time()

# Total latency
total_latency = (t3 - start) * 1000  # In milliseconds

print(f"""
Latency breakdown:
- Query embedding: {(t1-start)*1000:.0f}ms
- Vector search: {(t2-t1)*1000:.0f}ms
- LLM generation: {(t3-t2)*1000:.0f}ms
- Total: {total_latency:.0f}ms
""")

Typical output (baseline RAG):

Latency breakdown:
- Query embedding: 52ms
- Vector search: 28ms (ChromaDB, 10K docs)
- LLM generation: 620ms (GPT-3.5-turbo, 150 output tokens)
- Total: 700ms

Latency targets by application type:

Application	Target latency	Why
Interactive chatbot	<1,000ms (1 sec)	The user expects an immediate response
Document search	<2,000ms (2 sec)	Traditional search is ~1 sec, RAG can be 2x
Email assistant	<3,000ms (3 sec)	Async, doesn't block the UI
Batch processing	<10,000ms (10 sec)	Background jobs, latency isn't critical

The typical decision for a chatbot:

# Target: <1,000ms total
if total_latency > 1000:
    # Optimizations:
    # - Use gpt-3.5-turbo (not gpt-4) → -300ms
    # - Cache the embeddings of common queries → -50ms
    # - Reduce top-K from 10 to 5 → -10ms
    # - Parallel retrieval if multi-query → -100ms
    pass

Measuring latency with percentiles:

import statistics

# Measure 100 queries
latencies = []

for query in test_queries:
    start = time.time()
    answer = rag_system.query(query)
    latency = (time.time() - start) * 1000
    latencies.append(latency)

# Compute the percentiles
p50 = statistics.median(latencies)
p95 = statistics.quantiles(latencies, n=20)[18]  # 95th percentile
p99 = statistics.quantiles(latencies, n=100)[98]  # 99th percentile

print(f"""
Latency analysis (n=100):
- P50 (median): {p50:.0f}ms
- P95: {p95:.0f}ms
- P99: {p99:.0f}ms
""")

How to read it:

P50: 680ms → The typical user experience
P95: 1,200ms → 95% of users see <1.2s
P99: 2,300ms → 1% of users see >2s (outliers)

The typical target: P95 <1,000ms (95% of users get an answer in under 1 sec)

🎯 Metric 2: Accuracy (retrieval and generation precision)

Accuracy has 2 parts:

Retrieval accuracy: did we retrieve relevant documents?
Generation accuracy: is the answer correct and grounded?

2.1 Retrieval accuracy:

Precision:

# How many of the retrieved docs are relevant?
precision = relevant_retrieved / total_retrieved

# Example
retrieved_docs = 5  # Top-5
relevant_in_retrieved = 4  # 4 are relevant, 1 is junk

precision = 4 / 5  # 0.80 (80%)

How to read it:

High precision: little noise, the documents are relevant
Low precision: lots of noise, irrelevant documents confuse the LLM

The typical target: precision >0.70 (70%+ of the top-K are relevant)

Recall (coverage):

# How many of the relevant docs did we retrieve?
recall = relevant_retrieved / total_relevant_in_db

# Example
total_relevant_in_db = 10  # There are 10 relevant docs in total
relevant_retrieved = 4      # We retrieved 4 of those 10

recall = 4 / 10  # 0.40 (40%)

How to read it:

High recall: we found most of the relevant documents
Low recall: we left relevant documents behind

The typical target: recall >0.50 (we find 50%+ of the relevant docs)

The precision vs recall trade-off:

# Increasing K raises recall but lowers precision
top_k = 5:  Precision 0.80, Recall 0.40
top_k = 10: Precision 0.70, Recall 0.70
top_k = 20: Precision 0.60, Recall 0.90

# The decision: balance them based on your use case
if precision_critical:
    top_k = 5  # Less noise for the LLM
elif recall_critical:
    top_k = 20  # Don't leave relevant documents out
else:
    top_k = 10  # Balanced

2.2 Generation accuracy:

Faithfulness (groundedness):

# Is the answer grounded in the context?
# Score: 0.0 (made up) to 1.0 (fully grounded)

# Evaluate manually (baseline)
context = "FastAPI is a web framework."
answer = "FastAPI is a Python web framework."

# Is "Python" in the context? No → Faithfulness <1.0
# Is it a reasonable inference? Yes → Faithfulness ~0.90

Evaluating with RAGAS (Module 8):

from ragas.metrics import faithfulness

score = faithfulness.score(
    question="What is FastAPI?",
    answer="FastAPI is a Python web framework.",
    contexts=["FastAPI is a web framework."]
)

print(f"Faithfulness: {score:.2f}")  # 0.90

How to read it:

Score	Interpretation	Action
0.95+	Fully grounded	✅ Excellent
0.85-0.94	Mostly grounded	✅ Good
0.70-0.84	Some inferences	⚠️ Review the prompts
<0.70	Hallucinations	❌ Redesign the prompts

Answer relevancy:

# Does the answer actually answer the question?
# Score: 0.0 (irrelevant) to 1.0 (perfect)

# Example
question = "What is FastAPI?"
answer_relevant = "FastAPI is a web framework."  # Relevancy: 1.0
answer_partial = "FastAPI is popular."  # Relevancy: 0.60
answer_irrelevant = "Python is a language."  # Relevancy: 0.10

The typical target: relevancy >0.85 (the answer addresses the question well)

💰 Metric 3: Cost (operational cost)

The components of cost in RAG:

# Cost per query
embedding_cost = 0.0001  # $0.0001 per 1K tokens (OpenAI)
llm_cost = 0.002  # $0.002 per 1K tokens (GPT-3.5 input + output)
vector_db_cost = 70 / queries_per_month  # Pinecone $70/month
reranking_cost = 0.001  # Cross-encoder compute

total_cost_per_query = embedding_cost + llm_cost + vector_db_cost + reranking_cost

# For 10,000 queries/month
monthly_cost = total_cost_per_query * 10_000

print(f"Cost per query: ${total_cost_per_query:.4f}")
print(f"Monthly cost: ${monthly_cost:.2f}")

The detailed calculation:

# Example: RAG for documentation search
# 10K queries/day = 300K queries/month

# Indexing (one-time)
indexing_cost = (100_000_docs * 500_tokens / 1000) * 0.0001  # $5.00

# Query embeddings
query_embedding_cost = (300_000 * 20_tokens / 1000) * 0.0001  # $0.60/month

# LLM generation
llm_cost_per_query = (
    (1000_tokens_input / 1000) * 0.0015 +  # Input
    (200_tokens_output / 1000) * 0.002     # Output
) = 0.0019

llm_cost_monthly = 0.0019 * 300_000  # $570/month

# Vector DB
vector_db_cost = 70  # Pinecone serverless

# Total
total_monthly = 0.60 + 570 + 70  # $640.60/month
cost_per_query = 640.60 / 300_000  # $0.0021 per query

Cost optimization strategies:

Strategy	Saving	Trade-off	Module
Cache responses	50-70%	Freshness	Module 7
Use gpt-3.5 (not gpt-4)	90%	Quality -10%	Module 1
Reduce top-K	10-20%	Recall -15%	Module 2
Local embeddings	100% (embeddings)	Quality -20%	Module 1
Semantic cache	60-80%	Complexity	Module 7

The typical decision:

# For a startup on a tight budget
if monthly_budget < 500:
    # Aggressive optimizations:
    use_gpt_3_5 = True  # Not GPT-4 ($570 → $57)
    cache_enabled = True  # -60% queries ($57 → $23)
    top_k = 3  # Not 5 ($23 → $20)
    # Total: $90/month (indexing + queries + vector DB)

📊 Benchmarking: measuring your system

Benchmark setup:

# benchmark_rag.py
import time
from dataclasses import dataclass

@dataclass
class BenchmarkResult:
    latency_p50: float
    latency_p95: float
    precision: float
    recall: float
    faithfulness: float
    cost_per_query: float

def benchmark_rag_system(
    rag_system,
    test_queries: list[str],
    ground_truth: dict
) -> BenchmarkResult:
    """
    A complete benchmark of a RAG system.
    
    Input:
    - rag_system: your RAG system
    - test_queries: a list of test queries (30-50)
    - ground_truth: a dict with the correct answers and the relevant docs
    
    Output: a BenchmarkResult with the metrics
    """
    
    latencies = []
    precisions = []
    recalls = []
    
    for query in test_queries:
        # Measure latency
        start = time.time()
        result = rag_system.query(query)
        latency = (time.time() - start) * 1000
        latencies.append(latency)
        
        # Measure precision/recall
        retrieved = set(result['doc_ids'])
        relevant = set(ground_truth[query]['relevant_docs'])
        
        relevant_retrieved = retrieved & relevant
        precision = len(relevant_retrieved) / len(retrieved) if retrieved else 0
        recall = len(relevant_retrieved) / len(relevant) if relevant else 0
        
        precisions.append(precision)
        recalls.append(recall)
    
    # Compute the aggregates
    return BenchmarkResult(
        latency_p50=statistics.median(latencies),
        latency_p95=statistics.quantiles(latencies, n=20)[18],
        precision=statistics.mean(precisions),
        recall=statistics.mean(recalls),
        faithfulness=0.0,  # Measured with RAGAS in Module 8
        cost_per_query=0.0021  # Computed from actual usage
    )

# Usage
results = benchmark_rag_system(
    rag_system=my_rag,
    test_queries=test_queries,
    ground_truth=ground_truth_dict
)

print(f"""
Benchmark Results:
- Latency P50: {results.latency_p50:.0f}ms
- Latency P95: {results.latency_p95:.0f}ms
- Precision: {results.precision:.2%}
- Recall: {results.recall:.2%}
- Cost per query: ${results.cost_per_query:.4f}
""")

Expected output (baseline):

Benchmark Results:
- Latency P50: 680ms
- Latency P95: 1,120ms
- Precision: 68%
- Recall: 52%
- Cost per query: $0.0021

Interpreting the results:

Latency:

✅ P50 <1,000ms: good user experience
⚠️ P95 >2,000ms: 5% of users see a slow response
❌ P99 >5,000ms: problematic outliers

Precision:

✅ >70%: most of the retrieved docs are relevant
⚠️ 60-70%: there's noise, but the LLM can handle it
❌ <60%: too much noise, inconsistent answers

Recall:

✅ >60%: we find most of the relevant docs
⚠️ 40-60%: we're leaving relevant docs out
❌ <40%: retrieval is very poor

🎯 Targets by RAG type

RAG type 1: General chatbot

Use case: a conversational assistant (FAQs, support)

targets = {
    "latency_p95": 1_500,  # <1.5s
    "precision": 0.65,      # 65%+ (the LLM tolerates noise)
    "recall": 0.50,         # 50%+ (find enough info)
    "faithfulness": 0.80,   # 80%+ (grounded in the docs)
    "cost_per_query": 0.003 # <$0.003 (budget-friendly)
}

RAG type 2: Technical documentation search

Use case: searching technical docs (code, APIs)

targets = {
    "latency_p95": 2_000,   # <2s (not interactive)
    "precision": 0.85,      # 85%+ (precision is critical)
    "recall": 0.70,         # 70%+ (find every reference)
    "faithfulness": 0.95,   # 95%+ (don't invent APIs)
    "cost_per_query": 0.005 # <$0.005 (quality > cost)
}

RAG type 3: Legal/medical search

Use case: searching critical documents

targets = {
    "latency_p95": 5_000,   # <5s (precision > speed)
    "precision": 0.95,      # 95%+ (zero false positives)
    "recall": 0.85,         # 85%+ (don't miss relevant info)
    "faithfulness": 0.98,   # 98%+ (zero hallucinations)
    "cost_per_query": 0.010 # <$0.01 (maximum quality)
}

Decision matrix: does your RAG hit its targets?

def evaluate_against_targets(results: BenchmarkResult, targets: dict) -> dict:
    """Compares the results against the targets"""
    
    evaluation = {}
    
    # Latency
    if results.latency_p95 < targets['latency_p95']:
        evaluation['latency'] = "✅ PASS"
    else:
        evaluation['latency'] = f"❌ FAIL ({results.latency_p95:.0f}ms > {targets['latency_p95']}ms)"
    
    # Precision
    if results.precision >= targets['precision']:
        evaluation['precision'] = "✅ PASS"
    else:
        evaluation['precision'] = f"❌ FAIL ({results.precision:.2%} < {targets['precision']:.2%})"
    
    # Recall
    if results.recall >= targets['recall']:
        evaluation['recall'] = "✅ PASS"
    else:
        evaluation['recall'] = f"❌ FAIL ({results.recall:.2%} < {targets['recall']:.2%})"
    
    return evaluation

# Usage
evaluation = evaluate_against_targets(results, targets)

for metric, status in evaluation.items():
    print(f"{metric}: {status}")

Output:

latency: ✅ PASS
precision: ❌ FAIL (68% < 85%)
recall: ❌ FAIL (52% < 70%)

Action: you need to improve precision and recall → Modules 2-6 teach you the techniques for exactly that.

🔄 Metric-driven improvements

If latency is the problem:

# Latency optimizations
optimizations = {
    "Use gpt-3.5 instead of gpt-4": -300,  # ms
    "Cache query embeddings": -50,
    "Reduce top-K from 10 to 5": -20,
    "Use a smaller embedding model": -30,
    "Parallel retrieval": -100
}

# Apply them and re-measure

If precision is the problem:

# Precision improvements
improvements = {
    "Add re-ranking (Module 4)": +20,  # % improvement
    "Better chunking (Module 2)": +15,
    "Query optimization (Module 3)": +10,
    "Metadata filtering (Module 6)": +12
}

# Apply them and re-measure

If recall is the problem:

# Recall improvements
improvements = {
    "Query expansion (Module 3)": +25,  # % improvement
    "Increase top-K from 5 to 10": +20,
    "Hybrid search (Module 5)": +18,
    "Better embeddings": +10
}

# Apply them and re-measure

🎯 Summary

Key concepts:

✅ Latency: response speed (P50, P95, P99) - typical target: P95 <1,000ms
✅ Retrieval accuracy: precision (relevance) and recall (coverage) - target: >70% and >50%
✅ Generation accuracy: faithfulness (grounded) and relevancy (answers the question) - target: >85% and >85%
✅ Cost: cost per query (embeddings + LLM + vector DB) - target: typically <$0.005
✅ Targets vary by use case: chatbot vs technical search vs legal all have different targets
✅ Benchmarking: measure with test queries + ground truth → identify the gaps → apply improvements
✅ Metric-driven improvements: latency problem → optimize components; precision problem → re-ranking

What's next:

Capsule 05 shows you real-world use cases: how Perplexity, Notion AI and ChatGPT implement RAG in production, which techniques they use, and what you can learn from their architectures.

📚 Additional resources

RAGAS Metrics - The official metrics documentation
Retrieval Metrics Explained - Precision, Recall, MRR, NDCG
Benchmarking RAG Systems - Evaluation methodology
Cost Optimization for LLMs - OpenAI best practices
Latency Optimization - LangChain's performance guide
RAG Evaluation Framework - A HuggingFace blog post

Created: February 6, 2026
Version: 1.0

← PreviousRAG Architecture Decisions Next →Real-World RAG Use Cases in Production