Module 8: RAG Evaluation + The Capstone Project

Final Project: An Evaluated Advanced RAG System

Project description

This is the capstone project of the entire guide. It closes the full loop: it integrates the technical components you built in modules 2-7 (intelligent chunking, query optimization, hybrid search, re-ranking, metadata filtering, Pinecone production) with the operational evaluation discipline you just learned in M08.

It isn't "gluing components together". It's proving that the complete pipeline works end-to-end, scales under real load, holds a quality measured in numbers, and operates with the rigor that separates a professional system from a demo. It's the portfolio piece a senior interviewer will look at and say "this candidate really knows how to operate RAG in production".

By the end you'll have a public, production-ready repository that tells a quantitative story: "the baseline system had faithfulness 0.74 and p95 latency 800ms; after applying the guide's techniques, faithfulness is 0.92 and p95 is 350ms, validated against a golden dataset of 100 queries with CI/CD that blocks regressions automatically".

That story, told with data, is what separates "I studied RAG" from "I can build and operate RAG professionally".

The project's objective

Build and integrate a complete RAG system with six operational dimensions:

A working end-to-end RAG pipeline: query → expansion → hybrid retrieval → re-ranking → metadata filtering → generation
Production infrastructure on Pinecone with multi-tenancy and observability
A versioned golden dataset with 50-100 representative queries and validated ground truth
An automated evaluation pipeline with RAGAS, segmented metrics and JSON+MD reports
Quality gates with calibrated thresholds, regression testing and a versioned baseline
Active CI/CD in GitHub Actions with smoke on the PR, full nightly, and diff comments on pull requests

The complete system's architecture

┌──────────────────────────────────────────────────────────────────┐
│                      User Query (with TenantContext)              │
└──────────────────────────┬───────────────────────────────────────┘
                           │
                           ▼
                ┌──────────────────────┐
                │ Query Optimization   │  M03
                │ (expand, rewrite)    │
                └──────────┬───────────┘
                           │
                           ▼
            ┌──────────────────────────────┐
            │ Hybrid Retrieval             │  M05
            │ (BM25 + semantic, RRF fuse)  │
            └──────────┬───────────────────┘
                       │
                       ▼
        ┌─────────────────────────────────────┐
        │ Metadata Filtering                  │  M06
        │ (FilterSpec → namespace + filter)   │
        └──────────┬──────────────────────────┘
                   │
                   ▼
              ┌─────────────────────┐
              │ Re-ranking          │  M04
              │ (cross-encoder)     │
              └──────────┬──────────┘
                         │
                         ▼
                  ┌─────────────────┐
                  │ Generation      │
                  │ (gpt-4o-mini)   │
                  └────────┬────────┘
                           │
                           ▼
                ┌──────────────────────┐
                │ Evaluation Logging   │  M08
                │ (trace + metrics)    │
                └──────────────────────┘

Every box is a component you saw built in the previous modules. Your job is to integrate them while keeping the interfaces clean.

The mandatory stack

Layer	Technology	Source capsule
Vector DB	Pinecone serverless	M07
Embeddings	OpenAI text-embedding-3-small	M02, M07
Generation	OpenAI gpt-4o-mini	—
BM25	rank-bm25	M05
Re-ranker	sentence-transformers cross-encoder	M04
Eval framework	RAGAS	M08
API	FastAPI	M07
Observability	structlog + Prometheus	M07
CI/CD	GitHub Actions	M08
Tests	pytest	—

The repository's structure

advanced-rag-system/
├── app/
│   ├── pipeline.py              # The AdvancedRAGSystem orchestrator
│   ├── retrieval/
│   │   ├── hybrid.py            # M05 hybrid search
│   │   ├── reranker.py          # M04 cross-encoder
│   │   └── pinecone_backend.py  # M07
│   ├── query/
│   │   └── optimizer.py         # M03 expansion
│   ├── tenant.py                # TenantContext from M07
│   ├── metadata.py              # FilterSpec from M06
│   ├── generation.py            # LLM generation with a grounding prompt
│   ├── observability.py         # structlog + traces
│   └── api.py                   # The FastAPI endpoints
├── eval/
│   ├── loader.py                # The golden dataset loader
│   ├── runner.py                # The pipeline executor
│   ├── metrics.py               # The RAGAS wrapper
│   ├── thresholds.py            # The quality gates
│   ├── reporter.py              # JSON + MD reports
│   └── baseline.json            # The versioned baseline
├── golden_dataset/
│   ├── v1.0.0.json              # 50+ queries
│   ├── META.json
│   ├── PROCESS.md
│   └── CHANGELOG.md
├── scripts/
│   ├── evaluate.py              # The main CLI
│   ├── migrate_from_chroma.py
│   ├── benchmark.py
│   ├── compare_baseline.py
│   └── update_baseline.py
├── tests/
│   ├── test_isolation.py        # Multi-tenant safety
│   ├── test_pipeline.py
│   └── test_quality_gates.py
├── .github/workflows/
│   ├── rag-eval-smoke.yml
│   ├── rag-eval-nightly.yml
│   └── rag-eval-compare.yml
├── docker-compose.yml
├── pyproject.toml
├── BENCHMARK.md
├── DECISION.md
└── README.md

Mandatory features

1) The integrated RAG pipeline

# app/pipeline.py
from typing import Awaitable

class AdvancedRAGSystem:
    def __init__(
        self,
        query_optimizer,
        hybrid_retriever,
        reranker,
        generator,
        tracer,
    ):
        self.query_optimizer = query_optimizer
        self.hybrid_retriever = hybrid_retriever
        self.reranker = reranker
        self.generator = generator
        self.tracer = tracer

    async def answer_with_context(
        self,
        query: str,
        tenant: TenantContext,
        filter_spec: FilterSpec | None = None,
    ) -> RAGResponse:
        trace = self.tracer.start_trace(query=query, tenant_id=tenant.tenant_id)

        expanded = await self.query_optimizer.expand(query)
        trace.record("query_expansion", {"variants": expanded})

        candidates = await self.hybrid_retriever.retrieve(
            queries=expanded,
            tenant=tenant,
            filter_spec=filter_spec or FilterSpec(),
            top_k=50,
        )
        trace.record("retrieval", {"n_candidates": len(candidates)})

        reranked = await self.reranker.rerank(query=query, documents=candidates, top_k=5)
        trace.record("rerank", {"final_top_k": 5})

        answer = await self.generator.generate(query=query, context=reranked)
        trace.record("generation", {"answer_length": len(answer)})

        return RAGResponse(
            answer=answer,
            sources=reranked,
            trace_id=trace.trace_id,
        )

Why this design:

Dependency injection: every component is testable in isolation
Tracing built in: every stage records metrics, latency and decisions
Async native: parallelizable under load without a rewrite
A mandatory TenantContext: isolation guaranteed by types, not by convention

2) The golden dataset and the evaluation

At minimum 50 queries with the distribution from M08/04. Your pipeline must run python scripts/evaluate.py --mode smoke in under 2 minutes.

3) Quality gates in CI

A smoke evaluation on every PR, blocking the merge if:

Faithfulness < 0.85
Answer Relevancy < 0.80
Context Recall < 0.80
Or a regression > 2% vs the baseline on any core metric

4) Observability

Every API query generates:

{
  "trace_id": "...",
  "tenant_id": "...",
  "query": "...",
  "stages": {
    "expansion": {"latency_ms": 180, "variants": 3},
    "retrieval": {"latency_ms": 120, "candidates": 50},
    "rerank": {"latency_ms": 250, "final": 5},
    "generation": {"latency_ms": 850, "tokens": 142}
  },
  "total_latency_ms": 1400,
  "cost_usd": 0.0023
}

Mandatory validations

The list of invariants your system must guarantee:

Every query requires a valid TenantContext
There's no direct call to index.query outside of secure_query
The cache key includes the tenant_id (a passing test: two tenants don't share a cache entry)
CI blocks the merge if the quality gates fail
Structured JSON logs with trace_id, tenant_id, latency_ms per stage
The test suite includes test_isolation.py, test_pipeline.py, test_quality_gates.py
The README includes an M1 vs M8 comparison table with real metrics

Success criteria

✅ Precision@5 and Recall@20 improve ≥30% vs the simple RAG baseline (M01-style)
✅ The RAGAS metrics clear the thresholds in the evaluation against the golden dataset
✅ p95 latency ≤500ms at concurrency=50
✅ CI/CD runs with no manual intervention; at least 5 PRs went through the pipeline
✅ The final README explains the architecture with a diagram, the technical decisions, and the quantitative results
✅ The repo is demonstrable as a portfolio project

The final M1 → M8 comparison table (mandatory)

This table closes the narrative of the whole guide. It's the first thing a reviewer looks at:

Module	Technique added	Precision@5	Recall@20	Faithfulness	p95 (ms)
M1 (baseline)	Simple RAG	0.65	0.58	0.74	800
M2	Chunking optimization	0.71	0.62	0.79	760
M3	Query expansion	0.74	0.78	0.82	870
M4	Cross-encoder rerank	0.85	0.78	0.88	990
M5	Hybrid + RRF	0.88	0.84	0.89	940
M6	Metadata filtering	0.89	0.84	0.90	720
M7	Pinecone production	0.89	0.84	0.91	350
M8	Eval-driven tuning	0.91	0.86	0.93	350

Replace these numbers with your real ones. The structure matters more than the specific values: every row must show a module's quantitative contribution.

Grading rubric (100 points)

Technical integration (40 pts)

(15 pts) A working end-to-end pipeline with every stage
(10 pts) Multi-tenancy with TenantContext and tested isolation
(10 pts) Hybrid + reranking + filtering correctly composed
(5 pts) Observability with per-stage tracing

Evaluation (30 pts)

(10 pts) A versioned golden dataset with 50+ balanced queries
(10 pts) An automated evaluation pipeline, smoke + full modes
(10 pts) Regression testing with a baseline and calibrated thresholds

Operations (20 pts)

(10 pts) Working CI/CD: smoke on the PR, full nightly, diff comments
(10 pts) Error handling with retry, timeouts, fallbacks

Technical communication (10 pts)

(10 pts) A professional README with the comparison table, documented decisions, diagrams

Extra credit (+15)

(+5 pts) A simple dashboard of historical metrics (Grafana or Streamlit)
(+5 pts) Segmented evaluation (easy/medium/hard, by category)
(+5 pts) An automatic Slack report when the nightly eval fails

Common mistakes and how to avoid them

Integrating modules without measuring the real impact → every technique must add a row to the comparison table with its delta. If you can't show the quantitative gain, the technique isn't contributing.
A trivial dataset → 50% of the golden dataset must be sampled from real traffic, not invented. Perfect metrics on a trivial dataset are theater.
Made-up thresholds → calibrate empirically with 5+ stable runs (M08/06). Thresholds copied from blogs don't apply to your system.
CI/CD with no blocking → if the quality gate doesn't block the merge, it isn't a gate, it's decoration. --enforce-thresholds must be active.
A README with no quantitative narrative → "the system improved" is useless. What counts is "faithfulness 0.74 → 0.93, validated against 100 queries". Pin the comparison table to the top of the README.
No rollback plan → if a PR passes the quality gates but breaks in production, how do you go back? Document the procedure in DECISION.md.
Ignoring segmentation → global metrics hide problems. Report by difficulty: if "hard" drops 15% but "easy" rises 5%, the average is lying to you.

Documents to deliver

`README.md`

The architecture diagram
The M1 → M8 comparison table
Setup in 5 minutes
How to run the evaluation
How to deploy

`BENCHMARK.md` (from M07)

ChromaDB vs Pinecone latency
Throughput under concurrency
A 12-month cost estimate

`DECISION.md` (from M07)

The trade-offs evaluated
The recommendation with quantitative reasons
The rollback plan

`EVAL.md` (new in M08)

How the golden dataset was built
The threshold calibration methodology
Historical quality reports

`golden_dataset/PROCESS.md`

Who annotated it, when, and by what criteria
How it gets maintained (the quarterly cycle)

Resources for the project

RAGAS Documentation - The evaluation framework.
Pinecone Production Guides - Operating the vector DB.
GitHub Actions Documentation - CI/CD.
FastAPI Production Patterns - Deploying APIs.
structlog - Structured logging.
LangChain Retrieval Concepts - Retrieval patterns.
Anthropic on Building Eval-Driven Products - The operational philosophy.

Closing the guide

With this project you close a complete progression. You started by building a simple RAG. You learned why it fails in production. You applied advanced techniques: intelligent chunking, query optimization, hybrid search, re-ranking, metadata filtering. You migrated to production infrastructure. And you added the evaluation discipline that turns a system "that works" into a system "that keeps working".

What you have now is a concrete professional capability: designing, implementing, scaling and evaluating advanced RAG systems with objective quality criteria. That capability is scarce in the industry — most teams are stuck on simple RAG without knowing why it fails and without metrics to diagnose it.

The natural next step is to specialize in the frontier extensions:

Agentic RAG: systems where the LLM decides when and how to retrieve
Multimodal retrieval: images, video, audio in the same pipeline
Adaptive retrieval: the system learns from feedback and adjusts its strategies
Multi-step reasoning: queries that require multiple chained retrievals
Self-RAG and reflective retrieval: the system evaluates its own quality and retries

But before jumping to the frontier, prove you've mastered the fundamentals by publishing this project: a clear README, clean code, honest metrics, active CI/CD. It's the strongest proof of your ability.

Congratulations on making it here. The difference between the person who started the guide and the one who finishes it is real and measurable — the numbers in your final comparison table tell the story.

Created: March 13, 2026
Version: 2.0

← PreviousEvaluation CI/CD with GitHub Actions