Module 8: RAG Evaluation + The Capstone Project
Module Introduction: RAG Evaluation
Capsule description
You've reached the guide's final module. You already have an advanced RAG system: intelligent chunking, query optimization, hybrid search, re-ranking, metadata filtering, production infrastructure on Pinecone with multi-tenancy. Your system works, it serves real traffic, it scales, and it respects isolation.
But there's one question you still can't answer: does it answer well?
Not "is it fast?" — you answered that with benchmarks in M07. Not "is it secure?" either — you covered that with TenantContext and namespaces. The question is the most fundamental one, and the hardest to measure: are the answers it returns correct, grounded in the right documents, and useful to the user?
Without an evaluation system, the answer to "does it answer well?" is an opinion. "I tried three queries and it looked good" isn't evaluation. "The product team says it's nice" isn't evaluation. "The users aren't complaining" is definitely not evaluation — quiet users aren't satisfied users, they're users who already left for another product.
This module teaches you to transform your RAG system from an impressive demo into a professional product with continuous evaluation, reproducible metrics, versioned golden datasets, and CI/CD quality gates that block degradations before they touch production.
By the end you'll have the operational discipline that separates teams who ship models to production and pray, from teams who ship models with data-backed confidence.
Module 8's objectives
By the close of this module you'll be able to:
- ✅ Explain why manual evaluation doesn't scale and what "rigorous evaluation" looks like for a RAG system
- ✅ Implement RAGAS's core metrics: faithfulness, answer relevancy, context precision, context recall
- ✅ Build a golden dataset representative of the real traffic, not of made-up queries
- ✅ Automate the evaluation pipeline so it runs in minutes, not hours
- ✅ Define defensible thresholds that block real degradations without generating noise
- ✅ Integrate evaluation into GitHub Actions so every PR shows its impact on quality
- ✅ Deliver a capstone project that brings together everything you learned in the guide
Module roadmap
| Capsule | Topic | The outcome |
|---|---|---|
| 01 | Introduction | The closing map of the system |
| 02 | Evaluation fundamentals | The mental framework: what to measure and why |
| 03 | Core RAGAS metrics | Faithfulness, relevancy, precision, recall |
| 04 | The golden dataset | A versioned dataset with ground truth |
| 05 | The automated pipeline | A reproducible evaluation script |
| 06 | Regression testing | Thresholds and quality gates |
| 07 | CI/CD with GitHub Actions | Evaluation on every PR |
| 08 | The final project | An Advanced RAG System with evaluation |
The guide's closing context
Your progress through the guide covers four phases:
- M1-M3: the fundamentals of advanced RAG (architecture, chunking, query optimization)
- M4-M6: retrieval precision (re-ranking, hybrid search, metadata filtering)
- M7: production infrastructure (Pinecone, multi-tenancy, benchmarking)
- M8: systematic measurement and operational discipline
Without this last module, everything before it is pretty code with no guarantees. With this module, you have a system that improves on evidence, not on hope.
The problem this module solves
Consider the typical scenario of a team with no systematic evaluation:
Monday: someone opens a PR changing the generation prompt. CI is green, the unit tests pass. Merge.
Wednesday: users report the answers feel "vaguer".
Thursday: the team bisects git looking for the change. Three suspicious PRs. Nobody knows which one.
Friday: they revert blindly and pray. The following Monday: the problem comes back with a different PR.
Versus a team with systematic evaluation:
Monday: a PR changing the prompt. CI runs the evaluation against the golden dataset.
Monday 14:23: the bot comments "faithfulness dropped 0.87 → 0.79 (-9%)". PR blocked.
Monday 14:25: the author reviews it, finds the new prompt accepts unsupported inferences. Adjusts it.
Monday 14:40: faithfulness 0.88. Green. Merge.
The difference isn't talent or effort: it's operational discipline translated into automation.
Connection with the final project
Your module project is the Advanced RAG System: the consolidation of the whole guide into a production-ready repository.
The deliverables:
- The complete RAG pipeline integrated (chunking + expansion + hybrid + rerank + filter + generate)
- A versioned golden dataset with 100+ representative queries and ground truth
- An automated evaluation pipeline with RAGAS
- Calibrated thresholds and quality gates in GitHub Actions
- A dashboard of historical metrics to catch drift
This is the portfolio piece that separates "I studied RAG" from "I operated RAG with discipline".
The module's limits
To keep the module focused, we don't cover:
- ❌ Training your own evaluators (a fine-tuned LLM-as-judge). We'll use GPT-4o-mini as the judge, which is enough for 95% of cases.
- ❌ Academic multi-dataset benchmarking (BEIR, MTEB). Your golden dataset is specific to your product.
- ❌ Full-stack infrastructure observability (distributed tracing, profiling). We cover RAG quality metrics, not system metrics.
- ❌ Human evaluation at scale (annotation with panels). We assume you maintain 100-200 queries with manual ground truth; tools like Label Studio are out of scope.
Evidence of success
You'll know you've mastered the module when:
- ✅ You can run the full evaluation in under 5 minutes over 100 queries
- ✅ You catch quality degradations before the merge, not after the deploy
- ✅ Your final README reports baseline metrics (a simple RAG) vs advanced (this system) with concrete numbers
- ✅ You can tell the guide's "quantitative story": "M3 query expansion +12% recall, M4 reranking +8% precision, M5 hybrid +5% recall on technical queries..."
- ✅ Any PR that touches retrieval or generation triggers an automatic evaluation
Summary
- Evaluation is part of the product, not an optional final task
- Reproducible metrics + a golden dataset + automation = operational discipline
- CI/CD with quality gates prevents silent degradations
- This module closes the guide with a portfolio-ready, professional system
- The difference between an "impressive demo" and a "serious product" is exactly this
Additional resources
- RAGAS Documentation - The official RAG evaluation framework.
- RAG Evaluation - Humanloop - A practical approach for production.
- OpenAI Evals - A general LLM evaluation framework.
- GitHub Actions Documentation - Automation in your repository.
- Hidden Technical Debt in ML Systems - Why evaluation matters more than architecture.
- Evaluating LLM Applications - Eugene Yan's exhaustive guide.
Created: March 13, 2026
Version: 2.0