Evaluation CI/CD with GitHub Actions

Capsule description

You have an evaluation pipeline, a golden dataset and calibrated quality gates. But there's a law of human nature in software: if the evaluation doesn't run automatically, eventually it stops being run. Three weeks after implementing it, someone has an urgent delivery, skips the evaluation "just this once", nothing bad appears to happen, and it becomes a habit. Three months later nobody runs the evaluation and you're back to the initial state with no safety net.

The solution is to eliminate the option of skipping the evaluation. The professional way is to integrate it into CI/CD: every pull request automatically triggers the evaluation, the results get commented on the PR, and degradations block the merge. It doesn't depend on anyone remembering — it's part of the infrastructure.

In this capsule you're going to configure GitHub Actions so your RAG system has the same level of operational rigor as a traditional software system: smoke tests on every PR, a full nightly evaluation, reports published as browsable artifacts, automatic comments with the metric changes vs main.

By the end you'll have a repository with RAG quality evaluation running automatically, with no way to skip it. It's the operational close that separates "I have evaluation" from "my team operates with evaluation discipline".

The philosophy: CI layers by speed and coverage

Not every check can run on every PR. Cost, time and rate limits matter. Structure it in layers:

This structure means:

• The developer gets feedback in 2 min for normal PRs
• Risky changes (to main) get a deeper evaluation
• Releases go through an evaluation with a premium judge
• Drift in production gets caught weekly

The base workflow: smoke on every PR

# .github/workflows/rag-eval-smoke.yml
name: RAG Eval - Smoke

on:
  pull_request:
    paths:
      - 'app/**'
      - 'eval/**'
      - 'golden_dataset/**'
      - 'pyproject.toml'

jobs:
  smoke:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install dependencies
        run: pip install -e .[eval]

      - name: Run smoke evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
          PINECONE_INDEX: ${{ vars.PINECONE_INDEX_TEST }}
          OPENAI_SEED: '42'
        run: python scripts/evaluate.py --mode smoke --enforce-thresholds

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: smoke-report-${{ github.event.pull_request.number }}
          path: eval_reports/
          retention-days: 30

The key decisions:

• The paths filter: it only fires if the relevant directories change. Changes to the docs don't need to run the eval (it saves OpenAI costs).
• timeout-minutes: 5: it blocks infinite workflows from a bug in the code.
• if: always() on the upload: it uploads the report even if the run failed, so you can debug.
• OPENAI_SEED: '42': reproducibility between runs.
• A separate index for testing: PINECONE_INDEX_TEST keeps you from touching the production index.

The full workflow: a full nightly evaluation

# .github/workflows/rag-eval-nightly.yml
name: RAG Eval - Nightly Full

on:
  schedule:
    - cron: '0 3 * * *'  # 3am UTC daily
  workflow_dispatch:  # manual trigger

jobs:
  full:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      - run: pip install -e .[eval]

      - name: Run full evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
          PINECONE_INDEX: ${{ vars.PINECONE_INDEX_PROD }}
          OPENAI_SEED: '42'
        run: python scripts/evaluate.py --mode full --enforce-thresholds

      - name: Compare against baseline
        run: python scripts/compare_baseline.py --report eval_reports/latest.json

      - name: Upload to S3
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: |
          aws s3 cp eval_reports/ s3://rag-eval-reports/$(date +%Y-%m-%d)/ --recursive

      - name: Notify on failure
        if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "🚨 RAG nightly evaluation failed",
              "channel": "${{ secrets.SLACK_CHANNEL }}"
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

The differences from smoke:

• A cron schedule instead of a PR trigger
• The production index (PINECONE_INDEX_PROD) so you evaluate against the real state
• An upload to S3 for long retention (GitHub artifacts get deleted after 90 days)
• A Slack notification on failure — because it ran overnight and nobody is going to see the red in GitHub

The comparison workflow against main

To show the developer how the quality changed vs main, not just whether it passes the thresholds:

# .github/workflows/rag-eval-compare.yml
name: RAG Eval - Compare with main

on:
  pull_request:

jobs:
  compare:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - run: pip install -e .[eval]

      - name: Run smoke on PR branch
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
        run: |
          python scripts/evaluate.py --mode smoke
          mv eval_reports/latest.json eval_reports/pr.json

      - name: Run smoke on main
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
        run: |
          git checkout main
          python scripts/evaluate.py --mode smoke
          mv eval_reports/latest.json eval_reports/main.json

      - name: Generate comparison
        run: python scripts/diff_reports.py --pr eval_reports/pr.json --main eval_reports/main.json --output diff.md

      - name: Comment on PR
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: diff.md
          header: rag-eval-diff

The PR comment looks like this:

## 📊 RAG Quality Diff

| Metric | main | this PR | Δ |
|--------|------|---------|---|
| Faithfulness | 0.91 | 0.93 | ↑ +2.2% |
| Answer Relevancy | 0.87 | 0.86 | ↓ -1.1% |
| Context Precision | 0.82 | 0.85 | ↑ +3.7% |
| Context Recall | 0.85 | 0.84 | ↓ -1.2% |

**Verdict**: ✅ Net improvement (no regressions beyond tolerance)

This is what makes evaluation visceral for the developer — it isn't an abstract test, it's direct feedback on their PR.

Handling secrets and variables

GitHub distinguishes between secrets (encrypted, the value isn't visible) and variables (visible in the logs). Use them appropriately:

Configuration:

# Repo settings → Secrets and variables → Actions
# The "Secrets" tab:
gh secret set OPENAI_API_KEY
gh secret set PINECONE_API_KEY

# The "Variables" tab:
gh variable set PINECONE_INDEX_TEST --body "rag-eval-test"
gh variable set PINECONE_INDEX_PROD --body "rag-prod-v1"

The critical anti-patterns:

• Hardcoding keys in the YAML → automatic model expulsion and a key rotation
• Logging secrets with echo $OPENAI_API_KEY → they end up in the workflow's public logs
• Using production's OPENAI_API_KEY for CI → CI can exhaust the production quota. Use a separate key with its own quota.

Reducing CI costs

Evaluation with an LLM-as-judge costs money per run. For an active repo with 30 PRs/month:

Specific optimizations:

# Cancel old runs when a new push lands on the same PR
concurrency:
  group: rag-eval-${{ github.ref }}
  cancel-in-progress: true

# Pip cache to speed up the setup
- uses: actions/setup-python@v5
  with:
    python-version: '3.11'
    cache: 'pip'

# Embedding cache (if you reuse the golden set's queries)
- uses: actions/cache@v4
  with:
    path: .cache/embeddings/
    key: embeddings-${{ hashFiles('golden_dataset/v1.0.0.json') }}

Connection with the final project

Your Advanced RAG System must deliver:

.github/workflows/
├── rag-eval-smoke.yml         # On every PR
├── rag-eval-nightly.yml       # A daily cron
└── rag-eval-compare.yml       # A comment with the diff vs main

scripts/
├── evaluate.py                # The evaluation CLI
├── compare_baseline.py        # The regression check
└── diff_reports.py            # A comparison between runs

And in the README:

## Quality Gates

Every PR runs RAG smoke evaluation automatically. See latest results:
[![RAG Eval](https://github.com/org/repo/actions/workflows/rag-eval-smoke.yml/badge.svg)](...)

Nightly full evaluation: [latest report on S3](s3://rag-eval-reports/latest.html)

The badge in the README is the visible signal that the repo operates with evaluation discipline.

Comparison: CI without evaluation vs CI with RAG evaluation

Troubleshooting

Problem 1: "The workflow fails on a timeout"

The cause: a large dataset in the PR, or low concurrency on the runner.
The fix: smoke mode on the PR (not full). Raise timeout-minutes only if the full eval really does take more than 30 min — and if so, optimize the concurrency in execute_batch.

Problem 2: "The secret isn't found in CI"

The cause: the secret isn't configured, or the name is wrong.
The fix: check with gh secret list and compare the exact names. Remember that organization secrets aren't inherited automatically; they have to be granted to the repo.

Problem 3: "Unstable metrics: it passes locally, fails in CI"

The cause: different dependencies or a different seed.
The fix: pin exactly in pyproject.toml (ragas==0.1.x, not ragas>=0.1). Set OPENAI_SEED=42 and temperature=0 in the eval code. Verify the dataset_version is the same.

Problem 4: "The OpenAI costs exploded"

The cause: the workflow runs on every push (not just PRs), or it runs the full eval instead of smoke.
The fix: path filters, smoke on the PR, full only nightly. Use a separate key with a quota cap for CI. Consider gpt-4o-mini as the default judge.

Problem 5: "The diff comments flood the PR"

The cause: multiple runs comment separately.
The fix: use marocchino/sticky-pull-request-comment@v2 with a unique header — it updates the existing comment instead of creating new ones.

Problem 6: "The workflow runs on external forks with no secrets"

The cause: PRs from forks don't get access to the secrets, for security reasons.
The fix: the workflow detects this and skips the eval with a clear message. For PRs from external contributors, the eval runs when a maintainer types /eval or when it's merged into a feature branch.

Exercises

Exercise 1: A basic workflow for a PR

Create a minimal workflow that runs the smoke eval on every PR.

# .github/workflows/rag-eval.yml
name: RAG Eval

on:
  pull_request:
    paths:
      - 'app/**'
      - 'eval/**'
      - 'golden_dataset/**'

jobs:
  smoke:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      - run: pip install -e .[eval]
      - run: python scripts/evaluate.py --mode smoke --enforce-thresholds
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          OPENAI_SEED: '42'

Exercise 2: Uploading the report as an artifact

Extend the workflow to upload the JSON report with 30-day retention.

- name: Upload report
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: eval-report-pr-${{ github.event.pull_request.number }}
    path: |
      eval_reports/*.json
      eval_reports/*.md
    retention-days: 30

Exercise 3: An automatic PR comment with the diff

Implement a job that posts the diff vs main on every PR.

- name: Post diff comment
  uses: marocchino/sticky-pull-request-comment@v2
  with:
    header: rag-eval
    message: |
      ## 📊 RAG Eval Results
      
      | Metric | This PR | Threshold | Status |
      |--------|---------|-----------|--------|
      | Faithfulness | ${{ steps.eval.outputs.faithfulness }} | 0.85 | ${{ steps.eval.outputs.faithfulness_status }} |
      | Relevancy | ${{ steps.eval.outputs.relevancy }} | 0.80 | ${{ steps.eval.outputs.relevancy_status }} |

- id: eval
  run: |
    python scripts/evaluate.py --mode smoke --output-format github
    echo "faithfulness=$(jq .ragas.faithfulness eval_reports/latest.json)" >> $GITHUB_OUTPUT

Exercise 4: A schedule for the nightly full eval

Create a workflow that runs the full eval every night and notifies Slack if it fails.

name: RAG Eval Nightly

on:
  schedule:
    - cron: '0 3 * * *'
  workflow_dispatch:

jobs:
  full:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -e .[eval]
      - run: python scripts/evaluate.py --mode full --enforce-thresholds
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}

      - if: failure()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "🚨 RAG nightly eval failed: ${{ github.workflow_url }}"
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Summary

• CI/CD turns evaluation into permanent infrastructure, not a voluntary practice
• Layers by speed/coverage: smoke (PR), full (nightly), exhaustive (release), drift (weekly)
• Path filters, concurrency cancellation and caching cut the CI cost 10×
• Diff comments on the PR make evaluation visceral for the developer
• Secrets vs variables: secrets for credentials, variables for names
• A Slack notification on nightly failures: what you don't get told about doesn't get fixed
• The README badge is a public signal of operational maturity

Additional resources

1. GitHub Actions Documentation - The complete official reference.
2. Workflow Syntax - The YAML syntax in detail.
3. Encrypted Secrets - Handling credentials safely.
4. Sticky PR Comment Action - Comments that update in place.
5. Slack GitHub Action - Notifications to Slack.
6. Reusable Workflows - DRY across multiple repos.

Created: March 13, 2026
Version: 2.0