8. Final Project — Production AI Pipeline

Overview

This is THE deliverable of the whole guide. A production-grade CI/CD pipeline that takes your AI application from a commit all the way to production, fully automated. It's not an academic exercise — it's a functional pipeline that you'll run end-to-end in your repository.

The pipeline has to work in two scenarios:

1. The happy path: Commit → lint → test → AI checks → Docker build → push → deploy staging → smoke tests → approval → deploy production → validation → success
2. The error path: Commit → ... → deploy production → the validation fails → an automatic rollback → a notification

If both scenarios work, you have a portfolio-worthy pipeline that demonstrates competence in CI/CD for AI systems.

The project's objectives

By completing this project:

• ✅ A complete, functional pipeline YAML that runs end-to-end
• ✅ At least one successful run (a documented happy path)
• ✅ At least one run with a failure + rollback (a documented error path)
• ✅ Documentation: the pipeline README, a flow diagram, a runbook
• ✅ A cost report as an artifact on every deployment
• ✅ Slack notifications for failures, rollbacks, and production deploys
• ✅ A nightly health check and a weekly report as auxiliary workflows

Prerequisites

The prior verification

# 1. Verify that your repo has the necessary tools
git --version
docker --version
python --version  # 3.12+

# 2. Verify the GitHub CLI
gh auth status

# 3. Verify the project's structure
ls -la .github/workflows/
ls -la scripts/
ls -la src/
ls -la tests/
cat Dockerfile
cat docker-compose.yml
cat requirements.txt

The required secrets

The required environments

Your repo → Settings → Environments

1. Click "New environment" → Name: staging
   → Add variable: STAGING_HOST = your-staging-server.com
   → Save

2. Click "New environment" → Name: production
   → Add variable: PRODUCTION_HOST = your-server.com
   → Required reviewers: add your username
   → Save

The project's final structure

my-ai-project/
├── .github/
│   ├── actions/
│   │   └── setup-ai-project/
│   │       └── action.yml                 ← Composite action
│   ├── workflows/
│   │   ├── production-pipeline.yml        ← Main pipeline
│   │   ├── nightly-health.yml             ← Nightly health check
│   │   ├── weekly-report.yml              ← Weekly report
│   │   └── manual-rollback.yml            ← Manual rollback
│   └── PIPELINE.md                        ← Pipeline documentation
├── scripts/
│   ├── prompt_regression.py               ← Prompt regression check
│   ├── cost_estimation.py                 ← Cost estimation
│   └── pipeline_metrics.py                ← Health metrics
├── baselines/
│   └── current.json                       ← Prompt baselines
├── prompts.json                           ← Prompt definitions
├── src/
│   └── main.py
├── tests/
│   └── test_main.py
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── README.md

Step 1: The Composite Action

Create .github/actions/setup-ai-project/action.yml:

name: "Setup AI Project"
description: "Checkout, setup Python, install dependencies with caching"

inputs:
  python-version:
    description: "Python version"
    required: false
    default: "3.12"
  install-dev:
    description: "Install development dependencies"
    required: false
    default: "true"

runs:
  using: "composite"
  steps:
    - name: Checkout
      uses: actions/checkout@v4

    - name: Setup Python
      uses: actions/setup-python@v5
      with:
        python-version: ${{ inputs.python-version }}
        cache: pip

    - name: Install dependencies
      shell: bash
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        if [ "${{ inputs.install-dev }}" = "true" ]; then
          pip install ruff pytest pytest-cov
        fi

Step 2: The Main Pipeline

Create .github/workflows/production-pipeline.yml:

name: Production AI Pipeline

on:
  push:
    branches: [main]
    paths-ignore:
      - "*.md"
      - "docs/**"
      - ".github/PIPELINE.md"
  pull_request:
    branches: [main]
  workflow_dispatch:

env:
  PYTHON_VERSION: "3.12"
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

# ══════════════════════════════════════════════════════════════
# STAGE 1: CI CHECKS
# ══════════════════════════════════════════════════════════════
jobs:
  lint:
    name: "Lint & Format"
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: ./.github/actions/setup-ai-project
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      - run: ruff check src/
      - run: ruff format --check src/

  test:
    name: "Tests"
    runs-on: ubuntu-latest
    timeout-minutes: 15
    outputs:
      coverage: ${{ steps.cov.outputs.pct }}
    steps:
      - uses: ./.github/actions/setup-ai-project
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      - name: Run tests
        run: pytest tests/ -v --tb=short --cov=src --cov-report=term
      - name: Extract coverage
        id: cov
        run: |
          COV=$(pytest tests/ --cov=src --cov-report=term 2>/dev/null | \
            grep "TOTAL" | awk '{print $NF}' | tr -d '%' || echo "0")
          echo "pct=$COV" >> $GITHUB_OUTPUT

  ai-checks:
    name: "AI Quality Checks"
    runs-on: ubuntu-latest
    timeout-minutes: 15
    outputs:
      cost-estimate: ${{ steps.cost.outputs.estimate }}
    steps:
      - uses: ./.github/actions/setup-ai-project
        with:
          python-version: ${{ env.PYTHON_VERSION }}

      - name: Prompt regression
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python scripts/prompt_regression.py --mode check

      - name: Cost estimation
        id: cost
        run: |
          python scripts/cost_estimation.py \
            --prompts prompts.json \
            --threshold 0.20 \
            --output cost-report.json
          COST=$(python -c "
          import json
          with open('cost-report.json') as f:
              r = json.load(f)
          print(r['projections']['per_request'])
          ")
          echo "estimate=$COST" >> $GITHUB_OUTPUT

      - name: Upload cost report
        uses: actions/upload-artifact@v4
        with:
          name: cost-report-${{ github.run_number }}
          path: cost-report.json
          retention-days: 90

# ══════════════════════════════════════════════════════════════
# PR SUMMARY (only on pull requests)
# ══════════════════════════════════════════════════════════════
  pr-summary:
    name: "PR Summary"
    needs: [lint, test, ai-checks]
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - name: Generate summary
        run: |
          echo "## PR Check Results" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          echo "| Check | Result |" >> $GITHUB_STEP_SUMMARY
          echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
          echo "| Lint | ${{ needs.lint.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Tests | ${{ needs.test.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| AI Checks | ${{ needs.ai-checks.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Coverage | ${{ needs.test.outputs.coverage }}% |" >> $GITHUB_STEP_SUMMARY
          echo "| Cost/req | \$${{ needs.ai-checks.outputs.cost-estimate }} |" >> $GITHUB_STEP_SUMMARY

# ══════════════════════════════════════════════════════════════
# STAGE 2: DOCKER BUILD & PUSH (main only)
# ══════════════════════════════════════════════════════════════
  docker:
    name: "Docker Build & Push"
    needs: [lint, test, ai-checks]
    runs-on: ubuntu-latest
    timeout-minutes: 15
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    permissions:
      contents: read
      packages: write
    outputs:
      image-tag: ${{ steps.meta.outputs.tag }}
      image-full: ${{ steps.meta.outputs.full }}

    steps:
      - uses: actions/checkout@v4

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Image metadata
        id: meta
        run: |
          SHORT_SHA="${GITHUB_SHA::7}"
          TAG="sha-${SHORT_SHA}"
          FULL="${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${TAG}"
          echo "tag=$TAG" >> $GITHUB_OUTPUT
          echo "full=$FULL" >> $GITHUB_OUTPUT

      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: |
            ${{ steps.meta.outputs.full }}
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

# ══════════════════════════════════════════════════════════════
# STAGE 3: DEPLOY STAGING
# ══════════════════════════════════════════════════════════════
  deploy-staging:
    name: "Deploy Staging"
    needs: docker
    runs-on: ubuntu-latest
    timeout-minutes: 10
    environment:
      name: staging
      url: https://staging.your-app.com

    steps:
      - uses: actions/checkout@v4
      - name: Deploy to staging
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.STAGING_HOST }}
        run: |
          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && IMAGE_TAG=${{ needs.docker.outputs.image-tag }} \
             docker compose up -d --pull always"
          rm /tmp/key

# ══════════════════════════════════════════════════════════════
# STAGE 4: SMOKE TESTS
# ══════════════════════════════════════════════════════════════
  smoke-tests:
    name: "Smoke Tests"
    needs: deploy-staging
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - name: Health check
        run: |
          sleep 15
          for i in $(seq 1 20); do
            if curl -sf https://staging.your-app.com/health > /dev/null 2>&1; then
              echo "Staging healthy"
              exit 0
            fi
            sleep 3
          done
          echo "::error::Staging health check failed"
          exit 1

      - name: Prompt test
        run: |
          RESPONSE=$(curl -sf -X POST https://staging.your-app.com/api/chat \
            -H "Content-Type: application/json" \
            -d '{"message":"ping","max_tokens":5}' \
            --max-time 30 2>/dev/null || echo "")
          if [ -n "$RESPONSE" ]; then
            echo "Prompt test passed"
          else
            echo "::error::Prompt test failed"
            exit 1
          fi

# ══════════════════════════════════════════════════════════════
# STAGE 5: APPROVAL
# ══════════════════════════════════════════════════════════════
  approve:
    name: "Production Approval"
    needs: smoke-tests
    runs-on: ubuntu-latest
    environment:
      name: production
    steps:
      - run: echo "Approved by ${{ github.actor }}"

# ══════════════════════════════════════════════════════════════
# STAGE 6: DEPLOY PRODUCTION + VALIDATION + ROLLBACK
# ══════════════════════════════════════════════════════════════
  deploy-production:
    name: "Deploy Production"
    needs: [approve, docker]
    runs-on: ubuntu-latest
    timeout-minutes: 10
    outputs:
      previous-tag: ${{ steps.current.outputs.tag }}
      deploy-status: ${{ steps.result.outputs.status }}

    steps:
      - uses: actions/checkout@v4

      - name: Save current version
        id: current
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.PRODUCTION_HOST }}
        run: |
          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          CURRENT=$(ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && docker compose ps --format '{{.Image}}'" 2>/dev/null | \
            head -1 | cut -d: -f2 || echo "unknown")
          rm /tmp/key
          echo "tag=${CURRENT}" >> $GITHUB_OUTPUT
          echo "Current: ${CURRENT}"

      - name: Deploy new version
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.PRODUCTION_HOST }}
        run: |
          echo "Deploying: ${{ needs.docker.outputs.image-tag }}"
          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && IMAGE_TAG=${{ needs.docker.outputs.image-tag }} \
             docker compose up -d --pull always"
          rm /tmp/key

      - name: Post-deploy validation
        id: validate
        continue-on-error: true
        run: |
          sleep 15
          for i in $(seq 1 30); do
            STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
              https://your-app.com/health 2>/dev/null || echo "000")
            if [ "$STATUS" = "200" ]; then
              RESPONSE=$(curl -sf -X POST https://your-app.com/api/chat \
                -H "Content-Type: application/json" \
                -d '{"message":"ping","max_tokens":5}' \
                --max-time 30 2>/dev/null || echo "")
              if [ -n "$RESPONSE" ]; then
                echo "Validation passed"
                exit 0
              fi
              echo "Health OK but prompt failed"
              exit 1
            fi
            sleep 2
          done
          echo "Health check failed"
          exit 1

      - name: Rollback on failure
        if: steps.validate.outcome == 'failure'
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.PRODUCTION_HOST }}
        run: |
          PREV="${{ steps.current.outputs.tag }}"
          if [ "$PREV" = "unknown" ] || [ -z "$PREV" ]; then
            echo "::error::No previous version — cannot rollback"
            exit 1
          fi
          echo "ROLLBACK to: $PREV"
          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && IMAGE_TAG=$PREV docker compose up -d --pull always"
          rm /tmp/key
          sleep 15
          if curl -sf https://your-app.com/health > /dev/null 2>&1; then
            echo "Rollback successful"
          else
            echo "::error::CRITICAL — Rollback also failed!"
          fi
          exit 1

      - name: Set result
        id: result
        if: always()
        run: |
          if [ "${{ steps.validate.outcome }}" = "success" ]; then
            echo "status=success" >> $GITHUB_OUTPUT
          else
            echo "status=rolled-back" >> $GITHUB_OUTPUT
          fi

      - name: Summary
        if: always()
        run: |
          echo "## Production Deployment" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          if [ "${{ steps.validate.outcome }}" = "success" ]; then
            echo "✅ **Deployed:** \`${{ needs.docker.outputs.image-tag }}\`" >> $GITHUB_STEP_SUMMARY
          else
            echo "❌ **Failed:** \`${{ needs.docker.outputs.image-tag }}\`" >> $GITHUB_STEP_SUMMARY
            echo "⚠️ **Rolled back to:** \`${{ steps.current.outputs.tag }}\`" >> $GITHUB_STEP_SUMMARY
          fi

# ══════════════════════════════════════════════════════════════
# NOTIFICATIONS
# ══════════════════════════════════════════════════════════════
  notify:
    name: "Notifications"
    needs: [lint, test, ai-checks, docker, deploy-production]
    runs-on: ubuntu-latest
    if: always() && github.ref == 'refs/heads/main'
    steps:
      - name: Determine status
        id: status
        run: |
          if [ "${{ needs.deploy-production.result }}" = "success" ]; then
            echo "msg=🚀 Production deploy successful: ${{ needs.docker.outputs.image-tag }}" >> $GITHUB_OUTPUT
            echo "notify=true" >> $GITHUB_OUTPUT
          elif [ "${{ needs.deploy-production.result }}" = "failure" ]; then
            echo "msg=⚠️ Deploy failed — rollback executed" >> $GITHUB_OUTPUT
            echo "notify=true" >> $GITHUB_OUTPUT
          elif [ "${{ contains(needs.*.result, 'failure') }}" = "true" ]; then
            echo "msg=❌ CI Pipeline failed on main" >> $GITHUB_OUTPUT
            echo "notify=true" >> $GITHUB_OUTPUT
          else
            echo "notify=false" >> $GITHUB_OUTPUT
          fi

      - name: Send Slack notification
        if: steps.status.outputs.notify == 'true'
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "${{ steps.status.outputs.msg }}",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "${{ steps.status.outputs.msg }}\n*Commit:* `${{ github.sha }}`\n*Author:* ${{ github.actor }}\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Run>"
                  }
                }
              ]
            }

      - name: Final summary
        if: always()
        run: |
          echo "## Pipeline Summary" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
          echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
          echo "| Lint | ${{ needs.lint.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Test | ${{ needs.test.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| AI Checks | ${{ needs.ai-checks.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Docker | ${{ needs.docker.result || 'skipped' }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Deploy | ${{ needs.deploy-production.result || 'skipped' }} |" >> $GITHUB_STEP_SUMMARY

Step 3: The Auxiliary Workflows

The Nightly Health Check

Create .github/workflows/nightly-health.yml:

name: Nightly Health Check

on:
  schedule:
    - cron: "0 4 * * *"
  workflow_dispatch:

jobs:
  health:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt

      - name: Run tests
        id: tests
        continue-on-error: true
        run: pytest tests/ -v --tb=short

      - name: Check baselines
        id: baselines
        continue-on-error: true
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python scripts/prompt_regression.py --mode check

      - name: Pipeline metrics
        id: metrics
        continue-on-error: true
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          python scripts/pipeline_metrics.py \
            --owner ${{ github.repository_owner }} \
            --repo ${{ github.event.repository.name }} \
            --count 30 --json > health.json

      - name: Summary
        if: always()
        run: |
          echo "## Nightly Health" >> $GITHUB_STEP_SUMMARY
          echo "| Check | Status |" >> $GITHUB_STEP_SUMMARY
          echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
          echo "| Tests | ${{ steps.tests.outcome }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Baselines | ${{ steps.baselines.outcome }} |" >> $GITHUB_STEP_SUMMARY

      - name: Alert if issues
        if: steps.tests.outcome == 'failure' || steps.baselines.outcome == 'failure'
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "🚨 Nightly health check found issues"
            }

The Weekly Report

Create .github/workflows/weekly-report.yml:

name: Weekly Report

on:
  schedule:
    - cron: "0 9 * * 1"
  workflow_dispatch:

jobs:
  report:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Generate report
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          python scripts/pipeline_metrics.py \
            --owner ${{ github.repository_owner }} \
            --repo ${{ github.event.repository.name }} \
            --count 50

          python scripts/pipeline_metrics.py \
            --owner ${{ github.repository_owner }} \
            --repo ${{ github.event.repository.name }} \
            --count 50 --json > weekly.json

          echo "## Weekly Pipeline Report" >> $GITHUB_STEP_SUMMARY
      - uses: actions/upload-artifact@v4
        with:
          name: weekly-report-${{ github.run_number }}
          path: weekly.json
          retention-days: 90

The Manual Rollback

Create .github/workflows/manual-rollback.yml:

name: Manual Rollback

on:
  workflow_dispatch:
    inputs:
      image-tag:
        description: "Image tag to deploy (e.g., sha-abc1234)"
        required: true
        type: string
      environment:
        description: "Target environment"
        required: true
        type: choice
        options:
          - staging
          - production

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment:
      name: ${{ github.event.inputs.environment }}
    steps:
      - name: Deploy specified version
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.PRODUCTION_HOST }}
        run: |
          echo "Manual rollback: ${{ github.event.inputs.image-tag }} → ${{ github.event.inputs.environment }}"
          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && IMAGE_TAG=${{ github.event.inputs.image-tag }} docker compose up -d --pull always"
          rm /tmp/key

      - name: Verify
        run: |
          sleep 15
          HOST="${{ github.event.inputs.environment == 'production' && vars.PRODUCTION_HOST || vars.STAGING_HOST }}"
          for i in $(seq 1 15); do
            if curl -sf "https://${HOST}/health" > /dev/null 2>&1; then
              echo "Rollback successful"
              exit 0
            fi
            sleep 2
          done
          echo "::error::Health check failed after rollback"
          exit 1

      - name: Notify
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "🔄 Manual rollback to ${{ github.event.inputs.image-tag }} on ${{ github.event.inputs.environment }} by @${{ github.actor }}"
            }

Step 4: The Pipeline Documentation

Create .github/PIPELINE.md:

# Production AI Pipeline

## Overview

A production-grade CI/CD pipeline for [project-name]. It automates
from the commit all the way to production with AI quality checks, a Docker build,
a staged deployment, an automatic rollback, and monitoring.

## Architecture

```
     Lint ──┐
     Test ──┼──► Docker ──► Staging ──► Smoke ──► Approve ──► Production
  AI Checks ┘                                                     │
                                                             fail → Rollback
```

## Workflows

| File | Trigger | Description |
|------|---------|-------------|
| production-pipeline.yml | push main, PR | Full CI/CD pipeline |
| nightly-health.yml | cron 4am UTC | Tests + baselines check |
| weekly-report.yml | cron Mon 9am | Pipeline health metrics |
| manual-rollback.yml | manual | Deploy specific version |

## Secrets

| Name | Required | Description |
|------|----------|-------------|
| OPENAI_API_KEY | Yes | For AI checks |
| SLACK_WEBHOOK_URL | Yes | For notifications |
| DEPLOY_SSH_KEY | Yes | For deployment |

## Troubleshooting

See [Runbook section in the full documentation].

Step 5: End-to-End Validation

Test 1: The Happy Path

# 1. Make a valid change in src/main.py
echo "# Updated: $(date)" >> src/main.py

# 2. Commit and push
git add .
git commit -m "test: validate production pipeline (happy path)"
git push origin main

# 3. Watch it in GitHub Actions:
#    - Lint ✅
#    - Test ✅
#    - AI Checks ✅
#    - Docker Build ✅
#    - Deploy Staging ✅
#    - Smoke Tests ✅
#    - Approval Gate ⏸️ (waiting for the approval)

# 4. Approve the deploy in the GitHub Actions UI

# 5. Watch:
#    - Deploy Production ✅
#    - Validation ✅
#    - Notification: "🚀 Production deploy successful"

Test 2: The Error Path (Rollback)

To test the rollback, you need the post-deploy validation to fail. The safest way is to temporarily make the health check fail:

# Add this temporarily to the deploy-production job:
      - name: Force validation failure (TESTING)
        run: exit 1

# 1. Add the force-failure step
# 2. Commit and push
git add .
git commit -m "test: validate rollback path"
git push origin main

# 3. Approve the deploy when it reaches the approval gate

# 4. Watch:
#    - Deploy Production → Validation ❌ → Rollback ✅
#    - Notification: "⚠️ Deploy failed — rollback executed"

# 5. IMPORTANT: Revert the change
git revert HEAD
git push origin main

Test 3: The Nightly Health Check

# Run it manually
gh workflow run nightly-health.yml

# Verify it
gh run list --workflow=nightly-health.yml --limit 1

Success criteria

The mandatory checklist

The quality criteria

The project's troubleshooting

"The pipeline fails on the first step"

The most common cause: The composite action can't be found.

Solution: Verify that the file structure is:

.github/actions/setup-ai-project/action.yml

And that the reference is uses: ./.github/actions/setup-ai-project (with the ./ at the start).

"The Docker build fails with permission denied"

Cause: The job doesn't have permissions: packages: write.

Solution: Add this to the docker job:

permissions:
  contents: read
  packages: write

"The approval gate doesn't show up"

Cause: The production environment doesn't have required reviewers.

Solution: Settings → Environments → production → Required reviewers → add at least 1 reviewer.

"The notifications don't arrive"

Cause: The SLACK_WEBHOOK_URL secret doesn't exist or is invalid.

Solution:

# Verify the webhook
curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"test"}' \
  $SLACK_WEBHOOK_URL

"The rollback doesn't activate"

Cause: The validation step doesn't have continue-on-error: true or the rollback step doesn't have if: steps.validate.outcome == 'failure'.

Solution: Verify both conditions in the YAML.

"The pipeline takes more than 20 minutes"

Cause: A Docker build without a cache or AI checks with too many test cases.

Solution:

• Docker: Verify cache-from: type=gha and cache-to: type=gha,mode=max
• AI checks: Reduce the test cases or use gpt-4o-mini (faster)

What you built

BEFORE this guide:
  push → manual tests → a manual Docker build → ssh → docker compose up → 🤞

AFTER this guide:
  push → automatic lint → automatic tests → AI quality checks →
  Docker build with caching → push to GHCR →
  deploy staging → smoke tests → manual approval →
  deploy production → post-deploy validation →
  auto-rollback if fails → Slack notifications → cost tracking →
  nightly health checks → weekly reports → pipeline documentation

  All automatic. All monitored. All documented.

What comes next

You've completed the CI/CD for AI Systems guide. Your pipeline is production-grade within the scope of GitHub Actions + Docker Compose. The next steps:

Guide #17: Deployment & Cloud Infrastructure

Replace Docker Compose with a real deploy to the cloud:

THIS GUIDE:                   GUIDE #17:
Local Docker Compose    →     AWS ECS / GCP Cloud Run
An SSH deploy           →     Infrastructure as Code (Terraform)
A single server         →     Auto-scaling

Guide #18: Monitoring & Observability

Add monitoring of the application at runtime:

THIS GUIDE:                   GUIDE #18:
Monitor the pipeline    →     Monitor the app in production
Detect failures         →     Detect latency, errors, cost drift
Slack alerts            →     Dashboards, alerting, SLOs

Your pipeline integrates directly with both guides. What you built here is the foundation — the following guides add capabilities on top of this base.

Summary

• ✅ The complete pipeline: lint → test → AI checks → Docker → staging → smoke → approve → production
• ✅ Automatic rollback: the validation fails → a rollback to the previous version → notify
• ✅ Cost monitoring: an estimate per deployment, a comparison, alerts
• ✅ Notifications: Slack for failures, rollbacks, and production deploys
• ✅ The auxiliary workflows: nightly health, the weekly report, a manual rollback
• ✅ Documentation: the pipeline README, a diagram, a runbook
• ✅ A composite action: a centralized, reusable setup
• ✅ Tested end-to-end: the happy path (success) + the error path (rollback)
• ✅ Portfolio-worthy: it demonstrates competence in CI/CD for AI systems

Additional resources

1. GitHub Actions — Complete Reference - The complete reference
2. DORA Metrics - The standard metrics for evaluating DevOps maturity
3. docker/build-push-action - A Docker build optimized for Actions
4. slackapi/slack-github-action - The official notifications
5. GitHub Actions Security Best Practices - Security in Actions
6. Continuous Delivery by Jez Humble - The fundamental CD book