Module 8: Capstone Project — Production AI Pipeline

3. Full Pipeline Implementation

Overview

This is the module's central lesson — and probably the most important one in the whole guide. Here you implement the complete pipeline in a single YAML file: lint → test → AI checks → Docker build → push → deploy staging → smoke tests → approve → deploy production. It's not a sketch or a diagram — it's the functional workflow you'll run in your repository.

The YAML is long (~200 lines), but every section corresponds to a stage you already know from previous modules. What's new is how they connect: the outputs that flow from one job to another, the conditions that determine what runs and what gets skipped, and the error handling that allows recovery without manual intervention.

The complete pipeline

# .github/workflows/production-pipeline.yml
name: Production AI Pipeline

on:
  push:
    branches: [main]
    paths-ignore:
      - "*.md"
      - "docs/**"
      - ".github/workflows/nightly-*"
  pull_request:
    branches: [main]
  workflow_dispatch:

env:
  PYTHON_VERSION: "3.12"
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

# ══════════════════════════════════════════════════════════
# STAGE 1: CI CHECKS (parallel)
# ══════════════════════════════════════════════════════════
jobs:
  lint:
    name: "Lint & Format"
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}

      - name: Install linting tools
        run: pip install ruff

      - name: Check code style
        run: ruff check src/

      - name: Check formatting
        run: ruff format --check src/

  test:
    name: "Tests"
    runs-on: ubuntu-latest
    timeout-minutes: 15
    outputs:
      coverage: ${{ steps.coverage.outputs.pct }}

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: pip

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run test suite
        run: pytest tests/ -v --tb=short --cov=src --cov-report=term

      - name: Extract coverage
        id: coverage
        run: |
          COV=$(pytest tests/ --cov=src --cov-report=term 2>/dev/null | \
            grep "TOTAL" | awk '{print $NF}' | tr -d '%' || echo "0")
          echo "pct=$COV" >> $GITHUB_OUTPUT

  ai-checks:
    name: "AI Quality Checks"
    runs-on: ubuntu-latest
    timeout-minutes: 15
    outputs:
      cost-estimate: ${{ steps.cost.outputs.estimate }}

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: pip

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Prompt regression check
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python scripts/prompt_regression.py --mode check

      - name: Cost estimation
        id: cost
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          ESTIMATE=$(python scripts/cost_estimation.py --threshold 0.10 2>&1 | \
            grep "estimated_cost" | cut -d= -f2 || echo "0.00")
          echo "estimate=$ESTIMATE" >> $GITHUB_OUTPUT

# ══════════════════════════════════════════════════════════
# STAGE 2: DOCKER BUILD & PUSH (only on main)
# ══════════════════════════════════════════════════════════
  docker:
    name: "Docker Build & Push"
    needs: [lint, test, ai-checks]
    runs-on: ubuntu-latest
    timeout-minutes: 15
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    outputs:
      image-tag: ${{ steps.meta.outputs.tag }}
      image-full: ${{ steps.meta.outputs.full }}

    permissions:
      contents: read
      packages: write

    steps:
      - uses: actions/checkout@v4

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Generate image metadata
        id: meta
        run: |
          SHORT_SHA="${GITHUB_SHA::7}"
          TAG="sha-${SHORT_SHA}"
          FULL="${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${TAG}"
          echo "tag=$TAG" >> $GITHUB_OUTPUT
          echo "full=$FULL" >> $GITHUB_OUTPUT
          echo "Image: $FULL"

      - name: Build Docker image
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: |
            ${{ steps.meta.outputs.full }}
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

# ══════════════════════════════════════════════════════════
# STAGE 3: DEPLOY STAGING
# ══════════════════════════════════════════════════════════
  deploy-staging:
    name: "Deploy Staging"
    needs: docker
    runs-on: ubuntu-latest
    timeout-minutes: 10
    environment:
      name: staging
      url: https://staging.your-app.com

    steps:
      - uses: actions/checkout@v4

      - name: Deploy to staging
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.STAGING_HOST }}
        run: |
          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && IMAGE_TAG=${{ needs.docker.outputs.image-tag }} docker compose up -d --pull always"
          rm /tmp/key

      - name: Wait for startup
        run: sleep 15

# ══════════════════════════════════════════════════════════
# STAGE 4: SMOKE TESTS
# ══════════════════════════════════════════════════════════
  smoke-tests:
    name: "Smoke Tests"
    needs: deploy-staging
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
      - name: Health check
        run: |
          for i in $(seq 1 20); do
            STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
              https://staging.your-app.com/health 2>/dev/null || echo "000")
            if [ "$STATUS" = "200" ]; then
              echo "Staging healthy after $((i*3))s"
              exit 0
            fi
            echo "Attempt $i/20 — HTTP $STATUS"
            sleep 3
          done
          echo "::error::Staging health check failed"
          exit 1

      - name: Prompt smoke test
        run: |
          RESPONSE=$(curl -sf -X POST https://staging.your-app.com/api/chat \
            -H "Content-Type: application/json" \
            -d '{"message":"ping","max_tokens":10}' \
            --max-time 30 2>/dev/null)

          if [ -z "$RESPONSE" ]; then
            echo "::error::AI endpoint returned empty response"
            exit 1
          fi
          echo "AI endpoint responding: ${RESPONSE:0:100}..."

# ══════════════════════════════════════════════════════════
# STAGE 5: APPROVAL GATE
# ══════════════════════════════════════════════════════════
  approve:
    name: "Production Approval"
    needs: smoke-tests
    runs-on: ubuntu-latest
    environment:
      name: production

    steps:
      - name: Approval confirmed
        run: |
          echo "Production deploy approved by ${{ github.actor }}"
          echo "Image: ${{ needs.docker.outputs.image-tag || 'inherited' }}"

# ══════════════════════════════════════════════════════════
# STAGE 6: DEPLOY PRODUCTION
# ══════════════════════════════════════════════════════════
  deploy-production:
    name: "Deploy Production"
    needs: [approve, docker]
    runs-on: ubuntu-latest
    timeout-minutes: 10
    outputs:
      previous-tag: ${{ steps.current.outputs.tag }}
      deploy-status: ${{ steps.verify.outcome }}

    steps:
      - uses: actions/checkout@v4

      - name: Get current production version
        id: current
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.PRODUCTION_HOST }}
        run: |
          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          CURRENT=$(ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && docker compose ps --format '{{.Image}}'" 2>/dev/null | \
            head -1 | cut -d: -f2 || echo "unknown")
          rm /tmp/key
          echo "tag=${CURRENT}" >> $GITHUB_OUTPUT
          echo "Current production: ${CURRENT}"

      - name: Deploy new version
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.PRODUCTION_HOST }}
        run: |
          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && IMAGE_TAG=${{ needs.docker.outputs.image-tag }} docker compose up -d --pull always"
          rm /tmp/key

      - name: Verify deployment
        id: verify
        continue-on-error: true
        run: |
          sleep 15
          for i in $(seq 1 30); do
            STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
              https://your-app.com/health 2>/dev/null || echo "000")
            if [ "$STATUS" = "200" ]; then
              echo "Production healthy after $((15 + i*2))s"
              exit 0
            fi
            echo "Attempt $i/30 — HTTP $STATUS"
            sleep 2
          done
          echo "::error::Production health check failed"
          exit 1

      - name: Rollback on failure
        if: steps.verify.outcome == 'failure'
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.PRODUCTION_HOST }}
        run: |
          PREV="${{ steps.current.outputs.tag }}"
          if [ "$PREV" = "unknown" ] || [ -z "$PREV" ]; then
            echo "::error::Cannot rollback — no previous version (first deploy?)"
            exit 1
          fi
          echo "ROLLBACK: Reverting to $PREV"
          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && IMAGE_TAG=$PREV docker compose up -d --pull always"
          rm /tmp/key
          sleep 15
          if curl -sf https://your-app.com/health > /dev/null 2>&1; then
            echo "Rollback OK — running $PREV"
          else
            echo "::error::CRITICAL — Rollback also failed!"
          fi
          exit 1

# ══════════════════════════════════════════════════════════
# NOTIFICATIONS
# ══════════════════════════════════════════════════════════
  notify:
    name: "Notifications"
    needs: [lint, test, ai-checks, docker, deploy-production]
    runs-on: ubuntu-latest
    if: always()

    steps:
      - name: Determine final status
        id: status
        run: |
          if [ "${{ needs.deploy-production.result }}" = "success" ]; then
            echo "result=deploy-success" >> $GITHUB_OUTPUT
            echo "message=🚀 Production deploy successful" >> $GITHUB_OUTPUT
          elif [ "${{ needs.deploy-production.result }}" = "failure" ]; then
            echo "result=deploy-failed" >> $GITHUB_OUTPUT
            echo "message=❌ Production deploy failed — rollback executed" >> $GITHUB_OUTPUT
          elif [ "${{ contains(needs.*.result, 'failure') }}" = "true" ]; then
            echo "result=ci-failed" >> $GITHUB_OUTPUT
            echo "message=❌ CI checks failed" >> $GITHUB_OUTPUT
          else
            echo "result=success" >> $GITHUB_OUTPUT
            echo "message=✅ Pipeline completed" >> $GITHUB_OUTPUT
          fi

      - name: Notify Slack
        if: steps.status.outputs.result != 'success'
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "${{ steps.status.outputs.message }}",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "${{ steps.status.outputs.message }}\n*Branch:* ${{ github.ref_name }}\n*Commit:* `${{ github.sha }}`\n*Author:* ${{ github.actor }}\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Run>"
                  }
                }
              ]
            }

      - name: Pipeline summary
        if: always()
        run: |
          echo "## Production AI Pipeline" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
          echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
          echo "| Lint | ${{ needs.lint.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Test | ${{ needs.test.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| AI Checks | ${{ needs.ai-checks.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Docker | ${{ needs.docker.result || 'skipped' }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Deploy | ${{ needs.deploy-production.result || 'skipped' }} |" >> $GITHUB_STEP_SUMMARY

The pipeline's anatomy

Why `paths-ignore`?

paths-ignore:
  - "*.md"
  - "docs/**"

If you only change documentation, you don't need to run the complete pipeline. paths-ignore avoids unnecessary runs that burn compute and API calls.

Why `timeout-minutes` on every job?

timeout-minutes: 15

Without a timeout, a job can keep running indefinitely (an API call that doesn't respond, an infinite loop). For AI pipelines this is especially important because LLM API calls can hang with no timeout.

Why `cache: pip`?

uses: actions/setup-python@v5
with:
  cache: pip

Without a cache, every job downloads and installs all the dependencies from scratch. With a cache, the dependencies get stored between runs. The difference: ~30s with a cache vs ~2min without one.

Why `continue-on-error: true` on the health check?

- name: Verify deployment
  id: verify
  continue-on-error: true

Without continue-on-error, if the health check fails, the job stops immediately. With continue-on-error, the step can fail but the job continues to the next step — where you can do the rollback.

The execution flow

The happy path (everything works)

1. lint ✅        (22s)
2. test ✅        (1m 45s)     ← Parallel with lint
3. ai-checks ✅   (2m 15s)     ← Parallel with lint and test
4. docker ✅      (3m 10s)     ← Waits for 1-2-3
5. deploy-staging ✅ (45s)     ← Waits for 4
6. smoke-tests ✅  (30s)       ← Waits for 5
7. approve ✅      (manual)    ← Waits for 6
8. deploy-prod ✅  (1m 30s)    ← Waits for 7
9. notify: "🚀 Success"       ← Waits for 8

Total (without the approval): ~8 minutes

The failure path (CI fails)

1. lint ✅        (22s)
2. test ❌        (45s, failure)  ← The tests fail
3. ai-checks ✅   (2m 15s)
4. docker ⚪      (skipped)       ← It doesn't run because test failed
5-8. ⚪           (skipped)
9. notify: "❌ CI failed"

Total: ~2 minutes

The rollback path (the deploy fails)

1-7. ✅           (everything ok up to the approval)
8. deploy-prod:
   - Deploy new version ✅
   - Health check ❌ (the app doesn't respond)
   - Rollback to previous ✅
   - exit 1 (the job fails)
9. notify: "❌ Deploy failed — rollback executed"

Total: ~10 minutes

Comparisons

A minimal vs a production-grade pipeline

Aspect	Minimal	Production-grade
CI Checks	Tests only	Lint + test + AI checks
Docker	A local build	Build + push + cache + GHCR
Deploy	Staging only	Staging → approval → production
Error handling	None	Automatic rollback
Notifications	The default email	Slack with context
YAML lines	~30	~200
Time	~2 min	~8 min (without the approval)

Troubleshooting

"The Docker job gets skipped on every run"

Cause: The condition if: github.ref == 'refs/heads/main' && github.event_name == 'push' isn't met. You're probably running from a PR or a workflow_dispatch.

Solution: The Docker build only runs on a push to main. For testing, use workflow_dispatch with an adjusted condition or push to main.

"The deploy-production job doesn't have access to the image-tag"

Cause: needs doesn't include the docker job that produces the output.

Solution: Verify that deploy-production has needs: [approve, docker] — it needs docker to access needs.docker.outputs.image-tag.

"The approval gate doesn't show up"

Cause: The production environment doesn't have required reviewers configured.

Solution: In your repo → Settings → Environments → production → Required reviewers → add at least one reviewer.

"The pipeline takes more than 20 minutes"

Cause: The AI checks or the Docker build are slow.

Solution:

AI checks: Reduce the number of prompt regression test cases
Docker build: Verify that the layer cache is working (cache-from: type=gha)
Tests: Use markers to separate the fast tests from the slow ones

Exercises

Exercise 1: Identify the pipeline's outputs

Read the complete YAML and list every output each job produces.

See solution

Job	Output	Value
`test`	`coverage`	The coverage percentage (e.g. "85")
`ai-checks`	`cost-estimate`	The estimated cost (e.g. "0.05")
`docker`	`image-tag`	The image's tag (e.g. "sha-abc1234")
`docker`	`image-full`	The complete image (e.g. "ghcr.io/org/app:sha-abc1234")
`deploy-production`	`previous-tag`	The previous version's tag
`deploy-production`	`deploy-status`	The health check's result

Exercise 2: Add a security step

Add a trivy image scanning step to the Docker job, before the push.

See solution

  docker:
    steps:
      # ... (the build steps)

      - name: Security scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ steps.meta.outputs.full }}
          format: table
          exit-code: 1
          severity: CRITICAL,HIGH

      # the push goes after the scan
      - name: Push image
        if: success()
        run: docker push ${{ steps.meta.outputs.full }}

With exit-code: 1, the step fails if it finds CRITICAL or HIGH vulnerabilities, blocking the push.

Exercise 3: Modify the pipeline for PRs

The current pipeline only builds Docker on a push to main. Modify it so that on PRs, the CI checks run but it also generates a summary with the results.

See solution

Add a summary job that runs on PRs:

  pr-summary:
    name: "PR Summary"
    needs: [lint, test, ai-checks]
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    steps:
      - name: Generate PR summary
        run: |
          echo "## PR Check Results" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          echo "| Check | Result |" >> $GITHUB_STEP_SUMMARY
          echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
          echo "| Lint | ${{ needs.lint.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Tests | ${{ needs.test.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| AI Checks | ${{ needs.ai-checks.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Coverage | ${{ needs.test.outputs.coverage }}% |" >> $GITHUB_STEP_SUMMARY
          echo "| Est. Cost | \$${{ needs.ai-checks.outputs.cost-estimate }} |" >> $GITHUB_STEP_SUMMARY

This job only runs on PRs and generates a summary visible in the Actions UI.

Summary

✅ The complete pipeline in one YAML: lint → test → AI checks → Docker → staging → smoke → approve → production
✅ Parallel CI: lint, test, and ai-checks run at the same time (~3 min vs ~7 min sequentially)
✅ A conditional Docker build: it only builds on a push to main, not on PRs
✅ The data flow with outputs: image-tag flows from docker → deploy-staging → deploy-production
✅ An integrated rollback: the health check fails → redeploy the previous version → notify
✅ Smart notifications: only failures + a production deploy success
✅ The step summary: a report visible in the Actions UI without downloading artifacts
✅ ~200 lines of YAML that cover the complete commit-to-production cycle

Additional resources

GitHub Actions — Complete Workflow Reference - The syntax reference
docker/build-push-action - The official action for a Docker build
GitHub Actions — Environments - Approval gates
GitHub Actions — Job Outputs - The data flow between jobs
aquasecurity/trivy-action - Image security scanning
GitHub Actions — Step Summary - Generating summaries

← Previous2. Pipeline Architecture Next →4. Automatic Rollback

3. Full Pipeline Implementation

Overview

The complete pipeline

The pipeline's anatomy

Why paths-ignore?

Why timeout-minutes on every job?

Why cache: pip?

Why continue-on-error: true on the health check?

The execution flow

The happy path (everything works)

The failure path (CI fails)

The rollback path (the deploy fails)

Comparisons

A minimal vs a production-grade pipeline

Troubleshooting

"The Docker job gets skipped on every run"

"The deploy-production job doesn't have access to the image-tag"

"The approval gate doesn't show up"

"The pipeline takes more than 20 minutes"

Exercises

Exercise 1: Identify the pipeline's outputs

Exercise 2: Add a security step

Exercise 3: Modify the pipeline for PRs

Summary

Additional resources

Why `paths-ignore`?

Why `timeout-minutes` on every job?

Why `cache: pip`?

Why `continue-on-error: true` on the health check?