Module 7: Monitoring, Notifications, and Advanced Patterns

8. Project — Advanced CI/CD

Next →

Overview

This is Module 7's capstone project. You're going to take the existing pipeline from the previous modules and add three operational capabilities to it: Slack notifications for failures, a scheduled nightly health check, and a reusable workflow that centralizes your CI flow. The result is a pipeline that doesn't just work, but monitors itself, maintains itself, and is reusable.

You're not going to rewrite the pipeline — you're going to extend it. Each new capability is a new workflow file or a modification to an existing one. In the end, you'll have a system of 3-4 workflows that work together: the main CI/CD pipeline, a notifications workflow, a nightly health check, and a reusable workflow that other repos can consume.

Project objectives

By completing this project:

  • ✅ Your main pipeline sends notifications to Slack when it fails
  • ✅ A scheduled workflow runs nightly and verifies that your pipeline is healthy
  • ✅ You have a reusable workflow that encapsulates your CI flow
  • ✅ A composite action centralizes the Python + dependency setup
  • ✅ Your matrix strategy uses include to differentiate between test and build

Prerequisites

  • ✅ A working CI/CD pipeline (from modules 1-6)
  • ✅ A Slack webhook configured as a secret (SLACK_WEBHOOK_URL)
  • ✅ GitHub Secrets: OPENAI_API_KEY, SLACK_WEBHOOK_URL
  • ✅ Lessons 02-07 of this module completed

The project's structure

my-ai-project/
├── .github/
│   ├── actions/
│   │   └── setup-ai-project/
│   │       └── action.yml           ← Composite action (new)
│   └── workflows/
│       ├── ci.yml                    ← Main pipeline (modified)
│       ├── nightly-health.yml        ← Nightly health check (new)
│       ├── reusable-ai-ci.yml        ← Reusable workflow (new)
│       └── weekly-report.yml         ← Weekly report (new)
├── scripts/
│   └── pipeline_metrics.py           ← Metrics script (from the module)
├── src/
├── tests/
├── Dockerfile
├── docker-compose.yml
└── requirements.txt

Step 1: The composite action — Setup AI Project

Create the file .github/actions/setup-ai-project/action.yml:

name: "Setup AI Project"
description: "Checkout, setup Python, install dependencies, verify API"

inputs:
  python-version:
    description: "Python version"
    required: false
    default: "3.12"
  install-dev-deps:
    description: "Install development dependencies"
    required: false
    default: "true"

outputs:
  python-version:
    description: "Actual Python version installed"
    value: ${{ steps.python.outputs.python-version }}

runs:
  using: "composite"
  steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Setup Python
      id: python
      uses: actions/setup-python@v5
      with:
        python-version: ${{ inputs.python-version }}
        cache: pip

    - name: Install dependencies
      shell: bash
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        if [ "${{ inputs.install-dev-deps }}" = "true" ]; then
          pip install ruff pytest pytest-cov
        fi

Step 2: The reusable workflow — AI CI Pipeline

Create the file .github/workflows/reusable-ai-ci.yml:

name: Reusable AI CI Pipeline

on:
  workflow_call:
    inputs:
      python-version:
        type: string
        default: "3.12"
      run-ai-checks:
        type: boolean
        default: true
      run-docker-build:
        type: boolean
        default: false
      cost-threshold:
        type: string
        default: "0.10"
    secrets:
      openai-api-key:
        required: false
      slack-webhook-url:
        required: false
    outputs:
      test-result:
        description: "Test result"
        value: ${{ jobs.test.outputs.result }}

jobs:
  lint:
    name: "Lint & Format"
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ inputs.python-version }}
      - run: pip install ruff
      - run: ruff check src/
      - run: ruff format --check src/

  test:
    name: "Tests"
    runs-on: ubuntu-latest
    timeout-minutes: 15
    outputs:
      result: ${{ steps.test.outputs.result }}
    strategy:
      fail-fast: false
      matrix:
        python-version: ["${{ inputs.python-version }}"]
        include:
          - python-version: "${{ inputs.python-version }}"
            build-docker: ${{ inputs.run-docker-build }}

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip
      - run: pip install -r requirements.txt
      - name: Run tests
        id: test
        run: |
          if pytest tests/ -v --tb=short; then
            echo "result=passed" >> $GITHUB_OUTPUT
          else
            echo "result=failed" >> $GITHUB_OUTPUT
            exit 1
          fi

  ai-checks:
    name: "AI Quality Checks"
    runs-on: ubuntu-latest
    timeout-minutes: 15
    if: ${{ inputs.run-ai-checks }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ inputs.python-version }}
          cache: pip
      - run: pip install -r requirements.txt

      - name: Prompt regression check
        env:
          OPENAI_API_KEY: ${{ secrets.openai-api-key }}
        run: python scripts/prompt_regression.py --mode check

      - name: Cost estimation check
        env:
          OPENAI_API_KEY: ${{ secrets.openai-api-key }}
        run: python scripts/cost_estimation.py --threshold ${{ inputs.cost-threshold }}

  notify:
    name: "Notify on Failure"
    needs: [lint, test, ai-checks]
    runs-on: ubuntu-latest
    if: always() && contains(needs.*.result, 'failure') && inputs.run-ai-checks
    steps:
      - name: Identify failures
        id: failures
        run: |
          FAILED=""
          [ "${{ needs.lint.result }}" = "failure" ] && FAILED="${FAILED}Lint, "
          [ "${{ needs.test.result }}" = "failure" ] && FAILED="${FAILED}Test, "
          [ "${{ needs.ai-checks.result }}" = "failure" ] && FAILED="${FAILED}AI Checks, "
          FAILED="${FAILED%, }"
          echo "jobs=$FAILED" >> $GITHUB_OUTPUT

      - name: Send Slack notification
        if: ${{ secrets.slack-webhook-url != '' }}
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.slack-webhook-url }}
          payload: |
            {
              "text": "❌ CI Failed: ${{ steps.failures.outputs.jobs }}",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "❌ *CI Pipeline Failed*\n*Branch:* ${{ github.ref_name }}\n*Failed:* ${{ steps.failures.outputs.jobs }}\n*Author:* ${{ github.actor }}\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Run>"
                  }
                }
              ]
            }

Step 3: The main pipeline (modified)

Modify .github/workflows/ci.yml to use the reusable workflow:

name: CI Pipeline

on:
  push:
    branches: [main]
    paths-ignore:
      - "*.md"
      - "docs/**"
  pull_request:
    branches: [main]
  workflow_dispatch:

jobs:
  ci:
    uses: ./.github/workflows/reusable-ai-ci.yml
    with:
      python-version: "3.12"
      run-ai-checks: true
      run-docker-build: ${{ github.ref == 'refs/heads/main' }}
      cost-threshold: "0.10"
    secrets:
      openai-api-key: ${{ secrets.OPENAI_API_KEY }}
      slack-webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}

  docker:
    needs: ci
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build and push Docker image
        run: |
          echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
          docker build -t ghcr.io/${{ github.repository }}:sha-${{ github.sha }} .
          docker push ghcr.io/${{ github.repository }}:sha-${{ github.sha }}

      - name: Notify Docker build success
        if: success()
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "🐳 Docker image built and pushed: sha-${{ github.sha }}"
            }

Step 4: The nightly health check

Create .github/workflows/nightly-health.yml:

name: Nightly Health Check

on:
  schedule:
    - cron: "0 4 * * *"  # 4am UTC daily
  workflow_dispatch:

jobs:
  health-check:
    name: "Pipeline & App Health"
    runs-on: ubuntu-latest
    timeout-minutes: 10

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run test suite
        id: tests
        continue-on-error: true
        run: pytest tests/ -v --tb=short

      - name: Check prompt baselines
        id: baselines
        continue-on-error: true
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python scripts/prompt_regression.py --mode check

      - name: Generate pipeline metrics
        id: metrics
        continue-on-error: true
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          python scripts/pipeline_metrics.py \
            --owner ${{ github.repository_owner }} \
            --repo ${{ github.event.repository.name }} \
            --count 30 \
            --json > health-report.json

          python -c "
          import json
          with open('health-report.json') as f:
              m = json.load(f)
          print(f'health={m[\"health\"]}')
          print(f'success_rate={m[\"success_rate\"]}')
          " >> $GITHUB_OUTPUT

      - name: Generate summary
        if: always()
        run: |
          echo "## 🏥 Nightly Health Check" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          echo "| Check | Status |" >> $GITHUB_STEP_SUMMARY
          echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
          echo "| Tests | ${{ steps.tests.outcome }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Prompt Baselines | ${{ steps.baselines.outcome }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Pipeline Health | ${{ steps.metrics.outputs.health || 'N/A' }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Success Rate | ${{ steps.metrics.outputs.success_rate || 'N/A' }} |" >> $GITHUB_STEP_SUMMARY

      - name: Upload health report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: nightly-health-${{ github.run_number }}
          path: health-report.json
          retention-days: 30

      - name: Alert if unhealthy
        if: steps.tests.outcome == 'failure' || steps.baselines.outcome == 'failure'
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "🚨 Nightly health check found issues",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "🚨 *Nightly Health Check Failed*\n• Tests: ${{ steps.tests.outcome }}\n• Baselines: ${{ steps.baselines.outcome }}\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Details>"
                  }
                }
              ]
            }

Step 5: The weekly report

Create .github/workflows/weekly-report.yml:

name: Weekly Pipeline Report

on:
  schedule:
    - cron: "0 9 * * 1"  # Monday 9am UTC
  workflow_dispatch:

jobs:
  report:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Generate weekly report
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          python scripts/pipeline_metrics.py \
            --owner ${{ github.repository_owner }} \
            --repo ${{ github.event.repository.name }} \
            --count 50 \
            --json > weekly-report.json

          echo "## 📊 Weekly Pipeline Report" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY

          python -c "
          import json
          with open('weekly-report.json') as f:
              m = json.load(f)
          h = m['health']
          emoji = {'HEALTHY':'✅','DEGRADED':'⚠️','UNHEALTHY':'🔴','CRITICAL':'🚨'}.get(h,'❓')
          print(f'| Metric | Value |')
          print(f'|--------|-------|')
          print(f'| Status | {emoji} {h} |')
          print(f'| Success Rate | {m[\"success_rate\"]*100:.0f}% |')
          print(f'| Total Runs | {m[\"total_runs\"]} |')
          print(f'| Failures | {m[\"failures\"]} |')
          print(f'| Avg Duration | {m[\"avg_duration_seconds\"]:.0f}s |')
          print(f'| Consecutive Failures | {m[\"current_consecutive_failures\"]} |')
          " >> $GITHUB_STEP_SUMMARY

      - name: Upload report
        uses: actions/upload-artifact@v4
        with:
          name: weekly-report-${{ github.run_number }}
          path: weekly-report.json
          retention-days: 90

Additional exercises

Exercise 1: Add a success notification only for production deploys

Modify the main pipeline's notify job so that it sends a success message only when the Docker build (which only runs on main) finishes correctly.

Solution 
  notify:
    needs: [ci, docker]
    runs-on: ubuntu-latest
    if: always()
    steps:
      - name: Notify failure
        if: needs.ci.result == 'failure'
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "❌ CI failed on ${{ github.ref_name }}"
            }

      - name: Notify production build success
        if: needs.docker.result == 'success'
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "🚀 Docker image built and pushed from main"
            }

The key: needs.docker.result == 'success' is only true when the docker job existed and passed. On PRs, docker doesn't run because it has if: github.ref == 'refs/heads/main', so needs.docker.result is 'skipped' — it generates no notification.

Exercise 2: Add a duration alert to the health check

Modify the nightly health check so that it sends an alert if the pipeline takes more than 10 minutes on average.

Solution

Add this step after generating the report:

      - name: Check duration alert
        id: duration
        run: |
          AVG=$(python -c "
          import json
          with open('pipeline-health.json') as f:
              m = json.load(f)
          print(m['avg_duration_seconds'])
          ")

          echo "avg_seconds=$AVG" >> $GITHUB_OUTPUT

          if (( $(echo "$AVG > 600" | bc -l) )); then
            echo "alert=true" >> $GITHUB_OUTPUT
          else
            echo "alert=false" >> $GITHUB_OUTPUT
          fi

      - name: Alert slow pipeline
        if: steps.duration.outputs.alert == 'true'
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "⏱️ Slow pipeline: ${{ steps.duration.outputs.avg_seconds }}s on average (threshold: 600s)"
            }

Exercise 3: Version the reusable workflow

Create a v1 tag of the reusable workflow and modify the main pipeline to use the tag instead of @main.

Solution 
# In the terminal:
git tag v1-workflows
git push origin v1-workflows
# In ci.yml, change:
# uses: ./.github/workflows/reusable-ai-ci.yml@main
# to:
uses: ./.github/workflows/reusable-ai-ci.yml
# Note: for local repos, you can't use @tag.
# It only works with external repos:
# uses: org/shared-workflows/.github/workflows/ai-ci.yml@v1

For cross-repo setups, the approach is:

  1. In the workflows repo, create the tag: git tag v1 && git push origin v1
  2. In the consuming repos, reference it: uses: org/shared-workflows/.github/workflows/ai-ci.yml@v1

Verification

The verification checklist

Confirm that everything works:

  • Composite action: Your pipeline uses .github/actions/setup-ai-project without errors
  • Reusable workflow: ci.yml calls reusable-ai-ci.yml and the jobs run correctly
  • Slack notification: Introduce an intentional error → you receive a notification in Slack
  • Nightly health check: Run it manually (workflow_dispatch) → it generates a summary and an artifact
  • Weekly report: Run it manually → it generates a report with metrics
  • Matrix strategy: The test jobs show the Python version in the name

Testing the notifications

# Add this temporarily to a job to force a failure:
- name: Force failure for testing
  run: exit 1
  1. Push with this change
  2. Verify that the Slack notification arrives
  3. Revert the change
  4. Verify that with no failure there's no notification

Testing the scheduled workflows

Don't wait for the cron to trigger — use workflow_dispatch:

Repo → Actions → Nightly Health Check → Run workflow → Run

Verify that the summary and the artifact are generated correctly.

The project's troubleshooting

"The reusable workflow can't find the secrets"

Verify that you're passing the secrets correctly:

# ❌ Incorrect
jobs:
  ci:
    uses: ./.github/workflows/reusable-ai-ci.yml
    # Missing: secrets

# ✅ Correct
jobs:
  ci:
    uses: ./.github/workflows/reusable-ai-ci.yml
    secrets:
      openai-api-key: ${{ secrets.OPENAI_API_KEY }}
      slack-webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}

"The composite action fails with 'shell is required'"

Every step with run in a composite action needs shell: bash. An example:

# ❌ Missing shell
runs:
  using: "composite"
  steps:
    - name: Run something
      run: echo "hello"

# ✅ With an explicit shell
runs:
  using: "composite"
  steps:
    - name: Run something
      shell: bash
      run: echo "hello"

"The nightly workflow doesn't run"

Remember: scheduled workflows only run on the default branch (main). If your file is on a feature branch, it won't run until you merge it.

A temporary solution: use workflow_dispatch to test manually before merging.

"Slack doesn't show the nice formatting"

Verify that the payload's JSON is valid. Use the Slack Block Kit Builder to design and validate the messages.

Common errors:

  • Trailing commas in the JSON (JSON doesn't allow commas after the last element)
  • Unescaped quotes inside the payload
  • GitHub Actions variables with special characters

"The matrix shows 'Test (1 of 3)' instead of the name"

The name with the matrix variable is missing:

# ❌ No custom name
jobs:
  test:
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]

# ✅ With a name that shows the version
jobs:
  test:
    name: "Test Python ${{ matrix.python-version }}"
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]

"The reusable workflow can't access the repo's code"

A reusable workflow doesn't have an implicit actions/checkout. If it needs to access the code, it has to check out explicitly inside its jobs:

# Inside the reusable workflow:
jobs:
  ci:
    steps:
      - uses: actions/checkout@v4  # Necessary
      - run: pytest tests/

Optional extensions

If you finished and want to go further:

1. Add a second consumer of the reusable workflow

Create a second repo (or a second workflow in the same repo) that uses reusable-ai-ci.yml with different parameters:

# second-service-ci.yml
jobs:
  ci:
    uses: ./.github/workflows/reusable-ai-ci.yml
    with:
      python-version: "3.11"
      run-ai-checks: false
      run-docker-build: false

2. Add a badge to the README

![CI Pipeline](https://github.com/ORG/REPO/actions/workflows/ci.yml/badge.svg)
![Nightly Health](https://github.com/ORG/REPO/actions/workflows/nightly-health.yml/badge.svg)

3. A pipeline health dashboard

Create a dashboard.md that auto-updates with the weekly report's results using an auto-commit step.

Success criteria

CriterionMet?
The main pipeline uses a reusable workflow
The setup composite action works
The Slack notification arrives when there's a failure
The nightly health check runs and generates an artifact
The weekly report generates metrics
The matrix strategy differentiates test vs build
fail-fast: false in the matrix

If every criterion is checked → you completed Module 7. You're ready for Module 8 (the Capstone Project).

What you built

BEFORE (Module 6):
  A functional but closed pipeline:
  - ✅ CI/CD works
  - ❌ Nobody finds out when it fails
  - ❌ Baselines don't get updated
  - ❌ Duplicated YAML across repos

AFTER (Module 7):
  An operational pipeline:
  - ✅ CI/CD works
  - ✅ Slack notifications for failures
  - ✅ Automatic nightly health checks
  - ✅ Weekly pipeline reports
  - ✅ A centralized reusable workflow
  - ✅ A composite action for setup
  - ✅ An optimized matrix strategy

Summary

  • ✅ A composite action centralizes the Python + dependency setup in a single place
  • ✅ A reusable workflow encapsulates the complete CI flow, ready to be called from any repo
  • ✅ The main pipeline reduced to a call to the reusable workflow + a conditional Docker build
  • ✅ A nightly health check detects problems before they affect development
  • ✅ A weekly report generates pipeline health metrics automatically
  • ✅ Slack notifications only for failures — the team knows what happened in real time
  • ✅ Everything works together: 4 coordinated workflows that cover CI, monitoring, and operations

Additional resources

  1. GitHub Actions — Reusable Workflows - The complete reference
  2. GitHub Actions — Composite Actions - The complete reference
  3. slackapi/slack-github-action - Slack's official action
  4. GitHub Actions — Matrix Strategy - include, exclude, fail-fast
  5. GitHub Actions — Scheduled Events - Cron in Actions
  6. Slack Block Kit Builder - Designing Slack messages
← Previous7. Advanced Matrix StrategiesNext →1. Introduction: Production AI Pipeline