3. Notifications — Slack and Email

Overview

Your pipeline failed at 2am. Who found out? If the answer is "nobody until 10am," you have a visibility problem. Pipeline monitoring (the previous capsule) teaches you to analyze the run history. Notifications give you the real-time alert: the pipeline failed → 30 seconds later, a message in Slack → the on-call developer investigates → a fix in minutes, not hours.

But there's a trap: configuring notifications for everything creates notification fatigue. If you get a message for every push, every successful run, every completed step, you're going to mute the channel within a week. The fundamental rule is: notify failures, not successes. A pipeline that works doesn't need to announce that it works. A pipeline that fails needs someone to know immediately.

In this capsule you're going to configure notifications to Slack using slackapi/slack-github-action, understand when email is more appropriate, and build a complete workflow that notifies intelligently: failures only, with useful context, no spam.

Connection with the final pipeline: In the capstone pipeline (Module 8), the Slack notifications automatically report the result of each deployment — success, failure, or a rollback that was executed.

Slack vs Email: When to use each

The recommendation for AI pipelines:

• ✅ Slack for critical failures: The pipeline failed, the deploy failed, a rollback was executed
• ✅ Email for periodic reports: A weekly health report, a monthly cost summary
• ❌ Don't use Slack for successes: "The pipeline passed" × 20 times a day = a dead channel
• ❌ Don't use email for urgent alerts: Nobody checks email at 2am

Configuring Slack Incoming Webhooks

Step 1: Create a Slack Webhook

1. Go to api.slack.com/apps
2. Click Create New App → From scratch
3. Name: CI/CD Notifications, Workspace: your workspace
4. In the sidebar: Incoming Webhooks → Activate
5. Click Add New Webhook to Workspace
6. Select the channel (e.g. #ci-cd-alerts)
7. Copy the Webhook URL

The URL has this format:

https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

Step 2: Save it as a GitHub Secret

Your repo → Settings → Secrets and variables → Actions → New repository secret
  Name: SLACK_WEBHOOK_URL
  Value: https://hooks.slack.com/services/T00000000/B00000000/XXXX...

Step 3: Verify the webhook

curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"Test notification from CI/CD pipeline 🔔"}' \
  https://hooks.slack.com/services/T00000000/B00000000/XXXX

If you see the message in the Slack channel, the webhook works.

Notifications with slackapi/slack-github-action

The official action

GitHub and Slack maintain an official action: slackapi/slack-github-action. It's the recommended way to send notifications from GitHub Actions.

A basic failure notification

- name: Notify Slack on failure
  if: failure()
  uses: slackapi/slack-github-action@v2.0.0
  with:
    webhook-type: incoming-webhook
    webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
    payload: |
      {
        "text": "❌ Pipeline failed: ${{ github.workflow }}",
        "blocks": [
          {
            "type": "header",
            "text": {
              "type": "plain_text",
              "text": "❌ Pipeline Failure"
            }
          },
          {
            "type": "section",
            "fields": [
              {
                "type": "mrkdwn",
                "text": "*Workflow:*\n${{ github.workflow }}"
              },
              {
                "type": "mrkdwn",
                "text": "*Branch:*\n${{ github.ref_name }}"
              },
              {
                "type": "mrkdwn",
                "text": "*Commit:*\n`${{ github.sha }}`"
              },
              {
                "type": "mrkdwn",
                "text": "*Author:*\n${{ github.actor }}"
              }
            ]
          },
          {
            "type": "actions",
            "elements": [
              {
                "type": "button",
                "text": {
                  "type": "plain_text",
                  "text": "View Run"
                },
                "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
              }
            ]
          }
        ]
      }

What is if: failure()?

GitHub Actions has status check functions:

if: failure() is the key: the notification step only runs if something before it failed. If everything passes, the step gets skipped.

What to notify (and what NOT to notify)

The rule: Failures, not successes

GOOD PRACTICE:
  Pipeline fails  → Notify Slack ✅
  Deploy fails    → Notify Slack ✅
  Rollback        → Notify Slack ✅
  Deploy success  → Notify Slack ✅ (only for production)

BAD PRACTICE:
  Every push      → Notify Slack ❌ (spam)
  Tests pass      → Notify Slack ❌ (nobody needs to know)
  Lint passes     → Notify Slack ❌ (obvious)
  Every PR        → Notify Slack ❌ (too frequent)

Valid exceptions

There's one case where notifying a success makes sense: a deploy to production. When someone approves the deploy and the new version is live, the team should know:

- name: Notify production deploy
  if: success()
  uses: slackapi/slack-github-action@v2.0.0
  with:
    webhook-type: incoming-webhook
    webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
    payload: |
      {
        "text": "🚀 Deployed to production: ${{ github.sha }}",
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "🚀 *Production Deploy Successful*\nVersion `sha-${{ github.sha }}` is now live.\nApproved by @${{ github.actor }}"
            }
          }
        ]
      }

The complete workflow with a Slack notification

This is the recommended pattern: a dedicated notification job that always runs and decides what to send based on the result:

name: CI Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install ruff
      - run: ruff check src/

  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest tests/ -v

  ai-checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python scripts/prompt_regression.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

  notify:
    needs: [lint, test, ai-checks]
    runs-on: ubuntu-latest
    if: always()
    steps:
      - name: Determine status
        id: status
        run: |
          if [ "${{ needs.lint.result }}" = "failure" ] || \
             [ "${{ needs.test.result }}" = "failure" ] || \
             [ "${{ needs.ai-checks.result }}" = "failure" ]; then
            echo "result=failure" >> $GITHUB_OUTPUT
            echo "emoji=❌" >> $GITHUB_OUTPUT
            echo "color=#FF0000" >> $GITHUB_OUTPUT
          else
            echo "result=success" >> $GITHUB_OUTPUT
            echo "emoji=✅" >> $GITHUB_OUTPUT
            echo "color=#36A64F" >> $GITHUB_OUTPUT
          fi

      - name: Build failure details
        if: steps.status.outputs.result == 'failure'
        id: details
        run: |
          DETAILS=""
          [ "${{ needs.lint.result }}" = "failure" ] && DETAILS="${DETAILS}• Lint ❌\n"
          [ "${{ needs.test.result }}" = "failure" ] && DETAILS="${DETAILS}• Test ❌\n"
          [ "${{ needs.ai-checks.result }}" = "failure" ] && DETAILS="${DETAILS}• AI Checks ❌\n"
          echo "details=$DETAILS" >> $GITHUB_OUTPUT

      - name: Notify Slack (failure only)
        if: steps.status.outputs.result == 'failure'
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "${{ steps.status.outputs.emoji }} CI Pipeline failed on ${{ github.ref_name }}",
              "blocks": [
                {
                  "type": "header",
                  "text": {
                    "type": "plain_text",
                    "text": "${{ steps.status.outputs.emoji }} CI Pipeline Failed"
                  }
                },
                {
                  "type": "section",
                  "fields": [
                    {
                      "type": "mrkdwn",
                      "text": "*Branch:*\n${{ github.ref_name }}"
                    },
                    {
                      "type": "mrkdwn",
                      "text": "*Author:*\n${{ github.actor }}"
                    }
                  ]
                },
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Failed Jobs:*\n${{ steps.details.outputs.details }}"
                  }
                },
                {
                  "type": "actions",
                  "elements": [
                    {
                      "type": "button",
                      "text": {
                        "type": "plain_text",
                        "text": "View Run"
                      },
                      "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
                    }
                  ]
                }
              ]
            }

The anatomy of the pattern

1. needs: [lint, test, ai-checks] — The notify job depends on all the other jobs
2. if: always() — It always runs, even if a job failed (without this, it wouldn't run on failure)
3. Determine status — It checks each job's result with needs.<job>.result
4. Build failure details — It builds a summary of which jobs failed
5. Notify only on failure — It only sends the notification if there was a failure

Email notifications

GitHub's Built-in Notifications

GitHub has built-in email notifications. You don't need to configure anything extra — just make sure your notification settings are correct:

GitHub.com → Settings → Notifications → Actions
  → ✅ Email: "Only send notifications for failed workflows"

The limitations of the built-in email

• 📋 It only notifies the owner/contributor, not a team
• 📋 You can't customize the message
• 📋 You can't choose who to notify per job
• 📋 It has no rich formatting like Slack blocks

Custom email with GitHub Actions

If you need more control, you can send email from a workflow:

- name: Send failure email
  if: failure()
  uses: dawidd6/action-send-mail@v3
  with:
    server_address: smtp.gmail.com
    server_port: 587
    username: ${{ secrets.EMAIL_USERNAME }}
    password: ${{ secrets.EMAIL_PASSWORD }}
    subject: "❌ CI Pipeline Failed — ${{ github.repository }}"
    to: team@example.com
    from: ci-notifications@example.com
    body: |
      Pipeline failed on branch ${{ github.ref_name }}.

      Commit: ${{ github.sha }}
      Author: ${{ github.actor }}
      Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

When email vs Slack?

Notification fatigue: The real problem

The signs of notification fatigue

Week 1: The team reads every notification, investigates failures
Week 2: The team starts ignoring "repetitive" notifications
Week 3: Someone mutes the channel
Week 4: The pipeline has been broken for 3 days, nobody knows

How to avoid it

1. Failures only: Never notify successes (except a production deploy)
2. Useful context: The message must have the branch, the commit, what failed, and a link to the run
3. A dedicated channel: #ci-cd-alerts only for CI/CD, not for general chat
4. Group them: If there are 3 failures in 5 minutes, don't send 3 messages — the team already saw the first
5. Severity: Distinguish between "a test failed on a PR" vs "the production deploy failed"

The recommended channels

#ci-cd-alerts       → CI/CD failures (high noise is acceptable)
#deploys            → Only deploy events (staging and production)
#incidents          → Rollbacks and critical failures (low noise)

Comparisons

A direct Slack webhook vs slackapi/slack-github-action

A notification inside the job vs a dedicated job

Troubleshooting

"The notification doesn't reach Slack"

Cause 1: The SLACK_WEBHOOK_URL secret isn't configured or has a typo.

# Verify that the webhook works outside of Actions
curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"test"}' \
  $SLACK_WEBHOOK_URL

Cause 2: The notification step doesn't run because if: failure() or if: always() is missing.

Cause 3: The notify job has needs but not if: always(). Without always(), the job doesn't run if a dependency failed.

"I get duplicate notifications"

Cause: You have notification steps in multiple jobs AND a dedicated notify job.

Solution: Choose one: steps inside each job OR a dedicated job. Not both.

"The JSON payload has a syntax error"

Cause: Quotes inside the payload break the JSON. GitHub expressions ${{ ... }} that contain quotes cause invalid JSON.

Solution: Use intermediate variables:

- name: Prepare message
  id: msg
  run: |
    echo "text=Pipeline failed on ${{ github.ref_name }}" >> $GITHUB_OUTPUT

- name: Notify Slack
  if: failure()
  uses: slackapi/slack-github-action@v2.0.0
  with:
    webhook-type: incoming-webhook
    webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
    payload: |
      {
        "text": "${{ steps.msg.outputs.text }}"
      }

"The notification arrives but without the detail of which job failed"

Cause: The step that builds the details doesn't have access to needs.<job>.result because it isn't in a job with needs.

Solution: The notify job must have needs: [job1, job2, ...] to be able to access each job's results.

Exercises

Exercise 1: Configure a Slack webhook

Create a Slack Incoming Webhook, save it as a GitHub Secret, and send a test message.

curl -X POST -H 'Content-type: application/json' \
  --data '{"text":"✅ Webhook configured successfully!"}' \
  "https://hooks.slack.com/services/T.../B.../XXX..."

Exercise 2: Add a notification to your existing pipeline

Modify your CI workflow so it sends a notification to Slack when it fails. Use the dedicated-job pattern with needs and if: always().

  notify-failure:
    needs: [lint, test, ai-checks]
    runs-on: ubuntu-latest
    if: always() && contains(needs.*.result, 'failure')

    steps:
      - name: Identify failed jobs
        id: failed
        run: |
          FAILED=""
          [ "${{ needs.lint.result }}" = "failure" ] && FAILED="${FAILED}lint, "
          [ "${{ needs.test.result }}" = "failure" ] && FAILED="${FAILED}test, "
          [ "${{ needs.ai-checks.result }}" = "failure" ] && FAILED="${FAILED}ai-checks, "
          FAILED="${FAILED%, }"
          echo "jobs=$FAILED" >> $GITHUB_OUTPUT

      - name: Send Slack notification
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "❌ CI failed on ${{ github.ref_name }}: ${{ steps.failed.outputs.jobs }}",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "❌ *CI Pipeline Failed*\n*Branch:* ${{ github.ref_name }}\n*Failed:* ${{ steps.failed.outputs.jobs }}\n*Author:* ${{ github.actor }}\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Run>"
                  }
                }
              ]
            }

Exercise 3: A conditional notification by severity

Implement two notification levels: (1) Slack for failures on main, (2) only a GitHub Step Summary for failures on PRs.

  notify:
    needs: [lint, test, ai-checks]
    runs-on: ubuntu-latest
    if: always() && contains(needs.*.result, 'failure')

    steps:
      - name: Generate failure summary
        run: |
          echo "## ❌ Pipeline Failed" >> $GITHUB_STEP_SUMMARY
          echo "" >> $GITHUB_STEP_SUMMARY
          echo "| Job | Result |" >> $GITHUB_STEP_SUMMARY
          echo "|-----|--------|" >> $GITHUB_STEP_SUMMARY
          echo "| Lint | ${{ needs.lint.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| Test | ${{ needs.test.result }} |" >> $GITHUB_STEP_SUMMARY
          echo "| AI Checks | ${{ needs.ai-checks.result }} |" >> $GITHUB_STEP_SUMMARY

      - name: Notify Slack (main branch only)
        if: github.ref == 'refs/heads/main'
        uses: slackapi/slack-github-action@v2.0.0
        with:
          webhook-type: incoming-webhook
          webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "🚨 CI failed on MAIN branch — immediate attention needed",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "🚨 *CI Failed on main*\nThis is blocking production deploys.\n*Commit:* `${{ github.sha }}`\n*Author:* ${{ github.actor }}\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Run>"
                  }
                }
              ]
            }

Exercise 4: Simulate a failure and verify the notification

Introduce an intentional error in your pipeline to verify that the notification reaches Slack.

  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Intentional failure for testing
        run: |
          echo "This step will fail intentionally"
          exit 1

Summary

• ✅ Slack for real-time alerts, email for periodic reports
• ✅ The fundamental rule: Notify failures, not successes (except a production deploy)
• ✅ slackapi/slack-github-action is the official action for sending messages to Slack
• ✅ if: failure() runs a step only when something before it failed
• ✅ A dedicated notify job with needs + if: always() for the complete context of every job
• ✅ Notification fatigue is real: a dedicated channel, failures only, useful context, no spam
• ✅ Different levels: Slack for main (it blocks deploys), a Step Summary for PRs (informational)
• ✅ GitHub's built-in email is enough for personal notifications, Slack is for the team

Additional resources

1. Slack Incoming Webhooks - Official webhook documentation
2. slackapi/slack-github-action - Slack's official action for GitHub
3. Slack Block Kit Builder - A visual tool for designing messages
4. GitHub Actions Status Check Functions - success(), failure(), always()
5. GitHub Notifications Settings - Configuring email notifications
6. Notification Fatigue in DevOps - How to avoid notification fatigue