4. Automatic Rollback

Overview

The deploy to production completed. But the health check doesn't pass. Or it passes, but the LLM's responses are wrong. Or the cost per request tripled. What do you do? If you didn't design an automatic rollback before the deploy, you're improvising under pressure — and that never ends well.

In Module 6 you learned the three rollback strategies (redeploy the tag, git revert, workflow dispatch). In this lesson, you integrate automatic rollback directly into the production pipeline. The pipeline detects the post-deploy failure, gets the previous version's tag (saved before the deploy), and redeploys automatically. All without human intervention.

The edge cases are the hard part: what happens on the first deploy when there's no previous version? What happens if the rollback also fails? How do you notify the team? This lesson covers every scenario.

The automatic rollback pattern

The complete flow

1. Save current version    → "Production is on sha-prev123"
2. Deploy new version      → "Deploying sha-abc456"
3. Post-deploy validation  → Health check + prompt test
4a. Validation passes      → "Deploy successful" → Notify success
4b. Validation fails       → Rollback to sha-prev123 → Notify rollback

The implementation in the pipeline

deploy-production:
  name: "Deploy Production"
  needs: [approve, docker]
  runs-on: ubuntu-latest
  timeout-minutes: 10
  outputs:
    previous-tag: ${{ steps.current.outputs.tag }}
    status: ${{ steps.result.outputs.status }}

  steps:
    # STEP 1: Save the current version
    - name: Get current production version
      id: current
      env:
        SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        HOST: ${{ vars.PRODUCTION_HOST }}
      run: |
        echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
        CURRENT=$(ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
          "cd /app && docker compose ps --format '{{.Image}}'" 2>/dev/null | \
          head -1 | cut -d: -f2 || echo "unknown")
        rm /tmp/key
        echo "tag=${CURRENT}" >> $GITHUB_OUTPUT
        echo "Previous version: ${CURRENT}"

    # STEP 2: Deploy the new version
    - name: Deploy new version
      env:
        SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        HOST: ${{ vars.PRODUCTION_HOST }}
      run: |
        NEW_TAG="${{ needs.docker.outputs.image-tag }}"
        echo "Deploying: $NEW_TAG"
        echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
        ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
          "cd /app && IMAGE_TAG=$NEW_TAG docker compose up -d --pull always"
        rm /tmp/key

    # STEP 3: Post-deploy validation
    - name: Validate deployment
      id: validate
      continue-on-error: true
      run: |
        echo "Waiting for startup..."
        sleep 15

        echo "Running health check..."
        for i in $(seq 1 30); do
          STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
            https://your-app.com/health 2>/dev/null || echo "000")
          if [ "$STATUS" = "200" ]; then
            echo "Health check passed after $((15 + i*2))s"

            echo "Running prompt validation..."
            RESPONSE=$(curl -sf -X POST https://your-app.com/api/chat \
              -H "Content-Type: application/json" \
              -d '{"message":"hello","max_tokens":10}' \
              --max-time 30 2>/dev/null || echo "")

            if [ -n "$RESPONSE" ]; then
              echo "Prompt validation passed"
              exit 0
            else
              echo "::warning::Prompt validation failed — empty response"
              exit 1
            fi
          fi
          echo "Attempt $i/30 — HTTP $STATUS"
          sleep 2
        done
        echo "Health check failed after 75s"
        exit 1

    # STEP 4a: Rollback if the validation fails
    - name: Rollback on failure
      if: steps.validate.outcome == 'failure'
      env:
        SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        HOST: ${{ vars.PRODUCTION_HOST }}
      run: |
        PREV="${{ steps.current.outputs.tag }}"

        if [ "$PREV" = "unknown" ] || [ -z "$PREV" ]; then
          echo "::error::Cannot rollback — no previous version (first deploy)"
          echo "Manual intervention required."
          exit 1
        fi

        echo "ROLLBACK: Reverting to $PREV"
        echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
        ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
          "cd /app && IMAGE_TAG=$PREV docker compose up -d --pull always"
        rm /tmp/key

        sleep 15
        for i in $(seq 1 15); do
          if curl -sf https://your-app.com/health > /dev/null 2>&1; then
            echo "Rollback successful — running $PREV"
            echo "::error::Deploy failed for ${{ needs.docker.outputs.image-tag }}. Rolled back to $PREV."
            exit 1
          fi
          sleep 2
        done

        echo "::error::CRITICAL — Rollback also failed! Manual intervention needed."
        exit 1

    # STEP 4b: Determine the final result
    - name: Set result
      id: result
      if: always()
      run: |
        if [ "${{ steps.validate.outcome }}" = "success" ]; then
          echo "status=success" >> $GITHUB_OUTPUT
        else
          echo "status=rolled-back" >> $GITHUB_OUTPUT
        fi

    # STEP 5: Summary
    - name: Deploy summary
      if: always()
      run: |
        echo "### Production Deployment" >> $GITHUB_STEP_SUMMARY
        echo "" >> $GITHUB_STEP_SUMMARY
        if [ "${{ steps.validate.outcome }}" = "success" ]; then
          echo "**Status:** ✅ Deployed successfully" >> $GITHUB_STEP_SUMMARY
          echo "**Version:** \`${{ needs.docker.outputs.image-tag }}\`" >> $GITHUB_STEP_SUMMARY
        else
          echo "**Status:** ❌ Failed — Rolled back" >> $GITHUB_STEP_SUMMARY
          echo "**Failed version:** \`${{ needs.docker.outputs.image-tag }}\`" >> $GITHUB_STEP_SUMMARY
          echo "**Rolled back to:** \`${{ steps.current.outputs.tag }}\`" >> $GITHUB_STEP_SUMMARY
        fi

Edge cases

Edge case 1: The first deploy (there's no previous version)

The problem: It's the first deploy. There's no previous image on the server.
What do you roll back to?

The solution: Detect it and don't roll back.

if [ "$PREV" = "unknown" ] || [ -z "$PREV" ]; then
  echo "::error::First deploy — no previous version to rollback to"
  echo "Fix the code and push again."
  exit 1
fi

On the first deploy, if it fails, the solution is to fix forward: fix the code and do another push.

Edge case 2: The rollback also fails

The problem: Deploy v2 fails. Rollback to v1. But v1 also fails.
The likely cause: The problem isn't the image but the infrastructure.

echo "::error::CRITICAL — Rollback also failed!"
echo "Check: disk space, network, database, external services"
exit 1

If the rollback fails, the problem probably isn't the application. It's infrastructure: a full disk, a network issue, a downed database, an expired API key.

Edge case 3: The health check passes but the LLM doesn't respond

The problem: /health returns 200 but /api/chat doesn't generate responses.
The basic health check isn't enough.

The solution: Add a prompt test to the validation:

- name: Validate AI functionality
  run: |
    RESPONSE=$(curl -sf -X POST https://your-app.com/api/chat \
      -H "Content-Type: application/json" \
      -d '{"message":"What is 2+2?","max_tokens":10}' \
      --max-time 30)

    if echo "$RESPONSE" | grep -qi "4"; then
      echo "AI validation passed"
    else
      echo "::error::AI returned unexpected response: $RESPONSE"
      exit 1
    fi

Edge case 4: Image tags removed from the registry

The problem: The previous version's tag no longer exists in GHCR.
docker pull fails during the rollback.

The solution: A retention policy that keeps at least the last 10 images:

- name: Verify rollback image exists
  run: |
    if ! docker manifest inspect $REGISTRY/$IMAGE:$PREV > /dev/null 2>&1; then
      echo "::error::Previous image $PREV not found in registry"
      echo "::error::Cannot rollback — manual intervention needed"
      exit 1
    fi

The rollback notification

The team needs to know immediately when a rollback happens:

notify-rollback:
  name: "Rollback Notification"
  needs: deploy-production
  runs-on: ubuntu-latest
  if: always() && needs.deploy-production.outputs.status == 'rolled-back'

  steps:
    - name: Notify Slack
      uses: slackapi/slack-github-action@v2.0.0
      with:
        webhook-type: incoming-webhook
        webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
        payload: |
          {
            "text": "⚠️ ROLLBACK executed in production",
            "blocks": [
              {
                "type": "header",
                "text": {
                  "type": "plain_text",
                  "text": "⚠️ Production Rollback"
                }
              },
              {
                "type": "section",
                "fields": [
                  {
                    "type": "mrkdwn",
                    "text": "*Failed version:*\n`${{ needs.docker.outputs.image-tag }}`"
                  },
                  {
                    "type": "mrkdwn",
                    "text": "*Rolled back to:*\n`${{ needs.deploy-production.outputs.previous-tag }}`"
                  },
                  {
                    "type": "mrkdwn",
                    "text": "*Triggered by:*\n${{ github.actor }}"
                  },
                  {
                    "type": "mrkdwn",
                    "text": "*Commit:*\n`${{ github.sha }}`"
                  }
                ]
              },
              {
                "type": "actions",
                "elements": [
                  {
                    "type": "button",
                    "text": {
                      "type": "plain_text",
                      "text": "View Pipeline"
                    },
                    "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
                  }
                ]
              }
            ]
          }

Comparisons

Automatic vs manual rollback

A health check alone vs a health check + a prompt test

Troubleshooting

"The rollback runs but production is still down"

Cause: The problem isn't the image but the infrastructure (the database, the network, an API key).

Solution: The rollback only changes the Docker image. If the infrastructure is down, you need manual intervention. Check: the server's logs, the database's status, the network connectivity, the API key's balance.

"The health check passes but users report errors"

Cause: The health check is too simple — it only verifies that the app responds with HTTP 200.

Solution: Add a prompt test that verifies real AI functionality.

"The rollback takes too long"

Cause: The previous image isn't in the local cache and needs a full pull.

Solution: Keep the last 2-3 images on the server:

docker image prune --filter "until=168h" --force

This keeps images from the last 7 days, speeding up rollbacks.

"I don't want an automatic rollback for every failure"

Cause: Some failures are acceptable (e.g. a new endpoint that doesn't have a health check yet).

Solution: Use a flag to control the rollback:

- name: Rollback (if enabled)
  if: steps.validate.outcome == 'failure' && vars.AUTO_ROLLBACK == 'true'

Configure AUTO_ROLLBACK as an environment variable in GitHub.

Exercises

Exercise 1: Implement a rollback with image validation

Write the rollback step that first verifies the previous image exists in the registry before attempting the rollback.

- name: Rollback with image verification
  if: steps.validate.outcome == 'failure'
  env:
    SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
    HOST: ${{ vars.PRODUCTION_HOST }}
  run: |
    PREV="${{ steps.current.outputs.tag }}"

    if [ "$PREV" = "unknown" ] || [ -z "$PREV" ]; then
      echo "::error::No previous version — cannot rollback"
      exit 1
    fi

    PREV_IMAGE="${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:$PREV"
    echo "Verifying image exists: $PREV_IMAGE"

    if ! docker manifest inspect "$PREV_IMAGE" > /dev/null 2>&1; then
      echo "::error::Previous image not in registry: $PREV_IMAGE"
      echo "::error::Manual intervention required"
      exit 1
    fi

    echo "Image verified. Rolling back to $PREV"
    echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
    ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
      "cd /app && IMAGE_TAG=$PREV docker compose up -d --pull always"
    rm /tmp/key

    sleep 15
    if curl -sf https://$HOST/health > /dev/null 2>&1; then
      echo "Rollback successful"
    else
      echo "::error::Rollback health check failed!"
    fi
    exit 1

Exercise 2: Simulate a rollback

Add a step that forces a failure in the health check to test that the rollback works.

- name: Force validation failure (TESTING ONLY)
  if: false  # Change to true to test the rollback
  run: |
    echo "Forcing validation failure for rollback testing"
    echo "outcome=failure" >> $GITHUB_OUTPUT

Exercise 3: A rollback with a multi-channel notification

Implement a rollback that notifies both Slack and email.

- name: Notify rollback — Slack
  if: steps.validate.outcome == 'failure'
  uses: slackapi/slack-github-action@v2.0.0
  with:
    webhook-type: incoming-webhook
    webhook: ${{ secrets.SLACK_WEBHOOK_URL }}
    payload: |
      {
        "text": "⚠️ Rollback: ${{ needs.docker.outputs.image-tag }} → ${{ steps.current.outputs.tag }}"
      }

- name: Notify rollback — Email
  if: steps.validate.outcome == 'failure'
  uses: dawidd6/action-send-mail@v3
  with:
    server_address: smtp.gmail.com
    server_port: 587
    username: ${{ secrets.EMAIL_USERNAME }}
    password: ${{ secrets.EMAIL_PASSWORD }}
    subject: "⚠️ Production Rollback — ${{ github.repository }}"
    to: team@example.com
    from: ci@example.com
    body: |
      Production rollback executed.
      Failed: ${{ needs.docker.outputs.image-tag }}
      Reverted to: ${{ steps.current.outputs.tag }}
      Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

Rollback best practices

1. Always retain the last N images

Don't delete old Docker images aggressively. Your rollback depends on the previous image still being available:

# In your registry cleanup, keep at least the last 5 images:
- name: Cleanup old images
  run: |
    # List the tags, sort them by date, keep the last 5
    TAGS=$(gh api /user/packages/container/my-app/versions \
      --jq '.[5:] | .[].id')
    for TAG_ID in $TAGS; do
      gh api --method DELETE /user/packages/container/my-app/versions/$TAG_ID
    done

2. A manual rollback as a fallback

Always have a manual rollback workflow in case the automatic one fails or you need to revert hours later:

name: Manual Rollback
on:
  workflow_dispatch:
    inputs:
      target-tag:
        description: "Docker image tag to rollback to"
        required: true
        type: string
      environment:
        description: "Target environment"
        required: true
        type: choice
        options: [staging, production]

3. Log the rollback's full context

When a rollback happens, you need to answer: why did the deploy fail? Make sure the log includes:

• The version that was attempted
• The version it was rolled back to
• The health check's result (the HTTP status code, the response time)
• The prompt test's result (what it returned, what you expected)
• A precise timestamp

4. Don't roll back database migrations

If your deploy includes database migrations, rolling back the app doesn't revert the migrations. Design your migrations to be backwards-compatible:

✅ Adding a new column (backwards-compatible)
✅ Creating a new table (backwards-compatible)
❌ Renaming a column (it breaks the previous version)
❌ Dropping a table (it breaks the previous version)

5. Define a "rollback budget"

If the rollback takes more than 5 minutes, something more serious is going on. Define a timeout:

- name: Rollback
  timeout-minutes: 5
  run: |
    # ... the rollback logic ...

If the rollback fails on a timeout, the notification has to escalate to on-call.

6. Test the rollback regularly

Don't wait for a real incident to discover that your rollback doesn't work. Create a periodic test:

name: Test Rollback
on:
  schedule:
    - cron: "0 5 * * 6"  # Saturdays 5am UTC
  workflow_dispatch:

jobs:
  test-rollback:
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - name: Deploy known-good version
        run: |
          echo "Deploying test version to staging..."
          # Deploy a version we know passes the health check

      - name: Execute rollback
        run: |
          echo "Testing rollback mechanism..."
          # Run the rollback to the previous tag

      - name: Verify rollback worked
        run: |
          curl -sf https://staging.your-app.com/health || exit 1
          echo "Rollback test passed"

Summary

• ✅ Automatic rollback follows a clear pattern: save → deploy → validate → roll back if it fails
• ✅ continue-on-error: true lets the health check fail without stopping the job
• ✅ The edge cases: the first deploy with no previous version, a rollback that fails, an insufficient health check
• ✅ Complete validation: an HTTP health check + a prompt test for AI systems
• ✅ An immediate notification: a Slack alert when a rollback happens
• ✅ The rollback step always ends with exit 1 so that the job gets marked as a failure
• ✅ Fix forward vs rollback: roll back to stabilize immediately, fix forward afterwards
• ✅ A configurable flag AUTO_ROLLBACK to control whether the rollback is automatic

Additional resources

1. GitHub Actions — continue-on-error - Controlling the flow after errors
2. GitHub Actions — Status Check Functions - failure(), always(), success()
3. Docker Compose — Restart Policies - Automatic container restarts
4. GitHub Actions — Step Outputs - Passing data between steps
5. Blue-Green Deployment Pattern - The zero-downtime deployment pattern
6. GitHub Deployments API - Tracking deployments