6. Rollback Strategies

Overview

The deploy to production failed. The health check doesn't pass. Users are reporting errors. What do you do? If you didn't design a rollback plan before the deploy, this is the worst possible moment to improvise one.

Rollback strategies aren't an emergency plan you write while the incident is already happening. They're an integral part of the pipeline's design. If your pipeline can deploy, it must be able to un-deploy. If it can move forward, it must be able to go back. The tag system you built in Module 5 — the commit's SHA on every image — is exactly what enables fast rollbacks: every previous version is in the registry, ready to be re-deployed.

This capsule teaches you three rollback strategies, when to use each one, and how to integrate rollback into your pipeline so it's automatic when the deploy fails.

Why rollback is a first-class citizen

The scenario without a rollback

Pipeline: Deploy v2 to production → The health check fails

Developer: "Production is down!"
Developer: "What version was there before?"
Developer: "I don't know... latest? Yesterday's?"
Developer: *searches the docker compose logs*
Developer: *searches GHCR for the previous image's tag*
Developer: *ssh into the server, docker pull, docker compose up*
Developer: "Does it work now?"

Downtime: 15-30 minutes
Stress: Maximum
Confidence: "I hope this is the right version"

The scenario with a designed rollback

Pipeline: Deploy v2 to production → The health check fails

Pipeline (automatic):
  1. It detects the health check failure
  2. It gets the previous version's tag (saved in a previous step)
  3. It redeploys the previous version
  4. Health check of the previous version → OK
  5. It notifies: "Deploy failed, rolled back to sha-prev1234"

Downtime: 1-2 minutes
Stress: Low (automatic)
Confidence: "The previous version worked, it's back"

The principle

If you can deploy one version, you can deploy ANY version.
Rollback = deploying a version that already worked.
The tag system makes it possible.

The three rollback strategies

Strategy 1: Redeploy the previous tag

What it is: You take the previous version's image from the registry and deploy it again.

Registry:
  ghcr.io/user/ai-api:sha-abc1234  (v2 — the one that failed)
  ghcr.io/user/ai-api:sha-prev1234 (v1 — the one that worked)
  ghcr.io/user/ai-api:sha-old01234 (v0 — an earlier version)

Rollback:
  docker compose up with IMAGE_TAG=sha-prev1234

The advantages:

• 📋 Fast — The image is already in the registry, you just need a pull + up
• 📋 Predictable — You're deploying exactly the same image that worked
• 📋 No code changes — You don't need to touch the code or make commits

The disadvantages:

• 📋 It requires tracking — You need to know what the previous tag was
• 📋 It doesn't fix the problem — The new version is still broken, you just took it out of production

The implementation:

- name: Save current version before deploy
  id: current
  run: |
    CURRENT_TAG=$(ssh deploy@$HOST \
      "cd /app && docker compose ps --format json" | \
      python3 -c "
    import sys, json
    data = json.load(sys.stdin)
    if isinstance(data, list) and len(data) > 0:
        img = data[0].get('Image', '')
        print(img.split(':')[-1] if ':' in img else 'unknown')
    else:
        print('unknown')
    ")
    echo "tag=$CURRENT_TAG" >> $GITHUB_OUTPUT
    echo "Current version: $CURRENT_TAG"

- name: Deploy new version
  run: |
    ssh deploy@$HOST \
      "cd /app && IMAGE_TAG=sha-${{ github.sha }} docker compose up -d --pull always"

- name: Health check
  id: health
  continue-on-error: true
  run: |
    for i in $(seq 1 30); do
      if curl -sf https://your-app.com/health > /dev/null 2>&1; then
        echo "Healthy after ${i}s"
        exit 0
      fi
      sleep 2
    done
    exit 1

- name: Rollback to previous version
  if: steps.health.outcome == 'failure'
  run: |
    PREV_TAG="${{ steps.current.outputs.tag }}"
    echo "Rolling back to: $PREV_TAG"
    ssh deploy@$HOST \
      "cd /app && IMAGE_TAG=$PREV_TAG docker compose up -d --pull always"

    for i in $(seq 1 15); do
      if curl -sf https://your-app.com/health > /dev/null 2>&1; then
        echo "Rollback successful — running $PREV_TAG"
        exit 1
      fi
      sleep 2
    done
    echo "CRITICAL: Rollback also failed!"
    exit 1

Strategy 2: Revert the commit

What it is: You git revert the commit that caused the problem, which creates a new commit that undoes the changes. This triggers the pipeline normally.

Commits:
  abc1234 ← The broken commit (deployed, it fails)
  prev123 ← The previous commit (it worked)

Revert:
  git revert abc1234 → it creates a new commit def5678
  Push def5678 → Pipeline: test → build → deploy

The result: The code goes back to prev123's state,
  but with a new commit (def5678) documenting the revert.

The advantages:

• 📋 Documented — Git's history shows the revert explicitly
• 📋 It triggers the full pipeline — The tests run again, confirming that the revert works
• 📋 Clean — There are no inconsistent states between Git and production

The disadvantages:

• 📋 Slower — You need to revert, push, and wait for the whole pipeline
• 📋 It requires Git access — You have to be able to push to main
• 📋 It doesn't work in emergencies — If production is down, you need something faster

The implementation:

# From your local machine (or from a workflow)
git revert abc1234 --no-edit
git push origin main

Strategy 3: A manual redeploy via workflow_dispatch

What it is: You trigger the pipeline manually with a specific tag you want to deploy.

on:
  workflow_dispatch:
    inputs:
      image_tag:
        description: "Image tag to deploy"
        required: true
        type: string
      environment:
        description: "Target environment"
        required: true
        type: choice
        options:
          - staging
          - production

A manual rollback:
  GitHub → Actions → Deploy Pipeline → Run workflow
    image_tag: sha-prev1234
    environment: production
  
  → The pipeline deploys sha-prev1234 to production

The advantages:

• 📋 Flexible — You can deploy any tag (not just the previous one)
• 📋 It requires no code changes — You don't need to make commits
• 📋 Visual — The Actions UI shows exactly what was done

The disadvantages:

• 📋 Manual — Someone has to go to GitHub and trigger it
• 📋 It requires knowing the tag — You need to know which version to go back to

The implementation:

name: Deploy Pipeline

on:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      image_tag:
        description: "Image tag (e.g., sha-abc1234 or v1.2.3)"
        required: true
        type: string
      target_environment:
        description: "Target environment"
        required: true
        type: choice
        options:
          - staging
          - production

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: ${{ github.event.inputs.target_environment || 'staging' }}
    steps:
      - name: Determine image tag
        id: tag
        run: |
          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
            echo "tag=${{ github.event.inputs.image_tag }}" >> $GITHUB_OUTPUT
          else
            echo "tag=sha-${{ github.sha }}" >> $GITHUB_OUTPUT
          fi

      - name: Deploy
        run: |
          echo "Deploying ${{ steps.tag.outputs.tag }} to ${{ github.event.inputs.target_environment || 'staging' }}"

When to use each strategy

The decision tree

Is production down?
  ├─ YES → Redeploy the previous tag (Strategy 1) — IMMEDIATE
  │        Afterward: investigate, fix, do a revert or fix forward
  └─ NO → Do you want to document the rollback in Git?
           ├─ YES → Git revert (Strategy 2) — 5-10 min
           └─ NO → Do you know which version to go back to?
                    ├─ YES → Workflow dispatch (Strategy 3) — 2-3 min
                    └─ NO → Check the deployment history on GitHub

The comparison table

Automatic rollback in the pipeline

The complete pattern

deploy-production:
  needs: deploy-staging
  runs-on: ubuntu-latest
  environment:
    name: production
    url: https://your-app.com
  steps:
    - uses: actions/checkout@v4

    # Step 1: Save the current version
    - name: Get current version
      id: current
      env:
        SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        HOST: ${{ vars.DEPLOY_HOST }}
      run: |
        echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
        CURRENT=$(ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
          "cd /app && docker compose ps --format '{{.Image}}'" | head -1 | cut -d: -f2)
        echo "tag=${CURRENT:-unknown}" >> $GITHUB_OUTPUT
        echo "Current production version: ${CURRENT:-unknown}"
        rm /tmp/key

    # Step 2: Deploy the new version
    - name: Deploy new version
      env:
        SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        HOST: ${{ vars.DEPLOY_HOST }}
      run: |
        echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
        ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
          "cd /app && IMAGE_TAG=sha-${{ github.sha }} docker compose up -d --pull always"
        rm /tmp/key

    # Step 3: Health check
    - name: Verify deployment
      id: verify
      continue-on-error: true
      run: |
        sleep 10
        for i in $(seq 1 30); do
          STATUS=$(curl -sf -o /dev/null -w "%{http_code}" \
            https://your-app.com/health 2>/dev/null || echo "000")
          if [ "$STATUS" = "200" ]; then
            echo "Production healthy after $((10 + i*2))s"
            exit 0
          fi
          echo "Attempt $i/30 — HTTP $STATUS"
          sleep 2
        done
        echo "Health check failed after 70s"
        exit 1

    # Step 4: Roll back if it fails
    - name: Rollback on failure
      if: steps.verify.outcome == 'failure'
      env:
        SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        HOST: ${{ vars.DEPLOY_HOST }}
      run: |
        PREV="${{ steps.current.outputs.tag }}"
        echo "ROLLBACK: Reverting to $PREV"

        if [ "$PREV" = "unknown" ] || [ -z "$PREV" ]; then
          echo "::error::Cannot rollback — previous version unknown (first deploy?)"
          exit 1
        fi

        echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
        ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
          "cd /app && IMAGE_TAG=$PREV docker compose up -d --pull always"
        rm /tmp/key

        sleep 10
        for i in $(seq 1 15); do
          if curl -sf https://your-app.com/health > /dev/null 2>&1; then
            echo "Rollback successful — running $PREV"
            echo "::error::Deploy failed for sha-${{ github.sha }}. Rolled back to $PREV."
            exit 1
          fi
          sleep 2
        done

        echo "::error::CRITICAL — Rollback also failed! Manual intervention needed."
        exit 1

    # Step 5: Summary
    - name: Deploy summary
      if: always()
      run: |
        echo "### Production Deployment" >> $GITHUB_STEP_SUMMARY
        echo "" >> $GITHUB_STEP_SUMMARY
        if [ "${{ steps.verify.outcome }}" = "success" ]; then
          echo "**Status:** ✅ Deployed successfully" >> $GITHUB_STEP_SUMMARY
          echo "**Version:** \`sha-${{ github.sha }}\`" >> $GITHUB_STEP_SUMMARY
        else
          echo "**Status:** ❌ Failed — Rolled back" >> $GITHUB_STEP_SUMMARY
          echo "**Failed version:** \`sha-${{ github.sha }}\`" >> $GITHUB_STEP_SUMMARY
          echo "**Rolled back to:** \`${{ steps.current.outputs.tag }}\`" >> $GITHUB_STEP_SUMMARY
        fi

Edge cases

Edge case 1: The first deploy (there's no previous version)

The problem: It's the first deploy. There's no previous image on the server.
What do you roll back to?

The solution: Detect it and don't roll back.

if [ "$PREV" = "unknown" ] || [ -z "$PREV" ]; then
  echo "First deploy — no previous version to rollback to"
  echo "Manual intervention required"
  exit 1
fi

On the first deploy, if it fails, the solution is to fix forward: fix the code and make another push. There's no previous version to go back to.

Edge case 2: The rollback also fails

The problem: Deploy v2 fails. Rollback to v1. But v1 also fails.
The likely cause: The problem isn't the image but the infrastructure
(a full disk, a network issue, the database is down).

The solution: Escalate to manual intervention.

echo "CRITICAL: Rollback failed. Manual intervention needed."
echo "Check: disk space, network, database, external services"
exit 1

If the rollback fails, the problem probably isn't the application. It's infrastructure.

Edge case 3: Database migrations (schema changes)

The problem: Deploy v2 includes a migration that adds a column.
Roll back to v1, but v1 doesn't know about that column.

The solution: Migrations must be backwards-compatible.

v1: SELECT name, email FROM users
v2: ALTER TABLE users ADD COLUMN phone; SELECT name, email, phone FROM users
v1 (rollback): SELECT name, email FROM users → it works (it ignores phone)

The rule: Migrations must be additive (adding, not modifying or deleting).
If you need to delete a column, do it in a later deploy,
after v2 is stable and you're not going to roll back.

Edge case 4: The image tags got cleaned from the registry

The problem: You did a registry cleanup and deleted old images.
Now you need to roll back to an image that no longer exists.

The solution: A retention policy that keeps at least the last N images.

# In GHCR, you can configure retention:
# Settings → Packages → Package settings → Manage versions
# Keep at least the last 10 versions

# Or implement it in the pipeline:
- name: Keep last 10 images
  run: |
    VERSIONS=$(gh api /user/packages/container/ai-api/versions \
      --jq '.[].id' | tail -n +11)
    for V in $VERSIONS; do
      gh api --method DELETE /user/packages/container/ai-api/versions/$V
    done

Comparisons

An automatic rollback vs a manual rollback

Fix forward vs rollback

Revert vs redeploying a tag

Troubleshooting

1. "Cannot rollback — previous version unknown"

Symptom: The rollback step doesn't know which version to go back to.

Cause: The step that saves the previous version failed or didn't run.

Solution: Add a fallback that queries GitHub's deployment history:

- name: Get previous version (fallback)
  if: steps.current.outputs.tag == 'unknown'
  id: fallback
  run: |
    PREV=$(gh api repos/${{ github.repository }}/deployments \
      --jq '[.[] | select(.environment=="production" and .task=="deploy")] | .[1].sha' \
      | cut -c1-7)
    echo "tag=sha-$PREV" >> $GITHUB_OUTPUT

2. The rollback succeeds but the job gets marked as "success"

Symptom: The rollback works, but the workflow shows green instead of red.

Cause: The rollback step doesn't fail with exit 1.

Solution: The rollback step must always end with exit 1:

- name: Rollback
  if: steps.verify.outcome == 'failure'
  run: |
    # ... rollback commands ...
    echo "Rollback successful"
    exit 1    # The job must fail to indicate that the original deploy didn't work

3. The previous image was deleted from the registry

Symptom: docker pull fails with "manifest not found" for the previous tag.

Cause: The registry did a cleanup and deleted the image.

Solution: Implement a retention policy that keeps at least the last 10 images. And as a fallback, do a git revert + the full pipeline.

4. A rollback loop — the pipeline auto-triggers with the rollback

Symptom: The rollback triggers a new pipeline run that tries to deploy again.

Cause: The rollback doesn't trigger a new pipeline (it only re-deploys an image). But if you do a git revert, it does trigger the pipeline, which is correct.

Solution: This isn't a problem — the git revert creates a new, clean commit that goes through the whole pipeline normally. If the revert is correct, the deploy will succeed.

Exercises

Exercise 1: Implement the "save current version" pattern

Write a GitHub Actions step that connects via SSH to a server, gets the tag of the Docker image currently running with docker compose, and saves it as the step's output.

- name: Get current version
  id: current
  env:
    SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
    HOST: ${{ vars.DEPLOY_HOST }}
  run: |
    echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key

    CURRENT_IMAGE=$(ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
      "cd /app && docker compose ps --format '{{.Image}}'" 2>/dev/null | head -1)

    rm /tmp/key

    if [ -z "$CURRENT_IMAGE" ]; then
      echo "No container running (first deploy?)"
      echo "tag=unknown" >> $GITHUB_OUTPUT
    else
      CURRENT_TAG=$(echo "$CURRENT_IMAGE" | cut -d: -f2)
      echo "Current version: $CURRENT_TAG"
      echo "tag=$CURRENT_TAG" >> $GITHUB_OUTPUT
    fi

Exercise 2: A conditional rollback with the first-deploy edge case

Write a step that rolls back, but handles the first-deploy case (no previous version) correctly.

- name: Rollback on failure
  if: steps.verify.outcome == 'failure'
  env:
    SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
    HOST: ${{ vars.DEPLOY_HOST }}
  run: |
    PREV="${{ steps.current.outputs.tag }}"

    if [ "$PREV" = "unknown" ] || [ -z "$PREV" ]; then
      echo "::error::First deploy failed — no previous version to rollback to."
      echo "::error::Fix the issue and push again. The deploy will be retried."
      echo "" >> $GITHUB_STEP_SUMMARY
      echo "### ❌ Deploy Failed (First Deploy)" >> $GITHUB_STEP_SUMMARY
      echo "No previous version available for rollback." >> $GITHUB_STEP_SUMMARY
      echo "Fix the code and push a new commit." >> $GITHUB_STEP_SUMMARY
      exit 1
    fi

    echo "Rolling back to: $PREV"
    echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key

    ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
      "cd /app && IMAGE_TAG=$PREV docker compose up -d --pull always"
    rm /tmp/key

    sleep 10
    for i in $(seq 1 15); do
      if curl -sf https://$HOST/health > /dev/null 2>&1; then
        echo "Rollback OK — production running $PREV"
        echo "" >> $GITHUB_STEP_SUMMARY
        echo "### ⚠️ Deploy Failed — Rolled Back" >> $GITHUB_STEP_SUMMARY
        echo "- **Failed:** \`sha-${{ github.sha }}\`" >> $GITHUB_STEP_SUMMARY
        echo "- **Rolled back to:** \`$PREV\`" >> $GITHUB_STEP_SUMMARY
        exit 1
      fi
      sleep 2
    done

    echo "::error::CRITICAL: Rollback also failed!"
    exit 1

Exercise 3: Workflow dispatch for a manual rollback

Create a complete workflow that lets you roll back manually to any tag. The workflow must: (1) accept the tag and the environment as inputs, (2) verify that the image exists in the registry, (3) deploy, (4) verify health.

name: Manual Rollback

on:
  workflow_dispatch:
    inputs:
      image_tag:
        description: "Image tag to deploy (e.g., sha-abc1234, v1.2.3)"
        required: true
        type: string
      environment:
        description: "Target environment"
        required: true
        type: choice
        options:
          - staging
          - production

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment:
      name: ${{ github.event.inputs.environment }}
    steps:
      - uses: actions/checkout@v4

      - name: Verify image exists
        run: |
          IMAGE="${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.event.inputs.image_tag }}"
          echo "Checking: $IMAGE"
          if docker manifest inspect "$IMAGE" > /dev/null 2>&1; then
            echo "Image found: $IMAGE"
          else
            echo "::error::Image not found: $IMAGE"
            echo "Available tags:"
            gh api /user/packages/container/$(echo ${{ env.IMAGE_NAME }} | cut -d/ -f2)/versions \
              --jq '.[0:10] | .[].metadata.container.tags[]' 2>/dev/null || true
            exit 1
          fi

      - name: Deploy
        env:
          SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
          HOST: ${{ vars.DEPLOY_HOST }}
        run: |
          echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
          ssh -o StrictHostKeyChecking=no -i /tmp/key deploy@$HOST \
            "cd /app && IMAGE_TAG=${{ github.event.inputs.image_tag }} docker compose up -d --pull always"
          rm /tmp/key

      - name: Verify health
        run: |
          ENV="${{ github.event.inputs.environment }}"
          URL="${{ vars.DEPLOY_HOST }}"
          for i in $(seq 1 30); do
            if curl -sf "https://${URL}/health" > /dev/null 2>&1; then
              echo "Rollback successful: $ENV running ${{ github.event.inputs.image_tag }}"
              exit 0
            fi
            sleep 2
          done
          echo "::error::Health check failed after rollback"
          exit 1

      - name: Summary
        if: always()
        run: |
          echo "### Manual Rollback" >> $GITHUB_STEP_SUMMARY
          echo "**Environment:** ${{ github.event.inputs.environment }}" >> $GITHUB_STEP_SUMMARY
          echo "**Image tag:** \`${{ github.event.inputs.image_tag }}\`" >> $GITHUB_STEP_SUMMARY
          echo "**Triggered by:** @${{ github.actor }}" >> $GITHUB_STEP_SUMMARY

Exercise 4: Choose the rollback strategy

For each scenario, choose the most appropriate rollback strategy:

1. Production is down at 3 AM, the team is asleep
2. After a fix, you want to clean up Git's history
3. You need to go back to a version from 2 weeks ago
4. A prompt change generates incorrect responses but the app works

Summary

• ✅ Rollback is a first-class citizen — designed from the start, not as an afterthought
• ✅ Strategy 1: Redeploy the previous tag — fast, automatic, for emergencies
• ✅ Strategy 2: Git revert — documented, the full pipeline, for post-emergency cleanup
• ✅ Strategy 3: Workflow dispatch — flexible, manual, for any tag
• ✅ Module 5's tag system enables rollbacks — every version is in the registry
• ✅ The edge cases: a first deploy with no previous version, a rollback that also fails, migrations
• ✅ An automatic rollback in the pipeline: save the version → deploy → health check → roll back if it fails
• ✅ Fix forward vs rollback: rollback to stabilize, fix forward when the fix is quick

Additional resources

1. GitHub Actions — Rerunning workflows — Re-running workflows for a re-deploy
2. Git revert — Official git revert documentation
3. GitHub Actions — workflow_dispatch — Manual triggers for a rollback
4. Docker Compose — up — The docker compose up reference
5. GitHub Deployments API — The deployment history, to find previous versions
6. Backwards-compatible database migrations — Stripe on rollback-safe migrations