4. Deployment Strategies

Overview

Your pipeline already deploys automatically to staging and with approval to production. But how does the deploy itself get executed? When you say "deploy the new version," what happens to the current version? Does it shut down completely while the new one starts? Is there a moment where no version is available? Do users experience downtime?

Deployment strategies answer these questions. Each strategy defines how you transition from the current version to the new one. Some prioritize simplicity, others prioritize zero-downtime, and others prioritize the ability to test with a subset of users before deploying fully.

In this capsule you implement rolling update with docker-compose — the most universal strategy and the one you need for this guide. You also learn blue-green and canary conceptually so you understand when you would need more advanced infrastructure (guide #17).

The three main strategies

The overview

Rolling Update:
  Version A gets replaced gradually by Version B
  [A][A][A] → [A][A][B] → [A][B][B] → [B][B][B]
  
Blue-Green:
  Two complete environments — one active, one standby
  [A active] [B standby] → switch → [A standby] [B active]
  
Canary:
  A small percentage of the traffic goes to the new version
  [A 100%] → [A 90% | B 10%] → [A 50% | B 50%] → [B 100%]

Rolling Update — Implementation

What it is

Rolling update replaces instances of the current version with the new version gradually. At any moment during the deploy, there are instances of both versions running.

Start:         [v1] [v1] [v1]
Step 1:        [v1] [v1] [v2]  ← one new instance
Step 2:        [v1] [v2] [v2]  ← another new instance
Step 3:        [v2] [v2] [v2]  ← the deploy is complete

When to use it

• 📋 Stateless applications — Your FastAPI app doesn't keep state in memory between requests
• 📋 Simple infrastructure — docker-compose on a server, no Kubernetes
• 📋 Tolerance for two versions coexisting — Requests during the deploy can go to v1 or v2
• 📋 Limited resources — You don't need a second complete environment

Implementation with docker-compose

The base docker-compose.yml

# docker-compose.yml
services:
  ai-api:
    image: ghcr.io/user/ai-api:latest
    ports:
      - "8000:8000"
    environment:
      - APP_ENV=production
      - LOG_LEVEL=warning
    env_file:
      - .env
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 40s
    restart: unless-stopped

docker-compose.staging.yml (the staging override)

# docker-compose.staging.yml
services:
  ai-api:
    image: ghcr.io/user/ai-api:${IMAGE_TAG:-latest}
    ports:
      - "8001:8000"
    environment:
      - APP_ENV=staging
      - LOG_LEVEL=debug

docker-compose.prod.yml (the production override)

# docker-compose.prod.yml
services:
  ai-api:
    image: ghcr.io/user/ai-api:${IMAGE_TAG:-latest}
    ports:
      - "8000:8000"
    environment:
      - APP_ENV=production
      - LOG_LEVEL=warning

The deploy script

#!/bin/bash
# scripts/deploy.sh

set -euo pipefail

ENVIRONMENT=${1:?"Usage: deploy.sh <staging|production> <image_tag>"}
IMAGE_TAG=${2:?"Usage: deploy.sh <staging|production> <image_tag>"}

echo "=== Deploying $IMAGE_TAG to $ENVIRONMENT ==="

COMPOSE_FILE="docker-compose.yml"
OVERRIDE_FILE="docker-compose.${ENVIRONMENT}.yml"

if [ ! -f "$OVERRIDE_FILE" ]; then
  echo "Error: $OVERRIDE_FILE not found"
  exit 1
fi

export IMAGE_TAG

CURRENT_IMAGE=$(docker compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE" \
  ps --format json | python3 -c "
import sys, json
data = json.load(sys.stdin)
if isinstance(data, list) and len(data) > 0:
    print(data[0].get('Image', 'none'))
else:
    print('none')
" 2>/dev/null || echo "none")
echo "Current image: $CURRENT_IMAGE"
echo "New image: ghcr.io/user/ai-api:$IMAGE_TAG"

echo "Pulling new image..."
docker compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE" pull

echo "Deploying with rolling update..."
docker compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE" up -d --remove-orphans

echo "Waiting for health check..."
for i in $(seq 1 30); do
  HEALTH=$(docker compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE" \
    ps --format json | python3 -c "
import sys, json
data = json.load(sys.stdin)
if isinstance(data, list) and len(data) > 0:
    print(data[0].get('Health', 'unknown'))
else:
    print('unknown')
" 2>/dev/null || echo "unknown")

  if [ "$HEALTH" = "healthy" ]; then
    echo "Deploy successful: $ENVIRONMENT is healthy after ${i}s"
    exit 0
  fi
  echo "Waiting for health... ($i/30) Status: $HEALTH"
  sleep 2
done

echo "ERROR: Health check failed after 60 seconds"
echo "Rolling back to previous image: $CURRENT_IMAGE"

export IMAGE_TAG=$(echo "$CURRENT_IMAGE" | cut -d: -f2)
docker compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE" up -d --remove-orphans
exit 1

Rolling update in GitHub Actions

deploy-staging:
  needs: build
  runs-on: ubuntu-latest
  environment:
    name: staging
    url: https://staging.your-app.com
  steps:
    - uses: actions/checkout@v4

    - name: Deploy to staging
      env:
        SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        DEPLOY_HOST: ${{ vars.DEPLOY_HOST }}
        IMAGE_TAG: sha-${{ github.sha }}
      run: |
        echo "$SSH_KEY" > /tmp/deploy_key
        chmod 600 /tmp/deploy_key

        ssh -o StrictHostKeyChecking=no -i /tmp/deploy_key deploy@$DEPLOY_HOST \
          "cd /app && IMAGE_TAG=$IMAGE_TAG bash scripts/deploy.sh staging $IMAGE_TAG"

        rm /tmp/deploy_key

    - name: Smoke test
      run: |
        for i in $(seq 1 30); do
          if curl -sf https://staging.your-app.com/health > /dev/null 2>&1; then
            echo "Staging healthy after ${i}s"
            exit 0
          fi
          sleep 2
        done
        echo "Staging health check failed"
        exit 1

How the rolling update works with docker-compose

The initial state:
  The ai-api container (v1) is running, healthy

docker compose up -d runs:
  1. Pull the v2 image
  2. Stop the v1 container
  3. Create the v2 container
  4. Start the v2 container
  5. The healthcheck begins

The downtime window:
  Between step 2 and step 5 there's a moment with no service
  For a single instance: ~10-30 seconds typically
  With a load balancer and multiple replicas: zero downtime

For a single instance (this guide's case), there's a brief downtime. For zero-downtime with docker-compose, you need multiple replicas and a reverse proxy — that's covered in guide #17.

Blue-Green Deployment — Conceptual

What it is

Blue-green keeps two identical environments. "Blue" is the current version in production. "Green" is the new version. You deploy the new version to Green, verify it, and then switch the traffic from Blue to Green.

Before the deploy:
  [Blue: v1 — ACTIVE] ←── User traffic
  [Green: empty]

Deploy to Green:
  [Blue: v1 — ACTIVE] ←── User traffic
  [Green: v2 — deploying]

Verify Green:
  [Blue: v1 — ACTIVE] ←── User traffic
  [Green: v2 — healthy ✅]

Switch (the traffic change):
  [Blue: v1 — standby]
  [Green: v2 — ACTIVE] ←── User traffic

Rollback (if something fails):
  [Blue: v1 — ACTIVE] ←── User traffic  (switch back)
  [Green: v2 — standby]

The advantages

• 📋 Zero downtime — The traffic switch is instant
• 📋 Instant rollback — You just switch the traffic back to Blue
• 📋 Testing in production — You can verify Green with real traffic before the switch
• 📋 A clean environment — Green gets created from scratch, with no contamination from the previous one

The disadvantages

• 📋 Double infrastructure — You need two complete environments (2x the cost)
• 📋 Database complexity — If Blue and Green share a database, migrations get complicated
• 📋 It requires a load balancer — To switch traffic between Blue and Green

When to use it

You need blue-green when:
  ✅ Zero downtime is a requirement
  ✅ The rollback must be instant (seconds, not minutes)
  ✅ You have the budget for double infrastructure
  ✅ You have a load balancer or DNS-based routing

You don't need blue-green when:
  ❌ 10-30 seconds of downtime is acceptable
  ❌ Your infrastructure is a single server
  ❌ Your budget doesn't allow a double environment
  ❌ You're starting out and rolling update is enough

The conceptual implementation

# Conceptual — it requires the infrastructure of guide #17
deploy-green:
  steps:
    - name: Deploy the new version to Green
      run: |
        # Deploy to the Green environment (a separate server or container group)
        deploy_to_environment "green" "$IMAGE_TAG"

    - name: Verify Green
      run: |
        # A health check of the Green environment
        curl -sf https://green.your-app.com/health

    - name: Switch traffic to Green
      run: |
        # Change the load balancer / DNS to point to Green
        switch_traffic "green"

    - name: Verify production
      run: |
        # Verify that the traffic reaches Green
        curl -sf https://your-app.com/health

A real blue-green implementation requires a load balancer (Nginx, HAProxy, AWS ALB) or DNS-based routing. That's infrastructure covered by guide #17.

Canary Deployment — Conceptual

What it is

Canary sends a small percentage of traffic to the new version. If that version works well, you gradually increase the percentage until all the load goes to the new version.

Step 1: 5% canary
  [v1: 95% of the traffic] [v2: 5% of the traffic]
  → Monitor errors, latency, costs

Step 2: 25% canary
  [v1: 75% of the traffic] [v2: 25% of the traffic]
  → If the metrics are OK, increase

Step 3: 50% canary
  [v1: 50% of the traffic] [v2: 50% of the traffic]
  → Almost confirmed, one more verification

Step 4: 100% complete
  [v2: 100% of the traffic]
  → The deploy is complete

If there are problems at any step:
  [v1: 100% of the traffic] [v2: 0%]
  → An immediate rollback

The advantages

• 📋 Minimal risk — Only a small percentage of users see the new version
• 📋 Real data — You can compare metrics between v1 and v2 with real traffic
• 📋 A partial rollback — If v2 fails, it only affects the canary percentage

The disadvantages

• 📋 High complexity — You need traffic splitting at the load balancer level
• 📋 Monitoring is mandatory — Without comparative metrics, canary makes no sense
• 📋 The deploy's duration — A full canary can take hours

When to use it

You need canary when:
  ✅ Your application has significant traffic (thousands of requests/minute)
  ✅ You have robust monitoring (metrics for error rate, latency, cost)
  ✅ The changes are high-risk (a new model, a new prompt, a new feature)
  ✅ You need to validate with real traffic before a full commit

You don't need canary when:
  ❌ Your traffic is low (a few requests per minute)
  ❌ You don't have comparative monitoring
  ❌ Rolling update with a rollback is enough
  ❌ The configuration overhead doesn't justify the benefit

The relevance for AI

Canary is particularly relevant for AI systems because:

The scenario: You change the model from gpt-4o-mini to gpt-4o

The canary deploy:
  5% of the requests → gpt-4o (the new one)
  95% of the requests → gpt-4o-mini (the current one)

You monitor for 1 hour:
  - Response quality (are they better?)
  - Cost per request (gpt-4o is more expensive)
  - Latency (is it slower?)
  - Error rate (are there more failures?)

If everything is OK → increase to 50% → 100%
If not → roll back to 0%, everyone stays on gpt-4o-mini

Changes to the model, the prompt, and AI configuration are natural candidates for canary deployment. But the implementation requires traffic splitting — something guide #17 covers with infrastructure like Nginx, Istio, or AWS ALB.

Comparing the three strategies

The comparison table

When to use each one

                    Infrastructure complexity
                    Low ──────────────────── High
                    
Rolling Update ────── Blue-Green ────── Canary
     │                    │                │
     ▼                    ▼                ▼
  One server         Two environments  Traffic splitting
  docker-compose     A load balancer   + monitoring
  This guide         Guide #17         Guides #17+#18

The decision tree

Does your app tolerate 10-30s of downtime?
  ├─ YES → Rolling Update (this guide)
  └─ NO → Do you have the budget for double infrastructure?
           ├─ YES → Blue-Green (guide #17)
           └─ NO → Can you tolerate slower deploys?
                    ├─ YES → Canary (guide #17)
                    └─ NO → Blue-Green with minimal infra

For this guide, you implement rolling update. It's the strategy that works with one server and docker-compose. You'll implement the other strategies when you have more advanced infrastructure.

The progressive implementation of rolling update

Level 1: A basic deploy (functional but fragile)

deploy:
  steps:
    - name: Deploy
      run: |
        ssh deploy@server "cd /app && \
          docker compose pull && \
          docker compose up -d"

The problems: no health check, no rollback, no timeout.

Level 2: A deploy with a health check (better)

deploy:
  steps:
    - name: Deploy
      run: |
        ssh deploy@server "cd /app && \
          docker compose pull && \
          docker compose up -d"

    - name: Health check
      run: |
        for i in $(seq 1 30); do
          if curl -sf https://app.com/health; then
            echo "Healthy after ${i}s"
            exit 0
          fi
          sleep 2
        done
        exit 1

Better: it verifies that the app is healthy. But if it fails, it doesn't roll back.

Level 3: A deploy with a rollback (production)

deploy:
  steps:
    - name: Save current version
      id: current
      run: |
        CURRENT=$(ssh deploy@server "cd /app && \
          docker compose ps --format json | \
          python3 -c 'import sys,json; d=json.load(sys.stdin); print(d[0][\"Image\"].split(\":\")[1] if d else \"none\")'")
        echo "tag=$CURRENT" >> $GITHUB_OUTPUT

    - name: Deploy new version
      run: |
        ssh deploy@server "cd /app && \
          IMAGE_TAG=sha-${{ github.sha }} docker compose pull && \
          IMAGE_TAG=sha-${{ github.sha }} docker compose up -d"

    - name: Health check
      id: health
      continue-on-error: true
      run: |
        for i in $(seq 1 30); do
          if curl -sf https://app.com/health; then
            echo "Healthy after ${i}s"
            exit 0
          fi
          sleep 2
        done
        exit 1

    - name: Rollback on failure
      if: steps.health.outcome == 'failure'
      run: |
        echo "Health check failed — rolling back to ${{ steps.current.outputs.tag }}"
        ssh deploy@server "cd /app && \
          IMAGE_TAG=${{ steps.current.outputs.tag }} docker compose pull && \
          IMAGE_TAG=${{ steps.current.outputs.tag }} docker compose up -d"
        exit 1

This is the level you implement in this module's project. It saves the current version, deploys the new one, verifies health, and if it fails, rolls back to the previous one.

Troubleshooting

1. "Container unhealthy" after the deploy

Symptom: The container starts but the health check fails.

Cause: The healthcheck's start_period is shorter than your AI app's startup time (which can take 30-60s loading models).

Solution:

# docker-compose.yml
services:
  ai-api:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 60s    # Give the startup more time

2. Port already in use after a restart

Symptom:

Error: Bind for 0.0.0.0:8000 failed: port is already allocated

Cause: The previous container didn't stop completely before the new one started.

Solution:

docker compose down --remove-orphans && docker compose up -d

Or in the deploy script:

docker compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE" down --timeout 30
docker compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE" up -d

3. SSH connection refused on the deploy from Actions

Symptom:

ssh: connect to host staging.app.com port 22: Connection refused

Cause: The server's firewall doesn't allow SSH from GitHub Actions' runners, or the SSH key is wrong.

Solution:

# On the server, allow SSH from GitHub Actions' IPs
# Or use an SSH jump host / bastion
# Or use a self-hosted runner inside your network

# Verify the key:
ssh -i /tmp/deploy_key -o StrictHostKeyChecking=no deploy@staging.app.com echo "OK"

4. The image doesn't pull — "manifest unknown"

Symptom:

Error response from daemon: manifest for ghcr.io/user/ai-api:sha-abc1234 not found

Cause: The image wasn't pushed to the registry, or the tag is wrong.

Solution:

# Verify that the image exists
docker manifest inspect ghcr.io/user/ai-api:sha-abc1234

# If it doesn't exist, check the build job in Actions
# The tag must exactly match what metadata-action generated

Exercises

Exercise 1: Write a health check loop

Write a bash script that: (1) tries to curl a /health endpoint every 2 seconds, (2) has a maximum of 30 attempts (60 seconds), (3) prints the attempt number, (4) returns exit 0 if healthy or exit 1 on timeout.

#!/bin/bash
HEALTH_URL="${1:-http://localhost:8000/health}"
MAX_ATTEMPTS=30
INTERVAL=2

echo "Checking health at: $HEALTH_URL"

for i in $(seq 1 $MAX_ATTEMPTS); do
  HTTP_CODE=$(curl -sf -o /dev/null -w "%{http_code}" "$HEALTH_URL" 2>/dev/null || echo "000")

  if [ "$HTTP_CODE" = "200" ]; then
    echo "✅ Healthy after attempt $i (${i}x${INTERVAL}s = $((i * INTERVAL))s)"
    exit 0
  fi

  echo "Attempt $i/$MAX_ATTEMPTS — HTTP $HTTP_CODE — waiting ${INTERVAL}s..."
  sleep $INTERVAL
done

echo "❌ Health check failed after $MAX_ATTEMPTS attempts ($((MAX_ATTEMPTS * INTERVAL))s)"
exit 1

Exercise 2: A docker-compose with an image tag variable

Create a docker-compose.yml that uses an IMAGE_TAG environment variable for the image's tag, with a default of latest. Include a healthcheck and a restart policy.

# docker-compose.yml
services:
  ai-api:
    image: ghcr.io/user/ai-api:${IMAGE_TAG:-latest}
    ports:
      - "8000:8000"
    environment:
      - APP_ENV=${APP_ENV:-development}
    env_file:
      - .env
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 40s
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 2G

IMAGE_TAG=sha-abc1234 docker compose up -d
IMAGE_TAG=v1.2.3 docker compose up -d
docker compose up -d

Exercise 3: Choose the strategy for each case

For each scenario, choose rolling update, blue-green, or canary, and justify it:

1. A personal AI app with 10 users, one server, 30s of downtime acceptable
2. An e-commerce platform with 50k concurrent users, a 99.99% uptime SLA
3. A company's internal chatbot that completely changed its prompt system
4. An embeddings API receiving 100 requests/second, changing its model

Exercise 4: A deploy with a rollback in a workflow

Create a GitHub Actions job that: (1) saves the current version as an output, (2) deploys the new version, (3) runs a health check with continue-on-error, (4) rolls back if the health check fails.

deploy-production:
  needs: deploy-staging
  runs-on: ubuntu-latest
  environment:
    name: production
    url: https://your-app.com
  steps:
    - name: Save current version
      id: current
      run: |
        CURRENT_TAG=$(curl -sf https://your-app.com/health | \
          python3 -c "import sys,json; print(json.load(sys.stdin).get('version','unknown'))")
        echo "tag=$CURRENT_TAG" >> $GITHUB_OUTPUT
        echo "Current version: $CURRENT_TAG"

    - name: Deploy new version
      env:
        SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        HOST: ${{ vars.DEPLOY_HOST }}
      run: |
        echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
        ssh -i /tmp/key deploy@$HOST \
          "cd /app && IMAGE_TAG=sha-${{ github.sha }} docker compose up -d"
        rm /tmp/key

    - name: Health check
      id: health
      continue-on-error: true
      run: |
        for i in $(seq 1 30); do
          if curl -sf https://your-app.com/health > /dev/null; then
            echo "Healthy after ${i}s"
            exit 0
          fi
          sleep 2
        done
        echo "Health check failed after 60s"
        exit 1

    - name: Rollback on failure
      if: steps.health.outcome == 'failure'
      env:
        SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        HOST: ${{ vars.DEPLOY_HOST }}
      run: |
        PREV_TAG="${{ steps.current.outputs.tag }}"
        echo "Rolling back to: $PREV_TAG"
        echo "$SSH_KEY" > /tmp/key && chmod 600 /tmp/key
        ssh -i /tmp/key deploy@$HOST \
          "cd /app && IMAGE_TAG=$PREV_TAG docker compose up -d"
        rm /tmp/key
        echo "::error::Deploy failed — rolled back to $PREV_TAG"
        exit 1

Summary

• ✅ Rolling update is the strategy for this guide: it replaces the current version with docker-compose up -d
• ✅ Blue-green keeps two environments and switches the traffic — it requires a load balancer (guide #17)
• ✅ Canary sends a % of the traffic to the new version — it requires traffic splitting and monitoring (guide #17)
• ✅ Rolling update with docker-compose has a brief downtime (10-30s) that's acceptable for most apps
• ✅ Health checks verify that the new version is working after the deploy
• ✅ An automatic rollback redeploys the previous version if the health check fails
• ✅ The strategy depends on your requirements: downtime tolerance, budget, complexity, traffic
• ✅ The progression: rolling update (now) → blue-green (guide #17) → canary (guides #17+#18)

Additional resources

1. Docker Compose — Deploy — Best practices for compose in production
2. Kubernetes — Rolling Update — Rolling update in K8s (conceptual, for comparison)
3. Martin Fowler — Blue-Green Deployment — The original pattern
4. Canary Releases — Martin Fowler on canary deployment
5. GitHub Actions — SSH deploy — An action for running SSH commands
6. Docker Healthcheck — Docker's healthcheck reference