3. Deployment Automation — From Git Push to Production

Description

In this capsule you'll automate the complete deployment flow: from a git push to your AI system running in production. Not just the deploy — also environment promotion (local → staging → production) and a rollback strategy when something goes wrong. By the end, a push to main automatically triggers tests, build, deployment, and validation.

Context: The previous capsule defined the integration architecture. Now you implement it. Each pipeline step you used to do manually becomes an automated GitHub Actions job. Automation eliminates human error and makes the deployment reproducible.

The Principle: One Push, One Deploy

Why automate

Manual deployment:
  1. Run tests locally (sometimes forgotten)
  2. Build the Docker image
  3. Upload the image to the registry
  4. Connect to the platform
  5. Trigger the deploy
  6. Verify that it works
  Total: 10-20 min, error-prone, depends on YOU doing it

Automated deployment:
  1. git push main
  Total: 3-5 min, reproducible, happens the same way every time

In AI development, you iterate constantly: you tweak prompts, change parameters, experiment with models. If each deploy takes 20 manual minutes, you do 2-3 a day. With automation, you do 10+. Iteration speed directly impacts the quality of the system.

GitHub Actions: The Complete Pipeline

Workflow structure

# .github/workflows/deploy.yml
name: Deploy AI System to Production

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  workflow_dispatch:

concurrency:
  group: deployment-${{ github.ref }}
  cancel-in-progress: true

env:
  PYTHON_VERSION: "3.11"
  APP_NAME: "my-ai-app"

The concurrency block is important: if you make two quick pushes, the second cancels the first. You don't want two simultaneous deploys competing.

Job 1: Test

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: 'pip'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install pytest httpx pytest-asyncio

      - name: Run unit tests
        run: pytest tests/unit/ -v --tb=short -q
        env:
          ENVIRONMENT: test

      - name: Run integration tests
        run: pytest tests/integration/ -v --tb=short -q
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ENVIRONMENT: test

Job 2: Build and verify the Docker image

  build:
    needs: test
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Build Docker image
        run: docker build -t ${{ env.APP_NAME }}:${{ github.sha }} .

      - name: Verify image starts
        run: |
          docker run -d --name verify \
            -p 8000:8000 \
            -e OPENAI_API_KEY=test-key \
            -e ENVIRONMENT=test \
            ${{ env.APP_NAME }}:${{ github.sha }}
          sleep 5
          status=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8000/health || echo "000")
          docker logs verify
          docker stop verify && docker rm verify
          if [ "$status" != "200" ]; then
            echo "Image verification failed: status $status"
            exit 1
          fi
          echo "Image verified successfully"

Job 3: Deploy (per-platform variants)

Render:

  deploy:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    needs: build
    runs-on: ubuntu-latest
    timeout-minutes: 10
    environment: production
    steps:
      - name: Trigger Render deploy
        run: |
          response=$(curl -s -o /dev/null -w "%{http_code}" \
            -X POST "${{ secrets.RENDER_DEPLOY_HOOK_URL }}")
          if [ "$response" != "200" ] && [ "$response" != "201" ]; then
            echo "Deploy trigger failed: $response"
            exit 1
          fi
          echo "Deploy triggered successfully"

Railway:

  deploy:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    needs: build
    runs-on: ubuntu-latest
    timeout-minutes: 10
    environment: production
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install Railway CLI
        run: npm install -g @railway/cli

      - name: Deploy to Railway
        run: railway up --detach --service ${{ vars.RAILWAY_SERVICE_ID }}
        env:
          RAILWAY_TOKEN: ${{ secrets.RAILWAY_TOKEN }}

Fly.io:

  deploy:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    needs: build
    runs-on: ubuntu-latest
    timeout-minutes: 10
    environment: production
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup flyctl
        uses: superfly/flyctl-actions/setup-flyctl@master

      - name: Deploy to Fly.io
        run: flyctl deploy --remote-only
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

Job 4: Post-deploy validation

  validate:
    needs: deploy
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Wait for deployment to stabilize
        run: sleep 45

      - name: Health check with retry
        run: |
          MAX_RETRIES=5
          RETRY_DELAY=15
          for i in $(seq 1 $MAX_RETRIES); do
            status=$(curl -s -o /dev/null -w "%{http_code}" \
              "${{ vars.PRODUCTION_URL }}/health")
            if [ "$status" = "200" ]; then
              echo "Health check passed on attempt $i"
              exit 0
            fi
            echo "Attempt $i/$MAX_RETRIES: status $status. Retrying in ${RETRY_DELAY}s..."
            sleep $RETRY_DELAY
          done
          echo "Health check failed after $MAX_RETRIES attempts"
          exit 1

      - name: Smoke test - inference
        run: |
          response=$(curl -s -w "\n%{http_code}" -X POST \
            "${{ vars.PRODUCTION_URL }}/api/inference" \
            -H "Content-Type: application/json" \
            -d '{"prompt": "Respond with exactly: SMOKE_TEST_OK"}')
          http_code=$(echo "$response" | tail -1)
          body=$(echo "$response" | head -1)
          echo "Status: $http_code"
          echo "Body: $body"
          if [ "$http_code" != "200" ]; then
            echo "Smoke test failed: HTTP $http_code"
            exit 1
          fi
          echo "Smoke test passed"

      - name: Notify success
        if: success()
        run: echo "Deployment validated successfully at $(date)"

      - name: Notify failure
        if: failure()
        run: echo "DEPLOYMENT VALIDATION FAILED - manual intervention required"

Environment Promotion

The flow: local → staging → production

┌─────────────┐     ┌─────────────┐     ┌─────────────────┐
│   LOCAL      │────►│  STAGING    │────►│  PRODUCTION     │
│             │     │             │     │                 │
│ docker      │     │ Same image  │     │ Same image      │
│ compose up  │     │ staging     │     │ production      │
│             │     │ env vars    │     │ env vars        │
│ .env.local  │     │ Subset of   │     │ Full traffic    │
│             │     │ traffic     │     │                 │
└─────────────┘     └─────────────┘     └─────────────────┘
     develop             test               release
     branch           PR/staging            main branch

Implementation with GitHub Actions environments

# For staging (activates on PRs or push to develop)
deploy-staging:
  if: github.event_name == 'pull_request'
  needs: build
  runs-on: ubuntu-latest
  environment: staging
  steps:
    - uses: actions/checkout@v4
    - name: Deploy to staging
      run: |
        # Your platform may have a separate service for staging
        # Railway: different service, same project
        # Render: different service, same account
        # Fly.io: different app (my-app-staging)
        echo "Deploying to staging..."

# For production (only push to main, after staging)
deploy-production:
  if: github.ref == 'refs/heads/main' && github.event_name == 'push'
  needs: build
  runs-on: ubuntu-latest
  environment: production
  steps:
    - uses: actions/checkout@v4
    - name: Deploy to production
      run: echo "Deploying to production..."

Configuring environments in GitHub

GitHub → Settings → Environments:

Staging:
  - Protection rules: None (fast deploy for testing)
  - Secrets: OPENAI_API_KEY (staging key if you have one)
  - Variables: PRODUCTION_URL=https://staging.your-app.railway.app

Production:
  - Protection rules: Required reviewers (optional but recommended)
  - Secrets: OPENAI_API_KEY, PLATFORM_TOKEN
  - Variables: PRODUCTION_URL=https://your-app.railway.app

Rollback Strategy

When a deploy goes wrong

Not all deploys are successful. You need a plan to return to the previous version.

Option 1: Re-deploy the previous commit

# Identify the last good commit
git log --oneline -5
# abc1234 (HEAD -> main) feat: update prompt template  ← THIS ONE BROKE IT
# def5678 fix: adjust timeout                          ← THIS ONE WORKED

# Create a rollback branch and force a deploy
git checkout def5678
git checkout -b hotfix/rollback-to-def5678
git push origin hotfix/rollback-to-def5678

# Merge to main to trigger a deploy
# Or manually on the platform: deploy commit def5678

Option 2: Rollback from the platform

# Render: rollback from the dashboard
# Settings → Manual Deploy → select the previous deploy

# Railway: rollback
railway rollback

# Fly.io: rollback to a previous release
flyctl releases
flyctl deploy --image registry.fly.io/my-app:previous-version

# AWS Lambda: point the alias to a previous version
aws lambda update-alias \
    --function-name my-function \
    --name production \
    --function-version 5  # previous version

Option 3: Automated rollback workflow

# .github/workflows/rollback.yml
name: Rollback Deployment

on:
  workflow_dispatch:
    inputs:
      commit_sha:
        description: 'Commit SHA to rollback to'
        required: true
      reason:
        description: 'Reason for rollback'
        required: true

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Checkout specific commit
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.inputs.commit_sha }}

      - name: Log rollback
        run: |
          echo "Rolling back to ${{ github.event.inputs.commit_sha }}"
          echo "Reason: ${{ github.event.inputs.reason }}"
          echo "Triggered by: ${{ github.actor }}"
          echo "Time: $(date -u)"

      - name: Deploy rollback version
        run: |
          # Your deploy command here
          echo "Deploying rollback..."

      - name: Validate rollback
        run: |
          sleep 45
          status=$(curl -s -o /dev/null -w "%{http_code}" \
            "${{ vars.PRODUCTION_URL }}/health")
          if [ "$status" = "200" ]; then
            echo "Rollback successful"
          else
            echo "CRITICAL: Rollback failed, status: $status"
            exit 1
          fi

When to roll back

IMMEDIATE ROLLBACK (< 5 min):
├── Health check fails after the deploy
├── Error rate > 50% in the first 2 minutes
└── Inference smoke test fails

EVALUATE FIRST (5-30 min):
├── Latency increased but the system works
├── Error rate went up but < 10%
└── A specific endpoint fails, others work

DOESN'T REQUIRE ROLLBACK:
├── Logs show warnings but no errors
├── Marginal latency (< 20% increase)
└── Cosmetic issue (different response format)

Secrets Management

What secrets you need and where to configure them

GitHub Secrets (Settings → Secrets and variables → Actions):
├── OPENAI_API_KEY          # Your OpenAI API key
├── RENDER_DEPLOY_HOOK_URL  # If you use Render
├── RAILWAY_TOKEN           # If you use Railway
├── FLY_API_TOKEN           # If you use Fly.io
├── AWS_ACCESS_KEY_ID       # If you use AWS
└── AWS_SECRET_ACCESS_KEY   # If you use AWS

GitHub Variables (Settings → Secrets and variables → Actions → Variables):
├── PRODUCTION_URL          # https://your-app.platform.app
├── RAILWAY_SERVICE_ID      # If you use Railway
└── APP_NAME                # Your app's name

Platform (Render/Railway/Fly.io dashboard):
├── OPENAI_API_KEY          # The same key or a different one for prod
├── ENVIRONMENT             # "production"
├── LOG_LEVEL               # "WARNING" or "INFO"
└── [Other variables of your app]

Verify that the secrets are configured

# scripts/verify_secrets.py
"""Verifies that the required environment variables are configured."""
import os
import sys

REQUIRED = {
    "OPENAI_API_KEY": "API key for LLM inference",
    "ENVIRONMENT": "Current environment (development/staging/production)",
}

OPTIONAL = {
    "LOG_LEVEL": ("Logging level", "INFO"),
    "CORS_ORIGINS": ("Allowed CORS origins", '["*"]'),
}

def verify():
    errors = []
    warnings = []

    for var, description in REQUIRED.items():
        value = os.environ.get(var)
        if not value:
            errors.append(f"MISSING: {var} — {description}")
        else:
            masked = value[:4] + "..." + value[-4:] if len(value) > 8 else "***"
            print(f"  {var}: {masked}")

    for var, (description, default) in OPTIONAL.items():
        value = os.environ.get(var)
        if not value:
            warnings.append(f"OPTIONAL: {var} not set, using default: {default}")
        else:
            print(f"  {var}: {value}")

    if warnings:
        print(f"\nWarnings ({len(warnings)}):")
        for w in warnings:
            print(f"  ⚠️ {w}")

    if errors:
        print(f"\nErrors ({len(errors)}):")
        for e in errors:
            print(f"  ❌ {e}")
        sys.exit(1)

    print("\nAll required secrets verified.")

if __name__ == "__main__":
    verify()

Troubleshooting

Problem 1: "The pipeline stays pending"

Cause: GitHub Actions has a concurrent-jobs limit or there's an unavailable runner.

Solution:

# Add a timeout to all jobs
jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 10  # Kill after 10 min

Check in Actions → All workflows if there are queued jobs. Cancel obsolete ones manually.

Problem 2: "Deploy trigger works but the app doesn't update"

Cause: Platform cache, duplicate image tag, or the deploy completed with the old version.

Solution:

# Use the commit SHA as the tag to avoid cache
docker build -t my-app:${{ github.sha }} .

# On Railway: force a rebuild
railway up --detach

# On Render: verify in the dashboard that the deploy is the correct commit

Problem 3: "Rollback doesn't work — the previous commit also fails"

Cause: A change in environment variables or an external service, not in the code.

Solution:

# Verify whether the problem is the code or the env vars
# 1. Check if any secret was changed
# 2. Verify that the external APIs (OpenAI) are responding
curl -s https://api.openai.com/v1/models \
    -H "Authorization: Bearer $OPENAI_API_KEY" | head -c 200
# 3. If the external API is down, the rollback won't solve anything

Problem 4: "The validate job fails on timeout"

Cause: The platform takes longer than expected to deploy, or the container has a long cold start.

Solution:

# Increase the wait time and the retries
- name: Wait for deployment
  run: sleep 60  # Give more time

- name: Health check with extended retry
  run: |
    MAX_RETRIES=8
    RETRY_DELAY=20
    # ... (longer retry loop)

Problem 5: "Tests pass locally but fail in CI"

Cause: Version difference, unconfigured variables, or tests that depend on external services.

Solution:

# Use the same Python version
python-version: "3.11"  # Not "3.x" — be specific

# Freeze exact dependencies
pip freeze > requirements.txt

# Separate tests that need an API key
# tests/unit/ → without secrets
# tests/integration/ → with secrets

Hands-On Exercises

Exercise 1: Complete end-to-end pipeline

Create a GitHub Actions pipeline that does test → build → deploy → validate for your chosen platform.

# .github/workflows/deploy.yml
name: Deploy AI System

on:
  push:
    branches: [main]
  workflow_dispatch:

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

env:
  PYTHON_VERSION: "3.11"

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
          cache: 'pip'
      - run: pip install -r requirements.txt && pip install pytest httpx
      - run: pytest tests/ -v --tb=short
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ENVIRONMENT: test

  build:
    needs: test
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t my-app:${{ github.sha }} .
      - run: |
          docker run -d --name verify -p 8000:8000 \
            -e ENVIRONMENT=test -e OPENAI_API_KEY=fake \
            my-app:${{ github.sha }}
          sleep 5
          curl -f http://localhost:8000/health
          docker stop verify && docker rm verify

  deploy:
    if: github.ref == 'refs/heads/main'
    needs: build
    runs-on: ubuntu-latest
    timeout-minutes: 10
    environment: production
    steps:
      - uses: actions/checkout@v4
      - name: Deploy
        run: curl -X POST "${{ secrets.RENDER_DEPLOY_HOOK_URL }}"

  validate:
    needs: deploy
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - name: Wait and verify
        run: |
          sleep 60
          for i in 1 2 3 4 5; do
            if curl -sf "${{ vars.PRODUCTION_URL }}/health"; then
              echo "Validation passed"
              exit 0
            fi
            sleep 15
          done
          exit 1

Exercise 2: Rollback workflow

Create a workflow_dispatch workflow that allows rolling back to a specific commit.

# .github/workflows/rollback.yml
name: Rollback

on:
  workflow_dispatch:
    inputs:
      commit_sha:
        description: 'Commit to rollback to'
        required: true
      reason:
        description: 'Reason'
        required: true

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.inputs.commit_sha }}

      - name: Log
        run: |
          echo "Rollback to: ${{ github.event.inputs.commit_sha }}"
          echo "Reason: ${{ github.event.inputs.reason }}"
          echo "By: ${{ github.actor }} at $(date -u)"

      - name: Deploy previous version
        run: curl -X POST "${{ secrets.RENDER_DEPLOY_HOOK_URL }}"

      - name: Validate
        run: |
          sleep 60
          curl -sf "${{ vars.PRODUCTION_URL }}/health" || exit 1
          echo "Rollback validated"

Exercise 3: Secrets verification script

Create the scripts/verify_secrets.py script and run it in your pipeline before the deploy.

# scripts/verify_secrets.py
import os
import sys

REQUIRED_VARS = {
    "OPENAI_API_KEY": "Required for LLM inference",
    "ENVIRONMENT": "Must be development, staging, or production",
}

def main():
    missing = []
    for var, desc in REQUIRED_VARS.items():
        val = os.environ.get(var, "")
        if not val:
            missing.append(f"{var}: {desc}")
        else:
            safe = val[:3] + "***" if len(val) > 3 else "***"
            print(f"  {var}={safe}")

    if missing:
        print("Missing required variables:")
        for m in missing:
            print(f"  ❌ {m}")
        sys.exit(1)

    print("All required variables present.")

if __name__ == "__main__":
    main()

- name: Verify secrets
  run: python scripts/verify_secrets.py
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ENVIRONMENT: production

Exercise 4: Simulate a rollback

Make an intentional deploy that fails (change the health check endpoint to one that doesn't exist) and then roll back to the previous commit.

# 1. Create a commit that breaks the deploy
# In main.py, change the health check path:
# @app.get("/health") → @app.get("/healthz")
# The pipeline will look for /health, which no longer exists

# 2. Push and watch the pipeline fail at validate
git add -A && git commit -m "break: test rollback" && git push

# 3. The validate job will fail (health check returns 404)

# 4. Identify the previous commit
git log --oneline -3
# abc1234 break: test rollback     ← THIS ONE
# def5678 last working version     ← ROLLBACK TO THIS ONE

# 5. Rollback: go to GitHub → Actions → Rollback workflow
# Input: def5678
# Reason: "Health check endpoint changed accidentally"

# 6. Or revert the commit
git revert abc1234
git push  # This triggers a new deploy with the good code

# 7. Verify that the deploy works
curl https://your-app.platform.app/health
# {"status": "healthy"}

Summary

• One push, one deploy: git push main should automatically trigger test → build → deploy → validate
• Concurrency control: use concurrency in GitHub Actions to avoid simultaneous deploys
• Environment promotion: local → staging → production with the same image and different config
• Rollback strategy: have at least two methods (re-deploy previous commit + rollback from the platform)
• Secrets management: secrets in GitHub Secrets for CI/CD, on the platform for runtime
• Timeout on everything: each job needs timeout-minutes to avoid hung pipelines
• Always validate: a deploy without automatic post-deploy validation is a blind deploy

Additional Resources

1. GitHub Actions — Environments — Configuring environments
2. GitHub Actions — Concurrency — Concurrency control
3. GitHub Actions — Encrypted Secrets — Secrets management
4. Render — Continuous Deployment — Automatic deploys on Render
5. Railway — GitHub Deployments — Auto-deploy from GitHub on Railway
6. Fly.io — GitHub Actions Deploy — CI/CD with Fly.io