2. Deployment Strategies

Description

The moment of deploy is when things can go wrong for you in unexpected ways: different configuration in production, misconfigured secrets, different behavior with real traffic. Deployment strategies are not DevOps ceremonies — they're tools you choose to minimize the risk of each change. For your AI apps, there are additional considerations: prompts are part of the code, changing models is a change in behavior, and a failed deployment can be invisible if your system fails silently instead of raising clear exceptions.

The 3 main strategies

Rolling Update

Before the deploy:
  Instance 1: v1 (receiving traffic)
  Instance 2: v1 (receiving traffic)
  Instance 3: v1 (receiving traffic)

During the deploy (gradual):
  Instance 1: v2 (receiving traffic — new version)
  Instance 2: v1 (receiving traffic — previous version)
  Instance 3: v1 (receiving traffic — previous version)

After the deploy:
  Instance 1: v2 (receiving traffic)
  Instance 2: v2 (receiving traffic)
  Instance 3: v2 (receiving traffic)

Characteristics:
  - Minimal downtime (there are always instances serving)
  - Rollback: revert the image and do a rolling again
  - Risk: during the deploy, there are instances on v1 and v2 simultaneously
  - AI implication: if you change the prompt, there will be requests processed with v1 and v2 of the prompt
    → The logs will show inconsistent responses during the transition

Blue-Green

Normal state (blue active):
  BLUE (v1): receiving real traffic
  GREEN: inactive (last version that worked)

When there's a new deploy:
  1. Deploy v2 to GREEN
  2. Run smoke tests on GREEN
  3. Switch traffic: BLUE → GREEN
  4. GREEN (v2): receiving real traffic
  5. BLUE (v1): on standby (in case of rollback)

Instant rollback:
  If GREEN has problems → switch traffic back to BLUE
  Rollback time: seconds

Characteristics:
  - Faster rollback than rolling
  - Cost: two complete environments
  - No period of mixed versions
  - Ideal for important prompt changes

Canary

Initial state:
  MAIN: 100% of the traffic (v1)
  CANARY: 0% of the traffic

Canary deploy:
  MAIN: 90% of the traffic (v1)
  CANARY: 10% of the traffic (v2)

If the metrics are good (error rate, latency, cost):
  MAIN: 50% (v1)
  CANARY: 50% (v2)

If it's still fine:
  MAIN: 0% (v1 — deprecated)
  CANARY: 100% (v2 — now it's the main)

If the metrics worsen at any point:
  Rollback: MAIN goes back to 100%, CANARY to 0%

Characteristics:
  - The finest control over risk
  - Lets you compare v1 vs v2 in real production (A/B testing of prompts)
  - More complex to configure
  - Ideal when you change models or significant prompts

Considerations specific to AI apps

Prompts are part of the code

# ❌ Anti-pattern: prompt hardcoded in the code
async def analyze_sentiment(text: str) -> dict:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Analyze the sentiment..."},  # ← hardcoded
            {"role": "user", "content": text}
        ]
    )

# Problem: changing the prompt requires changing the code and doing a full deploy
# Problem: there's no history of prompt changes
# Problem: you can't revert the prompt without reverting the code

# ✅ Correct pattern: prompts in versioned files
# prompts/sentiment/v1.yaml:
#   version: "v1"
#   system: "You are a sentiment analysis expert..."

# In the code, load the prompt:
from src.prompts.loader import load_prompt
template = load_prompt("sentiment")  # Loads v1.yaml (or whichever is default)

# When making a prompt change:
# 1. Create prompts/sentiment/v2.yaml with the new prompt
# 2. Update settings: prompt_version: "v2"
# 3. Commit and deploy
# 4. If there's a problem, revert settings to "v1" (or revert the commit)

Changing models is a change in behavior

# Changing from gpt-4o to gpt-4o-mini is NOT just a config change
# The model can produce outputs of different quality
# This is a behavior change that needs testing

# Before changing the model in production:
# 1. Run the test suite with the new model
# 2. Compare outputs in staging (are the sentiment scores similar?)
# 3. Review the cost implications
# 4. Consider canary: 10% of the traffic to the new model, compare metrics

# In Settings:
class Settings(BaseSettings):
    openai_model: str = Field(
        default="gpt-4o",
        description="Changing this value is a behavior change — requires testing"
    )
    
    # For a model canary, you could have:
    canary_model: Optional[str] = Field(
        default=None,
        description="If set, X% of the traffic goes to this model"
    )
    canary_traffic_percent: int = Field(default=10)

Pre-deploy checklist specific to AI

# scripts/pre_deploy.sh

#!/bin/bash
set -e  # Exit if any command fails

echo "=== Pre-Deploy Checklist ==="

echo ""
echo "1. Unit tests..."
python -m pytest tests/unit/ -q
echo "   ✅ Unit tests passed"

echo ""
echo "2. Guardrails tests..."
python -m pytest tests/ -k "guardrail" -q
echo "   ✅ Guardrails tests passed"

echo ""
echo "3. No secrets in code..."
if grep -r "sk-" src/ 2>/dev/null; then
    echo "   ❌ FOUND POTENTIAL API KEYS IN CODE"
    exit 1
fi
echo "   ✅ No secrets found"

echo ""
echo "4. .env not in git..."
if git ls-files .env 2>/dev/null | grep -q ".env"; then
    echo "   ❌ .env IS TRACKED IN GIT — SECURITY RISK"
    exit 1
fi
echo "   ✅ .env not tracked"

echo ""
echo "5. Production config validation..."
ENVIRONMENT=production python -c "
from src.config import get_settings
try:
    s = get_settings()
    print(f'   Config valid: model={s.openai_model}, env={s.environment}')
except Exception as e:
    print(f'   ERROR: {e}')
    exit(1)
"
echo "   ✅ Production config valid"

echo ""
echo "=== All pre-deploy checks passed ==="

Post-deploy validation

# scripts/post_deploy_check.py
"""
Checks to run immediately after the deploy.
"""
import time
import urllib.request
import json
import sys

BASE_URL = os.environ.get("APP_URL", "http://localhost:8000")

def smoke_test_health():
    """The server responds to health checks."""
    url = f"{BASE_URL}/health/live"
    req = urllib.request.urlopen(url, timeout=10)
    assert req.status == 200
    print("  ✅ /health/live: 200 OK")

def smoke_test_ready():
    """The server is ready for traffic."""
    url = f"{BASE_URL}/health/ready"
    req = urllib.request.urlopen(url, timeout=10)
    assert req.status == 200
    print("  ✅ /health/ready: 200 OK")

def smoke_test_analyze():
    """The main endpoint responds correctly."""
    data = json.dumps({"text": "This product is amazing!"}).encode()
    req = urllib.request.Request(
        f"{BASE_URL}/api/v1/analyze",
        data=data,
        headers={"Content-Type": "application/json"}
    )
    response = urllib.request.urlopen(req, timeout=30)
    body = json.loads(response.read())
    
    assert response.status == 200
    assert "sentiment" in body
    assert "score" in body
    print(f"  ✅ /api/v1/analyze: {body['sentiment']} (score: {body['score']})")

def monitor_for_errors(duration_seconds: int = 60):
    """Monitors logs for errors during N seconds after the deploy."""
    print(f"  Monitoring for errors for {duration_seconds}s...")
    # In a real system, you would query your logging aggregator (Datadog, CloudWatch, etc.)
    # Simplified here:
    time.sleep(min(duration_seconds, 15))  # In CI, 15s is enough
    print("  ✅ No critical errors detected in monitoring window")

def run_post_deploy(monitor: bool = True):
    print("=== Post-Deploy Validation ===\n")
    
    checks = [
        ("Health liveness", smoke_test_health),
        ("Health readiness", smoke_test_ready),
        ("Analyze endpoint", smoke_test_analyze),
    ]
    
    if monitor:
        checks.append(("Error monitoring", lambda: monitor_for_errors(60)))
    
    for name, check in checks:
        try:
            check()
        except Exception as e:
            print(f"  ❌ {name} FAILED: {e}")
            print(f"\n⚠️  POST-DEPLOY FAILURE — consider rollback")
            sys.exit(1)
    
    print("\n=== All post-deploy checks passed ===")
    print("✅ Deploy successful")

if __name__ == "__main__":
    run_post_deploy(monitor="--no-monitor" not in sys.argv)

Configuration per environment

# The rule: never use "if env == 'production'" in the business code
# All the per-environment variation goes in Settings (config)

# .env.development
ENVIRONMENT=development
LOG_LEVEL=DEBUG
USE_MOCK_PROVIDER=true
OPENAI_MODEL=gpt-4o-mini      # Cheaper in dev
MAX_REQUESTS_PER_MINUTE=20    # Lower in dev

# .env.staging
ENVIRONMENT=staging
LOG_LEVEL=INFO
USE_MOCK_PROVIDER=false
OPENAI_MODEL=gpt-4o           # Same as prod for real testing
MAX_REQUESTS_PER_MINUTE=60

# .env.production
ENVIRONMENT=production
LOG_LEVEL=INFO
USE_MOCK_PROVIDER=false
OPENAI_MODEL=gpt-4o
MAX_REQUESTS_PER_MINUTE=400   # 80% of the API limit
CIRCUIT_BREAKER_THRESHOLD=5
MAX_RETRY_ATTEMPTS=4
DAILY_BUDGET_LIMIT_USD=50.0

# The validation in Settings (from M6) ensures that in production:
# - USE_MOCK_PROVIDER=false is mandatory
# - LOG_LEVEL != DEBUG is mandatory
# - OPENAI_API_KEY present is mandatory

Rollback procedure

# Rollback procedure for each strategy

# Rolling (with Docker/docker-compose):
docker-compose up -d --scale app=3  # Go back to the previous image
# Or if you use tags:
docker pull myapp:v1.2.3
docker-compose up -d

# Blue-Green (with nginx):
# In nginx.conf, change upstream from green to blue:
# upstream app { server blue:8000; }  # Revert to blue
nginx -s reload

# Kubernetes rolling:
kubectl rollout undo deployment/sentiment-app
kubectl rollout status deployment/sentiment-app  # Verify that it completed

# In all cases, run post-deploy after the rollback:
python scripts/post_deploy_check.py --no-monitor

Exercises

Exercise 1: Choosing the right strategy

For each scenario, what deployment strategy would you use?

1. Change the error message on a text endpoint
2. Update the sentiment analysis prompt with an important new instruction
3. Change the model from gpt-4o-mini to gpt-4o in production

Exercise 2: Canary script with metrics validation

Write a Python script that simulates a canary deployment: it starts with 10% of the traffic, checks metrics (error rate < 5%, p50 latency < 3s), and if it passes, it increases to 50% and then to 100%. If the metrics fail at any stage, it does a rollback.

import time
import random
from dataclasses import dataclass

@dataclass
class CanaryMetrics:
    error_rate: float
    p50_latency_ms: float
    total_requests: int

def get_canary_metrics(canary_percent: int) -> CanaryMetrics:
    """Simulates canary metrics (in real prod: Datadog/Prometheus)."""
    return CanaryMetrics(
        error_rate=random.uniform(0.01, 0.04),
        p50_latency_ms=random.uniform(800, 2500),
        total_requests=canary_percent * 10,
    )

def canary_deploy(stages: list[int] = [10, 50, 100]):
    max_error_rate = 0.05
    max_p50_ms = 3000
    observation_seconds = 5

    for stage_percent in stages:
        print(f"\n🔄 Canary: {stage_percent}% of traffic")
        print(f"   Observing for {observation_seconds}s...")
        time.sleep(observation_seconds)

        metrics = get_canary_metrics(stage_percent)
        print(f"   Error rate: {metrics.error_rate:.2%}")
        print(f"   p50 latency: {metrics.p50_latency_ms:.0f}ms")

        if metrics.error_rate > max_error_rate:
            print(f"   ❌ Error rate {metrics.error_rate:.2%} > {max_error_rate:.2%}")
            print(f"   🔙 ROLLBACK: returning to 0% canary")
            return False

        if metrics.p50_latency_ms > max_p50_ms:
            print(f"   ❌ p50 {metrics.p50_latency_ms:.0f}ms > {max_p50_ms}ms")
            print(f"   🔙 ROLLBACK: returning to 0% canary")
            return False

        print(f"   ✅ Metrics OK — advancing")

    print("\n✅ Canary complete — 100% on the new version")
    return True

if __name__ == "__main__":
    canary_deploy()

Exercise 3: Rollback procedure for a prompt change

Your team deployed a new v3 prompt for sentiment analysis. After 15 minutes, they notice that accuracy dropped from 92% to 78%. Write step by step the rollback procedure including the exact commands to revert the prompt, verify the reversion, and communicate to the team.

# 1. IDENTIFY: confirm that the problem is the prompt
jq 'select(.prompt_version == "v3") | {accuracy: .confidence, timestamp}' \
  logs/app.json | tail -5

# 2. REVERT: change the prompt version
#    Option A: revert the commit
git log --oneline -5
git revert <commit-hash> --no-edit
git push origin main

#    Option B: change the config directly (faster)
#    In .env.production: PROMPT_VERSION=v2

# 3. RE-DEPLOY with the previous prompt
#    Blue-green: instant traffic switch to the previous environment
#    Rolling:
docker-compose pull && docker-compose up -d

# 4. VERIFY that v2 is active
curl -s http://localhost:8000/api/v1/analyze \
  -H "Content-Type: application/json" \
  -d '{"text": "Great product!"}' | jq '.prompt_version'
# → Must return "v2"

# 5. MONITOR that accuracy returned to normal levels
jq 'select(.prompt_version == "v2") | .confidence' logs/app.json | \
  awk '{s+=$1; n++} END {print "Avg confidence:", s/n}'

# 6. COMMUNICATE to the team:
# "🔙 Rollback complete: prompt v3 → v2
#  Reason: accuracy dropped from 92% to 78% with v3
#  Current status: v2 active, metrics normalizing
#  Next steps: investigate what changed in v3 before retrying"

Exercise 4: Configure post-deploy validation with alerts

Write a function post_deploy_alert that runs the smoke tests, and if any fails, sends an alert (simulated) with the failure details and a link to the corresponding runbook.

from dataclasses import dataclass
from datetime import datetime
from typing import Optional
import json
import urllib.request

@dataclass
class DeployAlert:
    severity: str
    title: str
    detail: str
    runbook_url: str
    timestamp: str = ""

    def __post_init__(self):
        self.timestamp = datetime.utcnow().isoformat()

RUNBOOK_URLS = {
    "health_check": "https://docs.internal/runbook#health-check-failure",
    "smoke_test": "https://docs.internal/runbook#smoke-test-failure",
    "high_error_rate": "https://docs.internal/runbook#high-error-rate",
    "high_latency": "https://docs.internal/runbook#high-latency",
}

def post_deploy_alert(base_url: str = "http://localhost:8000") -> list[DeployAlert]:
    alerts = []

    # Check 1: Health
    try:
        req = urllib.request.urlopen(f"{base_url}/health/ready", timeout=10)
        if req.status != 200:
            alerts.append(DeployAlert(
                severity="critical",
                title="Health check failed post-deploy",
                detail=f"/health/ready returned HTTP {req.status}",
                runbook_url=RUNBOOK_URLS["health_check"],
            ))
    except Exception as e:
        alerts.append(DeployAlert(
            severity="critical",
            title="Server unreachable post-deploy",
            detail=str(e),
            runbook_url=RUNBOOK_URLS["health_check"],
        ))

    # Check 2: Smoke test of the main endpoint
    try:
        data = json.dumps({"text": "This is a great product"}).encode()
        req = urllib.request.Request(
            f"{base_url}/api/v1/analyze",
            data=data,
            headers={"Content-Type": "application/json"},
        )
        response = urllib.request.urlopen(req, timeout=30)
        body = json.loads(response.read())

        if "sentiment" not in body:
            alerts.append(DeployAlert(
                severity="critical",
                title="Analyze endpoint returns malformed response",
                detail=f"Missing 'sentiment' in response: {list(body.keys())}",
                runbook_url=RUNBOOK_URLS["smoke_test"],
            ))
    except Exception as e:
        alerts.append(DeployAlert(
            severity="critical",
            title="Analyze endpoint failed post-deploy",
            detail=str(e),
            runbook_url=RUNBOOK_URLS["smoke_test"],
        ))

    for alert in alerts:
        print(f"🚨 [{alert.severity.upper()}] {alert.title}")
        print(f"   Detail: {alert.detail}")
        print(f"   Runbook: {alert.runbook_url}")
        print(f"   Time: {alert.timestamp}")

    if not alerts:
        print("✅ Post-deploy: all checks passed, no alerts")

    return alerts

Troubleshooting

Problem: Rolling update causes inconsistent responses during the deploy

Symptom: During a rolling update, some requests return results with the v1 prompt and others with the v2 prompt, confusing the users.

Cause: In a rolling update, instances with v1 and v2 temporarily coexist. If the prompt changed significantly, the outputs will be different.

Solution:

# Option 1: Use blue-green instead of rolling for prompt changes
# → Eliminates the period of mixed versions

# Option 2: If you must use rolling, do the deploy during low-traffic hours
# → Minimizes the number of affected users

# Option 3: Include the prompt version in the response
# → The client can detect and handle the inconsistency
# In AnalyzeResponse: prompt_version: str = Field(...)

Problem: Blue-green doubles infrastructure costs

Symptom: You have two complete environments running, doubling server costs.

Cause: Blue-green requires two complete environments to work.

Solution:

# For AI apps, the main cost isn't infrastructure but the LLM calls
# The inactive environment doesn't make LLM calls → minimal additional cost

# If infrastructure cost does matter:
# 1. Inactive environment with minimal replicas (1 instance)
# 2. Auto-scale the active one based on demand
# 3. Shut down the inactive one after confirming stability (24-48h)

# In Docker Compose:
# blue (active): replicas: 3
# green (standby): replicas: 1  ← ready to scale if there's a rollback

Problem: The canary doesn't detect model quality degradation

Symptom: The canary's error rate and latency metrics are fine, but the response quality of the new model is worse.

Cause: Standard metrics (error rate, latency) don't capture the semantic quality of the LLM's responses.

Solution:

def check_canary_quality(canary_logs: list[dict]) -> bool:
    """Quality metrics specific to AI in your canary check."""
    avg_confidence = sum(l["confidence"] for l in canary_logs) / len(canary_logs)

    if avg_confidence < 0.75:
        print(f"⚠️ Average confidence: {avg_confidence:.2f} (threshold: 0.75)")
        return False

    degraded_pct = sum(1 for l in canary_logs if l.get("degraded")) / len(canary_logs)
    if degraded_pct > 0.1:
        print(f"⚠️ {degraded_pct:.0%} of requests used fallback")
        return False

    return True

Summary

• Rolling: the default — low risk, no downtime, gradual rollback
• Blue-Green: when you need instant rollback — ideal for prompt changes
• Canary: when you change something high-impact (model, critical prompt) — validate in real production with little traffic
• Prompts are code: they're versioned, tested, deployed, and reverted like any other code
• Changing the model = changing behavior: always requires testing before the full switch
• Pre-deploy script: executable before each deploy to verify the automatable checklist items
• Post-deploy monitoring: the first 30-60 minutes after the deploy are critical

Additional resources

1. Martin Fowler — Blue-Green Deployment — The canonical article
2. Martin Fowler — Canary Release — The pattern explained
3. Kubernetes Rolling Updates — Documentation
4. GitHub Actions — CI/CD to automate the pre-deploy checklist